Installing Unravel for GCP BigQuery

Before setting up Unravel for BigQuery, ensure that the installation requirements are completed.

Follow the instructions to install and set up Unravel to receive BigQuery data.

Install and set up Unravel on the GCP instance

Do the following to set up Unravel on the GCP instance.

1. Create and configure the GCE

On your GCP console, go to the GCE dashboard and click Create Instance.
Select the following options based on Unravel's instance requirements:
- Base OS
- Instance type and size
- Ports
- Networking
  The instance must be HTTPS and publicly accessible.
- Firewall rules or policies
  Sample inbound rule
  Type
  Protocol
  Port range
  Source
  All traffic
  All
  All
  For example, 10.10.0.0/16
  SSH
  TCP
  22
  0.0.0.0/0 or trusted public IP for SSH access
  Custom TCP Rule
  TCP
  3000
  Custom TCP Rule
  TCP
  4043
  Custom TCP Rule
  TCP
  4443
  Sample outbound rule
  Type
  Protocol
  Port range
  Source
  All traffic
  All
  All
  0.0.0.0/0
Note
The GCP instance should have all TCP access to the BigQuery cluster (server/parent or worker) nodes. You can grant access by inserting adding firewall rules of the BigQuery server/parent and worker with all TCP, all port range.
While creating the GCP instance add the Firewall properties, Enable the HTTP and HTTPS traffic Go to Network tab and add Network tags. (This is the firewall rule that is already created.)

Type	Protocol	Port range	Source
All traffic	All	All	For example, 10.10.0.0/16
SSH	TCP	22	0.0.0.0/0 or trusted public IP for SSH access
Custom TCP Rule	TCP	3000
Custom TCP Rule	TCP	4043
Custom TCP Rule	TCP	4443

Type	Protocol	Port range	Source
All traffic	All	All	0.0.0.0/0

Configure the GCE instance

Disable selinux.
```
sudo setenforce Permissive
```
Edit /etc/selinux/config to ensure the setting persists after reboot and ensure SELINUX=permissive.
```
sudo vi /etc/selinux/config
```
Install libaio.x86_64, lzop.x86_64, and ntp.x86_64.
```
sudo yum install -y libaio.x86_64
sudo yum install -y lzop.x86_64
sudo yum install -y ntp.x86_64
```
If you are using Unravel-managed MySQL or MariaDB, run sudo yum install numactl-devel.x86_64 command. Also refer to Integrating Database.
Set vm.max_map_count to 262144
Set nproc limit to unlimited.
Add unravel.conf file with the following settings to /etc/security/limits.d:
```
unravel    soft  nproc unlimited
```
Start ntpd and check the system time.
```
sudo service ntpd start
sudo ntpq -p
```
Create a new Unravel user, which should not be a root user. For example:
```
sudo useradd unravel
```

2. Download Unravel

Download Unravel onto the VM instance that you have created.

Important

Before you download Unravel for your platform, ensure to get the username and password from Unravel Support.

Download the Unravel package (RPM or Tar):

You can refer to Download section for the complete list of Unravel product downloads.

For example:

Tar

curl -v https://preview.unraveldata.com/unravel/RPM/4.7.x/unravel-4.7.x.x-cloud.tar.gz -u username:password

RPM

curl -v https://preview.unraveldata.com/unravel/RPM/4.7.x/unravel-4.7.x.x.rpm -o unravel-4.7.x.x.rpm -u username:password

3. Deploy Unravel

Deploy Unravel on the GCP instance that you have created.

Unravel binaries are available as a tar file or RPM package. You can deploy the Unravel binaries in any directory on the server. However, the user who installs Unravel must have the write permissions to the directory where the Unravel binaries are deployed.

After deploying the Unravel binaries, the directory layout for both the Tar and RPM will be unravel/versions/<Directories and files>. The binaries are deployed to <Unravel_installation_directory>, and Unravel will be available in <Unravel_installation_directory/unravel.

The following steps to deploy Unravel from a tar file should be performed by a user, who will run Unravel.

Create an Installation directory.
```
mkdir /path/to/installation/directory
```
For example: mkdir /opt/
Extract Unravel tar file to the installation directory, which you have created in the first step. After you extract the contents of the tar file, unravel directory is created within the installation directory.
```
tar zxf unravel-<version>tar.gz -C /path/to/installation/directory
```
For example: tar zxf unravel-4.7.x.x.tar.gz -C /opt
The unravel directory will be available within /opt
Grant ownership of the directory to a user who will run Unravel.
```
chown -R username:groupname /opt/unravel/
```
For example: chown -R hadoop:hadoop /opt/unravel/

Important

The following steps to deploy Unravel from an RPM package should be performed by a root user. After the RPM package is deployed, the remaining installation procedures should be performed by unravel user.

Create an installation directory.
```
mkdir /usr/local/unravel
```
Run the following command:
```
rpm -i unravel-<version>.rpm
```
For example: rpm -i unravel-4.7.0.0.rpm
The unravel directory will be available in /usr/local
If you want to provide a different location, use the --prefix command. For example:
```
mkdir /opt/unravel
rpm -i unravel-4.7.0.0.rpm --prefix /opt
```
The unravel directory will be available in /opt.
Grant ownership of the directory to a user who will run Unravel. This user executes all the processes involved in Unravel installation.
```
chown -R username:groupname /usr/local/unravel
```
For example: chown -R hadoop:hadoop /usr/local/unravel
Continue with the installation procedures as unravel user.

4. Install Unravel

The Interactive Precheck utility is run to validate the required configurations before installing Unravel. When you run the Interactive Precheck utility various checks are prompted for gathering configuration information. The responses you provide for these checks are used to generate a bootstrap configuration file. This file, which contains the configuration information, is then used to install Unravel.

Do the following to install and configure Unravel with Interactive Precheck.

After you download and deploy the Unravel, run the precheck.sh script from unravel/versions/X.Y.Z/healthcheck/.
For example:
/opt/unravel/versions/X.Y.Z/healthcheck/precheck.sh

Enter the necessary details when you are prompted for the following configuration information:

This section covers general information about your Unravel install. You are prompted for the following:

Data platform you want to monitor.
For Hadoop: type of Unravel node you want to configure.
For edge nodes: core node location and test connectivity.

You must answer the following prompts:

-- General information
Which data platform are you installing for?
   1- Hadoop
   2- EMR
   3- HDI
   4- Databricks
   5- Dataproc
   6- BigQuery

Select one of the above [Hadoop]: 

## You can choose a number corresponding to the platform.

This check allows you to configure database-related information and an external database for Unravel.

-- Database configuration
 Configure an external database? (y/n) [No]:

If you answer No, an Unravel-managed database is used for the installation.

If you answer Yes, you are further prompted for the type of external database that you want to configure.

-- Database configuration
   
   Configure an external database? (y/n) [No]: y

   Type
   1- PostgresQL
   2- MySQL
   3- MariaDB

If you choose a specific type of external database, you are prompted for the following database information and test connectivity to that database. Refer to Integrating Database for more details. For example:

-- Database configuration
  
   Configure an external database? (y/n) [No]: y

   Type
   1- PostgresQL
   2- MySQL
   3- MariaDB

   Select one of the above []: 1
   Selected: PostgresQL

   Database hostname [None]:

   Database port (integer) [None]:

   Database schema [None]:

   Does the database use TLS (y/n) [No]: 

   Database username [None]:

   Database password [None] (no echo):

   Do you wish to test connecting to the external database? (y/n) [Yes]:-- Database configuration
 Configure an external database? (y/n) [No]:

If you choose MySQL or MariaDB database you are further prompted for extra packages. If you answer Yes, the Extra packages section searches for the required JDBC drivers.
```
-- Database configuration
 Will Unravel connect to a MySQL or MariaDB database (ex: hive metastore) ? (y/n) [No]: 
```

The Extra packages check shows if you use Unravel-managed MySQL/MariaDB or need JDBC drivers. Else, this check is automatically skipped.

-- Extra package location

*** JDBC drivers are required for Unravel managed MySQL or MariaDB.
*** Database software package is required for Unravel managed MySQL or MariaDB.

External package location [None]: /<my-extra-packages>
##This is the path to the directory where the required packages are located.

If the required packages are located, then the following message is shown:

The following packages will be installed:
   Database server: /my-extra-packages/mysql-5.7.27-linux-glibc2.12-x86_64.tar.gz
   JDBC driver:
     - /my-extra-packages/mysql-connector-java-5.1.48.tar.gz

 External package: Ok

If the required packages are not found, then the following error message is showing:

External package: ERROR
 - ERROR: Couldn't find jdbc drivers in /my-extra-packages
 - ERROR: Looked for:
   mysql-connector-java-*.tar.gz
   mysql-connector-java-*.jar
   mariadb-java-client-*.jar
 - ERROR: Couldn't find database server package in /my-extra-packages
 - ERROR: Looked for:
   mysql-*-linux-glibc2.12-x86_64.tar.gz
   mariadb-*-linux-x86_64.tar.gz

This check allows you to configure and test HTTPS for the unravel UI. This check prompts you for the certificate, key, password, and hostname details that are used to access Unravel.

Use HTTPS to access unravel? (y/n) [Yes]:  
##If you answer “Yes”, you are prompted for the path to the certificate and key. Unravel uses this information to configure TLS during installation. 
##If you answer “No”, you are shown a warning message for confirmation.

The information provided is verified for the following:

If the Key and Certificate match
If the certificate is valid
If the certificate applies for the provided hostname

This check allows you to set the Unravel UI port and verify the connectivity.

-- Unravel default port
  
  Port number (integer) [3000]: 

  Do you want to test if the port is accessible? (y/n) [Yes]: 
    
  This will open port 3000 and listen for connection for 120 seconds.
  Use your browser to test if the Unravel UI will be accessible on that port.
    
  We have detected the following hostnames:
  - some.host.example
    
 Browse to: http://some.host.example:3000
    
 ATTENTION: This address is an example. You should test with the URL that will be used to access Unravel.

A connection on port 3000 is tried and established. If the connection is successful, Unravel Port Test: OK is shown on the browser, and Unravel port: Reached is shown on the server.

This check allows you to set a custom data directory and verify the access if the directories exist. You will always find the software location where you deploy the Unravel binaries. In this check, only the space and access are tested. That data location that you have configured will be used.

-- Unravel directories

  Software [/opt/unravel]: 

  Data [/opt/unravel/data]: 

  Directories: ERROR
  - OK: 33 GB of free disk space for software.
  - ERROR: SYSH0026: Space for data 33 GB is low, recommended minimum is 100 GB.

This check allows you to configure and test email. You are prompted for host and credentials, and the following items are tested:

Connectivity
Authentication, only if provided.
Optional: Send test mail.

Following is a sample:

-- Mail server (SMTP) configuration

Unravel can send notification and alert emails.
   This will allow you to configure and test connection to a SMTP server.
   Optionally, it can also send a test email.

   You will have to provide:
   - Protocol, hostname and port
   - Credentials if required



   Configure a SMTP server? (y/n) [No]: y

   SMTP hostname [None]: smtphostname.gmail.com

   SMTP port (usually 25 for clear text, 465 for SSL, 587 for STARTLS) (integer) [None]: 587

   Security protocol
   1- None
   2- SSL
   3- StartTLS

   Select one of the above [None]: 3
   Selected: StartTLS

   Authentication required? (y/n) [Yes]: y

   Username [None]: daemon@unraveldata.com

   Password [None] (no echo):

   From [None]: daemon@unraveldata.com

   To [None]: user@unraveldata.com

   Send test email (y/n) [No]: y

Note

For more information, refer to Using the Interactive Precheck utility.

The responses that you have provided for the configuration information are used to generate a configuration file. You can use this configuration file when you run the setup command to install and configure Unravel.

After you have completed the responses, you are prompted to confirm if you want to generate the bootstrap configuration file. Press ENTER if you want to generate the bootstrap configuration file.
```
-- Unravel bootstrap configuration

   Generate a unravel bootstrap configuration file? (y/n) [Yes]:
```
The bootstrap configuration file is generated and located at /tmp/unravel-interactive-precheck/unravel-bootstrap.yaml.

Install Unravel with the bootstrap configuration file.

<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --bootstrap /tmp/unravel-interactive-precheck/unravel-bootstrap.yaml

Apply the changes.

<Unravel installation directory>/unravel/manager config apply

Start all the services.

<unravel_installation_directory>/unravel/manager start

Check the status of services.
```
<unravel_installation_directory>/unravel/manager report 
```
The following service statuses are reported:
- OK: Service is up and running.
- Not Monitored: Service is not running. (Has stopped or has failed to start)
- Initializing: Services are starting up.
- Does not exist: The process unexpectedly disappeared. A restart will be attempted ten times.
You can also get the status and information for a specific service. Run the manager report command as follows:
```
<unravel_installation_directory>/unravel/manager report <service> 
```
For example: /opt/unravel/manager report auto_action

You can also manually install Unravel, Refer to 4. Run setup

5. Enable Transport Layer Security (TLS) for Unravel UI

Refer to Enabling Transport Layer Security (TLS) for Unravel UI.

6. Configure an HTTPS Load Balancer for Unravel Endpoint

Unravel LR endpoint should be available over a publically accessible HTTPS endpoint to receive messages from BigQuery PubSub. The Load Balancer is an easier and more secure method to push the log messages between Google Cloud Platform (GCP) and Unravel. Use the following instructions to configure an HTTPS load balancer for Unravel with public endpoint and SSL termination.

You must have the following information handy before you configure the Load Balancer:

Region and Zone where the Unravel VM is running.
Network and Subnet-network where the Unravel VM is running.
A valid SSL certificate in GCP.

Do the following to create a Load Balancer

Create an instance group. Refer to Create a managed instance group for detailed instructions.
- In the New unmanaged instance group page, ensure to keep the following items the same as that of Unravel VM.
  - Location > Region
  - Location > Zone
  - Network and Instances > Network
  - Network and Instances > SubNetwork
- Under Port Mapping, enter the following:
  - Port Name: http4043
  - Port Number: 4043
Set up an HTTPS Load Balancer. Refer to Set up an HTTPS Load Balancer for detailed instructions. Ensure to do the following:
- Under Name, update the name as unravel-loadbalancer.
- In Backends > New Backend > Instance groups, select the Unravel instance group that you had created in Step 1.
- Under Health check, do the following:
  - Select Create a health check, and then add the name as unravel-4043-hc
  - Update the Protocol as HTTP and Port as 4043.
  - Update the Request Path as /lr/status.
- Ensure that Port is set to 443 to allow HTTPS traffic.
After the Load Balancer is created, find the public IP address of the Load Balancer that is mentioned under Frontend section of the Load Balancer. Add the IP address of the Load Balancer to a valid DNS name.

7. Set up Unravel to receive BigQuery data automatically

Unravel can be set up to automatically create and configure resources in more than 100 projects at a time. You can either add single projects or multiple projects at a time for Unravel monitoring. These projects can be added either with customer-supplied credentials or with Unravel-generated credentials.

You must separately configure the BigQuery projects that you want to track from the Data page. The Data page on Unravel UI can show data for only up to 100 BigQuery projects.

Unravel ships the following resources, which are required to automatically set up Unravel to receive BigQuery data.

Terraform
Open-source software for infrastructure provisioning.
gcloud CLI
Set of tools to create and manage Google Cloud resources.

Note

You can also manually set up Unravel to receive BigQuery data. Refer to Set up Unravel to receive BigQuery data

Before you set Unravel to receive BigQuery data

BigQuery projects can be set in Unravel using either one of the following:

Unravel-generated credentials
Customer-supplied credentials

For projects added with Unravel-generated credentials, all the resources are created and handled by Unravel. However, for the customer-supplied credentials, you must create and provide all the resources.

Ensure to have the following handy before you add the projects:

Google account with required IAM role permissions for gcloud CLI authentication
IAM role permissions for gcloud CLI authentication
The Google account that is used to authenticate with gcloud CLI must have the following permissions to create and delete resources on GCP for tracking BigQuery. These permissions fit into the security principle of least privilege which means that these are the only permissions that are required to get the job done. If you do not have the following permissions, the Terraform script will not run properly.
Important
The IAM role permissions should be there for every project that the Google account user wants to integrate with Unravel.
iam.roles.create
iam.roles.delete
iam.roles.get
iam.roles.undelete
iam.roles.update
iam.serviceAccountKeys.create
iam.serviceAccountKeys.delete
iam.serviceAccountKeys.get
iam.serviceAccounts.create
iam.serviceAccounts.delete
iam.serviceAccounts.get
iam.serviceAccounts.getIamPolicy
iam.serviceAccounts.setIamPolicy
logging.sinks.create
logging.sinks.delete
logging.sinks.get
pubsub.subscriptions.create
pubsub.subscriptions.delete
pubsub.subscriptions.get
pubsub.topics.attachSubscription
pubsub.topics.create
pubsub.topics.delete
pubsub.topics.get
pubsub.topics.getIamPolicy
pubsub.topics.setIamPolicy
resourcemanager.projects.get
resourcemanager.projects.getIamPolicy
resourcemanager.projects.setIamPolicy
Execute the following commands from the Google Cloud shell to automate the creation of a role and bind it to the user.
Run the following to create the role.
gcloud iam roles create UnravelGoogleAuthRole --organization=<your-org-id-in-gcp> --title=UnravelGoogleAuthRole --description="Creation of role in GCP on an organization level for automated integrating bigquery projects with Unravel" --permissions=iam.roles.create,iam.roles.delete,iam.roles.get,iam.roles.undelete,iam.roles.update,iam.serviceAccountKeys.create,iam.serviceAccountKeys.delete,iam.serviceAccountKeys.get,iam.serviceAccounts.create,iam.serviceAccounts.delete,iam.serviceAccounts.get,iam.serviceAccounts.getIamPolicy,iam.serviceAccounts.setIamPolicy,logging.sinks.create,logging.sinks.delete,logging.sinks.get,pubsub.subscriptions.create,pubsub.subscriptions.delete,pubsub.subscriptions.get,pubsub.topics.attachSubscription,pubsub.topics.create,pubsub.topics.delete,pubsub.topics.get,pubsub.topics.getIamPolicy,pubsub.topics.setIamPolicy
For example:
gcloud iam roles create UnravelGoogleAuthRole --organization=592556919173 --title=UnravelGoogleAuthRole --description="Creation of role in GCP on an organization level for automated integrating bigquery projects with Unravel" --permissions=iam.roles.create,iam.roles.delete,iam.roles.get,iam.roles.undelete,iam.roles.update,iam.serviceAccountKeys.create,iam.serviceAccountKeys.delete,iam.serviceAccountKeys.get,iam.serviceAccounts.create,iam.serviceAccounts.delete,iam.serviceAccounts.get,iam.serviceAccounts.getIamPolicy,iam.serviceAccounts.setIamPolicy,logging.sinks.create,logging.sinks.delete,logging.sinks.get,pubsub.subscriptions.create,pubsub.subscriptions.delete,pubsub.subscriptions.get,pubsub.topics.attachSubscription,pubsub.topics.create,pubsub.topics.delete,pubsub.topics.get,pubsub.topics.getIamPolicy,pubsub.topics.setIamPolicy,resourcemanager.projects.get,resourcemanager.projects.getIamPolicy,resourcemanager.projects.setIamPolicy
Bind the role to a user.
gcloud organizations add-iam-policy-binding <your-org-id-in-gcp> --member='user:<user account>' --role='organizations/<unravel-organization-ID>/roles/UnravelGoogleAuthRole'
For example:
gcloud organizations add-iam-policy-binding 592556919173 --member='user:user@unraveldata.com' --role='organizations/592556919173/roles/UnravelGoogleAuthRole'
Project ID file in case you are integrating multiple projects at a time with Unravel.
Project ID text file
You can configure multiple projects with customer-supplied credentials. To implement this, you must create a text file containing the list of project IDs and provide the path of this file during integration. Each line in the text file must carry a single Project ID.
For example:
```
project-id-011
project-id-012
project-id-013
project-id-014
project-id-015
project-id-016
project-id-017
project-id-018
project-id-019
project-id-020
```
Customer-supplied credential files in case you are integrating projects using customer-supplied credentials.
Customer-supplied credentials directory
Each project, which you want to add for monitoring, must have a corresponding credentials file in the credentials directory with the same project ID and .json extension.
Important
The credentials file should have the same name as that of the project ID.
For example, you have the project-id-file with the following project IDs:
```
first-project
second-project
```
then the credential files in the credentials directory should have the service account keys JSON file as follows:
```
first-project.json
second-project.json
```
Log Receiver (LR) endpoint
This is required only when you are adding projects.
Subscription ID (Optional)
The subscriber ID that you want to configure with a pub/sub topic. This is optional. If this is not provided then the default subscription ID unravel-bigquery-sub is considered.
Note
If you have created the resources on GCP, then the Subscription ID is mandatory.

Setup Unravel to automatically configure BigQuery projects for Unravel monitoring

Stop Unravel.

<Unravel installation directory>/unravel/manager stop

Set the Log Receiver (LR) endpoint. This step is required only if you are using customer-supplied credentials to add the projects.
Important
Ensure that the LR end point supports HTTPS.
```
<Unravel installation directory>/unravel/manager config bigquery set-lr-endpoint <host> <port>
```
For example: /opt/unravel/manager config bigquery set-lr-endpoint myhostname.unraveldata.com 1234
Run <Unravel installation directory>/unravel/manager config bigquery show to verify. The following output is shown
```
— Running: config bigquery show
BigQuery support: Enabled
LR endpoint: https://myhostname.unraveldata.com:1234
No project configured.
```
Add BigQuery projects.
Configure the BigQuery projects that you want to be monitored by Unravel. Single or multiple BigQuery projects can be added for monitoring. You can configure the projects either with the customer-supplied credentials or with Unravel-generated credentials.
Also, refer to Remove BigQuery projects from Unravel
Add a single BigQuery project with customer-supplied credentials
When you configure a single project with customer-supplied credentials the credentials in the file are copied and encrypted.
Note
For customer-supplied credentials, you must configure the target endpoint of the Pub/Sub subscription ID to unravel LR HTTPS endpoint.
To add a single BigQuery project for Unravel monitoring with customer-supplied credentials, do the following:
Using a terminal, run the following command from the Unravel installation directory:
```
<Unravel installation directory>/unravel/manager config bigquery add <project-id> <subscription-id> <path/to/credentials-file>
```
For example:
/opt/unravel/manager config bigquery add my-project subscription-12345 /credentials/unravel.json
Note
You can use the same command to add more projects one at a time. In case you use the same project ID, the existing project gets updated.
Add a single BigQuery project with Unravel-generated credentials
Unravel can automatically create the appropriate resources and generate the credentials required to monitor BigQuery projects. To add a single BigQuery project for Unravel monitoring with Unravel managed credentials, do the following:
Using a terminal, run the following command from the Unravel installation directory:
```
<Unravel installation directory>/unravel/manager config bigquery add <project-id> <subscription-id> --create-credentials
```
For example:
/opt/unravel/manager config bigquery add my-project subcription-12345 --create-credentials
Add multiple BigQuery projects using customer-supplied credentials
You can configure multiple projects with customer-supplied credentials. To implement this, you must create a text file containing the list of project IDs and provide the path of this file in the following command. Each line in the text file must carry a single Project ID. Along with the project ID, you must also provide the subscription ID and the path to the customer-supplied credentials file. During the configuration, the credentials in the file are copied and encrypted.
Run the following command, to add multiple BigQuery projects for Unravel monitoring with customer-supplied credentials.
<Unravel installation directory>/unravel/manager config bigquery add --batch </path/to/project-id-file> [--subscription-id <sub>] --credentials <path/to/credentials-folder>
--subscription-id : This is an optional field when you add projects with Unravel-generated credentials. If you do not provide any value, the default unravel-bigquery-sub is considered.
Note
The --subscription-id is a mandatory field when you add projects with customer-supplied credentials since there is a different subscription ID.
</path/to/project-id-file>: The full path to the directory containing the project IDs.
<path/to/credentials-folder>: The full path to the directory containing the credentials files.
Each project must have a corresponding credentials file in the credentials directory, with the same name as that of the project ID and the .json extension.
Important
The credentials file should have the same name as that of project ID.
For example, you have the project-id-file with the following project IDs:
first-project second-project
then the credential files in the credentials directory should be named as follows:
first-project.json second-project.json
For example, you have a list of projects in project-id-file with a subscription ID as a-sub-id and your credentials file for each of the projects is listed in credentials-directory. Run the following command to add multiple projects with customer-supplied credentials:
manager config bigquery add --batch /opt/unravel/project-id-file --subscription-id a-sub-id --credentials /opt/unravel/credentials-directory
Run <Unravel installation directory>/unravel/manager config bigquery show to verify. The following output is shown:
$ /opt/unravel/manager config bigquery show -- Running: config bigquery show BigQuery support: Enabled LR endpoint: https://myhost.unraveldata.com:1234 Project: first-project credentials_file: /opt/unravel/data/conf/bigquery/credentials/first-project.json datapage: true subscription_id: a-sub-id Project: second-project credentials_file: /opt/unravel/data/conf/bigquery/credentials/second-project.json datapage: true subscription_id: a-sub-id Datapage projects: 2 out of 100. — Ok
Add multiple BigQuery projects with Unravel-generated credentials
Unravel can automatically create resources on the GCP side and generate the credentials required to monitor BigQuery projects.
Run the following command to add multiple BigQuery projects for Unravel monitoring with customer-supplied credentials:
<Unravel installation directory>/unravel/manager config bigquery add --batch </path/to/project-id-file> [--subscription-id <sub>] --create-credentials
Note
</path/to/project-id-file>: The full path to the directory containing the project IDs.
--subscription-id : This is an optional field. If you do not provide any value, the default unravel-bigquery-sub is considered.
For example, you have a list of projects in project-id-file with a subscription ID as a-sub-id. Run the following command to add multiple projects with Unravel-generated credentials:
manager config bigquery add --batch /opt/unravel/project-id-file --subscription-id a-sub-id --create-credentials
Run <Unravel installation directory>/unravel/manager config bigquery show to verify. The following output is shown:
$ /opt/unravel/manager config bigquery show -- Running: config bigquery show BigQuery support: Enabled LR endpoint: https://myhost.unraveldata.com:1234 Project: first-project credentials_file: /opt/unravel/data/conf/bigquery/credentials/first-project datapage: true integration: true subscription_id: a-sub-id Project: second-project credentials_file: /opt/unravel/data/conf/bigquery/credentials/second-project datapage: true integration: true subscription_id: a-sub-id Datapage projects: 2 out of 100. — Ok

Add projects to Data page.

You must separately configure the BigQuery projects that you want to track from the Data page. The Data page on Unravel UI can show data for only up to 100 BigQuery projects.

Also, refer to Remove BigQuery projects from Data page on Unravel UI.

To add single projects to Data page, run the following:
```
<Unravel installation directory>/unravel/manager config bigquery enable-datapage <project-id>
```
For example: /opt/unravel/manager config bigquery enable-datapage myproject
To add multiple projects to Data page, run the following:
```
<Unravel installation directory>/unravel/manager config bigquery enable-datapage --batch </path/to/project-id-file> 
```
For example: /opt/unravel/manager config bigquery enable-datapage --batch /opt/unravel/project-id-file

Run <Unravel installation directory>/unravel/manager config bigquery show to verify if the project IDs are enabled for Data page. The following sample output is shown:

$ /opt/unravel/manager config bigquery show
— Running: config bigquery show

BigQuery support: Enabled
LR endpoint: https://myhost.unraveldata.com:1234
Project: project-id-file
integration: true
subscription.id: a-sub-id

Project: first-project
credentials.file: /opt/unravel/data/conf/bigquery/credentials/first-project
subscription.id:  a-sub-id

Project: second-project
credentials.file: /opt/unravel/data/conf/bigquery/credentials/second-project
subscription.id:  a-sub-id

Datapage projects: 2 out of 100

Integrate BigQuery projects for Unravel monitoring. This is a mandatory step for Unravel-generated credentials.
Note
When the following command is run, you can configure all the projects added for Unravel monitoring at once i.e projects with customer-supplied credentials as well as with Unravel managed credentials.
```
<Unravel installation directory>/unravel/manager config bigquery integrate
```
A URL will be provided in the output.
Note
If you want to skip the interactive gcloud authentication by Unravel and handle the gcloud authentication on your own, then run the command as follows:
```
<Unravel installation directory>/unravel/manager config bigquery integrate --skip-authorization
```
Authenticate gcloud CLI.
1. On a Google Chrome browser, copy the URL provided in the output, and in the sign-in dialog box, click Allow. Ensure to sign in to the gcloud CLI from the account that is authenticated with the required permissions.
2. From the Sign in to the glcoud CLI box, click Copy button to copy the authorization code.
3. Go back to the terminal and paste the authorization code in the Enter authorization code field and press ENTER. This will run the following actions in the background:
  - Authenticate the user with Google Cloud.
  - Configure the required resources on the GCP.
  - Encrypt the credentials (service account keys) and then integrate them with Unravel.
  - Integrate all the added BigQuery projects with Unravel.
  - Securely sign out the end user from the gcloud session.

Apply the changes.

<Unravel installation directory>/unravel/manager config apply

Start Unravel.

<Unravel installation directory>/unravel/manager start

Also, see a complete list of manager commands for BigQuery installation.

Also, see Set up Unravel to receive BigQuery data manually.

Remove BigQuery projects from Unravel

Note

When you remove a BigQuery project from Unravel, then the associated data also gets deleted from ElasticSearch.

Stop Unravel.

<Unravel installation directory>/unravel/manager stop

Run the following command from the Unravel installation directory.
- For single project
```
<Unravel installation directory>/unravel/manager config bigquery remove <project-ID>
```
  For example: /opt/unravel/manager config bigquery remove my-project
- For multiple projects
```
<Unravel installation directory>/unravel/manager config bigquery remove --batch </path/to/project-id-file>
```
  For example: /opt/unravel/manager config bigquery remove --batch /opt/unravel/my-projects.txt

Apply the changes.

<Unravel installation directory>/unravel/manager config apply

Start Unravel.

<Unravel installation directory>/unravel/manager start

Remove BigQuery projects from Data page on Unravel UI

Note

When you remove a BigQuery project from Data page, then the associated data also gets deleted from ElasticSearch.

Stop Unravel.

<Unravel installation directory>/unravel/manager stop

Run the following command from the Unravel installation directory.
- For single project
```
<Unravel installation directory>/unravel/manager config bigquery delete-datapage <project-ID>
```
  For example: /opt/unravel/manager config bigquery delete-datapage --batch my-project
- For multiple projects
```
<Unravel installation directory>/unravel/manager config bigquery delete-datapage --batch </path/to/project-id-file>
```
  For example: /opt/unravel/manager config bigquery delete-datapage --batch /opt/unravel/my-projects.txt

Apply the changes.

<Unravel installation directory>/unravel/manager config apply

Start Unravel.

<Unravel installation directory>/unravel/manager start

8. Verify BigQuery integration

To verify BigQuery integration with Unravel, do the following:

On the GCP console, run test queries from the project integrated with Unravel.
Using a supported web browser, navigate to Unravel URL (For example, https://<unravel-host>:3000) and log onto Unravel UI using the credentials.
Navigate to Jobs tab > Applications , and under Application type, select BigQuery. The details of the test queries run from the GCP console will be listed under the All tab.

In this section:

Home

Installing Unravel for GCP BigQuery

Install and set up Unravel on the GCP instance

1. Create and configure the GCE

Note

Configure the GCE instance

2. Download Unravel

Important

3. Deploy Unravel

Important

4. Install Unravel

Note

5. Enable Transport Layer Security (TLS) for Unravel UI

6. Configure an HTTPS Load Balancer for Unravel Endpoint

7. Set up Unravel to receive BigQuery data automatically

Note

Before you set Unravel to receive BigQuery data

Important

Important

Note

Setup Unravel to automatically configure BigQuery projects for Unravel monitoring

Important

Note

Note

Note

Important

Note

Note

Note

Note

Note

8. Verify BigQuery integration

Search results