On the GCP Console, go to the Roles page.

Using the drop-down list at the top of the page, select the Unravel project in which you want to create a role.

Click Create Role and enter a Name, Title, and Descriptionfor the role. Preferably keep the role name as unravel for easy identification. The description is optional, which can be maintained as Unravel monitoring.

Select Role launch stage as General Availability.

Click Add Permissions.

In the Add Permissions dialog box, filter and select the following permissions for the role, and then click Add.

Click Create. The role is created.

On the GCP Console, go to the Create service account page.

Select the Unravel project.

Under Service Account Details, enter a service account name to display on the GCP Console. Preferably keep the service account name as unravel-service-account.

Optional: Enter a description of the service account. Preferably keep the description as Unravel monitoring.

Click CREATE AND CONTINUE.

Under Grant this service account access to this project, select the role, which was created in Step 1 with the associated permissions.

Click Done. The service account is created, and the role is now attached to the service account.

On the GCP Console, go to the Service accounts page.

Select the Unravel project.

Click the email address of the service account which you have created for Unravel.

Click the Keys tab.

Click the Add key drop-down menu, then select Create new key.

Select  JSON as the Key type and click Create. A service account key file is downloaded. You can download this only once. Store the key file securely and transfer this file to Unravel node.

Following is a sample of the downloaded key file:

{
  "type": "service_account",
  "project_id": "unravel-test-337406",
  "private_key_id": "b111a5ad112bf0000d5c03b28b9a1d1f8ac31af",
  "private_key": "-----BEGIN PRIVATE KEY-----\n<privatekey>\n-----END PRIVATE KEY-----\n",
  "client_email": "unravel-service-account@unravel-test-337406.iam.gserviceaccount.com",
  "client_id": "112180000016964232364",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-email"
}

The file path of the service account key must be set in Unravel using the manager utility. See Add BigQuery details in Unravel.

On the GCP console, go to thePub/Sub topics page.

Click Create topic.

In the Topic ID field, enter an ID for your topic and click Create Topic. Preferably specify unravel-bigquery as the topic ID. The topic is now listed in the list of topics.

On the GCP console, go to the Logs Router page.

Click Create Sink and provide a name and description for the sink. Preferably name the Sink as unravel-sink.

Under Sink Destination, from the Sink Service dropdown, select the Cloud Pub/Sub topic.

From Select a Cloud Pub/Sub topic, select the Cloud Pub/Sub topic that you had created for Unravel project in Step 5.

Under Choose logs to include in the Sink, add a filter to determine the logs that must be included in the log routing sink. For example

resource.type="bigquery_resource" AND (protoPayload.methodName="jobservice.insert" OR protoPayload.methodName="jobservice.jobcompleted")

In this example:

Click Create Sink.

On the GCP console, go to the Pub/Sub topics page.

Select the topic that was created in Step 5.

Scroll down to the Subscriptions tab and click Create Subscription > Create Subscription.

In the Subscription ID text box, provide a name for the subscription. Preferably keep the Subscription ID as unravel-bigquery-sub.

Under the Delivery type, select Push or Pull.

If you have selected Push, then in the Endpoint URL text box, provide the Log Receiver (LR) endpoint URL for PUSH messages. LR server receives the logs from google Pub/Sub and is the entry point to unravel. The criterion for the LR endpoint URLs are as follows:

You must specify the endpoint URL in the following format:

https://<unravel-hostname>:4443/logs/bigquery/<gcp-project-id>/bigquery/bigquery

For example:

https://playground-bq-4770.unraveldata.com:4443/logs/bigquery/unravelsaas-329506/bigquery/bigquery

Provide the expiry time for the subscription.

Under Message ordering, select the Order messages with an ordering key option. This will allow the messages tagged with the same ordering key to be received in the published order.

Optionally, you can filter the messages via Dead lettering and Retry Policy options.

Click Create.

Stop Unravel.

<Unravel installation directory>/unravel/manager stop

Run the following command from the Unravel installation directory.

<Unravel installation directory>/unravel/manager config bigquery add

Enter the following details when prompted:

For example:

Project id: unravel-test-331210
Subscription id: bigquery-sub
Credentials file: /tmp/unravel-test-331210-eoadad3feeea.json

Optional: Using the manager utility, set the frequency to poll Project data such as project name, state, etc. By default, it is one day that is 1440 minutes.

<Unravel installation directory>/unravel/manager config properties set com.unraveldata.bigquery.project.details.poll.delay.mins 720

Refer BigQuery properties to edit and set other BigQuery properties. BigQuery

Apply the changes.

<Unravel installation directory>/unravel/manager config apply

Start Unravel.

<Unravel installation directory>/unravel/manager start