Skip to main content

Home

Configuring BigQuery Data Page

You can configure the Data page for BigQuery after you install and set up Unravel for BigQuery monitoring.

Enable Data page for BigQuery

To enable the Data page, include the BigQuery projects you wish to monitor from the Data page. The Data page on Unravel UI can show data for up to 100 BigQuery projects. You can add single projects as well as multiple projects to the Data page. Data (BigQuery)

Also, refer to Remove BigQuery projects from the Data page on Unravel UI.

  1. Add projects to the Data page.

    • To add single projects to the Data page, run the following:

      <Unravel installation directory>/unravel/manager config bigquery enable-datapage <project-id>

      For example: /opt/unravel/manager config bigquery enable-datapage myproject

    • To add multiple projects to the Data page, run the following:

      <Unravel installation directory>/unravel/manager config bigquery enable-datapage --batch </path/to/project-id-file>

      For example: /opt/unravel/manager config bigquery enable-datapage --batch /opt/unravel/project-id-file

  2. Based on your chosen method of resource creation, (Manual, external terraform, Unravel managed resources), do the following:

    • Manual creation of resources

      If you have manually created resources, you should modify the IAM role by adding bigquery.tables.getData permissions to the role.

    • Creation of resources using external terraform

      For the resources created using an external terraform, you should add the project IDs to the input.tfvars file as elements in a list under the datapage_project_ids keyword and run the terraform apply command.

    • Unravel managed resources

      For Unravel managed resources, run the integrate command. The bigquery.tables.getData permission, which is required to enable the Data page, is added to the existing permission list.

      <Unravel installation directory>/unravel/manager config bigquery integrate

      /opt/unravel/manager config bigquery integrate

      Note

      An error may be shown when running Integrate or terraform apply command with external Terraform scripts. Refer to Error shown when running Integrate or terraform apply command with external Terraform scripts for the solution.

  3. Run <Unravel installation directory>/unravel/manager config bigquery show to verify if the project IDs are enabled for the Data page. The following sample output is shown:

    • API Pull/Push polling method 

      /opt/unravel/manager config bigquery show
-- Running: config bigquery show
BigQuery support: Enabled

Data access:
Mode: pull
Polling (seconds): Default

Billing data location:
Project id: unravel-flat-rate-test
Dataset id: all_billing_data
Table name: gcp_billing_export_resource_v1_016A85_0733E3_979331

Authentication:
Mode: multi
Project: bq-test-project
Terraform integration: Enabled

Projects:
test-admin:
Is admin: True
Is monitoring: False
Last modified: 2023-11-10 12:21:49
test-mon1:
Subscription: unravel-bigquery-sub
Is admin: False
Is monitoring: True
Last modified: 2023-11-10 12:22:05
test-mon2:
Subscription: unravel-bigquery-sub
Is admin: False
Is monitoring: True
Last modified: 2023-11-10 12:22:12
test-res:
Subscription: unravel-bigquery-sub
Is admin: True
Is monitoring: True
Last modified: 2023-11-10 12:21:57
unravel-flat-rate-test:
Subscription: mallik-test
Is admin: False
Is monitoring: True
Last modified: 2023-11-10 12:22:30

Datapage projects: 1 out of 100.
-- OK

    • INFORMATION_SCHEMA based polling method 

      `/opt/unravel/manager config bigquery show
-- Running: config bigquery show
BigQuery support: Enabled

Data access:
Mode: schema
Polling (seconds): 300
Polling threads: 5
Lookback days: 90
Max delay (seconds): 1800
Locations: US

Billing data location:
Project id: unravel-flat-rate-test
Dataset id: all_billing_data
Table name: gcp_billing_export_resource_v1_016A85_0733E3_979331

Authentication:
Mode: vm
Project: bq-test-project
Terraform integration: Enabled

Projects:
test-admin:
Is admin: True
Is monitoring: False
Last modified: 2023-11-10 10:31:56
test-mon1:
Is admin: False
Is monitoring: True
Last modified: 2023-11-10 10:32:14
test-mon2:
Is admin: False
Is monitoring: True
Last modified: 2023-11-10 10:32:22
test-res:
Is admin: True
Is monitoring: True
Last modified: 2023-11-10 10:32:07
unravel-flat-rate-test:
Is admin: False
Is monitoring: True
Last modified: 2023-11-10 10:32:27

Datapage projects: 1 out of 100.
-- OK
Disable BigQuery projects from the Data page

Note

When you remove a BigQuery project from the Data page, then the associated data also gets deleted from OpenSearch.

  1. Stop Unravel.

    <Unravel installation directory>/unravel/manager stop

  2. Run the following command from the Unravel installation directory.

    • For single project

      <Unravel installation directory>/unravel/manager config bigquery delete-datapage <project-ID>

      For example: /opt/unravel/manager config bigquery delete-datapage my-project

    • For multiple projects

      <Unravel installation directory>/unravel/manager config bigquery delete-datapage --batch </path/to/project-id-file>

      For example: /opt/unravel/manager config bigquery delete-datapage --batch /opt/unravel/my-projects.txt

  3. Based on your chosen method of resource creation (Manual, external terraform, Unravel managed resources), do the following:

    • Manual creation of resources

      If you have manually created resources, you should modify the IAM role by deleting the bigquery.tables.getData permissions from the role.

    • Creation of resources using external terraform

      For the resources created using an external terraform, you should remove the project IDs added to the input.tfvars file as elements in a list under the datapage_project_ids keyword and run the terraform apply command.

    • Unravel managed resources

      For Unravel managed resources, run the integrate command. The bigquery.tables.getData permission, which is required to enable the Data page, is removed from the existing permission list.

      <Unravel installation directory>/unravel/manager config bigquery integrate

      /opt/unravel/manager config bigquery integrate

      Note

      An error may be shown when running Integrate or terraform apply command with external Terraform scripts. Refer to Error shown when running Integrate or terraform apply command with external Terraform scripts for the solution.

  4. Apply the changes.

    <Unravel installation directory>/unravel/manager config apply

  5. Start Unravel.

    <Unravel installation directory>/unravel/manager start
In this section: