Skip to main content

Home

Configuring BigQuery Data Page

You can configure the Data page for BigQuery after you install and set up Unravel for BigQuery monitoring.

To enable the Data page, include the BigQuery projects you wish to monitor from the Data page. The Data page on Unravel UI can show data for up to 100 BigQuery projects. You can add single projects as well as multiple projects to the Data page.

Also, refer to Remove BigQuery projects from the Data page on Unravel UI.

  1. Add projects to the Data page.

    • To add single projects to the Data page, run the following:

      <Unravel installation directory>/unravel/manager config bigquery enable-datapage <project-id>

      For example: /opt/unravel/manager config bigquery enable-datapage myproject

    • To add multiple projects to the Data page, run the following:

      <Unravel installation directory>/unravel/manager config bigquery enable-datapage --batch </path/to/project-id-file> 

      For example: /opt/unravel/manager config bigquery enable-datapage --batch /opt/unravel/project-id-file

  2. Based on your chosen method of resource creation, (Manual, external terraform, Unravel managed resources), do the following:

    • Manual creation of resources

      If you have manually created resources, you should modify the IAM role by adding bigquery.tables.getData permissions to the role.

    • Creation of resources using external terraform

      For the resources created using an external terraform, you should add the project IDs to the input.tfvars file as elements in a list under the datapage_project_ids keyword and run the terraform apply command.

    • Unravel managed resources

      For Unravel managed resources, run the integrate command. The bigquery.tables.getData permission, which is required to enable the Data page, is added to the existing permission list.

      <Unravel installation directory>/unravel/manager config bigquery integrate

      /opt/unravel/manager config bigquery integrate

      Note

      An error may be shown when running Integrate or terraform apply command with external Terraform scripts. Refer to Error shown when running Integrate or terraform apply command with external Terraform scripts for the solution.

  3. Run <Unravel installation directory>/unravel/manager config bigquery show to verify if the project IDs are enabled for the Data page. The following sample output is shown:

    • API Pull/Push polling method

      /opt/unravel/manager config bigquery show
      -- Running: config bigquery show
      BigQuery support: Enabled
      
      Data access:
      Mode: pull
      Polling (seconds): Default
      
      Billing data location:
      Project id: unravel-flat-rate-test
      Dataset id: all_billing_data
      Table name: gcp_billing_export_resource_v1_016A85_0733E3_979331
      
      Authentication:
      Mode: multi
      Project: bq-test-project
      Terraform integration: Enabled
      
      Projects:
      test-admin:
      Is admin: True
      Is monitoring: False
      Last modified: 2023-11-10 12:21:49
      test-mon1:
      Subscription: unravel-bigquery-sub
      Is admin: False
      Is monitoring: True
      Last modified: 2023-11-10 12:22:05
      test-mon2:
      Subscription: unravel-bigquery-sub
      Is admin: False
      Is monitoring: True
      Last modified: 2023-11-10 12:22:12
      test-res:
      Subscription: unravel-bigquery-sub
      Is admin: True
      Is monitoring: True
      Last modified: 2023-11-10 12:21:57
      unravel-flat-rate-test:
      Subscription: mallik-test
      Is admin: False
      Is monitoring: True
      Last modified: 2023-11-10 12:22:30
      
      Datapage projects: 1 out of 100.
      -- OK
    • INFORMATION_SCHEMA based polling method

      `/opt/unravel/manager config bigquery show
      -- Running: config bigquery show
      BigQuery support: Enabled
      
      Data access:
      Mode: schema
      Polling (seconds): 300
      Polling threads: 5
      Lookback days: 90
      Max delay (seconds): 1800
      Locations: US
      
      Billing data location:
      Project id: unravel-flat-rate-test
      Dataset id: all_billing_data
      Table name: gcp_billing_export_resource_v1_016A85_0733E3_979331
      
      Authentication:
      Mode: vm
      Project: bq-test-project
      Terraform integration: Enabled
      
      Projects:
      test-admin:
      Is admin: True
      Is monitoring: False
      Last modified: 2023-11-10 10:31:56
      test-mon1:
      Is admin: False
      Is monitoring: True
      Last modified: 2023-11-10 10:32:14
      test-mon2:
      Is admin: False
      Is monitoring: True
      Last modified: 2023-11-10 10:32:22
      test-res:
      Is admin: True
      Is monitoring: True
      Last modified: 2023-11-10 10:32:07
      unravel-flat-rate-test:
      Is admin: False
      Is monitoring: True
      Last modified: 2023-11-10 10:32:27
      
      Datapage projects: 1 out of 100.
      -- OK

Note

When you remove a BigQuery project from the Data page, then the associated data also gets deleted from OpenSearch.

  1. Stop Unravel.

    <Unravel installation directory>/unravel/manager stop
  2. Run the following command from the Unravel installation directory.

    • For single project

      <Unravel installation directory>/unravel/manager config bigquery delete-datapage <project-ID>

      For example: /opt/unravel/manager config bigquery delete-datapage my-project

    • For multiple projects

      <Unravel installation directory>/unravel/manager config bigquery delete-datapage --batch </path/to/project-id-file>

      For example: /opt/unravel/manager config bigquery delete-datapage --batch /opt/unravel/my-projects.txt

  3. Based on your chosen method of resource creation (Manual, external terraform, Unravel managed resources), do the following:

    • Manual creation of resources

      If you have manually created resources, you should modify the IAM role by deleting the bigquery.tables.getData permissions from the role.

    • Creation of resources using external terraform

      For the resources created using an external terraform, you should remove the project IDs added to the input.tfvars file as elements in a list under the datapage_project_ids keyword and run the terraform apply command.

    • Unravel managed resources

      For Unravel managed resources, run the integrate command. The bigquery.tables.getData permission, which is required to enable the Data page, is removed from the existing permission list.

      <Unravel installation directory>/unravel/manager config bigquery integrate

      /opt/unravel/manager config bigquery integrate

      Note

      An error may be shown when running Integrate or terraform apply command with external Terraform scripts. Refer to Error shown when running Integrate or terraform apply command with external Terraform scripts for the solution.

  4. Apply the changes.

    <Unravel installation directory>/unravel/manager config apply
  5. Start Unravel.

    <Unravel installation directory>/unravel/manager start