Skip to main content

Home

Configuring Bigquery Data Page

You can configure the Data page for BigQuery when you install and set up Unravel for BigQuery monitoring.

Create a role for an Unravel project with the required permission for Unravel monitoring. This role is assigned to the Unravel service account that is used to authenticate Unravel applications.

  1. On the GCP Console, go to the Roles page.

  2. Using the drop-down list at the top of the page, select the Unravel project in which you want to create a role.

  3. Click Create Role and enter a Name, Title, and Description for the role. Preferably keep the role name as unravel for easy identification. The description is optional, which can be maintained as Unravel monitoring.

    Select Role launch stage as General Availability.

    Caution

    The role name cannot be changed after the role is created.

  4. Click Add Permissions.

  5. In the Add Permissions dialog box, filter and select the following permissions for the role, and then click Add.

    Note

    These permissions are only for fetching data from BigQuery to Unravel Data page.

    Permission

    Description

    bigquery.datasets.get

    bigquery.routines.get

    bigquery.routines.list

    bigquery.tables.getData

    bigquery.tables.list

    Permissions to get the tables and partitions metadata. These permissions are required for the Data page.

    bigquery.jobs.create

    Permissions to execute queries on BigQuery to fetch the metadata about the tables and partitions. These permissions are required for the Data page.

  6. Click Create. The role is created.

Refer to you Install and set up Unravel for BigQuery monitoring to complete the setup process for BigQuery, if not done already.

Queries executed by Unravel to fetch table and partitions details

Unravel executes the following queries to fetch the table and partition details. These queries are charged based on the pricing model of BigQuery. Bigquery charges for a minimum of 10 MB of data processed for the following queries. However, if the data processing is more than 10 MB it will be charged accordingly. More information about the pricing can be found here.

  • Query to fetch table information

    SELECT t.dataset_id, t.table_id, t.creation_time, t.last_modified_time, t.row_count,t.size_bytes, t.type,i.table_type,i.ddl 
FROM `<project_id>.<dataset_id>.__TABLES__` 
AS t JOIN `<project_id>.<dataset_id>.INFORMATION_SCHEMA.TABLES` AS i ON t.table_id = i.table_name and t.dataset_id = i.table_schema;

  • Query to fetch partition information

    SELECT table_catalog,table_schema,table_name,partition_id, total_rows,total_logical_bytes, total_billable_bytes, last_modified_time, storage_tier FROM `<project_id>.<dataset_id>.INFORMATION_SCHEMA.PARTITIONS`;
Enable/Disable BigQuery for Data page

By default, BigQuery is enabled for the Data page. To disable BigQuery for Data Page, do the following:

  1. Stop Unravel.

    <Unravel installation directory>/unravel/manager stop

  2. Set the following property to disable BigQuery for Datapage:

    <Unravel installation directory>/unravel/manager config properties set com.unraveldata.bigquery.tableinfo.enabled FALSE

  3. Apply the changes.

    <Unravel installation directory>/unravel/manager config apply

  4. Start Unravel.

    <Unravel installation directory>/unravel/manager start
In this section: