Unravel for BigQuery Setup Guide

Overview

Unravel for Google Cloud BigQuery delivers observability and AI-driven optimization for BigQuery environments. The platform analyzes job, query, and billing metadata to highlight actionable insights, uncover inefficiencies, and provide recommendations that improve performance and control cloud costs.

Architecture Overview

Unravel for Google Cloud BigQuery utilizes a secure, scalable SaaS architecture designed to enable the monitoring and optimization of BigQuery workloads without requiring the movement or exposure of sensitive data.

User access: Users interact with Unravel through the web interface, REST API clients, or receive notifications by email, Slack, Microsoft Teams, or GitHub Actions. Authentication uses single sign-on (SSO), local credentials, or tokens.
Unravel SaaS environment: Unravel runs on Google Kubernetes Engine (GKE) within a dedicated VPC on Google Cloud. Core services include the UI and API service, notification service, and worker processes. All collected metadata and metrics are stored securely in our internal data store.
Data collection: Unravel worker services run in the SaaS environment and connect securely to your BigQuery environment. Workers access information schema views, billing export tables, and REST API endpoints to collect metadata, job details, and billing information. All access uses only the necessary IAM roles and secure, authenticated channels.
Customer data: Unravel collects authorized metadata, job information, and billing details. Raw table data, query results, or sensitive content do not leave your environment. All communication uses HTTPS/TLS for security.

Before you begin

Before you begin onboarding, ensure you have:

Project IDs for all BigQuery projects you want to monitor.
Important
Project and Account Limits
Item
Limit
Monitoring projects
100 projects
Admin project
1 project
Billing account
1 per setup
Permissions for the required Google Cloud IAM roles
Choose your authentication method
Billing export information—Project ID, Dataset ID, and Table ID—for cost analytics (required only if you intend to use Unravel’s billing and FinOps features)

Item	Limit
Monitoring projects	100 projects
Admin project	1 project
Billing account	1 per setup

Permissions for the required Google Cloud IAM roles

Roles and permissions for the views in the information schema and APIs

The following predefined Google Cloud IAM roles must be granted to your authentication identity (service account or workload identity):

Monitoring projects Permissions

View / API Permissions

Role	Permissions Granted	Views/APIs
roles/bigquery.resourceViewer	bigquery.jobs.listAll bigquery.jobs.get	INFORMATION_SCHEMA.JOBS INFORMATION_SCHEMA.JOBS_TIMELINE
roles/bigquery.metadataViewer	bigquery.tables.get bigquery.tables.list bigquery.routines.get bigquery.routines.list bigquery.datasets.get	INFORMATION_SCHEMA.COLUMNS INFORMATION_SCHEMA.TABLE_STORAGE INFORMATION_SCHEMA.TABLES INFORMATION_SCHEMA.TABLE_OPTIONS INFORMATION_SCHEMA.SCHEMATA_OPTIONS
roles/bigquery.resourceViewer	resourcemanager.projects.get bigquery.reservationAssignments.search	Method: projects.get Method: projects.getAncestry Method: projects.locations.searchAllAssignments
roles/bigquery.user	bigquery.transfers.get	Method: transferConfigs.list

Operation Permissions

Role	Permissions Granted	Usage
roles/bigquery.user	bigquery.jobs.create serviceusage.services.use	Allows Unravel to run BigQuery jobs. Note You do not need to grant the bigquery.jobs.create permission to monitored projects if the polling project is already configured.
roles/bigquery.resourceAdmin	recommender.bigqueryCapacityCommitmentsInsights.get recommender.bigqueryCapacityCommitmentsInsights.list recommender.bigqueryCapacityCommitments Recommendations.get recommender.bigqueryCapacityCommitments Recommendations.list	Allows you to receive insights and recommendations from Google.
No predefined roles are available.	recommender.bigqueryPartitionClusterRecommendations.get recommender.bigqueryPartitionClusterRecommendations.list	Allows you to receive insights and recommendations from Google.

Polling projects Permissions

A polling project is a project Unravel uses to run queries and retrieve metadata.

Role	Permissions Granted	Usage
roles/bigquery.user	bigquery.jobs.create	Allows Unravel to run BigQuery jobs.

If Unravel polls jobs data from a custom table (for example, during data de-identification), you also need to grant the following permission to the polling project.

Role	Permissions Granted	Usage
roles/bigquery.dataViewer	bigquery.datasets.get bigquery.tables.getdata bigquery.tables.get	Allows you to retrieve metadata from a custom table.

Admin project Permissions

View / API Permissions

Role	Permissions Granted	Views/APIs
roles/bigquery.resourceViewer	bigquery.reservations.list bigquery.reservationAssignments.list	INFORMATION_SCHEMA.RESERVATIONS INFORMATION_SCHEMA.ASSIGNMENTS

Operation Permissions

Role	Permissions Granted	Usage
roles/bigquery.user	bigquery.jobs.create	Allows Unravel to run BigQuery jobs.

Billing project Permissions

View / API Permissions

Role	Permissions Granted	Usage
roles/bigquery.dataViewer	bigquery.datasets.get bigquery.tables.getdata bigquery.tables.get	Billing Export Table [Detailed usage cost] (gcp_billing_export_resource_v1_<BILLING_ACCOUNT_ID>)

Operation Permissions

Role	Permissions Granted	Usage
roles/bigquery.user	bigquery.jobs.create	Allows Unravel to run BigQuery jobs.

Service, Method, and Permission Requirements for VPC Service Controls in Unravel

Monitoring projects Permissions

View / API Permissions

Service	Method	Permissions Granted	Views/APIs
bigquery.googleapis.com		bigquery.jobs.listAll	INFORMATION_SCHEMA.JOBS INFORMATION_SCHEMA.JOBS_TIMELINE
bigquery.googleapis.com		bigquery.tables.get bigquery.tables.list bigquery.routines.get bigquery.routines.list bigquery.datasets.get	INFORMATION_SCHEMA.COLUMNS INFORMATION_SCHEMA.TABLE_STORAGE INFORMATION_SCHEMA.TABLES INFORMATION_SCHEMA.TABLE_OPTIONS INFORMATION_SCHEMA.SCHEMATA_OPTIONS
bigquery.googleapis.com	ReservationService.SearchAllAssignments		Method: projects.get Method: projects.getAncestry Method: projects.locations.searchAllAssignments
bigquerydatatransfer.googleapis.com	* (only option with GCP)		Method: transferConfigs.list
recommender.googleapis.com	* (only option with GCP)		INFORMATION_SCHEMA.RECOMMENDATIONS INFORMATION_SCHEMA.INSIGHTS

Operation Permissions

Service	Methods	Permissions Granted	Usage
bigquery.googleapis.com		bigquery.jobs.create	Allows Unravel to run BigQuery jobs.

Note

Although these services are not listed in the official VPC Service Controls supported method restrictions, they may still appear as selectable options in the service list.

bigquerydatatransfer.googleapis.com
recommender.googleapis.com

Polling projects Permissions

A polling project is a project Unravel uses to run queries and retrieve metadata.

Service	Methods	Permissions Granted	Usage
bigquery.googleapis.com		bigquery.jobs.create	Allows Unravel to run BigQuery jobs.

If Unravel polls jobs data from a custom table (for example, during data de-identification), you also need to grant the following permission to the polling project.

Service	Methods	Permissions Granted	Usage
bigquery.googleapis.com		bigquery.datasets.get bigquery.tables.getdata bigquery.tables.get	Allows you to retrieve metadata from a custom table.

Admin project Permissions

View / API Permissions

Service	Methods	Permissions Granted	Views/APIs
bigquery.googleapis.com	ReservationService.GetCapacityCommitment ReservationService.ListCapacityCommitments	bigquery.reservations.list bigquery.reservationAssignments.list bigquery.capacityCommitments.list	INFORMATION_SCHEMA.RESERVATIONS INFORMATION_SCHEMA.ASSIGNMENTS INFORMATION_SCHEMA.CAPACITY_COMMITMENT

Operation Permissions

Service	Methods	Permissions Granted	Usage
bigquery.googleapis.com		bigquery.jobs.create	Allows Unravel to run BigQuery jobs.

Billing project Permissions

View / API Permissions

Service	Methods	Permissions Granted	Usage
bigquery.googleapis.com		bigquery.datasets.get bigquery.tables.getdata bigquery.tables.get	Billing Export Table [Detailed usage cost] (gcp_billing_export_resource_v1_<BILLING_ACCOUNT_ID>)

Operation Permissions

Service	Methods	Permissions Granted	Usage
bigquery.googleapis.com		bigquery.jobs.create	Allows Unravel to run BigQuery jobs.

Choose your Authentication Method

Unravel supports two authentication methods when accessing your GCP resources for BigQuery monitoring. Choose one based on organizational policy.

Provide credentials via Workload Identity Federation or Single Key (Service Account Key).

Workload Identity Federation: Securely allows Unravel to access your GCP resources without sharing keys.
Single Key (Service Account JSON key): Provide Unravel with a GCP service account key file with the required permissions.

Setting up exporting of the GCP billing data for Unravel

Complete this step only if you have not already configured BigQuery billing export in your environment.

You need the following to set up the exporting of the GCP billing data for Unravel:

Either the Billing Accounts Costs Manager role or the Billing Account Administrator role on the target Cloud Billing account.
BigQuery user role for the project in which you want to export the billing data.

Run the following steps to set up the exporting of the GCP billing data for the projects monitored by Unravel.

From the GCP console, choose a project to export the GCP billing data.
Note
Ensure this is also the same project that you want Unravel to monitor.
Export the GCP billing data of the billing account you want to integrate with Unravel to the chosen project. Do the following to export the billing data:
1. Search for the Billing Exports page and select the Cloud billing account you want to integrate with Unravel.
2. In the Detailed Usage Cost section, click the Edit Settings button and choose the project you selected to export the GCP billing data.
3. From the Dataset ID drop-down, select a dataset to export the billing data or create a new dataset.
4. Confirm that the project and the dataset in which you want to export the billing data are correct, and then click Save.
5. In the Billing export page > Detailed usage cost section, check and ensure that the Detailed usage cost is shown as Enabled and your selected project ID and dataset are displayed.
The table creation process in the chosen dataset will take a few minutes.
Note the following billing export details and keep them handy:
- Dataset ID
- Table ID
- Billing Export project ID
- Billing Export Dataset ID
- Billing Export Table Name
- Note
  The administrator project manages the BigQuery Reservations resources and is the primary billing source for these resources. This project need not be the same project with BigQuery jobs. Google recommends creating a dedicated project for Reservations resources.

Getting started with Unravel for BigQuery

After your Unravel account is set, you will receive an email with the following details. Keep these handy to access your Unravel UI and integrate the BigQuery projects.
- URL to access Unravel (BigQuery) SaaS
- Login credentials
- Service account
Run the following steps to integrate your BigQuery projects for Unravel monitoring:
1. Click the URL that you have received in the email and access the Unravel UI.
2. Enter the login credentials received by email on the login page. The Unravel UI is displayed.
3. From the top right, click and select BigQuery configuration. The BigQuery account setting page is displayed.
4. Specify the following details:
  - In the Monitored projects section, specify the Project IDs you have integrated using Terraform.
  - In the Administrator Projects section, specify the admin project ID.
  - In the Google Cloud Billing Export Data section, specify the Project ID, Dataset ID, and Table ID. Refer to Prerequisites.Prerequisites for BigQuery SaaS setup
5. Click the Save button. The BigQuery projects are successfully integrated, and you can monitor these projects from the Unravel UI. In case any errors are shown, you must fix it and click the Save button again.

In this section:

Home