Prerequisites for Unravel BigQuery Standard (Free)

Home

Prerequisites for Unravel BigQuery Standard (Free)

After you sign up for BigQuery Standard (Free), ensure to complete the prerequisites required for integrating GCP projects. The integration of GCP projects with BigQuery consists of the following significant aspects:

Exporting the GCP billing data of the projects that you want to integrate with Unravel
Creating resources on GCP that are required for the integration.

This topic provides instructions on creating resources on GCP and setting the export of billing of data, which are the main prerequisites for the Unravel Standard (Free) account sign-up.

Setting exporting of the GCP billing data for Unravel

You need the following to set the exporting of the GCP billing data for Unravel:

Either the Billing Accounts Costs Manager role or the Billing Account Administrator role on the target Cloud Billing account.
BigQuery user role for the project in which you want to export the billing data.

Run the following steps to set up the exporting of the GCP billing data for the projects monitored by Unravel.

From the GCP console, choose a project to export the GCP billing data.
Note
Ensure this is also the same project that you want Unravel to monitor.
Export the GCP billing data of the billing account you want to integrate with Unravel to the chosen project. Do the following to export the billing data:
1. Search for the Billing Exports page and select the Cloud billing account you want to integrate with Unravel.
2. In the Detailed Usage Cost section, click the Edit Settings button and choose the project you selected to export the GCP billing data.
3. From the Dataset ID drop-down, select a dataset to export the billing data or create a new dataset.
4. Confirm that the project and the dataset in which you want to export the billing data are correct, and then click Save.
5. In the Billing export page > Detailed usage cost section, check and ensure that the Detailed usage cost is shown as Enabled and your selected project ID and dataset are displayed.
The table creation process in the chosen dataset will take a few minutes.
Note the following billing export details and keep them handy:
- Dataset ID
- Table ID
- Project ID of the monitoring project
- Project ID of the admin project
  Note
  The administrator project manages the BigQuery Reservations resources and is the primary billing source for these resources. This project need not be the same project with BigQuery jobs. Google recommends creating a dedicated project for Reservations resources.

Creating GCP resources for projects monitored by Unravel

You can add two monitoring projects and one administrator project for Unravel monitoring. Terraform scripts are used to facilitate the smooth integration of Unravel with BigQuery projects. These scripts help you set up various components and resources.

The following IAM permissions are required for creating the GCP resources. Whoever executes the Terraform script for creating GCP resources for integration must have these permissions.

iam.roles.create	iam.serviceAccountKeys.create	logging.sinks.create	pubsub.subscriptions.create	resourcemanager.projects.get
iam.roles.delete	iam.serviceAccountKeys.delete	logging.sinks.delete	pubsub.subscriptions.delete	resourcemanager.projects.getIamPolicy
iam.roles.get	iam.serviceAccountKeys.get	logging.sinks.get	pubsub.subscriptions.get	resourcemanager.projects.setIamPolicy
iam.roles.undelete	iam.serviceAccounts.create		pubsub.topics.attachSubscription
iam.roles.update	iam.serviceAccounts.delete		pubsub.topics.create
	iam.serviceAccounts.get		pubsub.topics.delete
	iam.serviceAccounts.getIamPolicy		pubsub.topics.get
	iam.serviceAccounts.getIamPolicy		pubsub.topics.get
	iam.serviceAccounts.setIamPolicy		pubsub.topics.getIamPolicy
			pubsub.topics.setIamPolicy

Follow the instructions to create GCP resources using Terraform scripts for projects that you want to be integrated with Unravel for monitoring:

Use a terminal and clone the following repository:
https://github.com/unraveldata-org/unravel-terraform-scripts/tree/develop
Go to unravel-terraform-scripts/saas/bigquery folder.
Duplicate the example input file named input.tfvars.example. Rename it to input.tfvars.
```
cp input.tfvars.example input.tfvars
```
Configure Terraform backend. Unravel recommends configuring a central storage for your Terraform state file.
1. Copy the backend.tf.example file using this command:
```
cp backend.tf.example backend.tf
```
2. Update the Google Storage Path, to which the user who runs Terraform has access, in the backend.tf file.
Using a VM editor, update the following in the input.tfvars file:
- Project ID
  Enter the project IDs that you had specified for exporting billing data in the monitoring_projects_list variable. You can add the following project IDs
  - Project ID in which you have exported the billing data.
  - Project ID that you want to be monitored by Unravel. You can enter a maximum of two project IDs.
- Administrator project ID
  Enter your administrator project ID in the admin_project_id variable.
- Service account
  Enter the service account for the variable service_account.
  Tip
  The service account is available from the Unravel UI > Manage icon > BigQuery configuration, which you can copy.
Navigate to the Terraform directory and run the following commands to log in to the gcloud application. A user with the essential IAM permissions can only log on to the gcloud application.
```
cd unravel-terraform-scripts/saas/bigquery
gcloud init
gcloud auth-application-default login
```
After the login, run the following commands:
```
terraform init
terraform plan --var-file=input.tfvars
terraform apply --var-file=input.tfvars
```
The GCP resources are created. You can now log in to the Unravel BigQuery account using the login credentials that you have received in the email and integrate the BigQuery projects.