A service principal (SPN) is required to authenticate Unravel to the Databricks API. This non-human identity enables secure, automated access for monitoring and data collection, following best practices for automation and security. Service principals provide a secure way for Unravel to interact programmatically with Databricks APIs. This allows Unravel to monitor and optimize your Databricks environment without relying on individual user credentials.

To create and configure a service principal:

After creating the service principal, configure essential permissions to ensure Unravel operates with the minimum necessary privileges.

To configure API permissions:

By default, non-admin tokens-including those used by service principals-do not have access to all users’ clusters and jobs in a Databricks workspace. Additionally, Databricks does not currently provide a built-in method to grant permissions to clusters and jobs that have not yet been created.

To address this, you must deploy a serverless function (AWS Lambda or Azure Function) in your environment. This function should run every minute with an admin token and automatically grant the following permissions to the Unravel service principal for any newly created resources:

For assistance with setting up this serverless function, contact Unravel support.

An initialization script deploys a JAR-based Unravel sensor onto all compute instances within the workspace. This deployment is managed automatically via a cluster-scoped policy, so developers do not need to modify their code.

{  "clusters": [    {      "init_scripts": [        {          "workspace": {            "destination": "$INIT_SCRIPT_PATH"          }        }      ]    }  ]}

A PrivateLink endpoint allows Unravel sensor data to reach the Unravel data plane without any internet exposure.

To set up a PrivateLink endpoint:

This configuration ensures that all Unravel sensor data is transmitted privately and securely, meeting strict network and compliance requirements.