Welcome to Unravel for Databricks Setup
Unravel for Databricks is a complete Data Observation platform that provides AI-driven Insights to help you tune and troubleshoot your data pipelines while ensuring data quality and cost-optimization policies are achieved. This guide is designed to assist you with integrating your Databricks workspaces to ensure secure and effective data monitoring.
Get Started
Sign up for Unravel for Databricks. Book a demo to see the product in action.
Keep the following information from your Databricks account:
Databricks workspace ID
Databricks Personal Access Token (PAT)
Databricks workspace instance URL
Workspace tier (Standard, Premium, Enterprises, Dedicated)
For information on how to get these, see here.
The reference architecture for Unravel SaaS with Databricks is illustrated in the diagram below. It highlights the main components and their interactions:
Unravel fetches cluster, job, and other required information with the help of Databricks API.
Unravel Sensor is deployed on each monitored cluster to collect cluster metrics.
Unravel UI displays aggregated results, recommendations, insights, and more for the users.
Connectivity Descriptions
Reference connections from the architecture diagram | Method | Authentication | Encryption-In-Transit |
|---|---|---|---|
1 | API | Choice of either Databricks Personal Access Token or Service Principal Name | TLS over HTTPS, port 443 Unravel connects to Databricks API endpoint |
2 | API | Choice of Unravel basic auth or Azure Active Directory (AAD) auth (via SAML 2) | TLS over HTTPS, port 443 Databricks connects to Unravel API endpoint |
3 | UI | Choice of Unravel basic auth or Active Directory (via SAML 2) for access authentication. Session-based JWT during usage | HTTPS, port 443 Client connects to Unravel UI |
Check that the Unravel sensor is deployed on all relevant clusters.
Monitor logs in the Unravel UI.
Verify that cluster and job data is being collected and displayed correctly.
Unravel supports setting role-based access controls.
Refer to the Unravel Documentation for help or contact Unravel Support.
Unravel Data is SOC2 compliant. Black Duck checks and validates vulnerabilities before the software is certified as Generally Available (GA).