v 4798 Release notes
Release information
Release date | April 2025 |
4798 Software |
New features
Cost Savings Dashboard
The Cost Savings Dashboard helps you monitor, track, and analyze cost reductions across jobs and interactive clusters. It detects key changes, such as configuration updates and code modifications, and calculates their effect on cost savings over different time frames. Unlike traditional cost monitoring tools that provide only high-level expense tracking, this app identifies specific optimizations that reduce costs. It provides insights at the job, cluster, and signature levels, allowing you to pinpoint the most effective cost-saving methods.
Databricks User Accountability Dashboard
The Databricks User Accountability Dashboard provides visibility into individual user activities, helping organizations track usage, identify inefficiencies, and reduce costs. It offers insights into user performance, cost impact, and resource utilization, enabling better decision-making. It includes a User Rating Analysis to categorize users based on optimization levels, a Workspace Analysis to track cost and performance across different workspaces, and User Drilldowns for detailed individual user insights. With these features, teams can optimize operations and improve accountability.
Auto Scale Insights
Introduced new Auto-Scale Insights that recommend minimum worker reductions based on CPU utilization metrics and task execution patterns, ensuring optimized resource allocation.
Role-Based Authentication for Billing
Unravel now supports AWS role-based authentication for billing. This feature enables Unravel to retrieve billing data using the assigned instance role.
Support for RHEL 8.x and later
This release supports RHEL 8 and later versions; RHEL 7.x and earlier are no longer supported.
The following table contains key issues addressed in the 4.7.9.8 release.
ID | Description |
|---|---|
Billing | |
DT-4973 | The cost displayed does not match the actual billing data in certain occasions. |
Home | |
DT-4872 | The homepage does not load due to an error with the 'utilization_df' variable. |
Jobs | |
DT-5266 | The Photon_Benefit insight is not being generated for jobs as expected. |
DT-5275 | The RightSizing insight does not provide a driver recommendation for applications with low usage. |
SQL | |
DT-4315 | The full description is not visible when viewing the query plan from the SQL tab in full-screen mode. |
DT-4348 | Large plans for certain SQL Execution IDs don't render and cause the UI to crash. |
DT-4753 | Stage IDs are not displayed in the SQL tab. |
Workflows | |
DT-5002 | CPU metric values are exceeding the expected limit of 100. |
Workspace Configuration | |
DT-5074 | Users are unable to edit the workspace when KeyVault is enabled. |
CUSTOMER-3073 | The domain name should replace the IP address as the host in NGUI log files. |
DT-5086 | The updater. frequency.minutes parameter should be unset when KeyVault is enabled to avoid conflicts in configuration. |
DT-5095 | The region list is not populating correctly in all scenarios when adding a workspace. |
The upcoming releases will include the following key fixes to enhance user experience. It is important to note that while these issues exist, there is no immediate critical impact on using the product, and users can continue to utilize its functionality with confidence.
ID | Description |
|---|---|
Cost | |
DT-2968, DT-2967 | Incorrect filters are applied while redirecting from budgets to chargeback. |
Compute | |
DT-4755 | The Job Name/ID filter displays all jobs regardless of status in the Running or Finished tabs. |
DT-1879 | Redirection from the DBX Budget page to the Compute page displays empty data for the selected tags. |
Home | |
DT-1853, DT-1871 | An incorrect date range is selected when redirecting from the Budget page to other pages. |
DT-2920, DT-3060, DT-2821 | Instead of displaying No data present for a selected date range when data is unavailable for a selected date range, empty widgets are shown. |
Insights | |
DT-2006 | Recommendations are provided for a failed pipeline when users utilize multiple tasks with shared job clusters, and one of the tasks fails. |
DT-3012 | The insight preview page displays incorrect costs for job signatures. |
Reports | |
DT-3122 | Incorrect count of events shown on TopX Report. |
Spark | |
DT-3104 | Multiple instances of the same events are being generated for a single Spark application cluster. |
DT-3103, DT-3026 | There is a mismatch in the executor count when garbage collection (GC) events occur. |
SaaS (Free) | |
DT-2037 | In the Databricks Standard (free) environment, there is an issue where the User Flow badge obstructs pagination. |
UI | |
UIX-6281 | The cost comparison for all the instances is not displayed on the Pipeline detail page. |
Workflows | |
DT-2104 | Sorting is incorrect when the list contains both strings starting with capital and small case letters, |
DT-4756 | The Cluster Name filter only supports prefix-based search and does not allow keyword search. |
Workspace Manager | |
DT-4766 | The Workspace table is missing the Region and SQL fields and columns. |
Azure bill integration
Actual bill data is not live; once configured, it will be available from Azure once every day. The cost for a few clusters may be updated after a couple of days in the bill.
One record per cluster per day is maintained, even if the cluster is restarted multiple times within a day or cluster sessions span across multiple days.
If there are issues with Azure billing, the cost data will not be updated on Unravel.
Tags on cost pages come from actual cost data, while tags on the compute page come from Spark configuration. There is a possibility that the tags on these pages don’t match. This issue will be fixed in the upcoming release.
Billing
Some discrepancies may occur in cost calculations due to differences between the user time zone displayed on the Compute page and the UTC-based aggregation on the Billing page. (DT-2350).
In certain scenarios, the budget status may inaccurately display as Ok even when the budget has been exceeded. This discrepancy occurs when Azure billing is enabled and is a known limitation. Notifications for the budget can be delayed by two or three days due to the minimum 24-hour delay in receiving bills. (DT-3091)
Compute
Jobs by status graphs in the Trends tab display spark application details and not the job details. Our development is actively looking into this design limitation and efforts are underway to address this in future updates to enhance the product's capabilities. (DT-2008)
Data is inconsistent between the Compute page and the Cost page in some cases when Azure bill is enabled. This is a known limitation because of Azure bill data not being live. Our team is actively looking into this limitation and efforts are underway to address this issue in future updates. (DT-3078)
Data
If tables are created with the same name, accessed, deleted, and re-created, and if those tables are re-accessed, then their query and app count do not match. (DATAPAGE-502)
Home
Home page does not display alerts on the UI when there is missing ROI data for a single day. (DT-2509)
Hovering on Total Cost Trend on the Summary tab of the Home page may display inaccurate date information. (DT-2408)
Insights Preview
In some cases, there is a discrepancy in the runs displayed on the Job Runs page for the selected job. When you click on View runs, only runs with
dbx_cluster kindhaving a Spark app ID are listed. The discrepancy is due to data inconsistency between entries ofdbx_cluster kindanddb kindand this is an expected behavior. (DT-3042)Workflows
The current implementation has a limitation where the wrong run count is displayed for the job ID when sorting by run count in the Workflows > Jobs section. This discrepancy is currently under investigation by our development team, and active efforts are being taken to resolve this issue. (UIX-6526)
Our development team is actively investigating the following Known issues and are working towards resolving them. It is important to note that while these issues exist, there is no immediate critical impact on using the product, and users can continue to utilize its functionality with confidence.
Bug ID | Description | Workaround |
|---|---|---|
Billing | ||
DT-2079 | The cluster cost for DBX Azure does not match the billing report for clusters with higher costs. | NA |
Compute | ||
PIPELINE-1636 | Inconsistent data is displayed for the cluster Duration and Start Time on the Compute page. | NA |
CUSTOMER-3017 | The job duration displayed in the TopX section of the "Longest Running Jobs" on the Job Compute graph is incorrect. | NA |
Cost | ||
UIX-5624 | Data is not displayed when you click the Optimize button corresponding to OTHERS for the Cost > Chargeback results shown in the table. | NA |
DT-1094 | The No data available message is displayed on the Compute page after navigating from the Trends and Chargeback pages with Tag filters. | NA |
Datapage | ||
DATAPAGE-473 | For Hive metastore 3.1.0 or earlier versions, the creation time of partitions is not captured if a partition is created dynamically. Therefore, the Last Day KPI for the partition section is not shown in Unravel. | NA |
Insights | ||
DT-1987 | There is a mismatch in the computation of costs for fleet and spot instances in Databricks clusters. This issue arises due to the unavailability of the exact node type in the cluster info response. | NA |
Performance | ||
ASI-933 | Duration update for clusters is delayed in case of lag in processing the data. | NA |
ASI-936 | In case of lag in processing the data App Time is missing in the timings tab of many applications. | NA |
Spark | ||
PIPELINE-1616 | If the Spark job is not running for Databricks, the values for the Duration and End time fields are not updated on the Databricks Run Details page. | NA |
DT-2012 | Incorrect details are displayed on the AppSummary > Job Run page when a user repairs a previously failed job. The displayed information may not accurately reflect the repaired job's details. | NA |
DT-1742 | The timezone for the NodeRightSizing insight event is inconsistent in the Spark details page. | NA |
DT-2029 | Applications in a success state may inaccurately display an associated job in a running state instead of transitioning to a failed state. | NA |
DT-3122 | The TopX Report displays an incorrect count of events. | NA |
DT-5551 | The SparkWorker encounters an exception when attempting to delete job data. This issue occurs specifically in Unravel deployments that use PostgreSQL as the backend database. | NA |
UI | ||
PIPELINE-1935 | In the Pipeline details page, when you select the data for a specific date, all instances are displayed instead of displaying only the instances within a selected date. | NA |
UIX-6263 | The | NA |
Workflows | ||
DT-1461, PIPELINE-1939, PIPELINE-1940, DT-1093 | The UI and data exhibit inconsistencies, including problems with job run details, issues related to multiple workflow runs and UTC timestamps , empty content in workflow job pages and issues with filter values and duration display. | NA |
