v 4.7.9.7 Release notes
Release information
Release date | November 2024 |
4797 Software | |
Configuration properties |
New features
AWS Billing and Databricks System Tables Integration
In this release, Unravel introduces AWS billing integration and System Tables integration, enabling precise cost tracking for Databricks entities based on accurate data directly from AWS billing. The insights now leverage accurate AWS DBU data, enhancing the precision of performance and cost analyses.
Databricks System Tables support in Azure Billing
Databricks System Tables DBU usage is now supported in Azure providing cost visibility across assets governed by system tables.
Databricks SQL Observability
This release introduces observability of Databricks SQL, expanding Unravel’s monitoring and insights to include SQL workloads. This provides users with performance tracking and optimization insights for SQL warehouses and queries.
Chageback trends
This release introduces significant updates and new capabilities to the Chargeback feature for more detailed cost tracking and visualization across business units and cost centers. The new Chargeback trends page provides a detailed view of cloud bill distribution by service, allowing users to identify major cost drivers across services. Additionally, it includes a total cost breakdown for compute resources, a daily trend view by cost center and business unit to track usage patterns, and a compute cost history table to support in-depth analysis and long-term budget planning.
RBAC Support
RBAC support is now expanded to Cost Explorer and Job Insights pages.
Cost Explorer
A new Cluster Type field has been added, allowing users to filter and display only the relevant clusters based on the selected type.
The following table contains key issues addressed in the 4.7.9.6 release.
ID | Description |
|---|---|
Compute | |
CLOUD-711 | The application status is incorrectly shown as "SUCCESS" even when multiple jobs are still in the "RUNNING" state. |
Data | |
DT-3123 | The default enabled checkbox in the Data page section is incorrectly disabled under tables. |
Home | |
DT-3096, DT-3095, DT-2837 | The Annualized savings under the sections highlighting the Most Savings in the Topx tab, is incorrectly displaying as zero dollars in certain scenarios. |
DT-2836 | Wastage cost is incorrectly showing more than the Cluster cost in some cases. |
SaaS | |
DT-4336 | Azure cost bills are not downloading as expected. |
Workflows | |
DT-4793 | The cost savings for Photon insights are calculated lower than expected for auto-scale clusters. |
DT-4801 | The Rightsizing event displays an incorrect number of runs. |
DT-4809 | The summary savings displayed on the Insights page are incorrect. |
DT-2125 | The UI shows a cost discrepancy for the Executor Idle time detected insight in the Databricks version 14.2 with Photon enabled. |
DT-3037 | Node rightsizing filters are not applied as expected on the Insights Preview page. |
The upcoming releases will include the following key fixes to enhance user experience. It is important to note that while these issues exist, there is no immediate critical impact on using the product, and users can continue to utilize its functionality with confidence.
ID | Description |
|---|---|
Cost | |
DT-2968, DT-2967 | Incorrect filters are applied while redirecting from budgets to chargeback. |
Compute | |
DT-4755 | The Job Name/ID filter displays all jobs regardless of status in the Running or Finished tabs. |
DT-1879 | Redirection from the DBX Budget page to the Compute page displays empty data for the selected tags. |
Home | |
DT-1853, DT-1871 | An incorrect date range is selected when redirecting from the Budget page to other pages. |
DT-2920, DT-3060, DT-2821 | Instead of displaying No data present for selected date range when data is unavailable for a selected date range, empty widgets are shown. |
Insights | |
DT-2006 | Recommendations are provided for a failed pipeline when users utilize multiple tasks with shared job clusters, and one of the tasks fails. |
DT-3012 | The insight preview page displays incorrect costs for job signatures. |
Reports | |
DT-3122 | Incorrect count of events shown on TopX Report. |
Spark | |
DT-3104 | Multiple instances of the same events are being generated for a single Spark application cluster. |
DT-3103, DT-3026 | There is a mismatch in the executor count when garbage collection (GC) events occur. |
SaaS (Free) | |
DT-2037 | In the Databricks Standard (free) environment, there is an issue where the User Flow badge obstructs pagination. |
UI | |
UIX-6281 | The cost comparison for all the instances is not displayed on the Pipeline detail page. |
Workflows | |
DT-2104 | Sorting is incorrect when the list contains both strings starting with capital and small case letters, |
DT-4756 | The Cluster Name filter only supports prefix-based search and does not allow keyword search. |
Workspace Manager | |
DT-4766 | The Workspace table is missing the Region and SQL fields and columns. |
Azure bill integration
Actual bill data is not live; once configured, it will be available from Azure once every day. The cost for a few clusters may be updated after a couple of days in the bill.
One record per cluster per day is maintained, even if the cluster is restarted multiple times within a day or cluster sessions span across multiple days.
If there are issues with Azure billing, the cost data will not be updated on Unravel.
Tags on cost pages come from actual cost data, while tags on the compute page come from Spark configuration. There is a possibility that the tags on these pages don’t match. This issue will be fixed in the upcoming release.
Billing
Some discrepancies may occur in cost calculations due to differences between the user time zone displayed on the Compute page and the UTC-based aggregation on the Billing page. (DT-2350).
In certain scenarios, the budget status may inaccurately display as Ok even when the budget has been exceeded. This discrepancy occurs when Azure billing is enabled and is a known limitation. Notifications for budget can be delayed by two or three days due to the minimum 24-hour delay in receiving bills. (DT-3091)
Compute
Jobs by status graphs in the Trends tab display spark application details and not the job details. Our development is actively looking into this design limitation and efforts are underway to address this in future updates to enhance the product's capabilities. (DT-2008)
Data is inconsistent between Compute page and Cost page in some cases when Azure bill is enabled. This is a known limitation because of Azure bill data not being live. Our team is actively looking into this limitation and efforts are underway to address this issue in the future updates. (DT-3078)
Data
If tables are created with the same name, accessed, deleted, and re-created, and if those tables are re-accessed, then their query and app count do not match. (DATAPAGE-502)
Home
Home page does not display alerts on the UI when there is missing ROI data for a single day. (DT-2509)
Hovering on Total Cost Trend on the Summary tab of the Home page may display inaccurate date information. (DT-2408)
Insights Preview
In some cases, there is a discrepancy in the runs displayed in the Job Runs page for the selected job. When you click on View runs, only runs with
dbx_cluster kindhaving a Spark app id are listed. The discrepancy is due to data inconsistency between entries ofdbx_cluster kindanddb kindand this is an expected behavior. (DT-3042)Workflows
The current implementation has a limitation where the wrong run count is displayed for the job ID when sorting by run count in the Workflows > Jobs section. This discrepancy is currently under investigation by our development team, and active efforts are being taken to resolve this issue. (UIX-6526)
Our development team is actively investigating the following Known issues and are working towards resolving them. It is important to note that while these issues exist, there is no immediate critical impact on using the product, and users can continue to utilize its functionality with confidence.
Bug ID | Description | Workaround |
|---|---|---|
Billing | ||
DT-2079 | The cluster cost for DBX Azure does not match the billing report for clusters with higher costs. | NA |
Compute | ||
PIPELINE-1636 | Inconsistent data is displayed for the cluster Duration and Start Time on the Compute page. | NA |
CUSTOMER-3017 | The job duration displayed in the TopX section of the "Longest Running Jobs" on the Job Compute graph is incorrect. | NA |
Cost | ||
UIX-5624 | Data is not displayed when you click the Optimize button corresponding to OTHERS for the Cost > Chargeback results shown in the table. | NA |
DT-1094 | The No data available message is displayed on the Compute page after navigating from the Trends and Chargeback pages with Tag filters. | NA |
Datapage | ||
DATAPAGE-473 | For Hive metastore 3.1.0 or earlier versions, the creation time of partitions is not captured if a partition is created dynamically. Therefore, the Last Day KPI for the partition section is not shown in Unravel. | NA |
Insights | ||
DT-1987 | There is a mismatch in the computation of costs for fleet and spot instances in Databricks clusters. This issue arises due to the unavailability of the exact node type in the cluster info response. | NA |
Performance | ||
ASI-933 | In the Lag setup, the Duration is not updated for running applications. The Duration should be updated every 15 minutes. | NA |
ASI-936 | In the Lag setup, the App Time data is missing in the Timing tab of many applications. | NA |
Spark | ||
PIPELINE-1616 | If the Spark job is not running for Databricks, the values for the Duration and End time fields are not updated on the Databricks Run Details page. | NA |
DT-2012 | Incorrect details are displayed on the AppSummary > Job Run page when a user repairs a previously failed job. The displayed information may not accurately reflect the repaired job's details. | NA |
DT-1742 | The timezone for the NodeRightSizing insight event is inconsistent in the Spark details page. | NA |
DT-2029 | Applications in a success state may inaccurately display an associated job in a running state instead of transitioning to a failed state. | NA |
DT-3122 | The TopX Report displays an incorrect count of events. | |
UI | ||
PIPELINE-1935 | In the Pipeline details page, when you select the data for a specific date, all instances are displayed instead of displaying only the instances within a selected date. | NA |
UIX-6263 | The | NA |
Workflows | ||
DT-1461, PIPELINE-1939, PIPELINE-1940, DT-1093 | The UI and data exhibit inconsistencies, including problems with job run details, issues related to multiple workflow runs and UTC timestamps , empty content in workflow job pages and issues with filter values and duration display. | NA |
