v4.7.9.5 Release notes

Release information

Release date	April 2024
Hotfix release date	July 2024
4795 Software	Downloads
4795 Hotfix Software
Configuration properties	Properties

New features

New Insights preview
A new Insights preview tab is introduced in the Workflow page. This tab offers visibility into your job performance, enabling you to identify inefficiencies, bottlenecks, failures, and optimization opportunities with precision. Equipped with detailed metrics and actionable insights, the insights tab enhances workflow efficiency and promotes operational success. For more details, see Insights.
Unravel Assistant
Introducing Unravel Assistant, your AI assistant for optimizing Databricks jobs. The Unravel Assistant is integrated with the Job Insights tab under the Workflow page and provides a convenient way to dive deeper into your workloads and spending to get insights and recommendations from Unravel to improve performance and reduce costs. Unravel Assistant answers questions in plain language, allowing you to effortlessly analyze performance and cost metrics and make informed decisions to enhance productivity and efficiency in your Databricks workflows.
Home page updates
- A new Productivity meter is introduced in the Summary tab, offering a detailed assessment of the operational efficiency within your workflow. This meter provides an indication of the productivity levels, ranging from Snail paced to Lightning, allowing you to identify areas for improvement and optimize resource utilization. The Productivity meter is a valuable tool for enhancing operational effectiveness and achieving peak performance across your workflow. For more information, see Summary.
- New dashboards are introduced in the Topx tab, helping you to efficiently manage resources by identifying under-utilized clusters and jobs, monitoring long-running processes, and optimizing costs within both All-Purpose Compute and Job Compute categories. For more information, see TopX.
- New metrics for Total ROI and Productivity boost are introduced in the Optimize tab, providing insights into the potential return on investment and productivity gains achievable through resource optimization. Additionally, new visuals for Code optimization allow you to visualize potential productivity boosts in hours, allowing you to make informed decisions, enhance performance, and maximize cost savings. For more information, see Optimize.
Unravel Billing page update
Unravel's billing page has been updated to introduce the new Pro Plan with Pay-In-Advance (PIA). This latest pricing model offers you the flexibility to purchase credits upfront with discounts. With PIA, you can monitor your purchased credits, opening and closing credit balances, total credit usage, and adjusted credits throughout the month. The transition to PIA ensures that you have access to more accurate billing, aligned with your individual usage patterns and requirements. For more details, see Unravel Billing for Databricks platform.

Improvements and enhancements

Cost Explorer page
Backend changes have been implemented in the Cost Explorer page to improve the accuracy of insights. You now receive more precise real-time costs in your trends, drill-downs, and budget tabs.

Issues fixed

The following table contains key issues addressed in the 4.7.9.5 release.

ID	Description
Cost
DT-2116	On the budget tab under the cost page, the search bar is displaying numeric values with the equal sign after selecting filters.
Home
DT-2672	The trends widget displays an incorrect savings percentage.
DT-2778	The ROI pipeline is failing for the last 7 days due to missing data.
DT-2851	Incorrect untapped / annualized savings displayed on the Optimize tab.
Spark
UIX-6523	The Sort by Write feature is currently not functioning as expected in the Spark details page.
DT-2123	On the Spark detail page, the date is overlapping with the drop-down icon.
Workflows
CUSTOMER-2712	RBAC filter is not functioning properly on the workflow - job runs page, job details page, and compute details page.
DT-2124	The timing displayed on the workflows page becomes incorrect when a user navigates from the chargeback page to the workflows page.
UIX-6274	In the Job Runs tab, the Cost and Duration filter values remain unchanged even after modifying filters in the left panel or switching between the All, Finished, and Running options.

Upcoming fixes

The upcoming releases will include the following key fixes to enhance user experience. It is important to note that while these issues exist, there is no immediate critical impact on using the product, and users can continue to utilize its functionality with confidence.

ID	Description
Auto Action
DT-3033	The date-time format in the policy-violation downloaded CSV file is incorrect.
Billing
DT-3186	The Billed Cost Over Time graph includes the first day of the next month on the X-axis, despite displaying only the data for the selected period. This issue does not affect functionality or data accuracy.
DT-3171	The downloaded CSV file name should be suffixed with the month of the year for which billing data is downloaded.
Cost
UIX-6305	The Others category is displayed twice in legends when the number of clusters exceeds 1000 in the Chargeback page.
DT-3130	In the Trends tab, the time displayed in the last x days selection has a time value greater than the current time on the trends page.
DT-2968, DT-2967	Incorrect filters applied while redirecting from budgets to chargeback.
Compute
DT-2083	The Total Allocated Key Performance Indicators (KPIs) for Vcore and memory are not visible in the Compute > Trends page.
UIX-6321	All jobs in the running status are displayed in the Finished tab under Job Runs instead of showing only the finished jobs.
Data
DT-3123	In the Data page section, the default enabled checkbox is incorrectly disabled under tables.
Home
DT-3096, DT-3095, DT-2837	The Annualized savings under the sections highlighting the Most Savings in the Topx tab, is incorrectly displaying as zero dollars in certain scenarios.
DT-2920, DT-3060, DT-2821	Instead of displaying No data present for selected date range when data is unavailable for a selected date range, empty widgets are shown.
DT-2836	Wastage cost is incorrectly showing more than the Cluster cost in some cases.
Insights
DT-2006	Recommendations are provided for a failed pipeline when users utilize multiple tasks with shared job clusters, and one of the tasks fails.
DT-2125	The UI shows a cost discrepancy for the Executor Idle time detected insight in the Databricks version 14.2 with Photon enabled.
DT-3037	Node right sizing filters are not being applied on the Insights Preview page.
DT-3012	The insight preview page displays incorrect costs for job signatures.
Reports
DT-3122	Incorrect count of events shown on TopX Report.
Spark
UIX-6567	Filters are not retained when switching between the Inefficient and All tabs.
DT-3104	Multiple instances of the same events are being generated for a single Spark application cluster.
DT-3103, DT-3026	There is a mismatch in the executor count when garbage collection (GC) events occur.
DT-3099	the Task Time and App Time fields are empty for certain jobs
DT-2981	Some cost deviations are observed between Spark hitdocs and Databricks clusters.
SaaS (Free)
DT-2037	In the Databricks Standard (free) environment, there is an issue where the User Flow badge obstructs pagination.
UI
UIX-6281	The cost comparison for all the instances is not displayed on the Pipeline detail page.
Unravel Assistant
AI-134, AI-138	The Unravel Assistant provides incorrect responses to certain questions related to productivity and data skew jobs. The team is actively working on resolving this issue.
Workflows
DT-2104	Sorting is incorrect when the list contains both strings starting with capital and small case letters,

Limitations

Azure bill integration
- Actual bill data is not live; once configured, it will be available from Azure once every day. The cost for a few clusters may be updated after a couple of days in the bill.
- One record per cluster per day is maintained, even if the cluster is restarted multiple times within a day or cluster sessions span across multiple days.
- If there are issues with Azure billing, the cost data will not be updated on Unravel.
- Tags on cost pages come from actual cost data, while tags on the compute page come from Spark configuration. There is a possibility that the tags on these pages don’t match. This issue will be fixed in the upcoming release.
Billing
- Some discrepancies may occur in cost calculations due to differences between the user time zone displayed on the Compute page and the UTC-based aggregation on the Billing page. (DT-2350).
- In certain scenarios, the budget status may inaccurately display as Ok even when the budget has been exceeded. This discrepancy occurs when Azure billing is enabled and is a known limitation. Notifications for budget can be delayed by two or three days due to the minimum 24-hour delay in receiving bills. (DT-3091)
Compute
- Jobs by status graphs in the Trends tab display spark application details and not the job details. Our development is actively looking into this design limitation and efforts are underway to address this in future updates to enhance the product's capabilities. (DT-2008)
- Data is inconsistent between Compute page and Cost page in some cases when Azure bill is enabled. This is a known limitation because of Azure bill data not being live. Our team is actively looking into this limitation and efforts are underway to address this issue in the future updates. (DT-3078)
Data
If tables are created with the same name, accessed, deleted, and re-created, and if those tables are re-accessed, then their query and app count do not match. (DATAPAGE-502)
Home
Home page does not display alerts on the UI when there is missing ROI data for a single day. (DT-2509)
Hovering on Total Cost Trend on the Summary tab of the Home page may display inaccurate date information. (DT-2408)
Insights Preview
In some cases, there is a discrepancy in the runs displayed in the Job Runs page for the selected job. When you click on View runs, only runs with dbx_cluster kind having a Spark app id are listed. The discrepancy is due to data inconsistency between entries of dbx_cluster kind and db kind and this is an expected behavior. (DT-3042)
Workflows
The current implementation has a limitation where the wrong run count is displayed for the job ID when sorting by run count in the Workflows > Jobs section. This discrepancy is currently under investigation by our development team, and active efforts are being taken to resolve this issue. (UIX-6526)

Known issues

Our development team is actively investigating the following Known issues and are working towards resolving them. It is important to note that while these issues exist, there is no immediate critical impact on using the product, and users can continue to utilize its functionality with confidence.

Bug ID	Description	Workaround
App Store
APP-614	App Store tasks fail to start with SSL enabled on the MySQL database.	Workaround
Compute
PIPELINE-1636	Inconsistent data is displayed for the cluster Duration and Start Time on the Compute page.	NA
Cost
UIX-5624	Data is not displayed when you click the Optimize button corresponding to OTHERS for the Cost > Chargeback results shown in the table.	NA
DT-1094	The No data available message is displayed on the Compute page after navigating from the Trends and Chargeback pages with Tag filters.	NA
Datapage
DATAPAGE-473	For Hive metastore 3.1.0 or earlier versions, the creation time of partitions is not captured if a partition is created dynamically. Therefore, the Last Day KPI for the partition section is not shown in Unravel.	NA
Insights
DT-1987	There is a mismatch in the computation of costs for fleet and spot instances in Databricks clusters. This issue arises due to the unavailability of the exact node type in the cluster info response.	NA
Performance
ASI-933	In the Lag setup, the Duration is not updated for running applications. The Duration should be updated every 15 minutes.	NA
ASI-936	In the Lag setup, the App Time data is missing in the Timing tab of many applications.	NA
Spark
PIPELINE-1616	If the Spark job is not running for Databricks, the values for the Duration and End time fields are not updated on the Databricks Run Details page.	NA
DT-2012	Incorrect details are displayed on the AppSummary > Job Run page when a user repairs a previously failed job. The displayed information may not accurately reflect the repaired job's details.	NA
DT-1742	The timezone for the NodeRightSizing insight event is inconsistent in the Spark details page.	NA
DT-2029	Applications in a success state may inaccurately display an associated job in a running state instead of transitioning to a failed state.	NA
UI
PIPELINE-1935	In the Pipeline details page, when you select the data for a specific date, all instances are displayed instead of displaying only the instances within a selected date.	NA
UIX-6321	In the Workflow section, instead of displaying only jobs completed within the selected time frame, it currently displays jobs running within the selected duration.	NA
UIX-6263	The cross button on the Pipeline details page does not close the detail page when you click the bars inside the Gantt chart.	NA
Workflows
DT-1461, PIPELINE-1939, PIPELINE-1940, DT-1093, PIPELINE-1924	The UI and data exhibit inconsistencies, including problems with job run details, issues related to multiple workflow runs and UTC timestamps , empty content in workflow job pages and issues with filter values and duration display.	NA

Workarounds

App Store

App Store tasks fail to start with SSL enabled on the MySQL database. (APP-614)

Stop Unravel.

<Unravel installation directory>/unravel/manager stop

Use an editor to open <Installation_directory>/unravel/data/conf/unravel.yaml file.
In the unravel.yaml file, under the database > advanced > python_flags block, enter the path to the trusted certificates. For example, if Unravel is installed at /opt/unravel, you must edit the unravel.yaml file as follows:
```
unravel:
...snip... database:
...snip... advanced: python_flags: ssl_ca: /opt/unravel/data/certificates/trusted_certs.pem
```
Use the manager utility to upload the certificates.
```
<Unravel installation director>/manager config tls trust add --pem /path/to/certificate
```
For example: /opt/unravel/manager config tls trust add --pem /path/to/certificate

Enable the Truststore.

<Unravel installation directory>/manager config tls trust enable

Apply the changes and restart Unravel.

<Unravel installation directory>/unravel/manager config apply --restart

4795 Hotfix Updates

Improvements

Implemented instance type whitelisting for enhanced control over the instance types used for receiving insights.
Enabled Service Principal Name (SPN) with Role-Based Access Control (RBAC) for seamless integration of Azure billing with Unravel.
Formatted DBX downloaded CSV file data to improve usability in configuring Spark jobs.
Redesigned the Diagnostics page UI and refined textual content for improved clarity and usability.
Enhanced authentication flexibility with a secondary NGUI for alternate authentication methods in Unravel support.
Improved readability of insights within Databricks Structured Streaming by formatting performance trend values.
Enhanced data filtering capabilities by synchronizing tag-based and date-based filtering options for improved data navigation.
Enhanced sensitivity of sunburst visualizations to accommodate all applied filters and features, providing comprehensive data insights.
Increased visibility of contended driver savings information on the homepage for improved user awareness and monitoring capabilities.
Optimized homepage user interface elements to improve navigation and user interaction.

Issues Fixed

Bug ID	Description
DT-3144	Not able to open apps on SAAS setup after the setup is upgraded.
DT-3662	Data unavailable on Jobs/Jobs Runs Page and Cluster status displayed as UNKNOWN state in Unravel UI.
DT-3661	Data not available on workflows Page whereas on Compute page data is available.
DT-3656	NullPointerException in connectors.DbxClusterConnector.
DT-3632	Data not available on workflows Page whereas on Compute page data is available.
DT-3630	DBX Mission Wired: Streaming data not available for first streaming query of Streaming job.
DT-3604	No data available for Workflows Page.
DT-3603	No data is shown in cost explorer and home page.
DT-3505	Resource utilization (RM data) is missing for all apps.
DT-3483	Multiple cluster sessions pointing to same spark application.
DT-3409	Node DownSizing Event is not generated for a particular job.
DT-3408	Auto-Action notification links are not working.
DT-3406	No data available for selected timeframes on the Home page.
DT-3404	Drill downs label shown incorrect for 90 days.
DT-3403	Unravel LLM app not working.
DT-3402	Signature ROI pipeline failing.
DT-3375	Issue in reflecting databricks jobs in workflow page as Kafka consumers were down due to 'java.util.ConcurrentModificationException'.
SUPPORT-3069	LDAP login fails when the user is part of hundreds of groups.
SUPPORT-3024	Billing page is not loading data.
SUPPORT-3021	Unravel_sensor taking time to start.
SUPPORT-3020	Home Page is not loading.
SUPPORT-3019 SUPPORT-3018	ElasticSearch causing stability and resource loading issues.
PLATFORM-3252	Cluster metrics not loading after upgrade.
PIPELINE-2077	When grouping cluster session by Tag Key, null values showed up on the chargeback page.
PIPELINE-2070	Improve Metrics cache performance in insightworker.
DT-3691	The Untapped Cost Savings (USD) value exceeds the Total Cost (USD) value for some signatures shown on the Insights Preview page.
DT-3669	Document Unravel property to restrict Insights for the Whitelisted instances.
DT-3661	Data not available on workflows Page whereas on Compute page data is available.
DT-3628	No data available for Workflows Page.
DT-3600	No Consumer Topic found for Consumer groups.
DT-3577	Analysis API error causing Analysis tab to be empty.
DT-3573	Jobs are not shown for some of the insights that are listed under Inefficient tab.
DT-3562	Majority of the clusters displayed under the list for 'Top 50 clusters with the selected inefficient events for Node Down Sizing event are not populated with Analysis tab.
DT-3414	Add support to the cost pipeline to process daily cost files from local directory
DT-3410	Out of order data is not handled properly for billing data generation.
DT-3391	Need to disable the migration savings top-X widget.

In this section:

Home