v4.7.9.2 Release notes
Software version
Release date: September 08, 2023
See 4.7.9.2 for download information.
See also Unity App release notes
Software upgrade support
The following upgrade paths are supported:
4.7.8.0 Hotfix → 4.7.9.2
4.7.8.0 → 4.7.9.2
For instructions to upgrade to Unravel v4.7.9.x, see Upgrading Unravel.
For fresh installations, see Installing Unravel.
Certified platforms
The following platforms are tested and certified in this release:
Databricks (Azure, AWS)
Review your platform's compatibility matrix before you install Unravel.
Updates to Unravel's configuration properties
See 4.7.x - Updates to Unravel properties.
Updates to upgrading Unravel to v4.7.9.2
The following updates are included in this release for upgrade:
The Global init script is now available within a workspace as install-unravel.sh at the following path:
/Workspace/Unravel/install-unravel.sh
The script in install-unravel.sh is automatically copied as the Global init script when you upgrade or when you add a new workspace. This script is applicable from Unravel version 4.7.8.0 hotfix and later. In case the script is not copied automatically, you can copy it from the /Unravel/install-unravels.sh file.
You can now configure the cluster init script from the workspace. Select the workspace as source and path as /unravel/install-unravel.sh
Note
You must delete the old cluster-level init script configuration that is pointing to the DBFS location. If you want to have a cluster-level init script, then you must configure it to the workspace file /unravel/install-unravel.sh.
Also, check Backward compatibility for the init scripts
If you have a premium workspace and the workspace access control is enabled, then you must provide any of the following tokens while adding a workspace:
Admin access token
Token of a user who has Can Manage permission on /Workspace
To ensure backward compatibility of the init scripts, you must run a manager command to copy the init scripts to the DBFS location (dbin folder). Typically the default behavior when adding or updating the workspace and when upgrading Unravel is to remove the dbin folder. However, if you run the following command, the dbin folder is not removed and will also get pushed into the DBFS.
<Installation_directory>/unravel/manager config databricks copy-to-dbfs --id <workspace-id> --unsafe-copy-init-to-dbfsFor example:
/opt/unravel/manager config databricks copy-to-dbfs --id 6679977360960347 --unsafe-copy-init-to-dbfs
Note
You can also use the --all option in the command instead of --id, which is used for a single workspace. The --all option performs the same step for all the workspaces that are currently registered or added in Unravel.
For example:
/opt/unravel/manager config databricks copy-to-dbfs --all --unsafe-copy-init-to-dbfs
New features
Support of CI/CD workflows
Unravel now integrates with DevOps tools like GitHub and Azure DevOps to optimize the efficiency of your development workflow. Your CI/CD pipeline gains actionable real-time insights (for example: node right-sizing recommendations) that developers can use to reduce bottlenecks, optimize cost, and improve performance and resource utilization.
Real-time cost savings insights
Real-time cost savings insights are now available on the Compute > App Summary page. You can now monitor and act on insights for potential cost-saving opportunities derived from resource usage analysis over specific time intervals. Also, you can make informed decisions about downsizing cluster instances to unlock potential operational cost savings and optimize resource management.
Support for Unity Catalog on Data page
Unity Catalog is now supported in Unravel's Data page along with Hive. You can now ingest data from Unity Catalog into the Data page.
Support for error analysis in Databricks Cost Anomaly Detection app
In addition to identifying abnormal cost fluctuations, the Databricks Cost Anomaly Detection app also supports the error analysis functionality.
Support for tagged workflows
Manage your workflows now with the Pipelines tab on the Workflows page. The tagged workflows are now displayed alongside jobs and job runs in the Pipelines tab.
Audit report app
The Audit Report app is included in this release in the App Store. The Audit Report app is used to get the details of the login count of all the users using Unravel.
Improvements and enhancements
Access to App Comparator and Unity App reports from Unravel UI
You can access the App Comparator and Unity app reports directly from the Unravel UI. Click
on the upper right corner of the Unravel UI to directly access the App Comparator and Unity app reports.Improved AutoAction
The following improvements are done to the AutoActions feature to improve clarity and usability:
Introduced real-time cost estimation for DBX jobs.
Addressed issues in displaying recent records.
Improved the presentation of cluster information.
Added cluster names to notifications.
Enhanced database query performance with cache support
Unravel now has cache support for its internal database queries. With the cache support, there is improved database performance and reduced bottlenecks.
Bug fixes
Configuration
An extra space in the property name resulted in the failure to retrieve the correct Elasticsearch URL. (APP-681)
Databricks
The cost optimization recommendation erroneously displayed a value of $0, despite resource usage analysis from recent runs suggesting potential savings. (DT-1519)
Node right-sizing report does not display relevant and current recommendations. It displays outdated recommendations after applying changes. (DT-1536)
Node right-sizing report does not provide recommendations for long-running jobs. (DT-1539)
Missing signature generation for Databricks long-running and slow SQL stage jobs resulting in incomplete node right-sizing reports. (DT-1540) and (DT-1550)
The DBU consumption report fails with an IndexError exception indicating positional indexers are out-of-bounds. (IMP-1012)
The Timings tab displays inconsistent application duration, with discrepancies between the Duration in Job and AppTime fields. (PIPELINE-1877)
Installation
Changing the default host_monitor port to a custom port (4025) results in the Monitoring page failing to load. Despite the port change, NGUI logs continue to reference the original port (4021). (INSTALL-3193).
AutoActions stop responding due to an invalid or unsupported HTTP URL or webhook. (AA-575)
On the Application details page, the original query link is missing for some cached queries due to the parallel processing of original and cached queries. (BIGQ-61)
Issue
Sometimes, when you process a large number of BigQuery projects with the manager config bigquery integrate command, you may see the following error:
Provider produced inconsistent result after applyWorkaround
Wait for a few minutes and re-run the command. (INSTALL-2860, INSTALL-2934)
If tables are created with the same name, accessed, deleted, and re-created, and if those tables are re-accessed, then their query and app count do not match. (DATAPAGE-502)
The query to fetch tableDailyKPIs is getting timed out when dealing with a huge table partition of 27 million records. From a threshold perspective, it has been verified that the API functions without issues for partition sizes up to 18 million. (DATAPAGE-740)
For Hive metastore 3.1.0 or earlier versions, the creation time of partitions is not captured if a partition is created dynamically. Therefore, the Last Day KPI for the partition section is not shown in Unravel. (DATAPAGE-473)
On the Data page, data size is missing for certain tables in databases, although the partition size is correctly displayed in the Partition Detail section. (DATAPAGE-695)
Issue
You can encounter a NoIndexFound exception for fresh installations of Unravel on GCP-BigQuery. (BIGQ-104)
Workaround
Run the following CURL command on the Unravel node after the installation.
curl -XPUT http://localhost:4171/app-19700101_07
Kerberos can only be disabled manually from the
unravel.yamlfile.kerberos: enabled: False
Task BreakDown option is not displayed in the Timeline tab of Task Status.(UIX-6332)
In GCP - BigQuery, for the Data page, a count of more than 100 projects is not supported.
For BigQuery pricing, Unravel only supports on-demand analysis pricing. Flat-rate analysis pricing and storage pricing (Active and Long-term storage) are not supported.