Skip to main content

Home

v4.7.9.2 Release notes

Software version

Release date: September 08, 2023

See 4.7.9.2 for download information.

See also Unity App release notes

Software upgrade support

The following upgrade paths are supported:

  • 4.7.8.0 Hotfix → 4.7.9.2

  • 4.7.8.0 → 4.7.9.2

For instructions to upgrade to Unravel v4.7.9.x, see Upgrading Unravel.

For fresh installations, see Installing Unravel.

Certified platforms

The following platforms are tested and certified in this release:

  • Databricks (Azure, AWS)

Review your platform's compatibility matrix before you install Unravel.

Updates to Unravel's configuration properties

Updates to upgrading Unravel to v4.7.9.2

The following updates are included in this release for upgrade:

Enable Global init script

The Global init script is now available within a workspace as install-unravel.sh at the following path:

/Workspace/Unravel/install-unravel.sh

dbx-new-4780hf-globalinit-script.png

The script in install-unravel.sh is automatically copied as the Global init script when you upgrade or when you add a new workspace. This script is applicable from Unravel version 4.7.8.0 hotfix and later. In case the script is not copied automatically, you can copy it from the /Unravel/install-unravels.sh file.

Configure Cluster init script

You can now configure the cluster init script from the workspace. Select the workspace as source and path as /unravel/install-unravel.sh

dbx-new-4780hf-cluster-level-init-script.png

Note

You must delete the old cluster-level init script configuration that is pointing to the DBFS location. If you want to have a cluster-level init script, then you must configure it to the workspace file /unravel/install-unravel.sh.

Also, check Backward compatibility for the init scripts

Manage workspace access token

If you have a premium workspace and the workspace access control is enabled, then you must provide any of the following tokens while adding a workspace:

  • Admin access token

  • Token of a user who has Can Manage permission on /Workspace

Set the backward compatibility for init scripts

To ensure backward compatibility of the init scripts, you must run a manager command to copy the init scripts to the DBFS location (dbin folder). Typically the default behavior when adding or updating the workspace and when upgrading Unravel is to remove the dbin folder. However, if you run the following command, the dbin folder is not removed and will also get pushed into the DBFS.

<Installation_directory>/unravel/manager config databricks copy-to-dbfs --id <workspace-id> --unsafe-copy-init-to-dbfs

For example:

/opt/unravel/manager config databricks copy-to-dbfs --id 6679977360960347 --unsafe-copy-init-to-dbfs

Note

You can also use the --all option in the command instead of --id, which is used for a single workspace. The --all option performs the same step for all the workspaces that are currently registered or added in Unravel.

For example:

/opt/unravel/manager config databricks copy-to-dbfs --all --unsafe-copy-init-to-dbfs

New features

  • Support of CI/CD workflows

    Unravel now integrates with DevOps tools like GitHub and Azure DevOps to optimize the efficiency of your development workflow. Your CI/CD pipeline gains actionable real-time insights (for example: node right-sizing recommendations) that developers can use to reduce bottlenecks, optimize cost, and improve performance and resource utilization.

  • Real-time cost savings insights

    Real-time cost savings insights are now available on the Compute > App Summary page. You can now monitor and act on insights for potential cost-saving opportunities derived from resource usage analysis over specific time intervals. Also, you can make informed decisions about downsizing cluster instances to unlock potential operational cost savings and optimize resource management.

  • Support for Unity Catalog on Data page

    Unity Catalog is now supported in Unravel's Data page along with Hive. You can now ingest data from the Unity Catalog into the Data page.

  • Support for error analysis in Databricks Cost Anomaly Detection app

    In addition to identifying abnormal cost fluctuations, the Databricks Cost Anomaly Detection app also supports the error analysis functionality.

  • Support for tagged workflows

    Manage your workflows now with the Pipelines tab on the Workflows page. The tagged workflows are now displayed alongside jobs and job runs in the Pipelines tab.

  • Audit report app

    The Audit Report app is included in this release in the App Store. The Audit Report app is used to get the details of the login count of all the users using Unravel.

Improvements and enhancements

  • Access to App Comparator and Unity App reports from Unravel UI

    You can access the App Comparator and Unity app reports directly from the Unravel UI. Click appstore-icon.png on the upper right corner of the Unravel UI to directly access the App Comparator and Unity app reports.

  • Improved AutoAction

    The following improvements are done to the AutoActions feature to improve clarity and usability:

    • Introduced real-time cost estimation for DBX jobs.

    • Addressed issues in displaying recent records.

    • Improved the presentation of cluster information.

    • Added cluster names to notifications.

  • Enhanced database query performance with cache support

    Unravel now has cache support for its internal database queries. With the cache support, there is improved database performance and reduced bottlenecks.

Bug fixes

  • Configuration

    • An extra space in the property name resulted in the failure to retrieve the correct Elasticsearch URL. (APP-681)

  • Databricks

    • The cost optimization recommendation erroneously displayed a value of $0, despite resource usage analysis from recent runs suggesting potential savings. (DT-1519)

    • Node right-sizing report does not display relevant and current recommendations. It displays outdated recommendations after applying changes. (DT-1536)

    • Node right-sizing report does not provide recommendations for long-running jobs. (DT-1539)

    • Missing signature generation for Databricks long-running and slow SQL stage jobs resulting in incomplete node right-sizing reports. (DT-1540) and (DT-1550)

    • The DBU consumption report fails with an IndexError exception indicating positional indexers are out-of-bounds. (IMP-1012)

    • The Timings tab displays inconsistent application duration, with discrepancies between the Duration in Job and AppTime fields. (PIPELINE-1877)

  • Installation

    Changing the default host_monitor port to a custom port (4025) results in the Monitoring page failing to load. Despite the port change, NGUI logs continue to reference the original port (4021). (INSTALL-3193).

Known issues

AutoActions

  • AutoActions stop responding due to an invalid or unsupported HTTP URL or webhook. (AA-575)

App Store

  • Issue

    App Store tasks fail to start with SSL enabled on the MySQL database. (APP-614)

    Workaround

    To resolve this issue, do the following:

    1. Stop Unravel.

      <Unravel installation directory>/unravel/manager stop

    2. Use an editor to open <Installation_directory>/unravel/data/conf/unravel.yaml file.

    3. In the unravel.yaml file, under the database > advanced > python_flags block, enter the path to the trusted certificates. For example, if Unravel is installed at /opt/unravel, you must edit the unravel.yaml file as follows:

      unravel:
...snip...
  database:
...snip...
    advanced:
      python_flags:
        ssl_ca: /opt/unravel/data/certificates/trusted_certs.pem

    4. Use the manager utility to upload the certificates.

      <Unravel installation directory>/manager config tls trust add --pem /path/to/certificate

      For example: /opt/unravel/manager config tls trust add --pem /path/to/certificate

    5. Enable the Truststore.

      <Unravel installation directory>/manager config tls trust enable

    6. Apply the changes and restart Unravel. 

      <Unravel installation directory>/unravel/manager config apply --restart
Compute

  • Inconsistent data is displayed for the cluster Duration and Start Time on the Compute page. (PIPELINE-1636)

Cost

  • Cost and DBUs are reported as zero for many clusters and applications, even with higher durations. (ASI-931)

  • Cost and DBUs data are not updating for clusters in the Unknown state. (ASI-930)

  • Data is not displayed when you click the Optimize button corresponding to OTHERS for the Cost > Chargeback results shown in the table. (UIX-5624)

  • In Cost > Chargeback results, for situations where there are more than 1000 clusters, the legend displays OTHERS twice. (UIX- 6305)

  • On the Cost > Trends and Cost > Chargeback pages, the tooltip for the Last <number of> days field includes more days than the displayed days. (UIX-5042)

  • The No data available message is displayed on the Compute page after navigating from the  Trends  and  Chargeback  pages with  Tag  filters. (DT-1094)

Data page

  • If tables are created with the same name, accessed, deleted, and re-created, and if those tables are re-accessed, then their query and app count do not match. (DATAPAGE-502)

  • The query to fetch tableDailyKPIs is getting timed out when dealing with a huge table partition of 27 million records. From a threshold perspective, it has been verified that the API functions without issues for partition sizes up to 18 million. (DATAPAGE-740)

  • For Hive metastore 3.1.0 or earlier versions, the creation time of partitions is not captured if a partition is created dynamically.  Therefore, the Last Day KPI for the partition section is not shown in Unravel. (DATAPAGE-473)

  • On the Data page, data size is missing for certain tables in databases, although the partition size is correctly displayed in the Partition Detail section. (DATAPAGE-695)

EMR

  • When you click the View Clusters link on the Cost-based pages and navigate to the Clusters page, the cluster numbers shown can vary. Sometimes fewer clusters are listed, and at times no clusters are shown. This is a known limitation due to differences in the definition of the time range selector for these pages. (UIX-5328)

    • Cost page

      Shows all the clusters that have accrued cost in the selected period, which may be running or terminated in the selected period, irrespective of their start date.

    • Cluster page

      Shows only those clusters that have started in the selected period.

  • Due to the known limitations, node downsizing recommendations are not suggested for the following scenarios. (EMR-519, EMR-513, EMR-424)

    • When only cluster recommendations are applied without applying Application recommendations.

    • When the workloads require high IO and partitioning.

    • When the Spark configuration spark.dynamicAllocation.enabled is true.

    • When the AWS EMR autoscaling is enabled.

    • When the workload must need parallelism (multiple CPU cores).

  • On the Clusters details page, the Insights chart is not synchronized with all other graphs displayed on the Cost tab. (EMR-618)

  • On the Cost > Trends page, users with readonly permission can create a budget. (EMR-576)

  • The workflow of multiple transient clusters (EMR) is not supported. (PIPELINE-1424, EMR-460)

  • Unable to run Spark applications on all the master nodes after Unravel bootstrap for high availability clusters. (EMR-49)

  • Support for high availability EMR master nodes. (EMR-31)

  • For the MapReduce failed job, error details are missing on the Errors tab. (UIX-5416)

Insights Overview

User interface and data presentation issues

  • Link re-direction issues, such as incorrect data filters for viewing Top Groups by Cost and Top Clusters by Cost, as well as missing re-direction links in the App Acceleration section. (UIX-5127, INSIGHTS-324,UIX-4176)

  • Inconsistencies in data presentation, including name tag mismatches in the Top Groups by Cost section and discrepancies in app counts between different sections and pages. (INSIGHTS-330, INSIGHTS-345, UIX-5622,UIX-6243)

Data accuracy and counting issues

  • Various app count problems on the Insights Overview page, including incorrect counts for tables in the App Acceleration section, issues with app counts after redirection, and inconsistencies in counts related to various app categories. (UIX-6193, INSIGHTS-372, INSIGHTS-371, INSIGHTS-375, INSIGHTS-376, UIX-6190,INSIGHTS-374)

Installation

  • An exception occurs when installing Unravel version 4.7.6.0 with the Azure MySQL database (SSL Enabled). (INSTALL-2799)

Kerberos

  • Kerberos can only be disabled manually from the unravel.yamlfile. 

     kerberos:
      enabled: False
Performance

  • The Insight Worker daemon is experiencing performance lag, causing delays in processing insights and data analytics tasks. (PIPELINE-1926)

  • In the Lag setup, the Duration is not updated for running applications. The Duration should be updated every 15 minutes. (ASI-933)

  • In the Lag setup, the App Time data is missing in the Timing tab of many applications. (ASI-936)

Reports

  • TopX report

    • The TopX report email links to the Unravel TopX report instead of showing the report content in the email as in the old reports.

Spark

  • Issue

    Jobs created for the PySpark application using User-Defined Functions on a job cluster fail after applying the recommendations for node downsizing. (DT-1404)

    Workaround

    1. In your Databricks workspace, go to Configure Cluster > Advanced Options > Spark config .

    2. Add and set the following property to true for spark.driver.extraJavaOptions and spark.executor.extraJavaOptions spark configurations:

      • -Dcom.unraveldata.metrics.proctree.enable=true

      For example:

      spark.executor.extraJavaOptions -Dcom.unraveldata.metrics.proctree.enable=true -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-3.0 spark.driver.extraJavaOptions -Dcom.unraveldata.metrics.proctree.enable=true -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,script=StreamingProbe.btclass,libs=spark-3.0

  • If the Spark job is not running for Databricks, the values for the Duration and End time fields are not updated on the Databricks Run Details page. (PIPELINE-1616)

  • You can see a lag for SQL Streaming applications. (PLATFORM-2764)

  • The timeline histogram needs to be generated correctly on the Spark application details page. (UX-632)

Spark insights

  • For PySpark applications, the processCPUTime and processCPULoad values are not captured properly. (PIPELINE-626)

  • Cost-saving insights and Host metrics are not available from Databricks version 13.0 onwards. (DT-1580)

Tez

  • SQL events generator generates SQL Like clause event if the query contains a like pattern even in the literals. (TEZLLAP-349)

UI

  • The job run count displayed on the Chargeback page differs from the job count shown on the Workflow page. (UIX-5581)

  • In the Pipeline details page, when you select the data for a specific date, all instances are displayed instead of displaying only the instances within a selected date. (PIPELINE-1935)

  • The cost comparison for all the instances is not displayed on the Pipeline detail page. (UIX-6281)

  • On the Pipeline details page, the arrows must point only to the latest run instead of all the runs. (PIPELINE-1934)

  • In the Workflow section, instead of displaying only jobs completed within the selected time frame, it currently displays jobs running within the selected duration. (UIX_6321)

  • The WhiteCross.png cross button on the Pipeline details page does not close the detail page when you click the bars inside the Gantt chart. (UIX-6263)

  • When a job fails to submit a Spark application, the failed Databricks job is missing from the Unravel UI. (PIPELINE-1427)

  • Task BreakDown option is not displayed in the Timeline tab of Task Status.(UIX-6332)

Unity Catalog

  • On the Unity Catalog Overview page, the Table Accessed, Queries, and Users KPIs for the last day do not reflect the latest data. This issue occurs specifically when using DBX Runtime version 13.2 for interactive clusters and job clusters. (DT-1582)

Workflows

  • User interface and data inconsistency

    The UI and data exhibit inconsistencies, including problems with job run details (DT-1461), issues related to multiple workflow runs and UTC timestamps (PIPELINE-1939, PIPELINE-1940), empty content in workflow job pages (DT-1093), and issues with filter values and duration display (UIX-6274, PIPELINE-1924).

  • Functionality and integration challenges

    The Unravel UI faces functional challenges such as the Cost slider issue (UIX-5508), missing Azure Databricks jobs (PIPELINE-1626), duplicate entries in Databricks workflow (DT-1461), and missing tags in the Pipeline details page (PIPELINE-1946).

Workflow

  • Jobs are falsely labeled as a Tez App for Oozie Sqoop and Shell actions. (PLATFORM-2403)

Unsupported

App Store

  • App-store does not support PostgreSQL over SSL.

Data

  • On the Data page, File Reports, Small File reports, and file size information are not supported for EMR clusters.

AutoActions

  • Sustained Violation is not supported in AutoActions for Databricks. This is a type of violation that triggers the AutoAction.

Amazon EMR

  • Unravel does not support the Insights Overview tab on the UI for the Amazon EMR platform.

Reports

  • All the reports, except for the TopX report, are not supported on Databricks.

  • All the reports, except for the TopX report, are not supported on EMR.

Support

For support issues, contact Unravel Support.

In this section: