Skip to main content

Home

v4.7.9.2 Release notes

Software version

Release date: September 08, 2023

See 4.7.9.2 for download information.v4.7.9.2

See also Unity App release notes

Software upgrade support

The following upgrade paths are supported:

  • 4.7.8.0 Hotfix → 4.7.9.2

  • 4.7.8.0 → 4.7.9.2

For instructions to upgrade to Unravel v4.7.9.x, see Upgrading Unravel.

For fresh installations, see Installing Unravel.

Certified platforms

The following platforms are tested and certified in this release:

  • Databricks (Azure, AWS)

Review your platform's compatibility matrix before you install Unravel.

Updates to Unravel's configuration properties

Updates to upgrading Unravel to v4.7.9.2

The following updates are included in this release for upgrade:

Enable Global init script

The Global init script is now available within a workspace as install-unravel.sh at the following path:

/Workspace/Unravel/install-unravel.sh

dbx-new-4780hf-globalinit-script.png

The script in install-unravel.sh is automatically copied as the Global init script when you upgrade or when you add a new workspace. This script is applicable from Unravel version 4.7.8.0 hotfix and later. In case the script is not copied automatically, you can copy it from the /Unravel/install-unravels.sh file.

Configure Cluster init script

You can now configure the cluster init script from the workspace. Select the workspace as source and path as /unravel/install-unravel.sh

dbx-new-4780hf-cluster-level-init-script.png

Note

You must delete the old cluster-level init script configuration that is pointing to the DBFS location. If you want to have a cluster-level init script, then you must configure it to the workspace file /unravel/install-unravel.sh.

Also, check Backward compatibility for the init scripts

Manage workspace access token

If you have a premium workspace and the workspace access control is enabled, then you must provide any of the following tokens while adding a workspace:

  • Admin access token

  • Token of a user who has Can Manage permission on /Workspace

Set the backward compatibility for init scripts

To ensure backward compatibility of the init scripts, you must run a manager command to copy the init scripts to the DBFS location (dbin folder). Typically the default behavior when adding or updating the workspace and when upgrading Unravel is to remove the dbin folder. However, if you run the following command, the dbin folder is not removed and will also get pushed into the DBFS.

<Installation_directory>/unravel/manager config databricks copy-to-dbfs --id <workspace-id> --unsafe-copy-init-to-dbfs

For example:

/opt/unravel/manager config databricks copy-to-dbfs --id 6679977360960347 --unsafe-copy-init-to-dbfs

Note

You can also use the --all option in the command instead of --id, which is used for a single workspace. The --all option performs the same step for all the workspaces that are currently registered or added in Unravel.

For example:

/opt/unravel/manager config databricks copy-to-dbfs --all --unsafe-copy-init-to-dbfs

New features

  • Support of CI/CD workflows

    Unravel now integrates with DevOps tools like GitHub and Azure DevOps to optimize the efficiency of your development workflow. Your CI/CD pipeline gains actionable real-time insights (for example: node right-sizing recommendations) that developers can use to reduce bottlenecks, optimize cost, and improve performance and resource utilization.

  • Real-time cost savings insights

    Real-time cost savings insights are now available on the Compute > App Summary page. You can now monitor and act on insights for potential cost-saving opportunities derived from resource usage analysis over specific time intervals. Also, you can make informed decisions about downsizing cluster instances to unlock potential operational cost savings and optimize resource management.

  • Support for Unity Catalog on Data page

    Unity Catalog is now supported in Unravel's Data page along with Hive. You can now ingest data from the Unity Catalog into the Data page.

  • Support for error analysis in Databricks Cost Anomaly Detection app

    In addition to identifying abnormal cost fluctuations, the Databricks Cost Anomaly Detection app also supports the error analysis functionality.

  • Support for tagged workflows

    Manage your workflows now with the Pipelines tab on the Workflows page. The tagged workflows are now displayed alongside jobs and job runs in the Pipelines tab.

  • Audit report app

    The Audit Report app is included in this release in the App Store. The Audit Report app is used to get the details of the login count of all the users using Unravel.

Improvements and enhancements

  • Access to App Comparator and Unity App reports from Unravel UI

    You can access the App Comparator and Unity app reports directly from the Unravel UI. Click appstore-icon.png on the upper right corner of the Unravel UI to directly access the App Comparator and Unity app reports.

  • Improved AutoAction

    The following improvements are done to the AutoActions feature to improve clarity and usability:

    • Introduced real-time cost estimation for DBX jobs.

    • Addressed issues in displaying recent records.

    • Improved the presentation of cluster information.

    • Added cluster names to notifications.

  • Enhanced database query performance with cache support

    Unravel now has cache support for its internal database queries. With the cache support, there is improved database performance and reduced bottlenecks.

Bug fixes

  • Configuration

    • An extra space in the property name resulted in the failure to retrieve the correct Elasticsearch URL. (APP-681)

  • Databricks

    • The cost optimization recommendation erroneously displayed a value of $0, despite resource usage analysis from recent runs suggesting potential savings. (DT-1519)

    • Node right-sizing report does not display relevant and current recommendations. It displays outdated recommendations after applying changes. (DT-1536)

    • Node right-sizing report does not provide recommendations for long-running jobs. (DT-1539)

    • Missing signature generation for Databricks long-running and slow SQL stage jobs resulting in incomplete node right-sizing reports. (DT-1540) and (DT-1550)

    • The DBU consumption report fails with an IndexError exception indicating positional indexers are out-of-bounds. (IMP-1012)

    • The Timings tab displays inconsistent application duration, with discrepancies between the Duration in Job and AppTime fields. (PIPELINE-1877)

  • Installation

    Changing the default host_monitor port to a custom port (4025) results in the Monitoring page failing to load. Despite the port change, NGUI logs continue to reference the original port (4021). (INSTALL-3193).

Known issues

Applications

  • On the Table Details page under the Applications tab, inaccurate data is displayed for a table. This issue occurs if a table is deleted and recreated multiple times and applications are executed to access the table before the next cycle of the table worker. (PG-156)

AutoActions

  • AutoActions stop responding due to an invalid or unsupported HTTP URL or webhook. (AA-575)

App Store

  • Issue

    App Store tasks fail to start with SSL enabled on the MySQL database. (APP-614)

    Workaround

    To resolve this issue, do the following:

    1. Stop Unravel.

      <Unravel installation directory>/unravel/manager stop

    2. Use an editor to open <Installation_directory>/unravel/data/conf/unravel.yaml file.

    3. In the unravel.yaml file, under the database > advanced > python_flags block, enter the path to the trusted certificates. For example, if Unravel is installed at /opt/unravel, you must edit the unravel.yaml file as follows:

      unravel:
...snip...
  database:
...snip...
    advanced:
      python_flags:
        ssl_ca: /opt/unravel/data/certificates/trusted_certs.pem

    4. Use the manager utility to upload the certificates.

      <Unravel installation directory>/manager config tls trust add --pem /path/to/certificate

      For example: /opt/unravel/manager config tls trust add --pem /path/to/certificate

    5. Enable the Truststore.

      <Unravel installation directory>/manager config tls trust enable

    6. Apply the changes and restart Unravel. 

      <Unravel installation directory>/unravel/manager config apply --restart
Data page

  • If tables are created with the same name, accessed, deleted, and re-created, and if those tables are re-accessed, then their query and app count do not match. (DATAPAGE-502)

  • The query to fetch tableDailyKPIs is getting timed out when dealing with a huge table partition of 27 million records. From a threshold perspective, it has been verified that the API functions without issues for partition sizes up to 18 million. (DATAPAGE-740)

  • For Hive metastore 3.1.0 or earlier versions, the creation time of partitions is not captured if a partition is created dynamically.  Therefore, the Last Day KPI for the partition section is not shown in Unravel. (DATAPAGE-473)

  • On the Data page, data size is missing for certain tables in databases, although the partition size is correctly displayed in the Partition Detail section. (DATAPAGE-695)

Insights Overview

User interface and data presentation issues

  • Link re-direction issues, such as incorrect data filters for viewing Top Groups by Cost and Top Clusters by Cost, as well as missing re-direction links in the App Acceleration section. (UIX-5127, INSIGHTS-324,UIX-4176)

  • Inconsistencies in data presentation, including name tag mismatches in the Top Groups by Cost section and discrepancies in app counts between different sections and pages. (INSIGHTS-330, INSIGHTS-345, UIX-5622,UIX-6243)

Data accuracy and counting issues

  • Various app count problems on the Insights Overview page, including incorrect counts for tables in the App Acceleration section, issues with app counts after redirection, and inconsistencies in counts related to various app categories. (UIX-6193, INSIGHTS-372, INSIGHTS-371, INSIGHTS-375, INSIGHTS-376, UIX-6190,INSIGHTS-374)

Installation
    Kerberos

    • Kerberos can only be disabled manually from the unravel.yamlfile. 

       kerberos:
      enabled: False
    Migration

    • WorkloadFit report

      • A large number of tags can cause the Workload Fit report to fail. (PG-265, CUSTOMER-2084)

      • WorkloadFit report > Heatmap: The job count has data, but Vcore and memory are empty. (MIG-262)

    • Inconsistency between the regions displayed on the Unravel user interface and the ones included in AWS EMR. (MIG-280, MIG-281)

    • The Cloud Mapping Per Host migration report fails for some regions. (MIG-303)

    Reports

    • Cluster discovery

      • The On-prem Cluster Identity might show an incorrect Spark version on CDH. The report may incorrectly show Spark 1 when Spark 2 is installed on the CDH cluster. (REPORT-1702)

    • Queue analysis:

      • The log file name (unravel_us_1.log) displayed in the error message is incorrect. The correct name of the log file is unravel_sensor.log. (REPORT-1663)

    • Cloud Mapping Per Host report scheduled in v4.6.1.x does not work in v4.7.1.0. Users must organize a new report. (REPORT-1886)

    • When using PostgreSQL, the percentage (%) sign is duplicated and displayed in the Workload Fit report > Map to single cluster tab. (MIG-42)

    Tez

    • SQL events generator generates SQL Like clause event if the query contains a like pattern even in the literals. (TEZLLAP-349)

    UI

    • Task BreakDown option is not displayed in the Timeline tab of Task Status.(UIX-6332)

    • In the App summary page for Impala, the Query> Operator view is visible after scrolling down. (UIX-3536).

    • When you click the hive query, which was executed as part of the Hive on the Spark application, a blank page is shown. (UIX-6037)

    • On the Clusters > Resources page, in the Group By drop-down list, the Application Type, User, and Queue options are duplicated for the YARN/IMPALA resource job type. The issue occurs if identical user-defined tags are used. (UIX-5898)

    Workflow

    • Jobs are falsely labeled as a Tez App for Oozie Sqoop and Shell actions. (PLATFORM-2403)

    Unsupported

    App Store

    • App-store does not support PostgreSQL over SSL.

    Billing

    • Unravel does not support Billing for on-prem platforms.

    Healthcheck

    • Monitoring the expiration of the SSL Certificates and Kerberos principals in Unravel multi-cluster deployments.

    AutoActions

    • Sustained Violation is not supported in AutoActions for Databricks. This is a type of violation that triggers the AutoAction.

    Migration planning

    • Migration planning is not supported for the following regions for Azure Data Lake:

      • Germany Central (Sovereign)

      • Germany Northeast (Sovereign)

    • Forecasting and Migration: In a multi-cluster environment, you can configure only a single cluster at a time. Hence, reports are generated only for that single cluster.

    MapR

    The following features are not supported for MapR:

    • Impala applications

    • Kerberos

    • The following features are not supported on the Data page:

      • Forecasting

      • Small Files Report

      • File Reports

    • The following reports are not supported on MapR:

      • File Reports

      • Small Files Report

      • Capacity Forecasting

      • Cloud Migration reports

    • AutoAction is not supported for Impala applications.

    • Billing

    • Insights Overview

    Multi-cluster deployment

    • Unravel does not support multi-cluster management of combined on-prem and cloud clusters.

    • Automatic FSImage processing is not supported for multi-cluster environments.

    Pipeline

    • Unravel does not support apps belonging to the same pipeline in a multi-cluster environment.

    Reports

    • All the reports, except for the TopX report, are not supported on Databricks.

    Support

    For support issues, contact Unravel Support.

    In this section: