Skip to main content

Home

v4.7.7.0 Release notes

Software version

Release date: 09/December/2022

See 4.7.7.0 for download information.

Software upgrade support

The following upgrade paths are supported:

  • v4.7.0.x, v4.7.1.x, v4.7.2.x, v4.7.3.x, v4.7.4.x, v4.7.5.x, v4.7.6.xv4.7.7.0

  • v4.6.1.9 v4.7.7.0

  • v4.6.1.8 or earlierv4.6.1.9v4.7.7.0

For instructions to upgrade to Unravel v4.6.1.9, see Upgrading Unravel server.

For instructions to upgrade to Unravel v4.7.7.x, see Upgrading Unravel.

For fresh installations, see Installing Unravel.

Certified platforms

The following platforms are tested and certified in this release:

  • Google Cloud Platform (BigQuery, Dataproc)

Review your platform's compatibility matrix before you install Unravel.Compatibility Matrix

Updates to Unravel's configuration properties

Updates to upgrading Unravel to v4.7.7.0

  • An existing license for any previous version (before 4.7.7.x) does not work with the newer version of Unravel. Therefore, before upgrading Unravel you must obtain a license file from Unravel Customer Support. For information about setting the license, see Upgrading Unravel from version 4.7.x to 4.7.7.x section in Upgrading Unravel.

  • In this release, Unravel supports the pull model, along with the push model, to get data from BigQuery projects. After the upgrade, the push model, which was used in previous releases, will be continued. However, the pull model can be implemented only with a fresh install of Unravel.

  • When you have enabled both Bigquery and Dataproc before upgrading to Unravel v4.7.7.0, then after the upgrade, you must enable Hadoop daemons (hive_worker, kafka_monitor, yarn_jc_worker, oozie_sensor) manually. see Upgrading Unravel from version 4.7.x to 4.7.7.x section in Upgrading Unravel for complete instructions.

  • For best performance, enable message ordering in GCP Cloud Pub/Sub. The Message ordering setting lets you receive the messages in the order they are published. Refer to Ordering messages for more details. After you upgrade, you cannot directly update the Message ordering setting.

    To enable the Message ordering setting after an upgrade, do the following:

    1. From the GCP console, delete the configured subscription for the respective project.

    2. Create a new subscription with the Message ordering setting on.

      Note

      The Message ordering setting can be configured only once when configuring the Pub/Sub service for the first time.

New features

  • License

    Replaced the license key with the license file for expandability and human readability. A license file contains product licensing information, the license validity, and the expiration date.

    Unravel no longer supports an automatic trial period. You must obtain a license file from Unravel Customer Support.

    For information about the Licensing documentation, see the following topics:

    Topics

    Guide

    • Setting Unravel license

    • Verifying Unravel license

    Configuration Guide

    • License: new property

    • License FAQ

    Reference Guide

    Licensing error messages and troubleshooting

    Troubleshooting Guide

    • Install Unravel on GCP Dataproc using Interactive precheck

    • Install Unravel on GCP Dataproc manually

    • Installing Unravel for GCP BigQuery

    • Manually installing Unravel for GCP BigQuery

    • Use the Interactive Precheck utility

    • Upgrading Unravel for Cloud platforms (v4.7.x to v4.7.7.0)

    Installation Guide

  • AutoActions for BigQuery

    Unravel now supports AutoActions for BigQuery. An updated user interface is provided for AutoActions in GCP with improved usability.

    For more information, refer to AutoActions > AutoActions (GCP - BigQuery, Dataproc) topic in User Guide.

  • BigQuery pull model

    Unravel now supports getting data about BigQuery from GCP with the pull model. Previously, only the push model was supported. The pull model is available only for fresh installations.

    Refer to the Installation Guide > Google Cloud platform - BigQuery installation section for more details.

  • Secure connection with Cloud SQL Auth proxy to external MySQL and PostgreSQL database

    For the Google Cloud Platform (GCP), you can enable a secure connection between Unravel server and an external MySQL or PostgreSQL database. This can be done using the Cloud SQL Auth proxy, which is a Cloud SQL connector that provides secure access to your instances without needing Authorized networks for configuring SSL. Refer to Enabling secure connection from Unravel GCP to external MySQL database with Cloud SQL Auth proxy section in the Configuration Guide.

Improvements and enhancements

  • Installation

    • Added support for the .xz archive format when configuring Unravel managed database with the --extra setup option. (INSTALL-2773)

    • When installing for BigQuery, the Interactive Precheck utility now prompts for the push and pull options. (INSTALL-2817)

  • BigQuery enhancements

    • Enhancements to the BigQuery Application Details page:

      • Added support for different job types displayed on the BigQuery details page. The supported job types are QUERY, LOAD, COPY, and EXTRACT. The new Type KPI shows the type of job run in BigQuery. (UIX-5003)

      • On the Tables tab, the tables are now categorized into Input and Output columns. (UIX-4939)

      • Added a new Child Jobs tab to display child jobs of the selected parent job. You can also view parent details in the child app. (ASP-1552)

      • Added a Close icon next to the selected stage. Click the icon to return to the App Summary page. (UIX-5045)

      For information, see the BigQuery Details page section in User Guide.

    • On-demand analysis pricing is configurable for different regions

      • You can now configure the on-demand analysis pricing for different regions, including custom prices, using the com.unraveldata.bigquery.ondemand.analysis.pricing.per.tb.usd.{region} property.

        For more information about this property, see BigQuery properties in Reference Guide.

        To configure on-demand pricing, see the Configuring on-demand pricing to process queries section in Configuration Guide.

  • Other enhancements

    • Upgraded the Node.js Javascript to the 16.18.0 version. (UIX-4461)

Unsupported

  • In GCP - BigQuery, for the Data page, a count of more than 100 projects is not supported.

  • For BigQuery pricing, Unravel only supports On-demand analysis pricing. Flat-rate analysis pricing and Storage pricing (Active and Long Term storage) is not supported.

Bug fixes

  • BigQuery

    • Incorrect column count for SelectStartEvent. (BIGQ-79)

  • Insights

    • The Elasticsearch cache issue is observed on the Insights Overview dashboard. (CDI-660)

    • After clicking the READ operator link on the Analysis tab, no action is triggered. (UIX-5067)

  • Workflow/Job page displays empty for Analysis, Resources, Daggraph, and Errors tabs. (DT-1093)

  • Event logs and YARN logs are not loaded for some applications in Google Dataproc clusters. (ASP-1372)

  • When multiple AutoActions policies are created with the Overlapping ruleset and scopes, only one of the AutoAction policies is triggered. (AA-498)

  • AutoActions stop responding due to an invalid or unsupported HTTP URL or webhook. (AA-575)

  • App store tasks fail to start with SSL. (APP-614)

    Workaround

    To resolve this issue, do the following:

    1. Stop Unravel.

      <Unravel installation directory>/unravel/manager stop
      
    2. Use an editor to open <Installation_directory>/unravel/data/conf/unravel.yaml file.

    3. In the unravel.yaml file, under the database > advanced > python_flags block, enter the path to the trusted certificates. For example, if Unravel is installed at /opt/unravel, you must edit the unravel.yaml file as follows:

      unravel:
      ...snip...
        database:
      ...snip...
          advanced:
            python_flags:
              ssl_ca: /opt/unravel/data/certificates/trusted_certs.pem
    4. Use the manager utility to upload the certificates.

      <Unravel installation directory>/manager config tls trust add --pem /path/to/certificate

      For example: /opt/unravel/manager config tls trust add --pem /path/to/certificate

    5. Enable the Truststore.

      <Unravel installation directory>/manager config tls trust enable
    6. Apply the changes and restart Unravel.

      <Unravel installation directory>/unravel/manager config apply --restart
      
  • On the Application details page, the original query link is missing for some cached queries due to the parallel processing of original and cached queries. (BIGQ-61)

  • Issue: Sometimes when you process a large number of BigQuery projects with the manager config bigquery integrate command, you may see the following error:

    Provider produced inconsistent result after apply

    Workaround: Wait for few minutes and re-run the command. (INSTALL-2860, INSTALL-2934)

  • If tables are created with the same name and are accessed, deleted, and re-created, and if those tables are accessed again, then their query and apps count does not match.(DATAPAGE-502)

  • For Hive metastore 3.1.0 or earlier versions, the create time of partitions is not captured if a partition is created dynamically.  Therefore, in Unravel, the “Last Day” KPIs for the partition section are not shown. (DATAPAGE-473)

  • Errors and Logs data are missing in specific scenarios for a failed Spark application, such as an application failing with OutOfMemoryError. (ASP-1624)

  • On the Chargeback page, when you group by clusters, Unravel has a limitation of only grouping a maximum of 1000 clusters. (SUPPORT-1570)

  • On the Cost > Trends and Cost > Chargeback pages, the tooltip for the Last <number of> days field includes more days than the displayed days. (UIX-5042)

  • When the Interactive cluster is restarted, the cluster count is increased on the Databricks Cost > Trends page. (DT-1275)

  • The duplicate job runs (with the same run IDs) are generated on the Job Runs page. (DT-1190)

  • After navigating from Trends and Chargeback pages with Tag filters, the No data available message is displayed on the Compute page. (DT-1094)

  • Inconsistent data is displayed for the cluster Duration and Start Time on the Compute page. (ASP-1636)

  • The DriverOOME and ExecutorOOME events are not generated for the Databricks notebook task. (DT-533)

  • When a job fails to submit a Spark application, the failed DataBricks job is missing from the Unravel UI. (ASP-1427)

  • In Databricks, when a job in a workflow fails and a new job is launched instead of a new attempt, the new job cannot be part of the same workflow. (PG-269)

  • In the Databricks view, the application is shown in a running state, even though the corresponding Spark application is marked as finished. (ASP-1436)

  • Google Cloud Dataproc: Executor Logs are not loaded for Spark applications. (ASP-1371)

  • The workflow of multiple transient clusters (EMR) is not supported. (ASP-1424, EMR-460)

  • Unable to run Spark applications on all the master nodes after Unravel bootstrap for high availability clusters. (EMR-49)

  • Unravel node fails to send email notifications. (INSTALL-1694)

  • Issue: You can encounter a NoIndexFound exception for fresh installations of Unravel on GCP-BigQuery. (BIGQ-104)

    Workaround: Run the following CURL command on the Unravel node after the installation.

    curl -XPUT http://localhost:4171/app-19700101_07
  • An exception occurs when installing Unravel version 4.7.6.0 with the Azure MySQL database (SSL Enabled). (INSTALL-2799)

  • During precheck and healthcheck, the Hadoop check fails for the MapR cluster. You can ignore the messages. (INSTALL-2603)

  • The Insights Overview tab uses UTC as the timezone, while other pages use local time. Hence, the date and time shown on the Insights Overview tab and the other pages after redirection can differ. (UIX-4176)

  • Kerberos can only be disabled manually from the unravel.yamlfile.

     kerberos:
          enabled: False
  • A blank page is displayed on the Databricks Run Details page for Spark structured streaming applications. (ASP-1629, UIX-5124)

  • If the Spark job is not running for Databricks, the values for the Duration and End time fields are not updated on the Databricks Run Details page. (ASP-1616)

  • You can see a lag for SQL Streaming applications. (PLATFORM-2764)

  • If the customer uses an active directory for Kerberos and the samAccountName and principal do not match, this can cause errors when accessing HDFS. (DOC-755)

  • For PySpark applications, the processCPUTime and processCPULoad values are not captured properly. (ASP-626)

  • SQL events generator generates SQL Like clause event if the query contains a like pattern even in the literals. (TEZLLAP-349)

  • After upgrading from v4.7.1.1 to v4.7.5.0, the Hive jobs running with the Tez application as an execution engine are not linked. (EMR-406)

  • After upgrading to v4.7.1.0, Notebooks do not work. You can configure them separately. (REPORT-1895)

  • After upgrading from v4.6.x to v4.7.1.0, the Tez application details page does not initially show DAG data. The DAG data is visible only after you refresh the page. (ASP-1126)

  • You can access the new user interface (UI) only from Chrome.

  • In the App summary page for Impala, the Query> Operator view is visible after scrolling down. (UIX-3536).

  • Issue: When you return from the application details > SQL tab> Stage page to the application details > Attempt page, the Duration, Data I/O, and Jobs Count fields are not displayed. (UIX-5048)

    Workaround: You can reload or refresh the application details > Attempt page to restore all the KPIs.

  • Jobs are falsely labeled as a Tez App for Oozie Sqoop and Shell actions. (PLATFORM-2403)

Support

For support issues, contact Unravel Support.