v4.7.7.0 Release notes
Software version
Release date: 09/December/2022
See 4.7.7.0 for download information.
Software upgrade support
The following upgrade paths are supported:
v4.7.0.x, v4.7.1.x, v4.7.2.x, v4.7.3.x, v4.7.4.x, v4.7.5.x, v4.7.6.x → v4.7.7.0
v4.6.1.9 → v4.7.7.0
v4.6.1.8 or earlier → v4.6.1.9 → v4.7.7.0
For instructions to upgrade to Unravel v4.6.1.9, see Upgrading Unravel server.
For instructions to upgrade to Unravel v4.7.7.x, see Upgrading Unravel.
For fresh installations, see Installing Unravel.
Certified platforms
The following platforms are tested and certified in this release:
Google Cloud Platform (BigQuery, Dataproc)
Review your platform's compatibility matrix before you install Unravel.
Updates to Unravel's configuration properties
See 4.7.x - Updates to Unravel properties.
Updates to upgrading Unravel to v4.7.7.0
An existing license for any previous version (before 4.7.7.x) does not work with the newer version of Unravel. Therefore, before upgrading Unravel you must obtain a license file from Unravel Customer Support. For information about setting the license, see Upgrading Unravel from version 4.7.x to 4.7.7.x section in Upgrading Unravel.
In this release, Unravel supports the pull model, along with the push model, to get data from BigQuery projects. After the upgrade, the push model, which was used in previous releases, will be continued. However, the pull model can be implemented only with a fresh install of Unravel.
When you have enabled both Bigquery and Dataproc before upgrading to Unravel v4.7.7.0, then after the upgrade, you must enable Hadoop daemons (hive_worker, kafka_monitor, yarn_jc_worker, oozie_sensor) manually. see Upgrading Unravel from version 4.7.x to 4.7.7.x section in Upgrading Unravel for complete instructions.
For best performance, enable message ordering in GCP Cloud Pub/Sub. The Message ordering setting lets you receive the messages in the order they are published. Refer to Ordering messages for more details. After you upgrade, you cannot directly update the Message ordering setting.
To enable the Message ordering setting after an upgrade, do the following:
From the GCP console, delete the configured subscription for the respective project.
Create a new subscription with the Message ordering setting on.
Note
The Message ordering setting can be configured only once when configuring the Pub/Sub service for the first time.
New features
License
Replaced the license key with the license file for expandability and human readability. A license file contains product licensing information, the license validity, and the expiration date.
Unravel no longer supports an automatic trial period. You must obtain a license file from Unravel Customer Support.
For information about the Licensing documentation, see the following topics:
Topics
Guide
Setting Unravel license
Verifying Unravel license
License: new property
License FAQ
Licensing error messages and troubleshooting
Install Unravel on GCP Dataproc using Interactive precheck
Install Unravel on GCP Dataproc manually
Installing Unravel for GCP BigQuery
Manually installing Unravel for GCP BigQuery
Use the Interactive Precheck utility
Upgrading Unravel for Cloud platforms (v4.7.x to v4.7.7.0)
AutoActions for BigQuery
Unravel now supports AutoActions for BigQuery. An updated user interface is provided for AutoActions in GCP with improved usability.
For more information, refer to AutoActions > AutoActions (GCP - BigQuery, Dataproc) topic in User Guide.
BigQuery pull model
Unravel now supports getting data about BigQuery from GCP with the pull model. Previously, only the push model was supported. The pull model is available only for fresh installations.
Refer to the Installation Guide > Google Cloud platform - BigQuery installation section for more details.
Secure connection with Cloud SQL Auth proxy to external MySQL and PostgreSQL database
For the Google Cloud Platform (GCP), you can enable a secure connection between Unravel server and an external MySQL or PostgreSQL database. This can be done using the Cloud SQL Auth proxy, which is a Cloud SQL connector that provides secure access to your instances without needing Authorized networks for configuring SSL. Refer to Enabling secure connection from Unravel GCP to external MySQL database with Cloud SQL Auth proxy section in the Configuration Guide.
Improvements and enhancements
Installation
Added support for the
.xzarchive format when configuring Unravel managed database with the--extrasetup option. (INSTALL-2773)When installing for BigQuery, the Interactive Precheck utility now prompts for the push and pull options. (INSTALL-2817)
BigQuery enhancements
Enhancements to the BigQuery Application Details page:
Added support for different job types displayed on the BigQuery details page. The supported job types are QUERY, LOAD, COPY, and EXTRACT. The new Type KPI shows the type of job run in BigQuery. (UIX-5003)
On the Tables tab, the tables are now categorized into Input and Output columns. (UIX-4939)
Added a new Child Jobs tab to display child jobs of the selected parent job. You can also view parent details in the child app. (ASP-1552)
Added a Close icon next to the selected stage. Click the icon to return to the App Summary page. (UIX-5045)
For information, see the BigQuery Details page section in User Guide.
On-demand analysis pricing is configurable for different regions
You can now configure the on-demand analysis pricing for different regions, including custom prices, using the com.unraveldata.bigquery.ondemand.analysis.pricing.per.tb.usd.{region} property.
For more information about this property, see BigQuery properties in Reference Guide.
To configure on-demand pricing, see the Configuring on-demand pricing to process queries section in Configuration Guide.
Other enhancements
Upgraded the
Node.jsJavascript to the 16.18.0 version. (UIX-4461)
Unsupported
In GCP - BigQuery, for the Data page, a count of more than 100 projects is not supported.
For BigQuery pricing, Unravel only supports On-demand analysis pricing. Flat-rate analysis pricing and Storage pricing (Active and Long Term storage) is not supported.
Bug fixes
BigQuery
Incorrect column count for SelectStartEvent. (BIGQ-79)
Insights
The Elasticsearch cache issue is observed on the Insights Overview dashboard. (CDI-660)
After clicking the READ operator link on the Analysis tab, no action is triggered. (UIX-5067)
Workflow/Job page displays empty for Analysis, Resources, Daggraph, and Errors tabs. (DT-1093)
Event logs and YARN logs are not loaded for some applications in Google Dataproc clusters. (ASP-1372)
When multiple AutoActions policies are created with the Overlapping ruleset and scopes, only one of the AutoAction policies is triggered. (AA-498)
AutoActions stop responding due to an invalid or unsupported HTTP URL or webhook. (AA-575)
On the Application details page, the original query link is missing for some cached queries due to the parallel processing of original and cached queries. (BIGQ-61)
Issue: Sometimes when you process a large number of BigQuery projects with the manager config bigquery integrate command, you may see the following error:
Provider produced inconsistent result after applyWorkaround: Wait for few minutes and re-run the command. (INSTALL-2860, INSTALL-2934)
If tables are created with the same name and are accessed, deleted, and re-created, and if those tables are accessed again, then their query and apps count does not match.(DATAPAGE-502)
For Hive metastore 3.1.0 or earlier versions, the create time of partitions is not captured if a partition is created dynamically. Therefore, in Unravel, the “Last Day” KPIs for the partition section are not shown. (DATAPAGE-473)
Errors and Logs data are missing in specific scenarios for a failed Spark application, such as an application failing with
OutOfMemoryError. (ASP-1624)On the Chargeback page, when you group by clusters, Unravel has a limitation of only grouping a maximum of 1000 clusters. (SUPPORT-1570)
On the Cost > Trends and Cost > Chargeback pages, the tooltip for the Last <number of> days field includes more days than the displayed days. (UIX-5042)
When the Interactive cluster is restarted, the cluster count is increased on the Databricks Cost > Trends page. (DT-1275)
The duplicate job runs (with the same run IDs) are generated on the Job Runs page. (DT-1190)
After navigating from Trends and Chargeback pages with Tag filters, the
No data availablemessage is displayed on the Compute page. (DT-1094)Inconsistent data is displayed for the cluster Duration and Start Time on the Compute page. (ASP-1636)
The DriverOOME and ExecutorOOME events are not generated for the Databricks notebook task. (DT-533)
When a job fails to submit a Spark application, the failed DataBricks job is missing from the Unravel UI. (ASP-1427)
In Databricks, when a job in a workflow fails and a new job is launched instead of a new attempt, the new job cannot be part of the same workflow. (PG-269)
In the Databricks view, the application is shown in a running state, even though the corresponding Spark application is marked as finished. (ASP-1436)
Google Cloud Dataproc: Executor Logs are not loaded for Spark applications. (ASP-1371)
Unravel node fails to send email notifications. (INSTALL-1694)
Issue: You can encounter a NoIndexFound exception for fresh installations of Unravel on GCP-BigQuery. (BIGQ-104)
Workaround: Run the following CURL command on the Unravel node after the installation.
curl -XPUT http://localhost:4171/app-19700101_07
An exception occurs when installing Unravel version 4.7.6.0 with the Azure MySQL database (SSL Enabled). (INSTALL-2799)
The Insights Overview tab uses UTC as the timezone, while other pages use local time. Hence, the date and time shown on the Insights Overview tab and the other pages after redirection can differ. (UIX-4176)
Kerberos can only be disabled manually from the
unravel.yamlfile.kerberos: enabled: False
TopX report
The TopX report email links to the Unravel TopX report instead of showing the report content in the email as in the old reports.
A blank page is displayed on the Databricks Run Details page for Spark structured streaming applications. (ASP-1629, UIX-5124)
If the Spark job is not running for Databricks, the values for the Duration and End time fields are not updated on the Databricks Run Details page. (ASP-1616)
You can see a lag for SQL Streaming applications. (PLATFORM-2764)
For PySpark applications, the
processCPUTimeandprocessCPULoadvalues are not captured properly. (ASP-626)
SQL events generator generates SQL Like clause event if the query contains a like pattern even in the literals. (TEZLLAP-349)
After upgrading from v4.7.1.1 to v4.7.5.0, the Hive jobs running with the Tez application as an execution engine are not linked. (EMR-406)
After upgrading to v4.7.1.0, Notebooks do not work. You can configure them separately. (REPORT-1895)
After upgrading from v4.6.x to v4.7.1.0, the Tez application details page does not initially show DAG data. The DAG data is visible only after you refresh the page. (ASP-1126)
You can access the new user interface (UI) only from Chrome.
In the App summary page for Impala, the Query> Operator view is visible after scrolling down. (UIX-3536).
Issue: When you return from the application details > SQL tab> Stage page to the application details > Attempt page, the Duration, Data I/O, and Jobs Count fields are not displayed. (UIX-5048)
Workaround: You can reload or refresh the application details > Attempt page to restore all the KPIs.
Jobs are falsely labeled as a Tez App for Oozie Sqoop and Shell actions. (PLATFORM-2403)