v4.7.8.0 Release notes
Software version
Release date: 20/February/2023
See 4.7.8.0 for download information.
Software upgrade support
The following upgrade paths are supported:
4.7.x.x → 4.7.8.0
4.6.1.9 → 4.7.8.0
4.6.1.8 or earlier → 4.6.1.9 → 4.7.8.0
For instructions to upgrade to Unravel v4.6.1.9, see Upgrading Unravel server.
For instructions to upgrade to Unravel v4.7.8.x, see Upgrading Unravel.
For fresh installations, see Installing Unravel.
Certified platforms
The following platforms are tested and certified in this release:
Amazon EMR
Databricks (Azure, AWS)
Review your platform's compatibility matrix before you install Unravel.
Updates to Unravel's configuration properties
See 4.7.x - Updates to Unravel properties.
Updates to upgrading Unravel to v4.7.8.0
An existing license for any previous version (before 4.7.7.x) does not work with the newer version of Unravel. Therefore, before upgrading Unravel , you must obtain a license file from Unravel Customer Support. For information about setting the license, see Upgrading Unravel from version 4.7.x to 4.7.8.x section in Upgrading Unravel.
Optionally, you can regroup multiple Spark worker instances for enhanced performance after upgrading to v4.7.8.0.
Caution
This task requires planning and can be performed only in collaboration with Unravel support team. This is a one-time task.
New features
Data quality integration with Great Expectations
Great Expectations is a product quality tool that enables you to run validations against your data asset by running an Expectation Suite (quality assertions) against it. Great Expectations when integrated with Unravel extends the measure of data quality into Unravel. At the same time, Unravel provides unified visibility of any expectations validated while running the Expectation Suite. Thus adding data quality insights to Unravel's current single-pane data monitoring aspect.
You can view the Data Quality insights of Great Expectations from the Unravel UI > Data > Tables detail page> Analysis tab and also from the Unravel UI > Jobs > Job details page > Analysis tab.
Multi-node deployment of Spark workers for high-volume data processing
You can deploy additional Spark workers on a separate server, other than the server where Unravel is installed, with services to process high-volume data.
Notification channels
A new Notification channels option has been added to the
Manage menu, using which you can set up notification channels to receive alerts when certain conditions are triggered. Use notifications to send alerts through email addresses or Slack messages to users or user groups.For information about the Notification channel, see the following topics:
Topics
Guide name
New topics
Notification channels
Creating a notification channel
Modifying the existing notification channel
Viewing notification channels
Updates to the existing topics
Cost Budget (EMR)
Creating a budget
Viewing a budget and its details
Updates to the existing topics
Cost Budget (Databricks)
Setting a budget
AutoActions support EMR apps and clusters to optimize cost
AutoActions can now monitor EMR apps and clusters. You can set the AutoAction policy to generate alerts for EMR apps and clusters. AutoActions can monitor EMR clusters based on cost, duration, and idle checks and send alerts.
For more information, refer to AutoActions > AutoActions (EMR) topic in User Guide.
Improvements and enhancements
Databricks enhancements
A Databricks Job can be associated with multiple clusters. Each job entry now corresponds to a Databricks job. The following enhancements have been made to the Databricks Workflows > Jobs page (DT-1187):
Pages
Changes
Workflows>Jobs
Removed the Clusters Name column
Workflows>Job Runs
Removed the Clusters Name and Cluster Type columns
Removed the Job name link from the Run Name / ID column.
Renamed the Run Name / ID to Job name / ID
Provided a link to the Run ID. After clicking the Run ID, the job run detail page is displayed.
Updated the Search by ID, Keyword field to Search by keyword. You can search for the job name by typing the keyword.
Changed the Filter by Cluster Name search to Filter by Job name or ID
The following enhancement is done for the Resources tab on the Spark details page. (DT-1456)
Pages
Changes
Compute>Spark>Resources>Host Metrics and Workflow>Job>Task>Resources>Host Metrics
The following new metrics are added to Host metrics:
Total memory
Free memory
You can use these metrics to evaluate the memory spent on processes other than those of Spark.
For information, see User Guide.
Other enhancements
Node count and duration values are provided for the aggregated cost savings for each recommended node type. (EMR-620)
The new Account Id column has been added to the AWS Account Settings page to view configured AWS account ID in the Unravel UI. (UIX-5469)
On the Clusters page, the ID filter has been relocated to the top and is separate. You cannot combine other filters (such as Date and time range) with an ID search. (UIX-5332)
For information, see the Monitor EMR clusters section in the User Guide.
The MySQL client library has been updated to the 12.0 version on the user interface. (UIX-5383)
Enhanced performance by reducing the lag in the Impala pipeline. (ASP-1677)
Support for downloading as CSV option for EMR Clusters and EMR AutoAction pages. (UIX-4853)
For information, see Viewing AutoAction and its details and Monitor EMR clusters sections in the User Guide.
Support for the EMR cluster
idlestate (EMR-465)Unravel now supports AutoAction for the
idlestate of the cluster. You can set AutoAction when the EMR cluster exceeds the idle duration threshold. For information, see Creating AutoActions in User Guide.
Unsupported
In GCP - BigQuery, for the Data page, a count of more than 100 projects is not supported.
For BigQuery pricing, Unravel only supports On-demand analysis pricing. Flat-rate analysis pricing and Storage pricing (Active and Long Term storage) is not supported.
Bug fixes
AutoActions
When multiple AutoActions policies are created with the Overlapping ruleset and scopes, only one of the AutoAction policies is triggered. (AA-498)
Databricks
The duplicate job runs (with the same run IDs) are generated on the Job Runs page. (DT-1190)
On the Compute page, inaccurate information is displayed for clusters in the Inefficient category. (UIX-5064)
The downloaded TopX Report (in JSON format) lists the incorrect type of Spark app. (REPORT-2094)
In Databricks, when a job in a workflow fails and a new job is launched instead of a new attempt, the new job cannot be part of the same workflow. (PG-269)
On the Chargeback page, when you group by clusters, Unravel has a limitation of only grouping a maximum of 1000 clusters. (SUPPORT-1570)
EMR
After clicking the Hive Query link on a cluster using the bootstrap script, the No apps found with the Id message is displayed. (CLOUD-532)
On the Clusters page, search by cluster name returns incomplete search results. (UIX-5345)
On the Clusters page, the Name and Cluster tags filters return incomplete search results. (EMR-595)
On the Clusters page, the following issues are observed (EMR-588):
The cluster list omits clusters with a zero cost when the custom date range is selected
The cluster list omits the latest cluster cost when the custom date range is selected
If clusters terminate with errors without generating NodeDownSizingEvent, then such clusters are displayed in the Inefficient category on the Clusters page. (EMR-542)
The Spark sensor fails to start. (EMR-485)
On the Clusters page, a mismatch in the cluster IDs displayed in the ID drop-down list with the selected cluster category in the left panel. (EMR-435)
For clusters terminated with errors, the node downsizing recommendations are shown. (EMR-422)
Insights
Clicking the links for operators and stages in the SQLTooManyGroupByEvent does not result in any action. (INSIGHTS-355)
An exception occurs when generating memory insights for a Spark application. (INSIGHTS-363)
Installation
Databricks Healthcheck App Store celery daemon fails to start. (INSTALL-2945)
Installing Unravel fails when connecting with SSL-enabled MariaDB. (INSTALL-3071)
Spark
A blank page is displayed on the Databricks Run Details page for Spark structured streaming applications. (ASP-1629, UIX-5124)
UI
On the Clusters page, a discrepancy exists between the cost of clusters and the minimum and maximum cost displayed in the left pane. (UIX-5270)
From the Clusters page, after clicking the Spark action, refreshing the Spark details page takes longer than expected. (UIX-5247)
When you return from the application details > SQL tab> Stage page to the application details > Attempt page, the Duration, Data I/O, and Jobs Count fields are not displayed. (UIX-5048)
AutoActions stop responding due to an invalid or unsupported HTTP URL or webhook. (AA-575)
On the Application details page, the original query link is missing for some cached queries due to the parallel processing of original and cached queries. (BIGQ-61)
Issue: Sometimes, when you process a large number of BigQuery projects with the manager config bigquery integrate command, you may see the following error:
Provider produced inconsistent result after applyWorkaround: Wait for a few minutes and re-run the command. (INSTALL-2860, INSTALL-2934)
If tables are created with the same name and are accessed, deleted, and re-created, and if those tables are re-accessed, then their query and app count does not match.(DATAPAGE-502)
For Hive metastore 3.1.0 or earlier versions, the create time of partitions is not captured if a partition is created dynamically. Therefore, in Unravel, the Last Day KPI for the partition section are not shown. (DATAPAGE-473)
Issue: You can encounter a NoIndexFound exception for fresh installations of Unravel on GCP-BigQuery. (BIGQ-104)
Workaround: Run the following CURL command on the Unravel node after the installation.
curl -XPUT http://localhost:4171/app-19700101_07
Kerberos can only be disabled manually from the
unravel.yamlfile.kerberos: enabled: False
After upgrading to v4.7.1.0, Notebooks do not work. You can configure them separately. (REPORT-1895)