v4.7.9.0 Release notes
Software version
Release date: 08/May/2023
See 4.7.9.0 for download information.
Software upgrade support
The following upgrade paths are supported:
4.7.x.x → 4.7.9.0
4.6.1.9 → 4.7.9.0
4.6.1.8 or earlier → 4.6.1.9 → 4.7.9.0
For instructions to upgrade to Unravel v4.6.1.9, see Upgrading Unravel server.
For instructions to upgrade to Unravel v4.7.9.x, see Upgrading Unravel.
For fresh installations, see Installing Unravel.
Certified platforms
The following platforms are tested and certified in this release:
On-premise platforms
Cloudera Distribution of Apache Hadoop (CDH)
Cloudera Data Platform (CDP)
Hortonworks Data Platform (HDP)
Review your platform's compatibility matrix before you install Unravel.
Updates to Unravel's configuration properties
See 4.7.x - Updates to Unravel properties.
Updates to upgrading Unravel to v4.7.9.0
An existing license for any previous version does not work with the newer version of Unravel. Therefore, before upgrading Unravel, you must obtain a license file from Unravel Customer Support. For information about setting the license, see Upgrading Unravel from version 4.7.x to 4.7.9.x section in Upgrading Unravel.
In the case of an Unravel version before v4.6.1.9, you have a customized UI port, for example, 4000, then after the upgrade, this port reverts to the default 3000. Hence, after you upgrade to 4.7.9.0, you must reconfigure the custom port.
Stop Unravel.
<Unravel installation directory>/unravel/manager stop
Display the list of ports and keys.
<Unravel installation directory>/unravel/manager config ports show
Set the port using the following command with the port and port key.
<Unravel installation directory>/unravel/manager config ports set <port key> <port>
For example, if you run the following command, NGUI listens on port 1234.
<Unravel installation directory>/unravel/manager config ports set /hosts/host_main/instances/ngui_1/config/network/port 1234
Note
Run manager config ports unset <port key> command to return the ports to their default values.
<Unravel installation directory>/unravel/manager config ports unset /hosts/host_main/instances/appstore_1/config/flask/port
Apply the changes.
<Unravel installation directory>/unravel/manager config apply
Start Unravel.
<Unravel installation directory>/unravel/manager start
After you have migrated from an Hortonworks Data Platform (HDP) to a Cloudera Data Platform (CDP) cluster, there are some properties you must unset and new properties that you must set on an edge node of the migrated cluster.
On the edge node, set the following property:
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.cluster.type CDPIn case HBase was configured earlier, then set the following property:
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.hbase.source.type=CDPUnset the following properties:
<Unravel installation directory>/unravel/manager config properties unset com.unraveldata.ambari.manager.url<Unravel installation directory>/unravel/manager config properties unset com.unraveldata.ambari.manager.username<Unravel installation directory>/unravel/manager config properties unset com.unraveldata.ambari.manager.password<Unravel installation directory>/unravel/manager config properties unset com.unraveldata.yarn.timeline-service.webapp.address<Unravel installation directory>/unravel/manager config properties unset com.unraveldata.yarn.timeline-service.portRun manager config auto command to automatically pull in all the Hadoop configurations. You will be prompted to provide the location and credentials for the CDP Cloudera manager URL.
<Unravel installation directory>/unravel/manager config autoIf there are more than one clusters that are handled by the CDP Cloudera manager, then you are prompted to enable the cluster that you want to monitor. Run the following command to enable a cluster:
<Unravel installation directory>/unravel/manager config cluster enable<CLUSTER KEY>Example: /opt/unravel/manager config cluster enable cluster1
Apply the changes.
<Unravel installation directory>/unravel/manager config applyStart Unravel.
<Unravel installation directory>/unravel/manager startEnsure to make the following changes in the Cloudera manager:
HDP/CDH has hive hooks under HIVE service but for CDP it is under HIVE_ON_TEZ Hence, you must update the properties accordingly.
Update CDP Version for sensor parcel in Parcel Repository & Network from Cloudera Manager to https://xxx.unraveldata.com:3000/parcels/cdh7.1/
Check for new parcels and distribute them.
In case of a multi-cluster environment, do the following:
Log in to the core node as an Unravel user.
Unset the following properties on the core node:
<Unravel installation directory>/unravel/manager config properties unset com.unraveldata.cluster.type<Unravel installation directory>/unravel/manager config properties unset com.unraveldata.ambari.manager.url<Unravel installation directory>/unravel/manager config properties unset com.unraveldata.ambari.manager.username<Unravel installation directory>/unravel/manager config properties unset com.unraveldata.ambari.manager.password<Unravel installation directory>/unravel/manager config properties unset com.unraveldata.ambari.manager.<EDGE KEY>.url<Unravel installation directory>/unravel/manager config properties unset com.unraveldata.ambari.manager.<EDGE KEY>.username<Unravel installation directory>/unravel/manager config properties unset com.unraveldata.ambari.manager.<EDGE KEY>.password<Unravel installation directory>/unravel/manager config properties unset com.unraveldata.ambari.manager.listTip
Run the following commands to obtain the
<EDGE KEY>.<unravel_installation_directory>/unravel/config edge showRun the following commands to obtain the
<CLUSTER KEY>.<unravel_installation_directory>/unravel/manager support show cluster_access_id/unravel/config edge showNote
For a multi-cluster environment, if your cluster name changes after the migration from HDP to CDP you must unset the following properties on the core node:
<Unravel installation directory>/unravel/manager config properties unset javax.jdo.option.<EDGE KEY>_<CLUSTER KEY>_HIVE.ConnectionURL<Unravel installation directory>/unravel/manager config properties unset javax.jdo.option.<EDGE KEY>_<CLUSTER KEY>_HIVE.ConnectionDriverName<Unravel installation directory>/unravel/manager config properties unset javax.jdo.option.<EDGE KEY>_<CLUSTER KEY>_HIVE.ConnectionUserName<Unravel installation directory>/unravel/manager config properties unset javax.jdo.option.<EDGE KEY>_<CLUSTER KEY>_HIVE.ConnectionPassword<Unravel installation directory>/unravel/manager config properties unset hive.metastore.<EDGE KEY>_<CLUSTER KEY>_HIVE.cluster.ids<Unravel installation directory>/unravel/manager config properties unset hive.metastore.<EDGE KEY>_<CLUSTER KEY>_HIVE.cluster.ids<Unravel installation directory>/unravel/manager config properties unset com.unraveldata.hive.metastore.listRun the following command:
<Unravel installation directory>/unravel/manager config edge auto<EDGE KEY>Apply the changes.
<Unravel installation directory>/unravel/manager config applyStart Unravel.
<Unravel installation directory>/unravel/manager start
New features
Horizontal scaling of Log Receiver
Unravel now supports deploying Log Receiver on a dedicated node. This support allows Log Receiver to process much more requests. In addition, you can deploy multiple instances of Log Receiver with a load balancer to distribute the workload efficiently.
For information, see the following topics:
Topics
Guide name
New topics
Log Receiver (LR) Load Balancer
Configuring load balancing for Log Receiver (LR)
Moving Log Receiver (LR) from core node to worker node
Log Receiver (LR) performance statistics
New topic
Load balancer FAQs
Updated topic
Log Receiver (LR) properties
Improvements and enhancements
Redesign of the user interface for AutoActions
The user interface of the AutoActions page has been improved to align with the AutoActions page of the cloud platform.
For more information, refer to AutoActions > AutoActions topic in User Guide.
API Token
The user interface of the API token page is enhanced for consistency. (UIX-5612)
API tokens can be generated for a user role. Moreover, you can select tags to associate with the API tokens generated for the user roles. (UIX-5854, UIX-5739)
Other enhancements
Support for downloading as a CSV option has been added on multiple pages of the Unravel UI. (CUSTOMER-2121, CUSTOMER-2139, CUSTOMER-2069)
You can now download the following details:
The cost summary, cost of clusters, and VM cost details from the Workload fit report.
Inefficient jobs from the Jobs > Applications page.
Topic summary from the Kafka > Metrics tab.
Export details of the resources used for the running jobs from the Clusters > Job Trends page and Clusters > Resources pages.
Provide AutoAction support for Impala-tagged workflows. (ASI-735)
Node.js has been upgraded to the 16.19.1 version. (UIX-5751)
Multi-cluster support is enabled (only for CDH and CDP platforms) for migration reports. (CUSTOMER-2402) and (CUSTOMER-2399)
To replicate AutoActions, a new Duplicate option has been provided on the AutoAction list and AutoAction Details pages. This option replaced the existing Expert Rule functionality. (CUSTOMER-2359)
In a multi-cluster setup, you can choose the cluster from where you want to generate the cloud migration reports. (CUSTOMER-2179)
On the Clusters > Overview, Clusters > Resources, and Clusters > Job Trends pages, for certain selected time ranges, you can now select an interval that you want for the data points from a drop-down list.
The Tez App details page has been revamped for the Hive pipeline improvements. For failed or killed apps, errors are displayed on the Errors tab instead of the Diagnostics tab. (PIPELINE-1764)
Unsupported
On the Data page, File Reports, Small File reports, and file size information are not supported for Dataproc clusters.
In GCP - BigQuery, for the Data page, a count of more than 100 projects is not supported.
For BigQuery pricing, Unravel only supports On-demand analysis pricing. Flat-rate analysis pricing and Storage pricing (Active and Long Term storage) is not supported.
Bug fixes
Report
The sensor log file name in the AppStore log needs to be updated for accuracy. (REPORT-2103)
UI
On the Clusters > Resources page, the selected cluster value is modified if the value in the Resource Usage/Resource Type list changes (from Impala to Yarn) or if the tabs are switched. (CUSTOMER-2161)
On the Clusters > Resources page, the page loading time is unusually long when a cluster is selected from the Cluster drop-down list. (CUSTOMER-2163)
Event logs and YARN logs are not loaded for some applications in Google Dataproc clusters. (ASP-1372)
AutoActions stop responding due to an invalid or unsupported HTTP URL or webhook. (AA-575)
On the Application details page, the original query link is missing for some cached queries due to the parallel processing of original and cached queries. (BIGQ-61)
Issue: Sometimes, when you process a large number of BigQuery projects with the manager config bigquery integrate command, you may see the following error:
Provider produced inconsistent result after applyWorkaround: Wait for a few minutes and re-run the command. (INSTALL-2860, INSTALL-2934)
If tables are created with the same name and are accessed, deleted, and re-created, and if those tables are re-accessed, then their query and app count does not match.(DATAPAGE-502)
For Hive metastore 3.1.0 or earlier versions, the creation time of partitions is not captured if a partition is created dynamically. Therefore, in Unravel, the Last Day KPI for the partition section is not shown. (DATAPAGE-473)
On the Data page, size data is missing for certain tables in databases, although the partition size is correctly displayed in the Partition Detail section. (DATAPAGE-695)
Google Cloud Dataproc: Executor Logs are not loaded for Spark applications. (ASP-1371)
Issue: You can encounter a NoIndexFound exception for fresh installations of Unravel on GCP-BigQuery. (BIGQ-104)
Workaround: Run the following CURL command on the Unravel node after the installation.
curl -XPUT http://localhost:4171/app-19700101_07
Kerberos can only be disabled manually from the
unravel.yamlfile.kerberos: enabled: False
If the customer uses an active directory for Kerberos and the samAccountName and principal do not match, this can cause errors when accessing HDFS. (DOC-755)
SQL events generator generates SQL Like clause event if the query contains a like pattern even in the literals. (TEZLLAP-349)
After upgrading from v4.7.1.1 to v4.7.5.0, the Hive jobs running with the Tez application as an execution engine are not linked. (EMR-406)
After upgrading to v4.7.1.0, Notebooks do not work. You can configure them separately. (REPORT-1895)
After upgrading from v4.6.x to v4.7.1.0, the Tez application details page does not initially show DAG data. The DAG data is visible only after you refresh the page. (ASP-1126)
Jobs are falsely labeled as a Tez App for Oozie Sqoop and Shell actions. (PLATFORM-2403)