v4.7.1.1 Release notes
Software version
Release Date: 17/Aug/2021
See 4.7.1.1 for download information.
Software upgrade support
The following upgrade paths are supported:
4.6.1.8 or earlier → 4.6.1.9 → 4.7.1.1
4.6.1.9 → 4.7.1.1
4.7.0.x→ 4.7.1.1
4.7.1.0 → 4.7.1.1
Sensor upgrade
Not applicable.
Certified platforms
The following platform is tested and certified in this release:
Hortonworks Data Platform (HDP)
Cloudera Distribution of Apache Hadoop (CDH)
Cloudera Data Platform (CDP)
Amazon Elastic MapReduce (EMR)
Databricks
Review your platform's compatibility matrix before you install Unravel.
Updates to Unravel's configuration properties
Refer to 4.7.x - Updates to Unravel properties.
New features
None
Improvements and enhancements
None
Unsupported
Billing is not supported on EMR and on-prem platforms.
The Data insights feature is not supported for EMR and Databricks clusters.
Other than Overview and Tables, the Data page is not supported for EMR and Databricks clusters.
Impala jobs are not supported on the HDP platform.
Monitoring the expiration of the SSL Certificates and Kerberos principals in Unravel multi-cluster deployments.
Migration Planning is not supported for the following regions for Azure Data Lake:
Germany Central (Sovereign)
Germany Northeast (Sovereign)
Forecasting and Migration: In a multi-cluster environment, only a single cluster can be configured at a time; hence reports are generated only for that single cluster.
Unravel does not support multi-cluster management of combined on-prem and cloud clusters.
Unravel does not support apps belonging to the same pipeline in a multi-cluster environment but is sourced from different clusters. A pipeline can only contain apps that belong to the same cluster.
Reports are not supported on Databricks and EMR.
In Jobs > Sessions, the feature of applying recommendations and then running the newly configured app is not supported.
Pig and Cascading applications are not supported.
Bug fixes
Applications, sensors, and pipeline
NPE in processing yarn logs when the log path is misconfigured. (ASP-1171)
Storing container data has issues if exit_status is populated. (ASP-1168)
Use generic/yarn log parser for spark container/executor log parsing. (ASP-1130)
DAG all vertices table KPI, in the Tez app, is shown as blank if the value is missing or 0. (ASP-1128)
Datapage
FSImage processing in multi-cluster fails. (DATAPAGE-504)
Add support for custom path for FSImage processing. (DATAPAGE-503)
Total Number of Tables KPI is not displayed unless apps are run on existing or new tables on a fresh start of a DataPage. (DATAPAGE-498)
On the Tables page, incorrect data is displayed in the Size column for tables. (DATAPAGE-496)
Size of Tables Created trend graph is not populated under Data Page > Overview > Tables KPIs and Trend section. (DATAPAGE-495)
Incorrect data is displayed in theNumber of Partitions Created trend graph under Data Page > Overview > Partitions KPIs and Trend section. (DATAPAGE-494)
Incorrect data displayed in Apps and Users KPIs in Tables page and Applications tab under Table Details page for a table which is dropped and recreated. (DATAPAGE-493)
Incorrect data displayed in Created and Owner column under Tables page. (DATAPAGE-492)
The data displayed for Number of Tables Accessed and Number of Users is not in sync with Number of Queries KPI data. (DATAPAGE-491)
FSimage feature is not getting enabled using manager config ondemand fsimage enable --automatic-fetch command. (INSTALL-1709)
Partitions column in Tables page displays incorrect data. (DATAPAGE-489)
Incorrect data displayed in the Label column for each of the tables in the downloaded
data.csvfile. (DATAPAGE-485)The Latest Access column is not getting updated with the latest access timestamp, thereby causing wrong data to be populated in Users and Apps trend charts under Data page > Tables page. (DATAPAGE-484)
Incorrect data displayed for Size of Tables Created KPI/trend graph and Total Size of Tables trend graph under Data page > Tables page. (DATAPAGE-483)
Running FSImage leaves some KPIs in an inconsistent state until the table worker daemon is restarted. (DATAPAGE-481)
The Incorrect count is displayed by the Total Number of Tables KPI under Overview on the Data page. (DATAPAGE-479)
Number of Tables Created KPI/Trend graph does not reflect the tables data for the last day even after restarting table worker daemon where rest all KPI's reflects the data correctly. (DATAPAGE-478)
Apps count is not getting displayed for the expected tables in the Table page for a selected table. (DATAPAGE-469)
The Latest Access column does not display the date and time when the table was last accessed on the Data page > Tables page. (DATAPAGE-466)
The Latest Access column does not display the Date/Time when the table was last accessed in the Data page> Tables > Details table. (DATAPAGE-466)
Apps count is not getting displayed for the expected tables in the Table Details page for a selected table. (DATAPAGE-469)
Wrong database name displayed for a hive application in App Summary page (DATAPAGE-474)
Number of Tables Created KPI/Trend graph does not reflect the tables data for the last day even after restarting the table worker daemon. (DATAPAGE-478)
The wrong count is displayed for the Total Number of Tables KPI/Trend graph under the Data >Overview page. (DATAPAGE-479)
Running FSImage leaves some KPIs in an inconsistent state until the table worker daemon is restarted. (DATAPAGE-481)
Incorrect data is displayed for Size of Tables Created and Total Size of Tables KPI/trend graphs under Data page > Overview > Tables KPIs > Trends section. (DATAPAGE-483)
In the Data page > Tables section, the Latest Access column is not updating with the latest accessed timestamp, which causes wrong data to be populated in the Users and Apps trend charts. (DATAPAGE-484)
Databricks
Chargeback ES Query Modification. (DT-918)
Install
Health Check: No logs found:
/opt/unravel/logs/pgsql/pgsql.log. (INSTALL-1756)Health Check: The error, Can't list: /tmp/logs/*/logs/ is fixed. (INSTALL-1755)
Autoconfig: File reporting properties do not get unset when you remove the cluster configuration. (INSTALL-1733)
Spark
Spark Errors are not displayed completely. (ASP-1147)
Errors and Logs tab is empty for all previous attempts except the last attempt in the multi-attempt spark app. (ASP-849)
UI
The Overview page does not reflect the details on the tables created using the Northwind schema, whereas the Tables page reflects the same. (DATAPAGE-505)
Chargeback Download CSV shows only 15 rows. (CUSTOMER-1784)
In a shared Tez session, the DAG level database displayed is shared. (ASP-1146)
Job Runs tab shows status as a success while Spark application details page shows the status as Running. (DT-920)
Databricks workload page shows Impala in the drop-down list. (DT-919)
Load logs not loading on Unravel UI (ondemand logs). (SUPPORT-614)
Per page option for list (20,50,100) etc. (SUPPORT-612)
Label column should be renamed to State in the downloaded CSV data as this column displays the state of the table. (DATAPAGE-476)
Wrong database name displayed for a hive application on App Summary page. (DATAPAGE-474)
Duplicate apps with the same cluster UID entry are found in Elastic Storage. (ASP-1044)
Event log of Spark apps fails to process if the app contains active or unfinished stages. (ASP-1175)
Create time of partitions does not get captured in hive meta-store if the partition is created dynamically. This limits Unravel to show Last Day KPIs for the partition section. (DATAPAGE-473)
Wrong data displayed for Number of Partitions Created KPI/trend graph under Partitions KPIs - Last Day section in THEData page. (DATAPAGE-473)
Unravel node fails to send email notifications. (INSTALL-1694)
Kerberos can only be disabled manually from the
unravel.yamlfile.kerberos: enabled: False
Cluster discovery
If the metric retrieval for a host fails, then the CPU and memory capacity/usage graphs and heatmaps are not displayed.
This happens on a CDH cluster when the Cloudera Manager agent of a host does not send any heartbeats to the Cloudera Manager server. Such a host is shown as Bad Health in Cloudera Manager. (REPORT-1706)
Workaround: Ensure that the Cloudera Manager agent sends heartbeats to the Cloudera Manager on all hosts and that none of the hosts are shown as Bad Health.
The On-prem Cluster Identity may show an incorrect Spark version on CDH. The report may incorrectly show Spark 1 when Spark 2 is installed on the CDH cluster. (REPORT-1702)
When using PostgreSQL, the % sign is duplicated and displayed in the Workload Fit report > Map to single cluster tab. (MIG-42)
Cloud Mapping Per Host report scheduled in v4.6.1.x will not work in v4.7.1.0. Users must schedule a new report. (REPORT-1886)
The TopX report email contains a link to the Unravel TopX report instead of showing the report content in the email as in the old reports.
Queue analysis: The log file name (
unravel_us_1.log) displayed in the error message is incorrect. The correct name of the log file isunravel_sensor.log. (REPORT-1663)
The sensor setup script fails with unrecognized arguments. (INSTALL-1667)
There is a lag seen for SQL Streaming applications. (PLATFORM-2764)
If the customer uses an active directory for Kerberos and the samAccountName and principal do not match, this can cause errors when accessing HDFS. (DOC-755)
You cannot access Unravel UI after you upgrade from Unravel version 4.6.2.0 and enable TLS for Unravel UI. (INSTALL-1760)
Workaround:
Use the following workaround to get access to Unravel UI.
Stop Unravel.
<unravel_installation_directory>/unravel/manager stop
Copy the key and certificates to a directory.
cp /usr/local/unravel/data/certificates/new/hkl20043024.hk.abcd.key /usr/local/unravel/data/certificates/server_key.pem cp /usr/local/unravel/data/certificates/new/hkl20043024.hk.abcd.pem /usr/local/unravel/data/certificates/server_cert.pem
Enable the Truststore.
<unravel_installation_directory>/unravel/manager config tls trust enable
Apply changes.
<unravel_installation_directory>/unravel/manager config apply
Start Unravel.
<unravel_installation_directory>/unravel/manager start
For PySpark applications, the processCPUTime and the processCPULoad are not captured properly. (USPARK-626)
SQL events generator generates SQL Like clause event if the query contains like pattern even in the literals. (TEZLLAP-349)
Dag additional input and output node is not plotted for 4710 Tez applications. (ASP-1114)
Notebooks will not work after upgrading to v4.7.1.0. You can configure them separately. (REPORT-1895)
After you upgrade from v4.6.x to v4.7.1.0, the Tez application details page does not initially show DAG data. The DAG data is visible only after you refresh the page. (ASP-1126)
The new user interface (UI) can be accessed only from Chrome.
In the App summary page for Impala, the Query> Operator view is visible after scrolling down. (UIX-3536).
Jobs getting falsely labeled as a Tez App for Oozie Sqoop and Shell actions. (PLATFORM-2403)