Release Date: 05/06/2019
See v4.5.0.7 for download information.
Please also review the Compatibility Matrix before upgrade/install.
CDH: On-premise CDH 6.0, 5.16, 5.15 , 5.13
HDP: On-premise HDP 3.0, 2.6.5, 2.6.4
MapR: 6.1.0, 6.0.0, 5.2.0
OS
RedHat/CentOS: 6.6-7.5
SELinux with Enforcement mode on Red Hat Enterprise Linux Server release 7.4 (Maipo)
Targeted Policy v.28
Database (PostgreSQL is bundled)
MySQL: 5.5, 5.6, 5.7 (recommended)
Database driver
MySQL: 5.1.47
Browsers
Chrome: 70.x
Internet Explorer: IE 11
ATSv2.0.
Small Files Report in Map:R.
Long Running Hive Query auto action not supported on HDP-3.0 +.
If you are upgrading from
4.3.x or 4.4.x you must upgrade the Unravel Sensor.
4.5.x you do not have to upgrade. However not upgrading limits some features and functionality, e.g., Unravel collects more metrics which improves the functionality of AutoActions.
If you doing a new/fresh install you must install the new sensors.
New Sensors
Parcel version: 1.0.4507001
Sensor version: 4.5.0.7rc0084
AutoActions
The policy violation messages can be delivered using SlackApp's incoming webhooks.
AutoActions
Only metrics which arrive before com.unraveldata.auto.action.metric.discard.ms are discarded and not published. (AA-30)
Hive-on-Spark and Hive-on-MR applications now alert in running state. (HIVE-72)
Added async polling of RM in jcs2. (PLATFORM-1348)
Fixed deduplication of metric stream in AA. (AA-37)
Spark
Made it clearer to users why Unravel does not load (BTrace) Resource Metrics until app completes.
Made general improvements to Containers, VCores, Memory Graphs.
Load Logs Improvements
Progress is captured in the UI while Load Logs is running.
Event logs are automatically loaded when not already loaded.
AppBadge messages under 'Load Diagnostics' explain reason for failure to load.
Made general improvements to Spark query table.
Spark BTrace Sensor Improvements
Split Spark SQL update from sensor into multiple messages.
Added new property, com.unraveldata.spark.shutdown.delay.ms, so that last messages are processed (allows BTrace sensor to send all the data before the spark driver exits).
Spark Enhancements
Spark live pipeline now updates job duration and data IO while the application is running.
Spark jobs and Hive query are linked for Spark-on- Hive applications.
Airflow
Airflow DAGs and Tasks processing are now scalable for large clusters.
Added new properties to control look-back, parsing, and whether to process logs in parallel or sequentially.
Sessions
Improved Session's algorithm for speeding up Spark applications.
Hive
Added support for running Hive-on-MR.
Added support for Hive-on-Spark.
AutoAction duration metric is now published for running Hive-on-MR and Hive-on-Sparks allowing alerts while running.
OnDemand
OnDemand is compatible with RHEL 7.1.
Kerberos
Added support for custom location for Kerberos client configuration. (CUSTOMER-774)
Customer reported
AutoActions for workflows now working per the condition specified in policy. (CUSTOMER-632)
Illegal Argument errors no longer logged in
unravel_tw.log
. (CUSTOMER-640)Queue analysis supports queue names with quotes or special characters. (CUSTOMER-657)
HBase: Connecting successfully to URL and the "hbase.HBaseMetricsHandler: Error connecting to url" no longer is thrown. (CUSTOMER-659)
Airflow DAGs are parsed correctly. (CUSTOMER-725)
Exceptions no longer cause disruptions in resource manager polling. (CUSTOMER-736)
User can name Top X reports and more comprehensively specify date ranges. (CUSTOMER-737)
Data Insights page is now correctly populated and displays all data. (CUSTOMER-754)
All MR jobs now have correct Container, Vcores, and Memory. (CUSTOMER-764)
Kafka Topic's KPIs now display cumulative value across all brokers for given time. (CUSTOMER-767, CUSTOMER-884)
Impala queries are now displayed in UI and data is correctly populated/display as user drills down (e.g., fragments). (CUSTOMER-803, CUSTOMER-860)
MySQL Schema Creation (using
db_schema_upgrade.sh
script) no longer fails. (CUSTOMER-806)
Queue analysis
Reports are now generated consistently and correctly for every request. (REPORT-316, REPORT-378)
Can generate reports for specific clusters. (REPORT-370)
Newly added queues are polled correctly. (PLATFORM-1363)
Spark
Improved Spark Session's reliability; specifically in terms of short term memory usage. (SESS-189)
MR and Tez
UI now handles unresponsive page and Containers, vCores, Memory graphs. User is informed users when metrics are not found. (UIX-1772, UIX-1773)
Special characters in passwords are not supported. (CUSTOMER-440, CUSTOMER-552)
Platform
appstatus.AppInfoAccessor: Unable to update ES document. (CUSTOMER-658)
Spark worker is missing Azure libraries which causes failure when loading event log and executor logs from Azure.
Upgrade from 4.5.0.3 to 4.5.0.6 on CDH-5.16 cluster 17 services stopped working. You must switch to user
hdfs
for services to restart. (INSTALL-145)RM Polling delayed when yarn kill and move actions are enabled in AutoActions.
YarnRMConnector is polling JMX from standby RM instead of active. (PLATFORM-1399)
Setting impalad as Impala data source raises errors for invalid http address. (PLATFORM-1444)
Security
Kerberos: kerberos.principal in
unravel.properties
does not take effect. (INSTALL-148)LDAP upgrade configs script not updating hive.server2.auth.ldap.base property to com.unraveldata.ldap.baseDN property. (INSTALL-170)
KB Article: https://unraveldata.zendesk.com/hc/en-us/articles/360026752672-Users-cannot-login-via-LDAP-after-upgrade-to-Unravel-4-5-0-6-from-Unravel-4-4-or-earlier-
Database
PostgreSQL DB is now usable across all features. However we do not recommend it for production usage at this time.
Unravel_tw
is not able to connect to Hive Metastore hosted on mariadb. (CUSTOMER-716)
AutoActions
Does not work correctly for Long Running Hive apps in MapR 6.1.0. (AA-102)
Violation badge does not display for a killed application. (AA-91)
OnDemand
Support needed for custom SSL cert for Ambari and Cloudera Manager in OnDemand reports. (CUSTOMER-649)
Forecasting
Throws error potentially due to SSL or bad configs. (CUSTOMER-655)
Is not working due to missing data in Cloudera Manager. (CUSTOMER-781)
Data Insights page
Does not always show data. (CUSTOMER-754)
Accessed Partition is missing in Data page. (DATAPAGE-105)
Created and Accessed Partition details are missing. (DATAPAGE-109)
Small Files report
Does not work when sentry is enabled in the cluster. (REPORT-323)
fs_image
download dies not trigger after install. (REPORT-262)
Cluster Optimization report is empty when one day interval is selected. (REPORT-565)
Cloud reports - You must select an instance type (< 30 types) when choosing EMR as the Cloud Provider.
MR
The Move to Queue option in Actions (MR APM) does not work. (PLATFORM-1195)
Missing cluster ID when the app has Data IO. (PLATFORM-1430)
MR Task Logs are empty as Indexed Format is not supported. (PLATFORM-1464)
HBase
Table pagination is incorrect. (HBASE-95)
RegionCount and AverageRegionSize column is empty for HBase tables on on-premise cluster. (HBASE-98)
Spaces in names are missing in HBase table name. (UIX-1736)
Queue Analysis
May not work in case of HA enabled RM. (PLATFORM- 1423)
Data not updating when secondary RM is active in HA configuration. (REPORT-393)
Graph zooming and resetting not working properly in Edge. (UIX-158)
Top X report
Hive-on-Spark queries show zero for vcore-seconds, memory-seconds and cluster usage duration. (PLATFORM-1425)
Parent app badges are missing for Hive-on-Spark apps. (REPORT-535)
Hive-on-Tez apps show vcore seconds and memory seconds values as zero. (TEZLLAP-249)
Impala Chargeback
Graph index does not have a tool tip to display the complete text. (UIX-1834)
UI should display "No Data Available" message when graph is not displayed. (UIX-1832)
Tez
Vertex name is null. (TEZLLAP-227)
Repeated events are showing up in the event box for the failed Tez apps. (TEZLLAP-259)
Number of events shown in annotation and UI don't match. (TEZLLAP-270)
Spark (Load Logs)
Users isn't notified if logs are already loaded. (UIX-1785)
Yarn logs might not get loaded if event logs are not loaded and sensor data is not available.
The live UI for Spark applications is auto refreshed every 60 seconds, but the live data Unravel collects is only updated upon Job start, and Stage and Job completion. This can make the Spark Navigation tab appear static, i.e., not updated, during long running stages.
Workflow - Subworkflow details are not getting loaded. (UIX-1838)
For support issues, contact Unravel Support.