Home

v4.5.1 Release notes

Software version

Release Date: 06/03/2019

See v4.5.1 for specific download information.

Software support

  • Upgrade from 4.5.0.x only. All that is required is an RPM upgrade.

Certified platforms

Please review your platform's compatibility matrix below before you upgrade/install.

CDH
HDP

Unsupported

  • ATSv2.0.

  • Small files report in MapR.

  • Long running Hive query auto action not supported on HDP-3.0 +.

  • Migration planning in MapR.

  • Impalad as data source for Impala queries.

  • Hive on Spark.

Unravel sensor upgrade

If you are upgrading from:

  • 4.5.0.x: you do not have to upgrade the sensor. However, it is recommended to ensure the Spark SQL app data is fully populated.

  • If you are doing a new/fresh install, you must install the new sensors.

New sensors

  • Parcel version: 1.0.4507001

  • Sensor version: 4.5.1.0rc0169

Updates to Unravel's configuration properties

New features

  • Migration planning

    • Unravel helps you plan your migration from an on-prem cluster to the cloud. Unravel begins with discovering your cluster, then finds the optimal cloud topology based on your business strategy (minimize risk/minimize costs) and computes the anticipated hourly costs.

    • Migration Planning is supported for CDH and HDP deployments.

    • Unravel currently supports mapping of only number of on-premise cores to vCPUs in the cloud.

    • Cloud Mappings

      • Ability to select the Target Cloud Product/Service for cloud mapping to be either one of: Azure (IaaS), Google Cloud Platform (IaaS), Amazon EC2 (IaaS), Amazon EMR, Azure HDInsight.

      • Ability to select Local Attached Storage or Object Storage for the cloud mapping.

  • TLS Support for Small Files, Sessions, Capacity Forecasting.

  • Spark Live Update Interval  from Sensor is now configurable. (com.unraveldata.spark.live.interval.sec).

  • Top X report

    • Added support Impala apps.

    • Displays custom tags for associated apps.

    • Can be filtered based on User/Queue/Tags.

  • Added support for YARN log aggregation IndexedFormat.

Improvements

  • Airflow

    • Added mode to stop the fetch an Airflow's workflow component apps. (PLATFORM-1340)

    • Improved scale of log parsing by using ThreadPool and Operator pruning. (PLATFORM-1369)

  • Capacity forecasting

    • When a Cluster Manager shows no data, the start time for the report is moved up until the manager contains data. (REPORT-373)

  • Impala

    • Added support for running Impala queries.

    • Running Impala queries now have the AutoActions metrics.

    • Improved efficiency and latency of Impala pipeline.

  • Reporting

    • Parsing Airflow DAGs no longer fails due to error with execution date format after upgrade to 4.5.0.3 with Airflow 1.8. (REPORT-339)

    • Support for encryption in-Transit for Sensor Communication.

  • Security

    • Added support for group based authentication to Active Directory in Unravel.

    • Added support for encryption in-Transit for Sensor Communication.

    • Elasticsearch Authentication: the internal Elasticsearch service used by Unravel now supports HTTP Basic Authentication, preventing access to Elasticsearch without appropriate credentials.

  • Spark

    • Load Logs: User is notified when attempting to load logs which are already loaded.

  • Tez

    • Tez applications are now tagged while running.

  • Top X

    • Parent app badges for Hive on Spark apps are displayed.

    • Displays an app status (running, killed, waiting, pending and unknown).

    • Tab has moved from Reports > Data Insights to Reports > Operational Insights.

Bug fixes

  • Customer reported

    • com.unraveldata.workflow.analytics.model.MRJob.initFV has an NPE. (Customer-256)

    • RBAC now works property for users specified in read-only admin groups property. (Customer-306)

    • unravel_tw working correct for Hive 1.2.1000 on HDP 2.6.4 and no longer throws an exception. (Customer-408)

    • Unravel_TD no longer fails on housekeeping.TidyDir NullPointerException. (Customer-507)

    • Log Receiver error/exceptions changed to warnings when running short-lived PI Spark job. (Customer-517)

    • LDAP Group search no longer fails upon connection to Unravel UI. (Customer-531)

    • Unravel_pg no longer fails to startup during installation. (Customer-567)

    • Unravel uses the right ResourceManager URL for single node RMs in TLS. (Customer-571)

    • The Diagnostics(Unravel Status) "==log_hdfs Tails==" are no longer populated; database table metrics and activities are now populated. (Customer-587)

    • unravel_all.sh script now includes unravel_db. (CUSTOMER-591)

    • End-user can now log in to UI after Unravel restart. (CUSTOMER-614)

    • Upgrade changes no longer changes ownership directory and files ownership back to unravel. (CUSTOMER-618)

    • Queue analysis tab is no longer throwing error SQL if queue names have quotes or special character. (CUSTOMER-657)

    • Unravel sensor captures sample immediately instead of waiting for first interval. (CUSTOMER-757)

  • Airflow

    • Parsing of DAGs no longer fails after upgrading to 4.5.0.3 with Airflow 1.8 due to error with execution date format. (REPORT-339)

    • Start time and end time for Workflow Instances are correct. Added annotations to functions, and the regex on Supported Operators more robust. (PLATFORM-1445)

  • Data insights: Impala queries inserting/updating a table are now accounted for. (DATAPAGE-86)

  • Impala

    • Peak memory usage per host is now correct. (IMPALA-175)

    • Peak memory usage per host is now populated on newer CDH versions. (IMPALA-179)

    • Impala queries are no longer stuck in RUNNING status. (IMPALA-178, IMPALA-192)

    • Killed Impala DML and DDL statements are no longer stuck in UNKNOWN status. (IMPALA-196)

    • Bytes written are now accounted in Data I/O. (UIX-1737)

    • UI should display "No Data Available" message when graph is not displayed. (UIX-1832)

    • Graph index does not have a tool tip to display the complete text. (UIX-1834)

  • Queue analysis

    • Queues can be excluded from metrics collection and reports altogether. (REPORT-192)

    • Better error reporting. (REPORT-213)

    • Report is no longer missing queues in HDP environments. (REPORT-294)

    • Analyzing queues for multiple clusters no longer causes overlapping of metrics. (REPORT-297)

    • Queue metric analysis reports average metric values rounded to lower bound. (REPORT-302)

    • Graph data is populated. (REPORT-316)

    • No longer fails when putting data to ES with “FAILED: (‘Connection aborted.’, error(104, ‘Connection reset by peer’))”. (REPORT-324)

    • Graphs are more legible. (REPORT-342)

    • No longer fails with list indices error when specific cluster is selected. (REPORT-370)

    • Reports are correctly populated. (REPORT-378)

    • Updates the data when secondary RM is active in HA configuration. (REPORT-393

    • Queue metric sensor works for remote clusters if RM is not accessible via HTTP. (PLATFORM-927)

    • Queue Metrics sensor polls newly added queues. (PLATFORM-1363)

    • YarnRMConnector no longer polls JMX from standby RM instead of active. (PLATFORM-1399)

Known issues

  • Airflow may still have scalability issues for clusters with a high number of DAGs.

  • AutoActions: Violation badge does not display for a killed application. (AA-91)

  • Cluster optimization report is empty when one day interval is selected. (REPORT-565)

  • Data insights

    • Accessed Partition is missing in Data page. (DATAPAGE-105)

    • Created and Accessed Partition details are missing. (DATAPAGE-109)

    • When switching view by labels (All, Warm, Cold, Hot) the page can become unresponsive. (UIX-1878)

    • Impala queries from 4.5.0.x are not included.

  • HBase:

    • Cluster names containing spaces are shown without the spaces, e.g., "A HBase Name" is displayed as "AHBaseName".

    • Region server details are missing when the cluster name contains special characters. (HBASE-87)

    • HBase OS metrics not visible in Ambari 2.7.3.0. (HBASE-101)

  • Hive

    • Hive table details page parameters are inconsistently null. (UIX-1874)

    • Timestamp is missing for Failed MR apps. (UIX-1873)

  • Impala chargeback

    • Impala Chargeback page will not show data for queries migrated from Unravel 4.3.1.x.

  • Kafka is not working when clusterID has a space. (UKAFKA-69)

  • Migration planning

    • Migration Planning Reports are not supported for MapR.

    • The following Cloud Product/Service configurations are not supported:

      • Cloud Product/Service: GCP

        Storage Type: Persistent disks

        Region: Asia

      • Cloud Product/Service: GCP

        Storage Type: Cloud Storage

        Region: asia-east1, asia, europe-north1, europe, australia, us-west1, asia-east, asia-southeast, and europe-west4

      • Cloud Product/Service: EMR and EC2

        Storage Type: S3, EBS General Purpose

        Region: Asia Pacific (Hong Kong)

    • Cloud Mapping per Instance report displays “unsupported operand type” error message when Unravel is unable to map the host to any VM instance. (REPORT-729)

    • Unlike the Cluster Discovery report the Cloud Mapping per Host report displays the cluster ID. (REPORT-698)

    • Cloud Mapping per Host report for Azure HDI and Azure (IaaS) cloud providers Unravel displays on-prem storage as total storage value and not the actual storage made available by Azure Managed Disks when Storage Type chosen is ‘Local Attached Storage’. (REPORT-728)

    • Cloud Product/Service name displayed in the reports for Cloud Mapping per Host and Cloud Mapping per Instance are not the complete names of the Cloud Product/Service provider. (REPORT-567)

    • Downloaded JSON file for cloud mapping per host and cloud mapping instance display info on Workload fit which is currently not supported. (REPORT-724)

    • The “New Report” page for both Cloud Mapping per Instance and Cloud Mapping per Host hangs if the user does not provide the Unravel property for cluster type (CDH/HDP) com.unraveldata.cluster.type. (REPORT-731)

    • Actual usage value utilized for figuring out cloud mapping for ‘Cost Reduction’ Strategy is based on last 6 days only of usage.

    • When Storage Type selected is Object Storage, the minimum disk space required on nodes is ignored when generating the cloud mapping. (REPORT-661)

    • There is no ability to pick the time for the Cluster Discovery Report (you can pick only the date). The time that gets chosen (automatically) does not match the time (automatically) picked for the Chargeback reports. So the data in the charts displayed By App Type, By User and By Queue on the Cluster Discovery Report don’t match the ones on the Chargeback reports. (REPORT-684)

    • The regions for Google Cloud Provider display extra regions Asia, Europe, US, Australia which should not be listed. (REPORT-663)

    • Object storage pricing is typically tiered, but is not taken into account while calculating the storage costs. (REPORT-596)

    • Unravel assumes one on-prem core is equivalent to one cloud vCPU and finds the best possible match without taking into account any performance differences. (CUSTOMER-926)

  • MR: The Move to Queue option in Actions (MR APM) does not work. (PLATFORM-1195)

  • Platform

    • appstatus.AppInfoAccessor: Unable to update ES document. (CUSTOMER-658)

    • Spark worker is missing Azure libraries which causes failure when loading event log and executor logs from Azure.

    • RM Polling is delayed when yarn kill and move actions are enabled in AutoActions.

    • Setting impalad as Impala data source raises errors for invalid HTTP address. (PLATFORM-1444)

    • On CDH 5.12 Unravel sometimes fetches the incorrect API version. (REPORT-735)

    • Security

      • OnDemand package is unable to connect to an Unravel database whose password contains the following special characters, --, \, ; or `. (The comma is the delimiter, not a special character.) (CUSTOMER-440, CUSTOMER-552)

  • Queue analysis: Graph zooming and resetting is not working properly in Edge. (UIX-158)

  • OnDemand Report generation fails when ES property max_result_window resets to its default of 10000. (PLATFORM-1605)

    Workaround:

    • Restart unravel UI daemon: service unravel_ngui restart.

  • Sessions

    • Limitation in HDP cluster: While applying on a Spark session sometimes we get error like “Could not run the Spark application on the cluster: java.lang.IllegalStateException: hdp.version is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file in conf with -Dhdp.version=xxx”. The exact reason for this bug is yet to be ascertained.

      Workaround:

      1. Open and edit spark-env.sh of the client where unravel is running.

      2. Add a line similar to export “HDP_VERSION=2.6.5.0-292". Make sure to put the proper HDP version in the line.

    • In a MapR cluster, Spark application's fail after turning recommendations are applied. (SESS-250)

    • Hive

      • Unable to apply hive sessions in HDP due to exception. (SESS-245)

      • Newly tuned app in MapR is not shown after applying recommendations. (SESS-251)

    • For some queries, Unravel does not properly capture the Hadoop configuration. However, when the query is re-run it works fine. (ESS-245)

  • Small files

    • "No Data Found" when we run small files report parallel (race condition). (REPORT-660)

    • get_hive_query_status(): internal error, message not found in unravel_ondemand.out. (REPORT-732)

  • Spark

    • Load logs: Yarn logs might not get loaded if event logs are not loaded and sensor data is not available.

    • Spark SQL apps are missing data. (USPARK-310) Note: Upgrading the sensor fixes this issue.

  • Tez:

    • Vertex name is null. (TEZLLAP-227)

    • Tool tip is missing for Appname/ID. (UIX-1806)

  • Top X report

    • Hive-on-Spark queries show zero for vcore-seconds, memory-seconds and cluster usage duration. (PLATFORM-1425)

    • Hangs for queryStrings that contain escape characters like '\n', '\t'. (REPORT-742)

    • Hive-on-Tez apps show vcore seconds and memory seconds values as zero. (TEZLLAP-249)

For support issues, contact Unravel Support.

In this section: