Home

v4.6.1.1 Release notes

Software version

Release Date: 06/15/2020

See v4.6.1.1 for download information

Software upgrade support

All that is required is an RPM upgrade. The following upgrade path is supported in v4.6.1.1:

  • 4.5.2.x to 4.6.1.1

Sensor upgrade

  • A sensor upgrade is mandatory.

Certified platforms

Review your platform's compatibility matrix before you upgrade or install.

Updates to Unravel's configuration properties

Unsupported

Unravel for Databricks (Azure and AWS)

  • AutoAction

  • Databricks jobs orchestration via services like ADF

  • Notebooks on interactive clusters

  • Spark Program / Query Graph for Notebook and Python tasks

  • Chargeback view by custom tags

  • Cost and Instance Recommendations for Jobs on AWS Databricks

  • Unravel's APIs

  • Sessions

  • Role-based Access Control (RBAC)

  • Support the HDFS path in displaying the Spark source code feature.

  • Data Insights for:

    • Workload

    • Spark

  • Reports

    • Small files

    • Cluster optimization

    • Notebooks

    • Top-X

    • Forecasting

    • Migration planning

    • Queue analysis

  • Datapage

    • Size created of the table

    • Total size

    • Accessed partitions

    • Size created of the partitions

EMR

  • Datapage

    • Size created of the table.

    • Total size

    • Size created of the partitions.

  • Oozie on EMR

HDP

  • Missing table and column statistics events.

Migration Planning

Migration Planning is not supported for the following regions for Azure Data Lake:

  • US DoD East

  • US DoD Central

  • Germany Central (Sovereign)

  • Germany Northeast (Sovereign)

New features

  • None

Improvements and enhancements

  • None

Bug fixes

Upgrade

The Unravel upgrade process can terminate prematurely when there is a large amount of data already collected in the queue_metrics table. This issue can be resolved using the available workaround.

The following workaround should be performed only by an Unravel admin:

Run /usr/local/unravel/install_bin/db_access.sh to get to the SQL prompt of the Unravel database. Inside the SQL prompt, execute the following:

SHOW INDEX FROM queue_metrics;

  • If you do not see "KEY `index_queue_metrics_on_time_se` (`time_se`)" in the output, execute the following:

    ALTER TABLE queue_metrics ADD INDEX `index_queue_metrics_on_time_se` (`time_se`);

  • If you do not see "queue_name` varchar(255) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL" in the output, execute the following:

    ALTER TABLE queue_metrics MODIFY `queue_name` varchar(255) NOT NULL collate utf8_bin;
ALTER TABLE queue_config MODIFY `queue_name` varchar(255) NOT NULL collate utf8_bin;

This can take some time to process based on the amount of data. It is highly recommended to run the commands such that it does not terminate when the SSH connection to Unravel host is disconnected. For example, use screen, tmux, nohup etc. (CUSTOMER-1455)

Known issues

Unravel for Databricks (Azure and AWS)

  • Operations dashboards do not support multi-cluster and have incorrect aggregations.

  • Operations Nodes Dashboard does not capture cluster inactivity in graphs.

  • Spark Application with the same application ID is captured as one.

  • Spark Program / Query graph for Notebook and Python tasks is not supported.

  • Spark default Databricks extraJavaOptions are overwritten by Unravel for spark-submit tasks.

  • DriverOOME and ExecutorOOME events are not generated for the Databricks notebook task.

  • Recommended Azure instances available in Cluster page but not at run time.

  • Recommended Azure instances could be in Beta mode only.

  • Instance recommendation is missing when EMDB is used.

AutoAction

  • The Violation Badge functionality for AutoAction is not working for Impala queries (Running, Killed). (AA-44)

  • EMR: Hive metrics are not published in the RUNNING state. (HIVE-135)

Data Page

  • INSERT statements are not included in the App count on HDI. (DATAPAGE-256)

  • Accessed partition does not support Hive on Tez etc. It only supports Hive on MR. (DATAPAGE-250)

Platforms

  • Latency in fetching the data for MR jobs. (PLATFORM-1613)

  • API connection error while Polling impalad metrics from CM. (PLATFORM-1567)

  • conflicted ephemeral node' or 'Corrupt index found'(PLATFORM-702)

  • gc load metric sensor for MR application will not load on EMR.

Reporting

  • All files deleted except fsimage.txt (parsed fsimage) after parse fsimage is processed. (REPORT-1195)

  • Certain users and queues are missing in the Cluster Compare report. (REPORT-1342)

  • In the Cluster Compare report, the trends of same values are highlighted instead of trends with different values.(REPORT-1479)

Spark Insights

  • For PySpark applications, the processCPUTime and the processCPULoad are not captured properly. (USPARK-626)

  • Partition size 0 is shown in the insight message on the timings tab. (USPARK-647)

Workflow

  • # of Apps is incorrect (PLATFORM-2403)

For support issues, contact Unravel Support

In this section: