Home

v4.6.1.3 Release notes

Software version

Release Date: 09/01/2020

See v4.6.1.3 for download information

Software upgrade support

v4.5.3.1 tarball upgrade. The following upgrade path is supported in v4.6.1.3:

  • 4.5.3.3 to 4.6.1.3

Sensor upgrade

  • A sensor upgrade is mandatory.

Certified platforms

Review your platform's compatibility matrix before you upgrade or install.

Updates to Unravel's configuration properties

Unsupported

Unravel for Databricks (Azure and AWS)

  • AutoAction

  • ADF and Airflow for interactive clusters

  • Notebooks on interactive clusters

  • Spark Program / Query Graph for Notebook and Python tasks

  • Chargeback view by custom tags

  • Cost feature for cluster type that includes spot instances. For clusters including spot instances, the predicted cost does not reflect the purchase price and outbidding events.

  • Unravel's APIs

  • Sessions

  • Role-based Access Control (RBAC)

  • Data Insights for:

    • Workload

    • Spark

  • Reports

    • Small files

    • Cluster optimization

    • Notebooks

    • Top-X

    • Forecasting

    • Migration planning

    • Queue analysis

  • Datapage

    • Size created of the table

    • Total size

    • Accessed partitions

    • Size created of the partitions

EMR

  • Datapage

    • Size created of the table.

    • Total size

    • Size created of the partitions.

  • Oozie on EMR

HDP

  • Missing table and column statistics events.

Migration Planning

Migration Planning is not supported for the following regions for Azure Data Lake:

  • US DoD East

  • US DoD Central

  • Germany Central (Sovereign)

  • Germany Northeast (Sovereign)

New features

  • Fully supported Unravel tarball installation.

Upgrade instructions

This section provides instructions to upgrade to Unravel v4.6.1.3.

Assumptions

The following assumptions are made in the upgrade instructions:

  • Unravel is installed to /unravel-ro. You can verify the correct directory name by looking for software directory in that location. For example: /unravel-ro/software.

  • Unravel is configured with /unravel-rw as the writable location. You can verify the correct directory name by looking for a data directory. For example: /unravel-rw/data.

Upgrading to Unravel v4.6.1.3

To install Unravel from a tar file, do the following:

  1. Change directories to /unravel-ro.

  2. Download Unravel.

    curl -v https://preview.unraveldata.com/unravel/RPM/4.6.1/unravel-4.6.1.3-CDH.tar.gz -o unravel-4.6.1.3-CDH.tar.gz -u username:password

  3. Untar unravel-4.6.1.3.

    tar zxf unravel-4.6.1.3.tar.gz -C /unravel-ro

  4. Stop Unravel.

    /unravel-rw/service/manager.sh stop

  5. Run the setup script.

    /unravel-ro/unravel/versions/4.6.1.3.333/setup --migrate-ms /unravel-rw/data

  6. (Optional) Refresh the cluster configuration.

    /unravel-rw/manager config auto

  7. Start unravel.

    /unravel-rw/manager start

  8. Check the status.

    /unravel-rw/manager report

Improvements and enhancements

  • Workflow

    • ID is added to the workflows that return from /api/v1/workflows. (CUSTOMER-1524)

    • Date time filter added in /api/v1/workflows/{workflow Id}/annotation endpoint to return a specific range of time. (CUSTOMER-1524)

Bug fixes

  • Reports

    • In Reports> Datasight, the table search is functioning appropriately. (CUSTOMER-1528)

  • Impala

    • In some cases, the unravel_tw daemon failed to deserialize Impala queries. This issue is resolved. (CUSTOMER-1518)

Known issues

Unravel for Databricks (Azure and AWS)

  • Job Run cost is estimated based on the total number of nodes present in the cluster and irrespective of whether all the nodes are in use by the Job Run or not.

  • AWS Databricks cost supports Standard and Premium tiers only. Enterprise and Dedicated are not supported.

  • Operations dashboards do not support multi-cluster and have incorrect aggregations.

  • Operations Nodes Dashboard does not capture cluster inactivity in graphs.

  • Spark Application with the same application ID is captured as one. It is captured correctly in the Billing Service.

  • Default Databricks Spark extraJavaOptions are overwritten by Unravel for spark-submit tasks.

  • DriverOOME and ExecutorOOME events are not generated for the Databricks notebook task.

  • Recommended Azure instances available in Cluster page but not at run time.

  • Recommended Azure instances could be in Beta mode only.

  • Instance recommendation is missing when EMDB is used.

AutoActions

  • The Violation Badge functionality for AutoAction is not working for Impala queries (Running, Killed). (AA-44)

  • EMR: Hive metrics are not published in the RUNNING state. (HIVE-135)

Data Page

  • INSERT statements are not included in the App count on HDI. (DATAPAGE-256)

  • Accessed partition does not support Hive on Tez etc. It only supports Hive on MR. (DATAPAGE-250)

Platforms

  • Latency in fetching the data for MR jobs. (PLATFORM-1613)

  • API connection error while Polling impalad metrics from CM. (PLATFORM-1567)

  • conflicted ephemeral node' or 'Corrupt index found'(PLATFORM-702)

  • gc load metric sensor for MR application will not load on EMR.

Reporting

  • All files deleted except fsimage.txt (parsed fsimage) after parse fsimage is processed. (REPORT-1195)

  • Certain users and queues are missing in the Cluster Compare report. (REPORT-1342)

  • In the Cluster Compare report, the trends of the same values are highlighted instead of trends with different values.(REPORT-1479)

  • FSImage process fails with an error message. (REPORT-1635)

    Workaround: Before running the etl fsi image process, check if the /opt/unravel/data/tmp/reports/spark_tmp directory exists. If it does not exist, manually create /opt/unravel/data/tmp/reports/spark_tmp directory.

Spark Insights

  • For PySpark applications, the processCPUTime and the processCPULoad are not captured properly. (USPARK-626)

  • Partition size 0 is shown in the insight message on the timings tab. (USPARK-647)

Workflow

  • The number of Apps is incorrect. (PLATFORM-2403)

  • Spark apps running in Oozie workflow is not visible under Oozie workflow in Unravel UI. (PLATFORM-2746)

Sessions

  • The Session apply feature does not function in a kerberized environment in Unravel v4.6.1.3 version. However, both Session creation and Session analyze features work as expected. (SESS-329)

    Workaround: Pick the recommendations from the session and apply them to the application manually.

For support issues, contact Unravel Support

In this section: