Home

v4.6.1.6 Release notes

Software version

Release Date: 01/26/2021

See v4.6.1.6 for download information.

Software upgrade support

In v4.6.1.6, the following paths are supported for Tarball and RPM upgrades:

Tarball supported upgrade path

RPM supported upgrade path

4.6.1.5 to 4.6.1.6

4.6.1.5 to 4.6.1.6

4.6.1.4 to 4.6.1.6

4.6.1.4 to 4.6.1.6

4.5.3.3 to 4.6.1.6

4.6.1.0 to 4.6.1.6

4.5.5.5 to 4.6.1.6

4.5.5.3 to 4.6.1.6

4.5.5.2 to 4.6.1.6

4.5.5.0 to 4.6.1.6

4.5.0.6 to 4.6.1.6

Sensor upgrade

  • A sensor upgrade is mandatory.

Certified platforms

Review your platform's compatibility matrix before you upgrade or install.

Updates to Unravel's configuration properties

Installation and Upgrade instructions

Refer to Unravel Tarball installation for instructions to install and configure Unravel using Tarball.

Refer to Upgrading Unravel server for instructions to upgrade Unravel.

New features

  • Billing Service

    The Billing tab shows the charges of Unravel for its support for Databricks. The following features are added to the existing Billing service in Unravel v4.6.1.6:

    • Unravel supports tracking usage in DBUs as an alternative to instance hours.

    • Unravel also supports a pay-in-advance pricing plan, where users can pay in advance for a specific number of units (either DBUs or Instance Hours). Credits will be taken based on the usage, with the remaining credits shown on the Billing tab daily. Thus users can monitor when they run out of credits.

    • The outstanding amount and the remaining credits are graphically represented.

  • Support Tez data collection using Unravel sensor instead of relying on Application Timeline Server (ATS)

    Before Unravel version v4.6.1.6, the Tez metrics were collected from Application Timeline Server APIs. Now, the BTrace sensor can be configured, for HDP and CDP platforms, to collect these metrics.

  • PostgreSQL is now supported for both On-prem and Databricks clusters

  • Azure Active Directory

    You can integrate Unravel with Azure Active Directory (AAD) to use this authentication service of Azure for Databricks.

  • Spark 3.0 support

    Support added to capture live data for Spark 3.0.

Improvements and enhancements

  • Currently, the streaming data is collected using Btrace. In some cases, the batch information is not collected and the Spark details page is not displayed appropriately for the streaming applications. Now a Listener is added to capture the metadata for streaming applications. (ASP-832)

  • Queue name from the RM data is used instead of the Spark conf to avoid listing separate names for the same queue in the cluster reports. (ASP-825)

  • Unravel sensors now support clusters with JDK versions greater than 8. (ASP-668)

  • Added PostgreSQL support for Capacity Forecasting report. (REPORT-1736)

  • Support Impala clusters with spaces in the cluster name. (IMPALA-288)

  • Support is added for Hive 3.1.3. (HIVE-148)

  • Program tab for Spark is supported for Notebooks and Python tasks on Databricks. (CUSTOMER-1526)

  • Clusters view for Databricks is enabled. (DT-754)

  • Resource metrics from the sensor is updated to support JDK versions greater or equal to 9. (CUSTOMER-1610)

Unsupported

  • Hive versions lesser than 1.10 are no longer supported.

  • On-premise platform MapR

  • Migration Planning is not supported for the following regions for Azure Data Lake:

    • US DoD East

    • US DoD Central

    • Germany Central (Sovereign)

    • Germany Northeast (Sovereign)

Bug fixes

  • Spark

    • For HDP 3.1.5, the Unravel UI does not load the Spark sensor data. (ASP-837)

  • Sessions

    • While creating a session, space is not allowed in the session name. (SESS-332)

  • Impala

    • Missed Impala nodes are visible from the base cluster as well as from the compute cluster (IMPALA-280)

  • Reports

    • Incorrect number of clusters displayed in Usage Trending. (DT-724)

    • JDBC drivers are looked at in the JAVA directory relative to the location pointed by the unravel.user_libraries.directory property, if the property exists. In case the property does not exist, it is looked for the connectors at /usr/local/unravel/share/java. (CUSTOMER-1607)

  • UI

    • When you click the Chargeback Clusters tab, you are logged out of Unravel UI. (CUSTOMER-1595)

Known issues

AutoAction

  • EMR: Hive metrics may not be published in the RUNNING state. (HIVE-135)

Data Page

  • INSERT statements are not included in the App count on HDI. (DATAPAGE-256)

  • The accessed partition does not support Hive on Tez etc. It only supports Hive on MR. (DATAPAGE-250)

  • Datapage does not load the total table and partition size KPI's (DATAPAGE-379)

Metrics Collection

  • Resource metrics will not be collected for JDK versions that are equal to or greater than 9 without using the extra JVM switches as mentioned below:

    Workaround: All metrics can be collected if the following options are used for sensor JVM

    --add-exports java.base/jdk.internal.perf=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED

    For example:

    JDK9_MODULE="--add-exports java.base/jdk.internal.perf=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED"
ENABLED_SENSOR_FOR_DRIVER="spark.driver.extraJavaOptions=$JDK9_MODULE -Dcom.unraveldata.client.rest.request.timeout.ms=500 -Dcom.unraveldata.client.rest.conn.timeout.ms=500 -Dcom.unraveldata.client.rest.queue=2000 -javaagent:unravel-agent-pack-bin.zip/btrace-agent.jar=libs=spark-2.3,config=driver,stdout=true,debug=true"
ENABLED_SENSOR_FOR_EXECUTOR="spark.executor.extraJavaOptions=$JDK9_MODULE -javaagent:unravel-agent-pack-bin.zip/btrace-agent.jar=libs=spark-2.3,config=executor"

    These JVM options are valid for JDK9 - JDK15. Without these JVM options, the following metrics are not collected:

    • ProcessCpuLoad, ProcessCpuTime, SystemCpuLoad

    • PerfCounters metrics

Platforms

  • gc load metric sensor for MR application will not load on EMR.

  • Spark applications that run in the local mode do not use YARN resources, so these applications will be shown using 0 resources in Unravel. (PLATFORM-2809)

  • Intermittent issue in fetching the Btrace data for MR applications. (PLATFORM-2807)

  • Killed MR apps do not display RM diagnostics. (PLATFORM-2815)

Reporting

  • Certain users and queues are missing in the Cluster Compare report. (REPORT-1342)

  • In the Cluster Compare report, the same value trends are highlighted instead of trends with different values. This is fixed in the New UX. (REPORT-1479)

Spark

  • There is a lag seen while streaming Spark SQL applications. (PLATFORM-2764)

Spark Insights

  • For PySpark applications, the processCPUTime and the processCPULoad are not captured properly. (USPARK-626)

Workflow

  • The number of TEZ applications in an Oozie workflow is sometimes incorrect. (PLATFORM-2403)

UI

  • For streaming applications, the global search with application ID does not work. (UIX-3312)

For support issues, contact Unravel Support

In this section: