Home

v4.3.1.6 Release notes

Software version

Release Date: 08/13/2018 for 4.3.1.6

For details on downloading updates see Downloads.

Certified platforms

  • CDH: On-premise 5.12, 5.13 & 5.14(including Spark 2.2.x and 2.3.x) with Kerberos

  • HDP: On-premise (up to v2.6.3) with Kerberos + spnego enabled.

  • MapR: 6.0.0 with MapR Expansion Packs 4.1.1

  • EMR: On-premise 5.13 & 5.14(including Hive 2.3.2 and Spark 2.3.0)

Unravel sensor upgrade

  • Required

    • if using Hive-2.3.2 or Spark-2.3.0, or

    • to see the Spark Version in Spark APM

  • Recommended for the customers who are on 4.3.1.4 versions or earlier.

New features

  • Added a queue column to Application | Application Tab. (Customer-59)

  • Tags can be added to running applications. (Customer-72)

Improvements and bug fixes
Improvements

  • jsoup timeout can now be set in unravel.properties. (CUSTOMER-160)

  • Added Hive Hook support for HDP 2.3.2

  • Spark sensor now supports Spark 2.3.0

  • Added CLI arguments to allow admins choose what to update: Spark, Hive, Tez or MapR. (INSTALL-68)

  • Added Unravel Hive Hook support for HDP 2.1 version. (Sensor-61)

  • Graph tooltips no longer overlap the y-axis. (UIX-1029)

  • Workflow page open apps and workflow instances in the same tab. (UIX-39)

Bug fixes

  • Upgraded ESLint version to 5.3.0 so Unravel no longer vulnerable to CWE400. (CUSTOMER-225)

  • Pushlog.sh creates the diagnostics file. (CUSTOMER-229)

  • Applications | Applications

    • AA/Tuning suggestions now sort properly. (UIX-918)

    • Username and Queue names are no longer truncated. (UIX-598)

  • Tez

    • Tags work correctly. (CUSTOMER-230)

    • Job status and DAG has finished. (TEZLLAP-69)

    • Task Attempts tab populating correctly. (TEZLLAP-70)

    • Tez app loading correctly. (TEZLLAP-106)

    • Graphs loading correctly. (TEZLLAP-811)

    • Resource x-axis name displays correctly. (UIX-911)

  • Spark

    • Execution tab populates and is displayed correctly. (CUSTOMER-259)

    • Logs populating correctly and scroll bars work. (UIX-1016, UIX-368)

    • All Stages are displayed. (UIX-824)

    • Spark-log driver shows Executor details. (UIX-112)

  • Auto Actions

    • Owner field populates and when selected the owner is notified when AA is triggered.(Customer-161)

    • Notification works properly when Workflow SLA is broken. (CUSTOMER-237)

    • Last run time and history updates correctly. (CUSTOMER-237)

    • AA exception no longer causes RM thread to die. (Platform-522)

    • History Runs Link count displays correctly. (UIX-904)

    • AutoAction email includes link to the violating job. (CUSTOMER-252)

  • MR

    • Available Memory graph populates correctly. (UIX-750)

    • MR/Hive apps no longer hanging when in Running mode. (PLATFORM-591)

    • Task logs display correctly in Internet Explorer. (UIX-903)Kafka

  • Kafka

    • kafka_create_topics.sh worked properly when RUN_AS and USE_GROUP are different in /etc/unravel_ctl. (INSTALL-70)

    • Log no longer has duplicated entries. (UKAFKA-6)

  • Impala

    • Usage graphs resizing correctly (UIX-543)

    • Operator search works. (UIX-838)

  • Reports | Operational Insights

    • Cluster Summary and Cluster Compare works correctly (Customer-194)

    • Chargeback report generates report correctly is leading 0 is missing, i.e., .16. (Customer -217)

    • Chargeback: queue list is no longer cut short. (UIX-619)

    • Chargeback: report download has correct memory hours. (UIX-886)

    • Chargeback: download has correct memory hours. (UIX-886)

    • Chargeback: display pagination works for all "group by" options. (UIX-887)

    • Cluster Summary: vCores and Memory tabs sorting corrected. (UIX-948)

    • Cluster Summary: sort by queue works. (UIX-952)

  • Admin | Manage | Daemon sorts correctly on uptime.

  • UIX

    • Graph titles no longer overlap graph. (UIX-1015)

    • Close tabs works correctly. (UIX-920)

    • Expand glyph in tiles now continually displays. (UIX-921)

    • Tool tips for minimize and maximize corrected. (UIX-922)

    • Blocks displays correctly when minimized. (UIX-941)

    • Event count shows correct in even panel. (UIX-964)

    • Infrastructure graphs alignment is correct in Internet Explorer. (UIX-967)

Known issues

  • None

Configuration properties
New

  • com.unraveldata.airflow.http.timeout.sec. See Monitoring Airflow for more information.

Deprecated

  • None

Software upgrade support

  • Support for upgrade from 4.2.6 (4.2.6-1128) or 4.2.7 (4.2.7-1154)

  • Support for upgrade from 4.3.0.X to 4.3.1.6

    Upgrading from 4.2.6 or 4.2.7 with this RPM causes a background process to run for about 15 minutes. During which data is migrated in the unravel_s_1 daemon. To see status of this migration, see /usr/local/unravel/install_bin/es_migrate_4.3.1.0_from42.out and /usr/local/unravel/logs/es_ migrate_4.3.1.0_from42.log ; this migration can be re-run using/usr/local/unravel/install_bin/es_migrate_4.3.1.0_from42.sh(d in the background with nohup. When the migration finishes successfully, disk space can be saved by running /usr/local/unravel/install_bin/es_migrate_4.3.1.0_from42_cleanup.sh.

    An RPM upgrade triggers a temporary background process so changes can take effect. You can monitor it or wait for these with /usr/local/unravel/install_bin/await_fixups.sh which prints DONE when it finishes. Tt takes anywhere from 2-15 minutes depending on your data size.

    After upgrade:

    Run /usr/local/unravel/install_bin/await_fixups.shand wait for it to finish. It can take up to 15 minutes. If shell is terminated, it can be run again at any time.

    There should be 47 tables. Check the table count manually with db_access.sh and 'show tables;'.If fewer tables are present, run: sudo /usr/local/unravel/dbin/db_schema_upgrade.sh.

Warning

Upgrade to 4.2.6 or later before upgrade to 4.3.x
Upgrading 4.2.x to 4.3.x, one-time potential must-do steps
Run-as

When upgrading from 4.2.x to 4.3.x, you need to be cognizant of the run-as user for Unravel server daemons. Unravel Server 4.2.x and earlier runs daemons with 2 different local users (usually 'unravel' and 'hdfs' or 'mapr'). In 4.3.x, Unravel server simplifies this to run-as the single user 'unravel' by default. During an upgrade from 4.2.x to 4.3.x, all the daemons are converted over to user 'unravel'. Other changes that occur because of this:/srv/unravel/log_hdfs/* logs are moved to /usr/local/unravel/logs

/srv/unravel/log_hdfs/ directory is removed

/srv/unravel/tmp_hdfs/ is no longer needed, so it is removed (/srv/unravel/tmp/ is used instead)

env vars HDFS_KEYTAB_PATH and HDFS_KERBEROS_PRINCIPAL in /usr/local/unravel/etc/unravel.ext.sh no longer used

For Kerberos, com.unraveldata.kerberos.principal and com.unraveldata.kerberos.keytab.path are used instead.

After the first 4.3.x RPM upgrade, you MUST evaluate whether the switch_to_user.sh must be run in order for Unravel Server to load logs via HDFS and to access kerberos-spnego protected REST endpoints. If MapReduce or Spark jobs stop loading after upgrade, that is a strong indication that you need to switch users. This script need only be run once. See Run Unravel Daemons with Custom User for more details. Subsequent upgrades 4.3.x to 4.3.y will not need to run switch_to_user.sh script.

Upgrade process

An RPM upgrade will trigger a temporary background process so changes can take effect. You can monitor or wait for these with /usr/local/unravel/install_bin/await_fixups.sh which prints "Done" when it finishes. The script waits and prints ps of the background process. It might take 2-5 min. depending on your data size.

The background fix ups will produce log file output in:

/tmp/unravel_post_upgrade_fixups.*.out

/tmp/unravel_schema.*.out

/usr/local/unravel/logs/unravel_schema.log

/usr/local/unravel/logs/es_migrate_4.3.1.0_from42.*

/usr/local/unravel/logs/unravel_esm.log

For further support, contact Unravel Support.

In this section: