Home

v4.3.1.5 Release notes

Software version

Release Date: 07/18/2018 for 4.3.1.5

For details on downloading updates see Downloads.

Certified platforms

  • CDH: On-premise5.13 & 5.14 (including Spark 2.2.x) with Kerberos

  • HDP: On-premise HDP (up to v2.6) with Kerberos + spnego enabled.

  • MapR: 6.0.0 with MapR Expansion Packs 4.1.1

Unravel sensor upgrade

  • Required

    • If using Hive-2.3.2 or Spark-2.3.0.

    • To see the Spark Version in Spark APM.

New features

  • Role Based Access Control (RBAC)

    • Reports available when turned on. (CUSTOMER-11)

    • Able to generate project and tenant tags from LDAP groups. See configuration properties. (CUSTOMER-63)

  • Pig application status no longer states unknown when not applicable. (CUSTOMER-122)

  • Added ability to archive data, i.e., chargeback, estimation models. (CUSTOMER-133

  • Spark - Added support for Spark 2.3 .(USPARK-71)

  • Hive -Added Support for hive hooks for Hive 2.3. (HIVE-23)

  • Workflow APM error logs display Oozie jobs less than 90 seconds. (CUSTOMER-158)

Improvements and bug fixes
Improvements

  • Data Insights | Details can be exported/downloaded into a CSV file. (CUSTOMER-124)

  • Applications | Workflow can be displayed by date range.(UIX-85)

  • Added HTPP security headers are present. (CUSTOMER-139)

  • Ability to see the Spark version in use. (CUSTOMER-100)

Bug fixes

  • Removed incorrect error message when running unravel_hdp_setup.sh. (CUSTOMER-13)

  • Charts correctly displaying with no overlaps. (CUSTOMER-132)

  • CVS downloads are no longer are vulnerable to JMX (CUSTOMER-138)

  • HDP setup script disables the Impala sensor by default. (CUSTOMER-145)

  • Table APM graphs displays information legibly. (CUSTOMER-146)

  • Airflow monitoring no longer cause unravel_jc2s to fail. (CUSTOMER-160).

  • Role Based Access Control (RBAC)

    • Supports 2 tags. (CUSTOMER-68)

    • IE browser correctly display tags in Application | Applications. (UIX-597)

    • Application | Applications correctly displays applications by user when RBAC is turned on. (UIX-910)

  • Kafka

    • Consumer shows all partitions. (UIX-104)

  • Operations | Usage Details

    • Infrastructure can be displayed on hourly basis. (CUSTOMER-140)

    • Infrastructure can be sorted for all columns. (UIX-843)

    • Jobs new_state shows correct data. (UIX-499)

  • Impala

    • Charts can be display using tags. (CUSTOMER-144)

    • APM row number consistent across Query Plan and Operator tabs. (UIX-827)

  • Reports | Operational Insights | Usage Details

    • Chargeback correctly populates the graphs. (CUSTOMER-153)

    • Chargeback correctly populates table for yarn applications. (UIX-861)

    • Cluster Compare and Summary data populated correctly. (PLATFORM-235) (UIX-621)

    • Cluster Workload reflects all apps running. (UIX-605)

    • Cluster Compare arrows sized correctly. (UIX-850)

  • Spark

    • Direct access to EMR works. (USPARK-73)

    • spark.executor.memorydisplays in Conf->Memory filter in Spark APM page. (UIX-80)

    • Event Count displays correctly in Events Panel. (UIX-847)

    • SparkSQL table entries sorted by duration. (USPARK-74)

    • EMR Sensor is no long sending data to spark topic on apps not running on. (SENSOR-43)

  • Tez

    • Successful Tez Apps show correct status. (TEZLLAP-64)

    • Diagnostics are collected and displayed in App and Dag tiles. (TEZLLAP-72)

    • EMR Sensor - metrics data for Tez application are seen. (SENSOR-51)

  • User Interface

    • Applications | Applications filter displays correctly. (UIX-596)

  • Resolved exception during Yarn parsing. (PLATFORM-353)

Known issues

  • PLATFORM-468: YARN logs won't be loaded into Unravel if yarn.log-aggregation.file-formats=IndexedFormat.

  • UIX-473: Support chat on app page not supported.

  • UIX-955: Role based access control - com.unraveldata.rbac.prefix config is not completely supported.

  • UIX-942: Role based access control - com.unraveldata.rbac.default=userName is not completely supported.

  • UIX-912: Corner case- Fix Data table plugin popup in the Unravel UI.

  • UIX-903: Intermittently task logs go empty while using Internet Explorer 11.

  • UIX-709: Improve accuracy in pending application count.

  • UIX-886: Improve Chargeback CSV report.

  • UIX-810, DATA PAGE-56: Improve DATA PAGE graphs and table KPI’s.

  • UIX-800, 798: Improvements in Kafka monitor partition page.

  • UIX-609: Improve Cluster workload graphs Hourly/day view.

  • TEZLLAP-83: Improvements to Hive/Tez Data Pipeline.

Configuration properties
New

RBAC

Please see Supported Roles and Role Based Access Control (RBAC) for a complete explanation of each property, including default settings and any caveats.

  • com.unraveldata.rbac.enabled: enables RBAC

  • com.unraveldata.rbac.ldap.tags: a comma separated list of the prefix of LDAP group, to be used as the Project tag in RBAC. .

  • com.unraveldata.rbac.ldap.tag.Tag_Name.regex.find:generates a list of Tenants tags for the Project, where Tag_Name was defined in com.unraveldata.rbac.ldap.tags

Deprecated

  • com.unraveldata.rbac.mode

  • com.unraveldata.rbac.user.operation

  • com.unraveldata.rbac.tag

  • com.unraveldata.rbac.prefix

Software upgrade support

  • Support for upgrade from 4.2.6 (4.2.6-1128) or 4.2.7 (4.2.7-1154).

  • Support for upgrade from 4.3.0.X to 4.3.1.4.

    Upgrading from 4.2.6 or 4.2.7 with this RPM causes a background process to run for about 15 minutes. During which data is migrated in the unravel_s_1 daemon. To see status of this migration, see /usr/local/unravel/install_bin/es_migrate_4.3.1.0_from42.out and /usr/local/unravel/logs/es_migrate_4.3.1.0_from42.log ; this migration can be re-run using/usr/local/unravel/install_bin/es_migrate_4.3.1.0_from42.sh the background with nohup. When the migration finishes successfully, disk space can be saved by running /usr/local/unravel/install_bin/es_migrate_4.3.1.0_from42_cleanup.sh.

    An RPM upgrade triggers a temporary background process so changes can take effect. You can monitor it or wait for these with /usr/local/unravel/install_bin/await_fixups.sh which prints DONE when it finishes. Tt takes anywhere from 2-15 minutes depending on your data size.

    After upgrade:

    Run /usr/local/unravel/install_bin/await_fixups.shand wait for it to finish. It can take up to 15 minutes. If shell is terminated, it can be run again at any time.

    There should be 47 tables. Check the table count manually with db_access.sh and 'show tables;'.If fewer tables are present, run: sudo /usr/local/unravel/dbin/db_schema_upgrade.sh.

Warning

Upgrade to 4.2.6 or later before upgrade to 4.3.x
Upgrading 4.2.x to 4.3.x, one-time potential must-so steps
Run-as

When upgrading from 4.2.x to 4.3.x, you need to be cognizant of the run-as user for Unravel server daemons. Unravel Server 4.2.x and earlier runs daemons with 2 different local users (usually 'unravel' and 'hdfs' or 'mapr'). In 4.3.x, Unravel server simplifies this to run-as the single user 'unravel' by default. During an upgrade from 4.2.x to 4.3.x, all the daemons are converted over to user 'unravel'. Other changes that occur because of this:/srv/unravel/log_hdfs/* logs are moved to /usr/local/unravel/logs

/srv/unravel/log_hdfs/ directory is removed

/srv/unravel/tmp_hdfs/ is no longer needed, so it is removed (/srv/unravel/tmp/ is used instead)

env vars HDFS_KEYTAB_PATH and HDFS_KERBEROS_PRINCIPAL in /usr/local/unravel/etc/unravel.ext.sh no longer used

For Kerberos, com.unraveldata.kerberos.principal and com.unraveldata.kerberos.keytab.path are used instead.

After the first 4.3.x RPM upgrade, you MUST evaluate whether the switch_to_user.sh must be run in order for Unravel Server to load logs via HDFS and to access kerberos-spnego protected REST endpoints. If MapReduce or Spark jobs stop loading after upgrade, that is a strong indication that you need to switch users. This script need only be run once. See Run Unravel Daemons with Custom User for more details. Subsequent upgrades 4.3.x to 4.3.y will not need to run switch_to_user.sh script.

Upgrade process

An RPM upgrade will trigger a temporary background process so changes can take effect. You can monitor or wait for these with /usr/local/unravel/install_bin/await_fixups.sh which prints "Done" when it finishes. The script waits and prints ps of the background process. It might take 2-5 min. depending on your data size.

The background fixups will produce log file output in:

/tmp/unravel_post_upgrade_fixups.*.out

/tmp/unravel_schema.*.out

/usr/local/unravel/logs/unravel_schema.log

/usr/local/unravel/logs/es_migrate_4.3.1.0_from42.*

/usr/local/unravel/logs/unravel_esm.log

For further support, contact Unravel Support.

In this section: