Home

v4.3.1.3 Release notes

Software version

Release Date: 05/21/2018 for 4.3.1.3

For details on downloading updates see.

Certified platforms

  • CDH: On-premise (up to v5.14)with Kerberos; includes Spark 2.2.X.

  • HDP: On-premise HDP (up to v2.6) with Kerberos + spnego enabled.

  • MapR: On-premise 5.2.2 MapR.

Unravel sensor upgrade

  • Optional

New features

  • Added ability to turn off support/comments panel. See New Configuration Properties below. (CUSTOMER-33)

  • Role Based Accessed Control

    • Added Tagged based support for Reports > Operational Insights > Chargeback page. (CUSTOMER-9)

    • Added ability to configure admin and read only RBAC role using LDAP. See New Configuration Properties below.

  • Data access information on Reports > Data Insights table access tables now report access by an Impala query or a Spark SQL access in addition to Hive queries.

  • Added storage and trend analysis for HDFS and S3 storage systems based upon historical and current usage. Trend analysis also done in different dimensions: overall storage on the cluster, storage by user, storage by app type.

  • Added default or on-demand small files analysis on the cluster. User can set parameters for the analysis which can be viewed in UI and downloaded as a file.

  • Added ability to interactively sync Unravel with the latest Metastore status, including database, tables and partitions. See New Configuration Properties below.

  • Heat map support has been added for Clusters (Reports > Operational Insights > Cluster Workload). (UIX-237)

  • When Impala queries run as part of an Oozie workflow it is shown as components of Oozie action nodes and Oozie insights reflect the underlying queries. (IMPALA-46)

Improvements and bug fixes
Improvements

  • Reports > Overview and Reports > Data Insights now has two-way synchronization. (DATAPAGE-3, DATAPAGE-7)

  • The Last Access field of the table entry is updated when a query reads from or writes to a table.

  • Whenever the Hot/Warm/Cold policy is changed, the Labels column in table list are immediately updated on Reports > Data Insights > Details.

Bug fixes

  • Pig Gantt chart shows correct elapsed time. (Customer-23)

  • Tables

    • Table labels reflect the set policy (definition of Hot, Warm, Cold for display). (DATAPAGE-13, UIX-419, UIX-262, UIX-284)

    • Tables are created after running queries. (DATAPAGE-15)

    • All tables now have ‘dt’ entry. (DATAPAGE-15)

  • Table/partition counts match on across displays. (DATAPAGE-9, DATAPAGE-14, UIX-256)

  • Fixed user name mismatch between CM and Impalad UI. (IMPALA-51)

  • Auto Actions badge is displayed for Tez apps. (PLATFORM-102)

  • When RBAC is turned on non-admin user can only reach their own apps. (UIX-477)

  • Operations > Usage Details

    • Displays Resource graph when weekly is selected. (UIX-450)

    • Shows correct data when Group by filter is used. (UIX-456)

  • Operations > Dashboard charts resize correctly upon window expansion/contraction. (UIX-41)

  • Cluster

    • Month and Day view show correct/same jobs. (UIX-457)

    • Summary reports can be downloaded as CSV/JSON. (UIX-242)

  • Spark Gantt chart properly reflects time period. (UIX-371)

  • Impala

    • Operations > Charts > Impala populating apps correctly for a particular time. (UIX-236)

    • Tool tip in Graph shows correct number of apps.

    • Insights ordered by created timestamp.

  • Fragment or Operators

    • Fragment and Operator view populates correctly. (UIX-470)

    • Merging-Exchange shows Instance View correctly. (UIX-387)

    • Descending sort on time works correctly. (UIX-311)

    • Horizontal Scroll works correctly. (UIX-445)

  • Reports > Data Insights columns sort correctly. (UIX-388)

  • Workflow pagination works. (UIX-425)

  • dump_bundled_db.shr fixed for large packet size. (INSTALL-32)

  • Follow symlinks for k_data. (INSTALL-44)

Known issues

  • None

New configuration properties
LDAP properties for Role Based Access Control

Please see Supported Roles and Role Based Access Control (RBAC) for a complete explanation of each property, including default settings and any caveats.

  • com.unraveldata.login.admins.ldap.groups: Specifies Admin with read and write privilege.

  • com.unraveldata.login.admins.readonly.ldap.groups: Specifies Admin with read only privilege.

Hive Metastore

Please see Hive Metastore Configuration for a complete explanation of each property, including default settings and any caveats.

  • com.unraveldata.metastore.db.c3p0.maxpoolsize: Maximum connections in the connection pool.

  • com.unraveldata.metastore.db.c3p0.maxconnectionage: Maximum number of seconds any connections were forced to be released from the pool.

  • com.unraveldata.metastore.db.c3p0.maxidletimeexcessconnections: The number of seconds that connections are permitted to remain idle in the pool before being released.

  • com.unraveldata.metastore.db.c3p0.breakafteracquirefailure: Allows you to mark data source as broken and permanently be closed if a connection cannot be obtained from database.

  • com.unraveldata.metastore.db.c3p0.acquireRetryAttempts: Controls how many times c3p0 tries to obtain a connection from the database before giving up.

  • com.unraveldata.metastore.db.c3p0.acquireRetryDelay: Controls how much waiting time is between each retry attempts in milliseconds.

Disable support/comment in UI

Please see Disabling Support/Comments Panel for information on how to set the property.

  • com.unraveldata.ngui.support.enabled: Specifies if Customer Support/Comments are available in UI.

Software upgrade support

  • Support for upgrade from 4.2.6 (4.2.6-1128) or 4.2.7 (4.2.7-1154).

  • Support for upgrade from 4.3.0.4 and 4.3.1.0.

  • Support for upgrade from 4.3.0.4 and 4.3.1.x to 4.3.1.

Upgrading 4.2.6 or later to 4.3.x

Warning

Upgrade to 4.2.6 or later before upgrade to 4.3.x
Upgrading 4.2.x to 4.3.x, one-time potential must-do steps
Run-as

When upgrading from 4.2.x to 4.3.x, you need to be cognizant of the run-as user for Unravel server daemons. Unravel Server 4.2.x and earlier runs daemons with 2 different local users (usually 'unravel' and 'hdfs' or 'mapr'). In 4.3.x, Unravel server simplifies this to run-as the single user 'unravel' by default. During an upgrade from 4.2.x to 4.3.x, all the daemons are converted over to user 'unravel'. Other changes that occur because of this:/srv/unravel/log_hdfs/* logs are moved to /usr/local/unravel/logs

  • /srv/unravel/log_hdfs/ directory is removed

  • /srv/unravel/tmp_hdfs/ is no longer needed, so it is removed (/srv/unravel/tmp/ is used instead)

  • env vars HDFS_KEYTAB_PATH and HDFS_KERBEROS_PRINCIPAL in /usr/local/unravel/etc/unravel.ext.sh no longer used

  • For kerberos, com.unraveldata.kerberos.principal and com.unraveldata.kerberos.keytab.path are used instead.

After the first 4.3.x RPM upgrade, you MUST evaluate whether the switch_to_user.sh must be run in order for Unravel Server to load logs via HDFS and to access kerberos-spnego protected REST endpoints. If MapReduce or Spark jobs stop loading after upgrade, that is a strong indication that you need to switch users. This script need only be run once. See Run Unravel Daemons with Custom User for more details. Subsequent upgrades 4.3.x to 4.3.y will not need to run switch_to_user.sh script.

Upgrade Process

An RPM upgrade will trigger a temporary background process so changes can take effect. You can monitor or wait for these with /usr/local/unravel/install_bin/await_fixups.sh which prints "Done" when it finishes. The script waits and prints ps of the background process. It might take 2-5 min. depending on your data size.

The background fixups will produce log file output in:

  • /tmp/unravel_post_upgrade_fixups.*.out

  • /tmp/unravel_schema.*.out

  • /usr/local/unravel/logs/unravel_schema.log

  • /usr/local/unravel/logs/es_migrate_4.3.1.0_from42.*

  • /usr/local/unravel/logs/unravel_esm.log

For further support, contact Unravel Support.

In this section: