v4.3.1.1 Release notes
Software version
Release Date: 04/25/2018 for 4.3.1.1 Upgrade from 4.2.6 and 4.2.7 is supported
For details on downloading updates see Unravel 4.3.1.1.
Certified platforms
Please also review the compatibility matrix before upgrade/installation.
CDH: On-premise CDH (up to v5.13) with Kerberos.
HDP: On-premise HDP (up to v2.6) with Kerberos enabled + spnego enable. New Installs only.
Unravel sensor upgrade
Optional
New features
Significant improvement in speed to UI based on new technology stack (Angular + Node.js).
Support for Spark Streaming applications.
Support for showing the program for SparkSQL applications. SQL queries can be easily mapped to the corresponding line of code of the program.
Backend support for Spark application badges. The application level annotation contains meta-information about each input data source (event log, executor log, resource manager information). If one of the data sources has not been loaded for any reason (e.g., permission issues to download the logs from HDFS), the error message will be present in the application badge.
Merged Operation | Reports Tab and Data tab and into the new Reports tab to locate all reports together. Added ability to filter apps and resource usage graphs based on tags, e.g., user groups, projects, department.
Added ability to choose what columns to display in APM tables. 4.3.1.3 will add the column selection for the Applications page.
Support for scope target transformations in Auto Action rules. (UNRAVEL-3449)
Support for target transformation for global cluster/queue/user filters in Auto Actions rules. (UNRAVEL-3595)
Improvements and bug fixes
Improvements
Automatic Management of Temporary File Spaces
Improved Hive’s and MR’s TooManyMapper event to produce more accurate recommendation. (UNRAVEL-2558 & UNRAVEL-2562)
Improved Hive’s and MR’s TooManyReducer event to produce more accurate recommendation. (UNRAVEL-2560 & UNRAVEL-2564)
Added Hive’s and MR’s TooFewMapper event to detect cases where too few mappers are used and provide recommendation. (UNRAVEL-2559 & UNRAVEL-2563)
Improved Hive’s and MR’s TooFewReducer event to produce more accurate recommendation. (UNRAVEL-2561 & UNRAVEL-2565)
Added ImpalaNonColumnarTablesEvent to detect tables that are in non-columnar format.
Bug fixes
Impala tagging via Python script is overridden by Impala Query Options on CDH 5.13 (IMPALA-20)
Shorten jhist file name when its length is longer than limit of local file system. (UNRAVEL-2168)
Consolidated Impala time skew event from one per fragment to one event per query. (UNRAVEL-3502)
Known issues
HiveTooManyMapEvent, HiveTooManyReduceEvent, MRTooManyMapEvent, and MRTooManyReduceEvent could recommend a new value less than the current value for the parameters in the event. In that case, these recommendations should be ignored. Fix is already available in 4.3.1.1.
Unravel's temporary folder might have to be deleted manually at re/start.
New configuration properties
Please see Spark Properties for Spark Worker daemon @ Unravel for a complete explanation of each property, including default settings and any caveats.
com.unraveldata.onprem: Specifies whether the deployment is on premise or on cloud.
com.unraveldata.spark.live.pipeline.enabled: Specifies when to process stages and jobs.
com.unraveldata.spark.live.pipeline.maxStoredStages: Maximum number of jobs/stages stored in the DB.
com.unraveldata.spark.hadoopFsMulti.useFilteredFiles: Specifies how to search the event log files.
com.unraveldata.spark.hadoopFsMulti.eventlog.suffixes: Specifies the event log file suffixes used for prefix+suffix search
com.unraveldata.spark.time.histogram: Specifies whether the timeline histogram is generated or not.
Software upgrade support
Upgrading 4.2.6 or later to 4.3.x
Warning
Upgrade to 4.2.6 or later before upgrade to 4.3.x
Upgrading 4.2.x to 4.3.x, one-time potential must-do steps
Run-as
When upgrading from 4.2.x to 4.3.x, you need to be cognizant of the run-as user for Unravel server daemons. Unravel Server 4.2.x and earlier runs daemons with 2 different local users (usually 'unravel' and 'hdfs' or 'mapr'). In 4.3.x, Unravel server simplifies this to run-as the single user 'unravel' by default. During an upgrade from 4.2.x to 4.3.x, all the daemons are converted over to user 'unravel'. Other changes that occur because of this: /srv/unravel/log_hdfs/* logs are moved to /usr/local/unravel/logs
/srv/unravel/log_hdfs/directory is removed/srv/unravel/tmp_hdfs/is no longer needed, so it is removed (/srv/unravel/tmp/is used instead)env vars
HDFS_KEYTAB_PATHandHDFS_KERBEROS_PRINCIPALin/usr/local/unravel/etc/unravel.ext.shno longer usedFor Kerberos, com.unraveldata.kerberos.principal and com.unraveldata.kerberos.keytab.path are used instead.
After the first 4.3.x RPM upgrade, you MUST evaluate whether the /srv/unravel/log_hdfs/* logs are moved to /usr/local/unravel/logsswitch_to_user.sh /srv/unravel/log_hdfs/* logs are moved to /usr/local/unravel/logs must be run in order for Unravel Server to load logs via HDFS and to access kerberos-spnego protected REST endpoints. If MapReduce or Spark jobs stop loading after upgrade, that is a strong indication that you need to switch users. This script need only be run once. See Run Unravel Daemons with Custom User for more details. Subsequent upgrades 4.3.x to 4.3.y will not need to run switch_to_user.sh script.
Upgrade process
An RPM upgrade will trigger a temporary background process so changes can take effect. You can monitor or wait for these with /usr/local/unravel/install_bin/await_fixups.sh which prints "Done" when it finishes. The script waits and prints ps of the background process. It might take 2-5 min. depending on your data size.
The background fix ups will produce log file output in:
/tmp/unravel_post_upgrade_fixups.*.out/tmp/unravel_schema.*.out/usr/local/unravel/logs/unravel_schema.log/usr/local/unravel/logs/es_migrate_4.3.1.0_from42.*/usr/local/unravel/logs/unravel_esm.log
For further support, contact Unravel Support