Upgrading Unravel from version 4.7.x to 4.7.5.x

This topic provides instructions to upgrade Unravel from version v4.7.0.x, v4.7.1.x, v4.7.2.x, 4.7.3.x, 4.7.4.x to v4.7.5.x for the following platforms:

On-prem platforms (single cluster and multi-cluster deployments): Cloudera Distribution of Apache Hadoop (CDH), Cloudera Data Platform (CDP), and Hortonworks Data Platform (HDP)
Cloud platforms: Amazon EMR, Databricks (Azure, AWS), GCP - Dataproc, GCP - BigQuery

The following upgrade paths are supported:

4.7.0.x → 4.7.5.x
4.7.1.x → 4.7.5.x
4.7.2.x → 4.7.5.x
4.7.3.x → 4.7.5.x
4.7.4.x → 4.7.5.x

On-prem platforms upgrade (v4.7.x to v4.7.5.0)

Cloudera Distribution of Apache Hadoop (CDH), Cloudera Data Platform (CDP), and Hortonworks Data Platform (HDP)

This section provides information about upgrading Unravel for on-prem platforms (CDH, CDP, HDP, and MapR)

Before you upgrade Unravel for on-prem platforms

Backup the data directory
Take a backup of the data directory and the external database if you use one. To back up the data, refer to Backing-up and recovering Unravel
Note
If an upgrade fails, you can roll back the upgrade to the release from which you had upgraded. For information, see Rollback after a failed upgrade. For this rollback to work, you must ensure to take a backup of the data directory and the external database before you upgrade.
Configure custom JDK, if you have a different version of JDK installed
Unravel requires Java Runtime Environment (JRE), and hence it is shipped with OpenJDK version 17.0.1. If you have a different version of JDK installed, you must configure Unravel to access the corresponding jre directory in that JDK. For instructions, refer to Configuring custom JDK.
In a multi-cluster deployment, you must configure the jre directory both on the edge node and the core node.
If you upgrade from 4.7.4.x to 4.7.5.0 version, then OpenJDK requires a high standard of encryption for the Kerberos keytab. If the default Kerberos encryption type is rc4-hmac and des3-hmac-sha1, add the following line in /etc/krb5.conf or any custom location of the Kerberos configuration file on the Unravel server:
```
allow allow_weak_crypto=true
```

Upgrade Unravel for on-prem platforms (v4.7.x to v4.7.5.0)

You can upgrade Unravel for the single cluster as well as multi-cluster deployments. Refer to the following:

Important

Ensure to perform the post-upgrade steps. Refer to Post upgrade steps for on-prem platforms (v4.7.x to v4.7.5.0).

Upgrade Unravel in a single cluster deployment (v4.7.x to v4.7.5.0)

Go to the Download section. The complete list of Unravel downloads is available in this section.
Select and download the Unravel version that you want to upgrade. The section provides the details and necessary instructions to download Unravel.
Extract the Unravel binaries of the Unravel version you want to upgrade.
```
tar zxf unravel-<version>tar.gz -C </path/to/installation/directory>
```
For example: tar zxf unravel-4.7.0.0.tar.gz -C /opt
Note
The default installation directory is /usr/local for RPM installation. This might have changed during the initial deployment. Make sure that you use the same installation directory when you upgrade Unravel.

Stop Unravel.

<Unravel installation directory>/unravel/manager stop

Activate Unravel version 4.7.5.0.
```
<Unravel installation directory>/unravel/manager activate <version>
```
Note
For on-prem platforms (CDH, CDP, HDP or MapR), If you have used an automatic Hadoop configuration, run auto-configuration and refresh to reload the updates automatically.
```
<unravel_installation_directory>/unravel/manager config auto --refresh
```

Apply the changes.

<unravel_installation_directory>/unravel/manager config apply

Start all the services.

<unravel_installation_directory>/unravel/manager start

Check the status of the services.
```
<unravel_installation_directory>/unravel/manager report 
```
The following service statuses are reported:
- OK: Service is up and running.
- Not Monitored: Service is not running. (Has stopped or has failed to start)
- Initializing: Services are starting up.
- Does not exist: The process unexpectedly disappeared. A restart will be attempted 10 times.
Validate the JAVA version that Unravel is using now.
Delete the previous installation directory from unravel/versions/<THE.OLD.VERSION>.

Upgrade Unravel in multi-cluster deployment (v4.7.x to v4.7.5.0)

To upgrade Unravel in a multi-cluster environment, you must upgrade all the edge nodes, upgrade the core node, and pull all the edge node updates to the core node.

Edge nodes
Run the following steps to upgrade Unravel on the edge nodes involved in Unravel monitoring.
1. Go to the Download page.
2. Click the Unravel version you want to upgrade to and run the commands provided to download Unravel.
3. Extract the Unravel binaries of the Unravel version you want to upgrade.
```
tar zxf unravel-<version>tar.gz -C </path/to/installation/directory>
```
  For example: tar zxf unravel-4.7.0.0.tar.gz -C /opt
  Note
  The default installation directory is /usr/local for RPM installation. This might have changed during the initial deployment. Make sure that you use the same installation directory when you upgrade Unravel.
4. Stop Unravel.
```
<Unravel installation directory>/unravel/manager stop
```
5. Activate the version that you want to upgrade.
```
<Unravel installation directory>/unravel/manager activate <version>
```
  Caution
  You may get a Network port failed precheck error if you run the manager activate command immediately after executing the manager stop command.
  To avoid this precheck error, stop Unravel and wait for a minute or two before executing the manager activate command.
  Retry the manager activate command in case of the Network port failed precheck error.
6. If you have used an automatic Hadoop configuration, run auto-configuration and refresh to reload the updates automatically.
```
<unravel_installation_directory>/unravel/manager config auto --refresh
```
7. Apply the changes.
```
<unravel_installation_directory>/unravel/manager config apply
```
8. Start all the services.
```
<unravel_installation_directory>/unravel/manager start 
```
9. Check the status of services.
```
<unravel_installation_directory>/unravel/manager report 
```
  The following service statuses are reported:
  - OK: Service is up and running.
  - Not Monitored: Service is not running. (Has stopped or has failed to start)
  - Initializing: Services are starting up.
  - Does not exist: The process unexpectedly disappeared. Restarts will be attempted 10 times.
10. Validate the JAVA version that Unravel is using now.

Core node

Run the following steps only on the core node.

Download Unravel.
Extract the Unravel binaries of the Unravel version you want to upgrade.
```
tar zxf unravel-<version>tar.gz -C </path/to/installation/directory>
```
For example: tar zxf unravel-4.7.0.0.tar.gz -C /opt

Stop Unravel.

<Unravel installation directory>/unravel/manager stop

Activate the version that you want to upgrade.

<Unravel installation directory>/unravel/manager activate <version>

Run auto-configuration on the core node and refresh to reload the updates automatically. Run this command only if you have previously run the config auto command on the cluster. This is an optional step.
```
<unravel_installation_directory>/unravel/manager config auto --refresh
```
Caution
If you have NOT run the manager config auto earlier on the core node, an error will be displayed, which you can ignore.

Find the edge key using the manager config edge show command, keep it handy and run the following command for each configured edge node. This refreshes the configurations on the edge nodes. Provide the edge key when prompted:

<unravel_installation_directory>/unravel/manager config edge auto --refresh

For example:

/opt/unravel/manager config edge auto --refresh
-- Running: config edge auto --refresh
2021-09-22 03:32:25 Archiving configuration ... Ok
Edge key: edge-tnode39

The Hive metastore database password can be recovered automatically only for a cluster manager with an administrative account. Otherwise, you must manually set the password as follows:

Run the manager config edge show command to get the <EDGE_KEY>, <HIVE_KEY>, and <CLUSTER_KEY>, which must be provided when you set the Hive metastore password.

<EDGE_KEY> is the label you provide to identify the edge node when you add the edge node in Step 3.
CLUSTER_KEY is the name of the cluster where you set the Hive configurations.
<HIVE_KEY> is the definition of the Hive service. In the output of the manager config edge show command, this is shown as the <SERVICE_KEY> for Hive.

-- Running: config edge show
------------ | ---------------------------------------- | ------------
    EDGE KEY | - edge-a                                 | Enabled
             |     Cluster manager:                     | Enabled
             |     Clusters:                            | 
 CLUSTER KEY |       - Cluster_Name                     | Enabled
             |           HBASE:                         | 
 SERVICE KEY |             - hbase                      | Enabled
             |           HDFS:                          | 
 SERVICE KEY |             - hdfs                       | Enabled
             |           HIVE:                          | 
 SERVICE KEY |             - hive                       | Enabled
 SERVICE KEY |             - hive2                      | Enabled
             |           IMPALA:                        | 
 SERVICE KEY |             - impala                     | Enabled
 SERVICE KEY |             - impala2                    | Enabled
             |           KAFKA:                         | 
 SERVICE KEY |             - kafka                      | Enabled
 SERVICE KEY |             - kafka2                     | Enabled
             |           SPARK_ON_YARN:                 | 
 SERVICE KEY |             - spark_on_yarn              | Enabled
             |           YARN:                          | 
 SERVICE KEY |             - yarn                       | Enabled
             |           ZOOKEEPER:                     | 
 SERVICE KEY |             - zookeeper                  | Enabled
------------ | ---------------------------------------- | ------------
-- OK

In a multi-cluster deployment, where edge nodes are monitoring, set the password on the core node as follows:

<Unravel installation directory>/unravel/manager config edge hive metastore password <EDGE_KEY> <CLUSTER-KEY> <HIVE-KEY> <password> 
##Example: /opt/unravel/manager config edge hive metastore password local-node cluster1 hive password

In case, the core node is monitoring the Hadoop cluster directly, run the following command from the core node.

<Unravel installation directory>/unravel/manager config hive metastore password <CLUSTER_KEY> <HIVE_KEY> <password> 
##Example: /opt/unravel/unravel/manager config edge hive metastore password cluster1 hive P@SSw0rd

Apply the changes.

<unravel_installation_directory>/unravel/manager config apply

Start all the services.

<unravel_installation_directory>/unravel/manager start

Check the status of services.

<unravel_installation_directory>/unravel/manager report

Delete the old install directory from unravel/versions/<THE.OLD.VERSION>.

Important

Ensure to perform the post-upgrade steps. Refer to Post upgrade steps for on-prem platforms (v4.7.x to v4.7.5.0).

Post upgrade steps for on-prem platforms (v4.7.x to v4.7.5.0)

After you upgrade Unravel, set the following:

Upgrade Unravel sensors

Refer to Upgrading sensors.

Important

In a multi-cluster deployment, upgrading the core node is mandatory before upgrading the sensors on the edge node.

Enable Migration reports (Optional)

If you had migrations reports on CDH or CDP clusters, you must run the following steps to continue using the migration reports.

Stop Unravel.

<Unravel installation directory>/unravel/manager stop

Enable the Migration reports as an admin user. Run the following command:

<Unravel installation directory>/unravel/manager config ondemand cloud-migration enable

Apply changes.

<Unravel installation directory>/unravel/manager config apply

Start Unravel
```
<Unravel installation directory>/unravel/manager start
```
The Migration tab will be visible on the Unravel UI. Also, refer to the Migrations topics for more details.

Cloud platforms upgrade (v4.7.x to v4.7.5.0)

Amazon EMR, Databricks, GCP-Dataproc, GCP-Bigquery

This section provides information about upgrading Unravel for cloud platforms (Amazon EMR, Databricks, GCP-Dataproc, GCP-Bigquery)

Important notice - Amazon EMR platform

For the Amazon EMR platform, the data collected for Clusters/Cost tabs in the previous versions of Unravel (v4.7.3.0, v4.7.4.0) will not be retained after you upgrade to Unravel v4.7.5.0.
Unravel does not support the Insights Overview tab on the UI for the EMR platform.

Before you upgrade Unravel for Cloud platforms (v4.7.x to v4.7.5.0)

Backup the data directory
Take a backup of the data directory and the external database if you use one. To back up the data, refer to Backing-up and recovering Unravel
Note
If an upgrade fails, you can roll back the upgrade to the release from which you had upgraded. For information, see Rollback after a failed upgrade. For this rollback to work, you must ensure to take a backup of the data directory and the external database before you upgrade.
Configure custom JDK, if you have a different version of JDK installed
Unravel requires Java Runtime Environment (JRE), and hence it is shipped with OpenJDK version 17.0.1. If you have a different version of JDK installed, you must configure Unravel to access the corresponding jre directory in that JDK. For instructions, refer to Configuring custom JDK.

Reset global init scripts (For Databricks platform only)

The global init script is not backward compatible, and therefore, before you upgrade, ensure to disable or remove the existing global init scripts and set the new global init scripts. Also, in Unravel version 4.7.5.0, the following tokens/passcodes will be encrypted. For this encryption to take effect too, you must disable or remove all the previously set up global init scripts and set up the new global init scripts.

DBX Personal Access Token (PAT)
LR Authentication password
Unravel trust store password
Azure Active Directory token

Do the following to reset the global init scripts.

On Databricks, go to Workspace > Settings > Admin Console > Global init scripts.
If you are upgrading from a previous version of Unravel, delete all the existing init scripts; unravel_cluster_init.sh, unravel_spark_init.sh.

Click +Add and set the following:

Item	Settings
Name	Enter the name as unravel_init.
Script	Copy and paste the following content in the Script box: #!/bin/bash # # Runs Unravel Init scripts COUNTER=1 while [ ! -d "/dbfs" ] && [ $COUNTER -le 20 ]; do echo "$(date) Waiting for dbfs mount: RetryCount = ${COUNTER} ....." ((COUNTER++)) sleep 0.1 done UD_ROOT=/dbfs/databricks/unravel CLUSTER_INIT=${UD_ROOT}/unravel-db-sensor-archive/dbin/unravel_cluster_init.sh SPARK_INIT=${UD_ROOT}/unravel-db-sensor-archive/dbin/unravel_spark_init.sh if [ ! -f "${CLUSTER_INIT}" ]; then echo "Unravel Cluster Init ${CLUSTER_INIT} doesn't exist!" exit 0 else cp ${CLUSTER_INIT} /tmp/ chmod a+x /tmp/unravel_cluster_init.sh /tmp/unravel_cluster_init.sh fi if [ ! -f "${SPARK_INIT}" ]; then echo "Unravel Spark Init ${SPARK_INIT} doesn't exist!" exit 0 else cp ${SPARK_INIT} /tmp/ chmod a+x /tmp/unravel_spark_init.sh /tmp/unravel_spark_init.sh fi
Enabled	Turn on the Enable toggle.

Item

Settings

Name

Enter the name as unravel_init.

Script

Copy and paste the following content in the Script box:

#!/bin/bash
#
# Runs Unravel Init scripts

COUNTER=1
while [ ! -d "/dbfs" ] && [ $COUNTER -le 20 ]; do
 echo "$(date) Waiting for dbfs mount: RetryCount = ${COUNTER} ....."
 ((COUNTER++))
 sleep 0.1
done

UD_ROOT=/dbfs/databricks/unravel
CLUSTER_INIT=${UD_ROOT}/unravel-db-sensor-archive/dbin/unravel_cluster_init.sh
SPARK_INIT=${UD_ROOT}/unravel-db-sensor-archive/dbin/unravel_spark_init.sh

if [ ! -f "${CLUSTER_INIT}" ]; then
 echo "Unravel Cluster Init ${CLUSTER_INIT} doesn't exist!"
 exit 0
else
 cp ${CLUSTER_INIT} /tmp/
 chmod a+x /tmp/unravel_cluster_init.sh
 /tmp/unravel_cluster_init.sh
fi

if [ ! -f "${SPARK_INIT}" ]; then
 echo "Unravel Spark Init ${SPARK_INIT} doesn't exist!"
 exit 0
else
 cp ${SPARK_INIT} /tmp/
 chmod a+x /tmp/unravel_spark_init.sh
 /tmp/unravel_spark_init.sh
fi

Enabled

Turn on the Enable toggle.

Click Add to save the settings.

Note

Cluster logging should be enabled at the cluster level. See Logging in Cluster init script for instructions.

Upgrade Unravel for Cloud platforms (v4.7.x to v4.7.5.0)

Go to the Download section. The complete list of Unravel downloads is available in this section.
Download the Unravel version that you want to upgrade. The section provides the details and necessary instructions to download Unravel.
Extract the Unravel binaries of the Unravel version you want to upgrade.
```
tar zxf unravel-<version>tar.gz -C </path/to/installation/directory>
```
For example: tar zxf unravel-4.7.0.0.tar.gz -C /opt

Stop Unravel.

<Unravel installation directory>/unravel/manager stop

Activate to Unravel version 4.7.5.0.

<Unravel installation directory>/unravel/manager activate <version>

Apply the changes.

<unravel_installation_directory>/unravel/manager config apply

Start all the services.

<unravel_installation_directory>/unravel/manager start

Check the status of the services.
```
<unravel_installation_directory>/unravel/manager report 
```
The following service statuses are reported:
- OK: Service is up and running.
- Not Monitored: Service is not running. (Has stopped or has failed to start)
- Initializing: Services are starting up.
- Does not exist: The process unexpectedly disappeared. A restart will be attempted 10 times.
Validate the JAVA version that Unravel is using now.
Delete the previous installation directory from unravel/versions/<THE.OLD.VERSION>.

Important

Ensure to perform the post-upgrade steps. Refer to Post upgrade steps for Cloud platforms (v4.7.x to v4.7.5.0).

Post upgrade steps for Cloud platforms (v4.7.x to v4.7.5.0)

After you upgrade Unravel, set the following:

For the Databricks platform, you must configure the following:
- Update Log Receiver (LR) endpoint
- Configure Log Receiver (LR) authentication (Optional)
For the GCP BigQuery platform, you must set up Unravel to receive BigQuery data automatically.
Enable SSL/TLS support for external database.

Update Log Receiver (LR) endpoint for Databricks

Update the Log Receiver (LR) endpoint if the Unravel LR is not accessible with the default hostname and port from Databricks.

Set LR hostname and LR port number to default based on the following scenarios:
- TLS is NOT enabled, run the following:
```
<Unravel installation directory>/unravel/manager config databricks set-lr-endpoint <hostname> 4043
```
- TLS is enabled, run the following:
```
<Unravel installation directory>/unravel/manager config databricks set-lr-endpoint <FQDN> 4443
```
- TLS is enabled, but if you want to keep using HTTP, run the following:
```
<Unravel installation directory>/unravel/manager config databricks set-lr-endpoint <FQDN> 4043 --no-tls
```
If TLS is enabled, the LR uses HTTPS protocol and listens on the 4443 port (by default). You must ensure that port 4443 to the unravel server is open.

Apply changes.

<Unravel installation directory>/unravel/manager config apply

Update Workspaces.

<Unravel installation directory>/unravel/manager refresh databricks

Restart Unravel.

<Unravel installation directory>/unravel/manager start

Examples:

TLS is NOT enabled

/opt/unravel/manager config databricks set-lr-endpoint 10.2.1.4 4043
/opt/unravel/manager stop then config apply then refresh databricks then start

TLS is enabled

/opt/unravel/manager config databricks set-lr-endpoint myhost.unraveldata.com 4443
/opt/unravel/manager stop then config apply then refresh databricks then start

TLS is enabled, but you want to continue using HTTPS

/opt/unravel/manager config databricks set-lr-endpoint myhost.unraveldata.com 4043 --no-tls
/opt/unravel/manager stop then config apply then refresh databricks then start

Configure Log Receiver (LR) authentication for Databricks (Optional)

Refer to Configuring LR authentication.

Set up Unravel to receive BigQuery data automatically

Refer to Set up Unravel to receive BigQuery data automatically

Note

The plain text service account keys on Unravel server are encrypted when you upgrade to Unravel v4.7.5.0.

Enable SSL/TLS support for external MySQL

After you upgrade, you can set the SSL/TLS support for external MySQL.

Important

You can enable the SSL/TLS support only for MySQL version 8.

Stop Unravel.

<Unravel installation directory>/unravel/manager stop

Use an editor to open <Installation_directory>/unravel/data/conf/unravel.yaml file.

In the unravel.yaml file, change tls value to true.

database:
    external: true
    hostname: <hostname>
    port: 'port'
    schema: unravel_db_prod
    tls: true
    type: mysql
    username: <username>
    password: <encrypted password>

Apply the changes.

<Unravel installation directory>/unravel/manager config apply

Start Unravel.

<Unravel installation directory>/unravel/manager start

Validate the JAVA version of Unravel

Go to Unravel node and login as unravel user. In the case of multi-cluster, go to Unravel edge node.

Run the following command:

ps -ef | grep unravel | grep java

For example, the following output is shown:

unravel 26871 1 1 Jul05 ? 00:30:43 /tmp/jdk1.8.0_112/jre/bin/java -server -Dident=unravel_taw_1 -Dunravel.log.dir=/opt/unravel/logs -Dhadoop.version= -Djdbc.driver.jar=com.mysql.jdbc.Driver -Dhadoop.conf.dir=/etc/hadoop/conf -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dnetworkaddress.cache.ttl=30 -Dsun.net.inetaddr.ttl=30 -Xmx6g -Xms1g -Dvertx.cacheDirBase=/opt/unravel/tmp/ver

In the above example, Unravel has selected the Java version /tmp/jdk1.8.0_112/jre/bin/java.

You can further validate the Java version. For example, run the following command:
```
/tmp/jdk1.8.0_112/jre/bin/java -version 
```

In this section:

Home

Upgrading Unravel from version 4.7.x to 4.7.5.x

On-prem platforms upgrade (v4.7.x to v4.7.5.0)

Cloudera Distribution of Apache Hadoop (CDH), Cloudera Data Platform (CDP), and Hortonworks Data Platform (HDP)

Before you upgrade Unravel for on-prem platforms

Note

Upgrade Unravel for on-prem platforms (v4.7.x to v4.7.5.0)

Important

Note

Note

Note

Caution

Caution

Important

Post upgrade steps for on-prem platforms (v4.7.x to v4.7.5.0)

Important

Cloud platforms upgrade (v4.7.x to v4.7.5.0)

Amazon EMR, Databricks, GCP-Dataproc, GCP-Bigquery

Important notice - Amazon EMR platform

Before you upgrade Unravel for Cloud platforms (v4.7.x to v4.7.5.0)

Note

Note

Upgrade Unravel for Cloud platforms (v4.7.x to v4.7.5.0)

Important

Post upgrade steps for Cloud platforms (v4.7.x to v4.7.5.0)

Note

Enable SSL/TLS support for external MySQL

Important

Validate the JAVA version of Unravel

Search results