Home

Instructions to upgrade Unravel from version 4.6.1.x to 4.7.x

This topic provides instructions to upgrade from Unravel version 4.6.x to 4.7.1.x.

Upgrade path

The following upgrade path is supported for upgrading from Unravel 4.6.x to 4.7.x

  • 4.6.1.8 or earlier → 4.6.1.9 → 4.7.1.0

  • 4.6.1.9 → 4.7.1.0

Upgrading Unravel (4.6.x to 4.7.x)

The following occurs when you upgrade:

  • All the configurations are preserved.

  • All the data along with the configuration files is restored to /srv/unravel

  • Kafka topics are updated.

  • Elastic templates are updated.

  • Database schemas are updated.

Note

All new 4.7.x features such as Chargeback EMR and Cluster Insights are based on new ES docs and therefore are not applicable for apps that had run in lower versions of Unravel.

Before you upgrade from 4.6.x to 4.7.x version, do the following:

  1. Download and extract the Unravel binaries to /usr/local/unravel location. The files will be deployed at /usr/local/unravel/versions/X.Y.Z.

    ##TAR 
tar zxf unravel-4.7.1.0.tar.gz -C /usr/local

##RPM
rpm -i unravel-4.7.1.0.rpm --prefix /usr/local

  2. Stop Unravel.

    /etc/init.d/unravel_all.sh stop

  3. Make a copy of the data directory (/srv/unravel).

    ##For example:
tar czf /tmp/unravel-data.tar.gz /srv/unravel

    If you have an external database, you can use database backup tools to back up the data. For example, if you are using MySQL:

    ##For example:
mysqldump -u [username] –p[password] [database_name] > [dump_file.sql]

  4. Set the ownership and permissions for the files. When you upgrade, ensure that the unravel user owns all the files and directories.

    chown -R <unravel user>:<unravel group> /usr/local/unravel /srv/unravel

## For example:
chown -R unravel:unravel /usr/local/unravel /srv/unravel

Run the following steps to upgrade:

  1. Before you begin the upgrade process, ensure to switch to the Unravel user if you have not done that already.

    su - <unravel user>

  2. Run setup based on the platform where you upgrade Unravel.

    • On-prem platforms

      /usr/local/unravel/versions/<unravel_version>/setup

    • Cloud

      • Databricks

        /usr/local/unravel/versions/<unravel_version>/setup --enable-databricks

      • Amazon Elastic MapReduce (EMR)

        /usr/local/unravel/versions/<unravel_version>/setup --enable-emr

  3. Check the Unravel version. In case of a failed upgrade, refer to Recovering a failed upgrade.

    /usr/local/unravel/manager version

  4. Start Unravel.

    /usr/local/unravel/manager start

  5. Check the status of services.

    /usr/local/unravel/manager report

    The following service statuses are reported:

    • OK: Service is up and running.

    • Not Monitored: Service is not running. (Has stopped or has failed to start)

    • Initializing: Services are starting up.

    • Does not exist: The process unexpectedly disappeared. Restarts will be attempted 10 times.

Manual configurations after upgrade (Optional)

Optionally, after the upgrade, you may require to do the following configurations manually.

  • Enable TLS certificates

  • Re-configure some set properties

  • Add multiple workers (daemons)

  • Restore Jupyter Notebooks

Enable TLS certificates

If you have manually added certificates to unravel’s JVM, those will not be imported when you upgrade. Therefore, you must add these certificates again. Refer to Configuring Truststore certificates and follow the instructions to add the Truststore certificates.

Re-configure some set properties

All the properties are imported during an upgrade. However, some properties may not carry the set value or may not work as intended. In such a case, you must re-configure such properties as follows:

  1. Run the unset command for the corresponding property.

    ./manager config properties unset <property name>

  2. Use the manager commands to reconfigure the settings. Refer to the Configuration section.

Adding multiple workers (daemons)

If you have configured additional workers, they are not enabled after the upgrade. Refer to Enabling multiple daemons for high-volume data to add more workers.

Restore Jupyter Notebooks after upgrading from Unravel v4.6.x to 4.7.1.0

This section provides information about restoring Juypter Notebooks after upgrading from Unravel v4.6.x to v4.7.1.0.

Before upgrading from v4.6.x

  • Kill jupyter-notebook process.

    Ps -elf | grep jupyter-notebook
Kill -9 <pid of jupyter-notebook>

  • If you want to preserve the outputs of previous runs of notebooks, you must back up the following directory:

    /usr/local/unravel/ondemand/unravel-python-1.0.0/reports/datalab/analysis/notebooks/reports/html/
After upgrading to 4.7.1.0

  1. Enable Notebook service.

    1. Change directory to /usr/local/unravel/:

      cd /usr/local/unravel/

    2. Stop Unravel.

      <Unravel_installation_directory>/unravel/manager stop

    3. Run the following using the manager service:

      <Unravel_installation_directory>/unravel/manager config notebook

    4. Apply changes.

      <Unravel_installation_directory>/unravel/manager config apply

    5. Start Unravel.

      <Unravel_installation_directory>/unravel/manager start

    6. Check the status:

      <Unravel_installation_directory>/unravel/manager status

  2. Deploy the latest notebooks tar bundle.

    Notice

    The notebooks tar bundle is not prepackaged with Unravel installer. The Unravel support team provides this.

    1. Place notebooks.tgz tarball in /srv/unravel/notebook directory. Create this directory if it is not there already: 

      mkdir -p /srv/unravel/notebook
mv notebooks.tar /srv/unravel/notebook
cd /srv/unravel/notebook

    2. Untar notebooks.tgz.

      tar xzvf notebooks.tgz

    3. Verify the notebook service logs.

      tail -f /usr/local/unravel/logs/notebook.log
[I 09:50:26.701 NotebookApp] Writing notebook server cookie secret to /home/unravel/.local/share/jupyter/runtime/notebook_cookie_secret
[I 09:50:31.861 NotebookApp] Serving notebooks from local directory: /srv/unravel/notebook
[I 09:50:31.862 NotebookApp] The Jupyter Notebook is running at:
[I 09:50:31.862 NotebookApp] http://(tnode12.unraveldata.com or 127.0.0.1):8888/
[I 09:50:31.862 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

    4. Copy the backup HTML folder to /srv/unravel/notebook/notebooks/reports/html.

Recovering from a failed upgrade

The following sections provide information about recovering from a failed upgrade in the following scenarios:

Pre-conversion verification failed

A verification failed and did not show the message Converting to the new installer . At this point, changes are not made, and therefore you can resolve the issue and rerun the setup.

New installer conversion failed

The Converting to new installer message is shown during the upgrade process, but the Starting upgrade message is not shown. This is an unlikely situation as the pre-conversion is covering for such failures. However, in the case of this scenario, you must manually revert to the original state. Depending on what and how things failed, some of these steps may not be required:

  1. Copy the old data files into the same location:

    mv /usr/local/unravel/versions/pre-conversion-can-be-deleted/* /usr/local/unravel/
rmdir /usr/local/unravel/versions/pre-conversion-can-be-deleted
mv /srv/unravel/zk_1_data.pre-conversion-can-be-deleted /srv/unravel/zk_1_data
mv /srv/unravel/zk_2_data.pre-conversion-can-be-deleted /srv/unravel/zk_2_data
mv /srv/unravel/zk_3_data.pre-conversion-can-be-deleted /srv/unravel/zk_3_data

  2. Remove the new installer based files:

    rm -rf /srv/unravel/zookeeper_1
rm -rf /srv/unravel/conf

  3. At this point, you can either start the old version of Unravel or resolve the cause of failure and rerun the setup.

Upgrade failed after displaying "Starting Upgrade"

The conversion to the new installer has been completed successfully, and the normal upgrade process failed.

Warning

Rerunning setup at this point without passing a configuration file can cause a new installation.

  1. Stop all the running Unravel processes.

  2. Check and resolve the cause of failure and rerun the upgrade using:

    • If manager service is available, run the following:

      <Unravel installation directory>/unravel/manager refresh all

    • If manager service is not yet available, run the following:

      /usr/local/unravel/versions/<unravel_version>/4.7.1.0/setup --config /srv/unravel/conf/unravel.yaml
Unable to roll forward with the upgrade

Caution

If the database or Elasticsearch were updated, the old version of Unravel would not run, therefore restoring a full backup is required.

If you are still unable to resolve the upgrade issue, you can return to the previous version of Unravel.

In case you see any of the following messages, restoring a full backup is required:

  • Loading elasticsearch templates

  • Creating schema objects

If the database and Elasticsearch are still in the pre-upgrade state, ensure that all Unravel processes are stopped. Follow the instructions in New installer conversion failed and bring the process to a state where you can start the old version.

Upgrading Unravel sensors

Refer to Upgrading sensors.

Important

In a multi-cluster deployment, it is mandatory to Upgrade the core node before you upgrade the sensors on the edge node.

In this section: