Home

Upgrading Unravel from version 4.6.1.x to 4.7.x

This topic provides instructions to upgrade from Unravel version 4.6.x to 4.7.x.

Upgrade path

The following upgrade paths are supported for upgrading from Unravel 4.6.1.x to 4.7.x

Upgrading Unravel (4.6.1.x to 4.7.x)

The following occurs when you upgrade:

  • All the configurations are preserved.

  • All the data along with the configuration files is restored to /srv/unravel

  • Kafka topics are updated.

  • Elastic templates are updated.

  • Database schemas are updated.

Note

All new 4.7.x features such as Chargeback EMR and Cluster Insights are based on new ES docs and therefore are not applicable for apps that had run in lower versions of Unravel.

Before you upgrade from the 4.6.1.x to the 4.7.x version, do the following:

  1. Download the latest Unravel 4.7.x version and extract the Unravel binaries to /usr/local/unravel location. The files will be deployed at /usr/local/unravel/versions/X.Y.Z.

    ##TAR 
tar zxfh unravel-<version>.tar.gz -C /usr/local

##RPM
rpm -i unravel-<version>.rpm --prefix /usr/local

  2. Stop Unravel.

    /etc/init.d/unravel_all.sh stop

  3. Make a copy of the data directory (/srv/unravel).

    ##For example:
tar czf /tmp/unravel-data.tar.gz /srv/unravel

    You can use database backup tools to back up the data if you have an external database. For example, if you are using MySQL:

    ##For example:
mysqldump -u [username] –p[password] [database_name] > [dump_file.sql]

  4. Set the ownership and permissions for the files. Ensure that the unravel user owns all the files and directories when upgrading.

    chown -R <unravel user>:<unravel group> /usr/local/unravel /srv/unravel

## For example:
chown -R unravel:unravel /usr/local/unravel /srv/unravel

Run the following steps to upgrade:

  1. Before you begin the upgrade process, ensure to switch to the Unravel user if you have not done that already.

    su - <unravel user>

  2. Run setup based on the platform where you upgrade Unravel.

    • On-prem platforms (CDH, HDP, CDP, and MapR)

      /usr/local/unravel/versions/<unravel_version>/setup

    • Cloud

      • Databricks

        /usr/local/unravel/versions/<unravel_version>/setup --enable-databricks

      • Amazon Elastic MapReduce (EMR)

        /usr/local/unravel/versions/<unravel_version>/setup --enable-emr

  3. Check the Unravel version. In case of a failed upgrade, refer to Recovering a failed upgrade.

    /usr/local/unravel/manager version

  4. Start Unravel.

    /usr/local/unravel/manager start

  5. Check the status of services.

    /usr/local/unravel/manager report

    The following service statuses are reported:

    • OK: Service is up and running.

    • Not Monitored: Service is not running. (Has stopped or has failed to start)

    • Initializing: Services are starting up.

    • Does not exist: The process unexpectedly disappeared. Restarts will be attempted 10 times.

  6. Remove the old version of Unravel. Do the following:

    1. Remove the pre-conversion-can-be-deleted directory in /usr/local/unravel/versions

    2. Remove the zk_*_data.pre-conversion-can-be-deleted directories in /src/unravel.

    3. Run rpm -e to delete the old RPM. That will also clean up the scripts in /etc/init.d.

Manual configurations after upgrade (Optional)

Optionally, after the upgrade, you can manually configure the following:

For more Unravel configurations, refer to Configurations.

Adding certificates to the Truststore

The Truststore certificates for Unravel v4.7.x.x are located at data/certificates.

If you have manually added certificates to Unravel's JVM (used keytool to add certificates to the cacert files), those will not be imported when you upgrade. Therefore, you must add these certificates again. Refer to Truststore certificates and follow the instructions to add the Truststore certificates.

Caution

The regular Truststore cacert files in the Unravel JRE directory are not used. Also, do not use keytool to import directory.

Re-configure key tab location

In case before upgrading you have stored the keytab for Unravel at /usr/local/unravel, you must re-configure the keytab location using the following command:

manager config kerberos set --keytab </path/to/keytab>.

The content of /usr/local/unravel can be recovered from /usr/local/unravel/versions/pre-conversion.

If the unravel server had Kerberos enabled before the upgrade, refer to Configuring Kerberos section to re-configure Kerberos.

Re-configure some set properties

All the properties are imported during an upgrade. However, some properties may not carry the set value or work as intended. In such a case, you must re-configure such properties as follows:

  1. Run the unset command for the corresponding property.

    ./manager config properties unset <property name>

  2. Use the manager commands to re-configure the settings. Refer to the Configuration section.

Re-configuring Data page for single cluster deployment

If you have configured a single cluster deployment for Unravel and the cluster name is not default, then the Datapage feature may not work correctly. Data

For this, you must explicitly set the hive.metastore.cluster.ids property after upgrading:

  1. Stop Unravel.

    <Unravel installation directory>/unravel/manager stop

  2. Set the hive.metastore.cluster.ids property. You can get the cluster ID from the Operations page. 

    <Unravel installation directory>/unravel/manager config properties set hive.metastore.cluster.ids=<cluster-name>

  3. Apply the changes

    <Unravel installation directory>/unravel/manager config apply

  4. Start Unravel. 

    <Unravel installation directory>/unravel/manager start
Adding multiple workers (daemons)

If you have configured additional workers, they are not enabled after the upgrade. Refer to Enabling multiple daemons for high-volume data to add more workers.Enabling multiple daemon workers for high-volume data

Restore Jupyter Notebooks after upgrading from Unravel v4.6.x to 4.7.1.0

This section provides information about restoring Juypter Notebooks after upgrading from Unravel v4.6.x to v4.7.1.0.

Before upgrading from v4.6.x

  • Kill jupyter-notebook process.

    Ps -elf | grep jupyter-notebook
Kill -9 <pid of jupyter-notebook>

  • If you want to preserve the outputs of previous runs of notebooks, you must back up the following directory:

    /usr/local/unravel/ondemand/unravel-python-1.0.0/reports/datalab/analysis/notebooks/reports/html/
After upgrading to 4.7.1.0

  1. Enable Notebook service.

    1. Change directory to /usr/local/unravel/:

      cd /usr/local/unravel/

    2. Stop Unravel.

      <Unravel_installation_directory>/unravel/manager stop

    3. Run the following using the manager service:

      <Unravel_installation_directory>/unravel/manager config notebook

    4. Apply changes.

      <Unravel_installation_directory>/unravel/manager config apply

    5. Start Unravel.

      <Unravel_installation_directory>/unravel/manager start

    6. Check the status:

      <Unravel_installation_directory>/unravel/manager status

  2. Deploy the latest notebooks tar bundle.

    Notice

    The notebooks tar bundle is not prepackaged with Unravel installer. The Unravel support team provides this.

    1. Place notebooks.tgz tarball in /srv/unravel/notebook directory. Create this directory if it is not there already: 

      mkdir -p /srv/unravel/notebook
mv notebooks.tar /srv/unravel/notebook
cd /srv/unravel/notebook

    2. Untar notebooks.tgz.

      tar xzvf notebooks.tgz

    3. Verify the notebook service logs.

      tail -f /usr/local/unravel/logs/notebook.log
[I 09:50:26.701 NotebookApp] Writing notebook server cookie secret to /home/unravel/.local/share/jupyter/runtime/notebook_cookie_secret
[I 09:50:31.861 NotebookApp] Serving notebooks from local directory: /srv/unravel/notebook
[I 09:50:31.862 NotebookApp] The Jupyter Notebook is running at:
[I 09:50:31.862 NotebookApp] http://(tnode12.unraveldata.com or 127.0.0.1):8888/
[I 09:50:31.862 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

    4. Copy the backup HTML folder to /srv/unravel/notebook/notebooks/reports/html.

Enable Migration reports

If you had migrations reports on CDH or CDP clusters, you must run the following steps to continue using the migration reports.

  1. Stop Unravel.

    <Unravel installation directory>/unravel/manager stop

  2. Enable the Migration reports as an admin user. Run the following command:

    <Unravel installation directory>/unravel/manager config ondemand cloud-migration enable

  3. Apply changes. 

    <Unravel installation directory>/unravel/manager config apply

  4. Start Unravel

    <Unravel installation directory>/unravel/manager start

    The Migration tab will be visible on the Unravel UI. Also, refer to the Migrations topics for more details.

Configure MapR ticket (only required for MapR platform)

This is configured only in the case of MapR platform.

  1. Stop Unravel.

    <Unravel installation directory>/unravel/manager stop

  2. Use an editor to open <Installation_directory>/unravel/data/conf/unravel.yaml file.

  3. Specify the path to MapR Ticket corresponding to MAPRTICKET_LOCATION and save the changes. 

    unravel:
  environment:
    MAPRTICKET_LOCATION: /path/to/MapR_Ticket

  4. Apply the changes. 

    <Unravel installation directory>/unravel/manager config apply

  5. Start Unravel

    <Unravel installation directory>/unravel/manager start
Upgrading Unravel sensors

Refer to Upgrading sensors and upgrade the sensors corresponding to your platforms.

Important

In a multi-cluster deployment, it is mandatory to Upgrade the core node before you upgrade the sensors on the edge node.

Recovering from a failed upgrade

The following sections provide information about recovering from a failed upgrade in the following scenarios:

Pre-conversion verification failed

A verification failed and did not show the message Converting to the new installer . At this point, changes are not made, and therefore you can resolve the issue and rerun the setup.

New installer conversion failed

The Converting to new installer message is shown during the upgrade process, but the Starting upgrade message is not shown. This is an unlikely situation as the pre-conversion is covering for such failures. However, in the case of this scenario, you must manually revert to the original state. Depending on what and how things failed, some of these steps may not be required:

  1. Copy the old data files into the same location:

    mv /usr/local/unravel/versions/pre-conversion-can-be-deleted/* /usr/local/unravel/
rmdir /usr/local/unravel/versions/pre-conversion-can-be-deleted
mv /srv/unravel/zk_1_data.pre-conversion-can-be-deleted /srv/unravel/zk_1_data
mv /srv/unravel/zk_2_data.pre-conversion-can-be-deleted /srv/unravel/zk_2_data
mv /srv/unravel/zk_3_data.pre-conversion-can-be-deleted /srv/unravel/zk_3_data

  2. Remove the new installer based files:

    rm -rf /srv/unravel/zookeeper_1
rm -rf /srv/unravel/conf

  3. At this point, you can either start the old version of Unravel or resolve the cause of failure and rerun the setup.

Upgrade failed after displaying "Starting Upgrade"

The conversion to the new installer has been completed successfully, and the normal upgrade process failed.

Warning

Rerunning setup at this point without passing a configuration file can cause a new installation.

  1. Stop all the running Unravel processes.

  2. Check and resolve the cause of failure and rerun the upgrade using:

    • If manager service is available, run the following:

      <Unravel installation directory>/unravel/manager refresh all

    • If manager service is not yet available, run the following:

      /usr/local/unravel/versions/<unravel_version>/4.7.1.0/setup --config /srv/unravel/conf/unravel.yaml
Unable to roll forward with the upgrade

Caution

If you have updated the database or Elasticsearch, the old version of Unravel will not run. Hence, restoring a full backup is required.

If you are still unable to resolve the upgrade issue, you can return to the previous version of Unravel.

In case you see any of the following messages, restoring a full backup is required:

  • Loading elasticsearch templates

  • Creating schema objects

If the database and Elasticsearch are still in the pre-upgrade state, you must ensure to stop all the Unravel processes. Follow the instructions in New installer conversion failed and bring the process to a state where you can start the old version.

In this section: