Home

Installing or upgrading OnDemand

The OnDemand service works with MySQL only.

Install MySQL on the Unravel host

  1. Install MySQL.

  2. Verify that MySQL is installed correctly by running netstats.

    The output should be as shown below.

    netstat -tunlp | grep :3306
tcp6       0      0 :::3306                 :::*                    LISTEN      28006/mysqld

  3. Run db_access.sh to verify that the ondemand_tasks, ondemand_sessions, and report_instances tables have been created:

    sudo /usr/local/unravel/install_bin/db_access.sh
mysql> show tables;
Install required packages on the Unravel host
Upgrades

Use rpm -q to see what packages are installed. It returns a list showing what packages are installed and which are not. Install the uninstalled packages.

rpm -q cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi cyrus-sasl-lib cyrus-sasl-plain python-saslwrapper openssl tkinter

For example

Check whether cyrus-sasl, open-ssl, and python are installed.

rpm -q cyrus-sasl openssl python
cyrus-sasl-2.1.26-23.el7.x86_64
package openssl is not installed
python-2.7.5-80.el7_6.x86_64

open-ssl is not installed, so install it.

yum install openssl

Verify open-ssl is installed.

rpm -q openssl

If you had to install tkinter and it failed to install, install it manually.

wget https://www.rpmfind.net/linux/centos/7.8.2003/os/x86_64/Packages/tk-8.5.13-6.el7.x86_64.rpm
yum install tk-8.5.13-6.el7.x86_64.rpm
systemctl restart unravel_ondemand.service

Important

If you have FSimage enabled see Configuring FSimage for necessary updates.

Fresh installs

Use the yum install command to install these packages:

  • cyrus-sasl

  • cyrus-sasl-devel

  • cyrus-sasl-gssapi

  • cyrus-sasl-lib

  • cyrus-sasl-plain

  • python-saslwrapper

  • saslwrapper

  • saslwrapper-devel

  • openssl

  • tkinter

yum install cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi cyrus-sasl-lib cyrus-sasl-plain python-saslwrapper openssl tkinter

Verify the packages have been installed.

rpm -q cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi cyrus-sasl-lib cyrus-sasl-plain python-saslwrapper openssl tkinter

If yum install tkinter fails, install it manually:

wget https://www.rpmfind.net/linux/centos/7.7.1908/os/x86_64/Packages/tk-8.5.13-6.el7.x86_64.rpm
yum install tk-8.5.13-6.el7.x86_64.rpm
systemctl restart unravel_ondemand.service
Install the OnDemand service on the Unravel host

  1. Download the OnDemand package to the /tmp directory on the Unravel host.

  2. If /usr/local/unravel/ondemand exists, save a copy of your (possibly customized) hive_properties.hive, and then delete /usr/local/unravel/ondemand.

    cd /usr/local/unravel
mv ondemand/unravel-python-1.0.0/scripts/hive_queries_reporting/hive_properties.hive /tmp/hive_properties.hive
sudo rm -rf ondemand

  3. In the /tmp directory, extract the contents of the tarball.

    cd /tmp
tar xvf ondemand-version-rhel[6|7].tar.gz

    This creates a new directory, /tmp/ondemand.

  4. Move the new directory, /tmp/ondemand, to /usr/local/unravel.

    sudo mv /tmp/ondemand/ /usr/local/unravel/

  5. Run the OnDemand installation script.

    cd /usr/local/unravel/ondemand/unravel-python-1.0.0
sudo ./install/ondemand_quick_install.sh
sudo /etc/init.d/unravel_all.sh restart

  6. (Optional) If you want Unravel to generate Small Files reports and you've customized hive_properties.hive, copy your customized hive_properties.hive to the ondemand directory:

    cp /tmp/hive_properties.hive /usr/local/unravel/ondemand/unravel-python-1.0.0/scripts/hive_queries_reporting/hive_properties.hive

  7. If your HiveServer2 is kerberized, add the unravel user to it.

  8. In /usr/local/unravel/etc/unravel.properties, specify the required properties for OnDemand:

  9. Restart Unravel Server.

    sudo /etc/init.d/unravel_all.sh restart

    Note

    If your host operating system is running a security-enhanced Linux (SELinux), you might get alerts like these after restarting Unravel Server, depending on your environment. You can ignore them:

    sealert -a /var/log/audit/audit.log
Alert 1: SELinux is preventing /usr/bin/bash from using the rlimitinh access on a process.
Alert 2: SELinux is preventing /usr/bin/python2.7 from using the rlimitinh access on a process.

    If you encounter problems, contact Unravel Support.

Start the OnDemand service

Important

If your HiveServer2 is kerberized, you should have already added the unravel user to it. Use the unravel user and its KRB ticket to run the commands below.

  1. Execute the following four commands in the order shown, replacing run-as-user and run-as-group with the username and group you used when running switch_to_user during Unravel server installation.

    For RHEL 7.x:

    sudo systemctl stop unravel_ondemand.service
sudo chown -RL run-as-user:run-as-group /usr/local/unravel/ondemand/
sudo service unravel_all.sh restart
sudo systemctl start unravel_ondemand.service

    For RHEL 6.x:

    sudo service unravel_ondemand stop
sudo chown -RL run-as-user:run-as-group /usr/local/unravel/ondemand/
sudo service unravel_all.sh restart
sudo service unravel_ondemand start

  2. Confirm that the OnDemand service is running:

    You should see output similar to the example below. Process IDs vary based on the number of processors in Unravel Server, number of current tasks, and so on.

    For RHEL 7.x:

    sudo systemctl status unravel_ondemand.service

    For RHEL 6.x:

    sudo service unravel_ondemand status

    You can also use ps command. You should see output similar to the example below.

    ps -ef | grep ondemand | grep -v grep root 11159 1 0 Sep21 ? 00:00:00 su - -c bash -c cd /usr/local/unravel/ondemand/; nohup unravel-python-*/bin/unravel_on_demand_start.sh root 11163 11159 0 Sep21 ? 00:00:00 -bash -c cd /usr/local/unravel/ondemand/; nohup unravel-python-*/bin/unravel_on_demand_start.sh root 11450 11176 0 Sep21 ? 00:00:42 /usr/local/unravel/ondemand/python/bin/python2.7 /usr/local/unravel/ondemand/python//bin/flask run hdfs 11452 11176 0 Sep21 ? 00:27:19 /usr/local/unravel/ondemand/python/bin/python2.7 /usr/local/unravel/ondemand/python//bin/celery -A celery_tasks worker --app=celery_tasks.celeryapp.celery --uid hdfs --gid hdfs --loglevel=info --workdir=/usr/local/unravel/ondemand/unravel-python-1.0.0/conf/.. --autoscale=24,3 hdfs 11670 11452 0 Sep21 ? 00:00:00 /usr/local/unravel/ondemand/python/bin/python2.7 /usr/local/unravel/ondemand/python//bin/celery -A celery_tasks worker --app=celery_tasks.celeryapp.celery --uid hdfs --gid hdfs --loglevel=info --workdir=/usr/local/unravel/ondemand/unravel-python-1.0.0/conf/.. --autoscale=24,3 hdfs 11671 11452 0 Sep21 ? 00:00:00 /usr/local/unravel/ondemand/python/bin/python2.7 /usr/local/unravel/ondemand/python//bin/celery -A celery_tasks worker --app=celery_tasks.celeryapp.celery --uid hdfs --gid hdfs --loglevel=info --workdir=/usr/local/unravel/ondemand/unravel-python-1.0.0/conf/.. --autoscale=24,3 hdfs 11672 11452 0 Sep21 ? 00:00:00 /usr/local/unravel/ondemand/python/bin/python2.7 /usr/local/unravel/ondemand/python//bin/celery -A celery_tasks worker --app=celery_tasks.celeryapp.celery --uid hdfs --gid hdfs --loglevel=info --workdir=/usr/local/unravel/ondemand/unravel-python-1.0.0/conf/.. --autoscale=24,3
Enable and configure OnDemand-based reports and features
In this section: