Home

Running AutoAction demos

The Demos program provides you a way to understand and experiment with AutoActions and their triggering. Example AutoActions are included for Map Reduce, Hive, Spark, and Workflow jobs. Scripts are provided that trigger a violation in order to demonstrate the Actions in "action".

Contact Unravel Support to in order to download AutoActions demos.

Text indicates where you must substitute your particular values for the text.

demo-path is the complete path of the auto-actions-demos directory.

Unpack and install the AutoAction demos

  1. Put the auto-actions-demos.tgz file in the directory Unravel Server host machine where you want to unpack it.

  2. Navigate to the directory and unpack the demos.

    # tar -xvzf auto-actions-demo.tgz

  3. Tar creates and unpacks the files into auto-actions-demos directory.

    The directory should contain the following files.

    # ls auto-actions-demos
demos/          setup/

  4. Go to demo-path/setup directory.

  5. Open ./settings file and enter the email address to receive violation notifications.

  6. Execute the ./setup-all script.

    # ./setup-all

    Note

    The AutoAction rules that include time specification are automatically adjusted to the current time period, for example, from CURRENT_HOUR:00 to CURRENT_HOUR+2:00.

  7. After running the script go to the Unravel Server UI and select Manage | AutoActions tab.

    You should see all the AutoAction demos listed under Active AutoActions. Each AutoAction is entitled AA-tag, for example, AA-Spark-1c, Map-1b.

Executing the demos

  1. Go to demo-path/demos directory. For each AutoActions rule listed in Manage | AutoActions there is a corresponding script in the demo's directory. Each script triggers the corresponding AutoAction demo.

    For example, in the UI the AutoAction named AA-Spark-1c is listed. The corresponding triggering script file is demo-Spark-1c. You should see the following files in the demos directory.

    # cd demo-path/setup 
# ls
demo-Hive-1a            demo-MR-1a              demo-MR-2b              demo-Spark-1b
demo-Hive-2a            demo-MR-1b              demo-MR-3a              demo-Spark-1c
demo-Hive-2b            demo-MR-1c              demo-MR-3b              run-all-demos
demo-Hive-3a            demo-MR-2a              demo-Spark-1a   scripts/

  2. Execute ./demo-tag script to trigger the corresponding AA-tag rule. Each script is designed to simulate violation conditions for the corresponding AutoAction on the target Hadoop cluster, for example, to trigger AA-Spark-1c you run the demo-Spark-1c script.

    Some of AutoAction's demo scripts trigger multiple AutoActions. This side effect can happen when running your own defined AutoActions due to AutoActions having overlapping definitions.

Cleaning up demos

Run ./clean-all script to remove all the demo AutoAction from the Unravel Server. If you want to run the demos again, simply follow the procedure starting with extracting the files (step 3 above)to

# cd demo-path/setup 
# ./clean-all

.

AutoActions demos list

Application and Alert Type

Use case

AutoAction Triggering Script

[empty]

Notes

MapReduce

Alert if a MapReduce job is grabbing majority of cluster resources and may affect other users jobs at any time.

Alert if any MapReduce job allocated memory > 20GB.

AA-MR-1a

Demo-MR-1a

Submits to “root.sla” queue.

Alert if any MapReduce job allocated vCores > 10.

AA-MR-1b

Demo-MR-1b

Submits to “root.sla” queue.

Alert if any MapReduce job is running for longer than 10 minutes.

AA-MR-1c

Demo-MR-1c

Submits to “root.sla” queue.

May trigger MR-1b.

MapReduce

Alert if a MapReduce job may affect any production SLA jobs running on a cluster.

Alert if any app is not in the queue ‘sla_queue’ and running between X and Y and allocated memory > 20GB.

AA-MR-2a

Demo-MR-2a

Also triggers MR-1a as well.

Alert if any app is not in the queue ‘sla_queue’ and running between X and Y and allocated vCores greater than 10.

AA-MR-2b

Demo-MR-2b

Also triggers MR-2a as well.

MapReduce

Alert if an ad hoc MapReduce job is grabbing majority of cluster resources and may affect cluster performance.

Alert if any MapReduce job allocated vCores > 10 between X and Y in queue ‘root.adhoc’.

AA-MR-3a

Demo-MR-3a

Submits to “root.adhoc” queue.

Also triggers MR-1a and MR-2a.

Alert if any MapReduce job allocated memory > 20GB between X and Y in queue ‘root.adhoc’.

AA-MR-3b

Demo-MR-3b

Submits to “root.adhoc” queue.

Also triggers MR-1b and MR-2b.

Spark

Alert if a Spark app is grabbing majority of cluster resources and may affect other users jobs at any time.

Alert if any Spark app has allocated more than 20GB of memory.

AA-Spark-1a

Demo-Spark-1a

Alert if any Spark app allocated vCores > 8.

AA-Spark-1b

Demo-Spark-1b

Alert if any Spark app is running longer than 10 minutes

AA-Spark-1c

Demo-Spark-1c

Spark

Alert if a Spark SQL query has unbalanced input vs output, which may point to an inefficient or “rogue” queries.

Alert if any Spark app is generating lots of rows in comparison with input,i.e. ‘outputToInputRowRatio’ > 1000.

TBD

Hive

Alert if a Hive query duration is running longer than expected.

Alert if a Hive query duration > 5 minutes.

AA-Hive-1a

Demo-Hive-1a

You can Ctrl-C the query once it triggers the AA.

Hive

Alert if SLA bound query is taking longer than expected.

Alert if a Hive query started between A:00 and B:00 in queue ‘root.prod’ and duration > 10 minutes.

AA-Hive-2a

Demo-Hive-2a

You can Ctrl-C the query once it triggers the AA.

Alert if any Hive query is started between A:00 and B:00 in any queue except ‘root.prod’.

AA-Hive-2b

Demo-Hive-2b

Very short query.

Hive

Alert if a Hive query is writing lots of data.

Alert if a Hive query writes out more than 200MB in total.

AA-Hive-3a

Demo-Hive-3a

Alert if a Hive query reads in more than 10GB in total.

AA-Hive-3b

Demo-Hive-3b

Hive

Detect inefficient and “stuck” Hive queries.

Alert if any Hive query has read less than 10MB in total and its duration is longer than 10 minutes.

AA-Hive-4a

Demo-Hive-4a

Alert if any Hive query in the queue 'root.adhoc' is running for longer than 2 minutes.

AA-Hive-4b

Demo-Hive-4b

Workflow

Alert if a workflow is taking longer than expected.

Alert if any workflow is running for longer than 10 minutes, might be stuck.

AA-WF-1a

Demo-WF-1a

You can Ctrl-C the query once it triggers the AA.

Alert if a SLA bound workflow named ‘market_report’ is running for longer than 5 minutes.

AA-WF-1b

Demo-WF-1b

You can Ctrl-C the query once it triggers the AA.

Workflow

Alert if a workflow is reading more data than expected.

In this section: