Skip to main content

Home

AutoActions (Databricks)

The AutoActions automate the monitoring of your compute cluster by allowing you to define complex, actionable policies for different cluster metrics.

autoaction-main.png

Alerts can be generated by setting AutoActions. For example, you can use an AutoAction to:

  • Notify you about a situation that requires manual intervention, such as resource contention or stuck jobs.

  • Automatically kill an app or move it to a different queue.

The Unravel server collects various metrics from clusters, which are aggregated based on the defined AutoAction policies and scope. Violations are detected based on the trigger conditions, and accordingly, set actions are executed.

autoactions-tasks.png

Each AutoAction policy consists of the following:

  • Trigger Conditions are the settings that determine when an AutoAction is violated, for example, apps using more than 2 GB of memory.

  • Scope Define refinements to various scopes. For example, the app is owned by User X and on cluster Y.

  • Actions Define the automated actions when user-defined trigger conditions are met. For example, email User X when the trigger conditions are met.

Note

Refer to the Limitations section to view the limitations of the AutoActions feature.

Also, see:

Creating AutoActions

  1. On the Unravel UI, click the AutoActions tab.

  2. Click new.png. The New AutoAction dialog box is displayed.

  3. Select any of the following templates. The New AutoAction page is displayed.

    Tip

    Templates are policies that you define to trigger an alert.

    • Databricks Clusters: This is a template where you can define an AutoAction policy for Databricks clusters that exceed the cost or duration threshold.

    • Databricks Job runs: This is a template where you can define an AutoAction policy for Databricks job runs that exceed the cost or duration threshold.

  4. Specify the following details:

    Item

    Description

    Policy Name

    Specify a policy name that reflects the purpose of the AutoAction policy.

    Description

    Provide a description of the policy. This is optional.

    Active

    Select the checkbox to activate the policy after you create a new AutoAction. It is not mandatory to activate a policy during creation.

    Trigger Conditions

    Specify the conditions when an alert should be triggered.

    • Metrics

      • totalDuration: Total duration of a Databricks job.

      • cost: The cost incurred for a Databricks job.

    • Operator: Use the operators (>, >=, ==, <, <=) to define the threshold level of the selected metrics as compared to the value.

    • Value: Specify the threshold value in numeric. The default value is 0 is considered if you leave the Value field empty.

    Refine Scope

    Click CyanWhitePlus.png and specify a scope to trigger the AutoAction. You can refine the scope based on User, Workspace, and Time

    For the Users and Workspaces option, you must specify if it affects All, Only, and Except. For example, if you choose User and select:

    • All, then, all the users are considered in the scope.

    • Only, then only that user is considered in the scope.

    • Except, then all the users except the specified user are considered in the scope.

    For Only and Except options, you need to do the following based on the scope that you have selected.

    • User: Select a user from the list. Click Add User, if you want to add more users. You can also specify regular expressions (regex) to specify the scope. Ensure to select a scope and then only define the regex.

    • Workspace: Select a user from the list. Click Add Workspace, if you want to add more workspaces. You can also specify regular expressions (regex) to specify the scope. Ensure to select a scope and then only define the regex.

    For the Time option, you must specify if it is always or daily. If you select the daily option, you must specify the timezone, start time, and end time during the day.

    Actions

    Click CyanWhitePlus.png and specify the action to be taken when the conditions defined in the AutoAction policy are met, and an AutoAction is triggered. You can choose from any or all of the following actions:

    • Email

      • Enter the email ID of at least one recipient. Click Add Email to add more recipients. Select Include Owner to include the app's owner as a recipient of the email.

    • HTTP Post

      • Enter at least one URL. For example: http://localhost:8000/?notify=xyz@unraveldata.com&amp;amp;notify_owners=false

        Ensure to open the corresponding port.

        You can add more URLs by clicking Add URL.

    • Post to Slack

      Unravel provides integration with SlackApp, which lets you post information to one more Slack channel and user. Specify the following:

      • Webhook URL: The incoming webhook URL configured in Slack for the channel or user. You can post to multiple webhooks.

      • Token: The OAuth access token for the SlackApp.

      See Slack's Incoming Webhooks for more information.

  5. Click Create.

Also, refer to the Templates section for more information.

Viewing AutoActions

You can view all the AutoActions created from the AutoActions tab. To filter and view the AutoActions, do the following:

  1. On the Unravel UI, click the AutoActions tab.

  2. From the following, select an option to filter the AutoActions:

    • Created Date

    • Cluster

    • Created By

    The AutoActions are listed in the following tabs:

    • All, where all the created AutoActions, irrespective of whether they are active or inactive, are listed. A green green.png bar adjacent to an AutoAction indicates that it is active. A red red.png bar adjacent to an AutoAction indicates that it is inactive.

    • Active, where only the active AutoActions are listed.

    • Inactive, where only the inactive Autoactions are listed.

    • Click the name link of the AutoAction to view the history of runs and the details of the applications that violated the policy.

      autoaction-history.png

    • Click the threedots.png icon corresponding to the AutoAction that you want to edit or delete. Select Edit to edit the AutoAction details and select Delete to delete the AutoAction.

    • Click the corresponding Deactivate/Activate link to activate or inactivate an AutoAction.

Also, refer to the Limitations section to view the limitations of the AutoActions feature.

Activating/Deactivating an AutoAction

  1. On the Unravel UI, click the AutoActions tab.

  2. From the following, select an option to filter the AutoActions:

    • Created Date: Select a date range to view the list of AutoActions. You can either specify a custom range or choose one of the date range options (Today, 7 days, 14 days, 1 month, 3 months, 6 months, 1 year)

    • Cluster: Select a cluster to view the AutoActions created for that specific cluster. You can only choose a single cluster at a time.

    • Created By: Type a user's name to view the list of AutoActions created by that user.

    The AutoActions are listed in the following tabs:

    • All, where all the created AutoActions, irrespective of whether they are active or inactive, are listed.

    • Active, where only the active AutoActions are listed.

    • Inactive, where only the inactive Autoactions are listed.

  3. Click the corresponding Deactivate/Activate link to activate or inactivate an AutoAction.

Editing an AutoAction

  1. On the Unravel UI, click the AutoActions tab.

  2. From the following, select an option to filter the AutoActions:

    • Created Date

    • Cluster

    • Created By

    The AutoActions are listed in the following tabs:

    • All, where all the created AutoActions, irrespective of whether they are active or inactive, are listed.

    • Active, where only the active AutoActions are listed.

    • Inactive, where only the inactive Autoactions are listed.

  3. In the Actions column, click the threedots.png icon corresponding to the AutoAction you want to edit.

  4. Select Edit to edit the AutoAction details.

Deleting an AutoAction

  1. On the Unravel UI, click the AutoActions tab.

  2. From the following, select an option to filter the AutoActions:

    • Created Date

    • Cluster

    • Created By

    The AutoActions are listed in the following tabs:

    • All, where all the created AutoActions, irrespective of whether they are active or inactive, are listed.

    • Active, where only the active AutoActions are listed.

    • Inactive, where only the inactive Autoactions are listed.

  3. In the Actions column, click the threedots.png icon corresponding to the AutoAction you want to delete.

  4. Select Delete to delete the AutoAction details.

Regular Expressions (regex) (Databricks)

You can add regular expressions (regex) to AutoActions templates when you narrow the scope of User and Workspace with Only or Except. You must first select the scope (User or Workspace) and only specify the regular expressions in the Transform box. Ensure always to select a scope before you specify the regex.

Note

Keyword regex is mandatory. The regex keyword is similar to substitute s in regular expressions. The regular expression must be in the following format:

regex/{reg_exp_string}/{transform}/

Here:

  • {reg_exp_string}: Replace with a Java regular expression. For documentation of Java regular expression, see Oracle documentation.

  • {transform}: Replace with a transform string, which typically contains a reference to a capturing group that can be referenced with $1. See Oracle documentation.

Example: regex/^.*(my_user).*/$1/

This example matches any string that contains the "my_user" string.

In the following example, regex is used to define the user scope.

{"enabled":true,
"admin":true,
"policy_name":"AutoAction2",
"policy_id":10,
"instance_id":"3955285931312492702",
"name_by_user":"Long Running Databricks Job with Include owner",
"description_by_user":"Long Running Databricks Job with include owner email",
"created_by":"admin",
"last_edited_by":"admin",
"created_at":1664884043240,
"updated_at":1664884043240,
"rules":[{"OR":[{"scope":"Databricks jobs",
"metric":"totalDuration",
"compare":">=","value":5000},{"scope":"Databricks jobs","metric":"cost",
"compare":">=","value":1}]}],
"actions":[],
"cluster_mode":0,
"cluster_list":[],
"cluster_transform":"",
"queue_mode":0,
"queue_list":[],
"queue_transform":"",
"user_mode":2,
"user_list":["vlad","sandip"],
"user_transform":"regex/^.*[.](.+)[.](.+)$/$2/",
"app_mode":1,
"app_list":[],
"app_transform":"",
"sustain_mode":0,
"sustain_time":0,
"time_mode":0}
In this section: