Skip to main content

Home

AutoActions (GCP - BigQuery, Dataproc)

AutoActions automate the monitoring of your BigQuery jobs / YARN apps by allowing you to define complex, actionable policies for different cluster metrics.

autoaction-main.png

Alerts can be generated by setting AutoActions. For example, you can use an AutoAction to:

  • Notify you about a situation that requires manual intervention, such as resource contention or stuck jobs.

The Unravel server collects various metrics from clusters, which are aggregated based on the defined AutoAction policies and scope. Violations are detected based on the trigger conditions, and accordingly, set actions are executed.

autoactions-tasks-BIGQUERY.png

Each AutoAction policy consists of the following:

  • Trigger Conditions are the settings that determine when an AutoAction is violated, for example, apps using more than 2 GB of memory.

  • Scope Define refinements to various scopes. For example, the app is owned by User X and on cluster Y.

  • Actions Define the automated actions when user-defined trigger conditions are met. For example, email User X when the trigger conditions are met.

Note

Refer to the Limitations section to view the limitations of the AutoActions feature.

Also, see:

Creating AutoActions
  1. On the Unravel UI, click the AutoActions tab.

  2. Click new.png. The New AutoAction dialog box is displayed.

  3. Select either BigQuery or Dataproc tab.

  4. Select any of the following templates. The New AutoAction page is displayed.

    Tip

    Templates are policies that you define to trigger an alert.

    • BigQuery

      • Long running BigQuery Job: An AutoAction policy to detect BigQuery jobs that exceed the threshold of the following metrics:

        • Total Duration: Duration in seconds for a BigQuery job.

        • Slot MS: A BigQuery slot is a virtual CPU used by BigQuery to execute SQL queries. BigQuery automatically calculates how many slots each query requires, depending on query size and complexity.

          Slot MS is the total amount of slots per milliseconds used by a query.

    • Dataproc

      • Resource contention in cluster (allocated memory)

        Auto action policy for a cluster where the application running in that cluster has allocated memory, which exceeds the specified threshold.

      • Resource contention in cluster (allocated vcores)

        Auto action policy for a cluster where the application running in that cluster has allocated vcores, which exceed the specified threshold.

      • Resource contention in queue (allocated memory)

        Auto action policy for a queue where the application running in this queue has allocated memory, which exceeds the specified threshold.

      • Resource contention in queue (allocated vcores)

        Auto action policy for a queue where the application running in this queue has allocated vcores, which exceed a specified threshold.

      • Rogue user (allocated memory)

        Auto action policy for a user where an application submitted by this user has allocated memory, which exceeds the specified threshold.

      • Rogue user (allocated vcores)

        Auto action policy for a user where application submitted by this user has allocated vcores, which exceed the specified threshold.

      • Rogue application (allocated memory)

        Auto action policy for an application with allocated memory, which exceeds the specified threshold.

      • Rogue application (allocated vcores)

        Auto action policy for an application with allocated vcores, which exceed the specified threshold.

      • Long running YARN application

        Auto action policy for an application with a running time, which exceed the specified threshold.

      • Long running Hive query

        Auto action policy for a hive application with a running time, which exceed the specified threshold.

      • Long running workflow

        Auto action policy for workflow application with running time, which exceed the specified threshold.

      • Too many mappers for an MR job

        AutoAction policy for MapReduce job with mappers, which exceed the specified threshold.

      • Too many reducers for an MR job

        AutoAction policy for MapReduce job with reducers that exceed the specified threshold.

  5. Specify the following details:

    Item

    Description

    Policy Name

    Specify a policy name that clearly reflects the purpose of the AutoAction policy.

    Description

    Provide a description of the policy. This is optional.

    Active

    Select the checkbox to activate the policy after you create a new AutoAction. It is not mandatory to activate a policy during creation.

    Trigger Conditions

    Specify the conditions when an alert should be triggered.

    • Metrics

      Selected one of the cluster metrics. Refer to Supported cluster metrics.

    • Operator: Use the operator (>=) to define the threshold level of the selected metrics as compared to the value.

    • Value: Specify the threshold value in numeric. The default value is 0 is considered if you leave the Value field empty.

    Refine Scope

    Click CyanWhitePlus.png and specify a scope to trigger the AutoAction. You can refine the scope based on User, Projects, and Time

    For the User and Projects option, you must specify if it affects All, Only, or Except. For example, if you choose User and select:

    • All, then all the users are considered in the scope.

    • Only, then only that user is considered in the scope.

    • Except, then all the users except the specified user are considered in the scope.

    For Only and Except options, you need to do the following based on the scope that you have selected.

    • User: Select a user from the list. Click Add User, if you want to add more users. You can also specify regular expressions (regex) to specify the scope. Ensure to select a scope and then only define the regex.

    • Project: Select a user from the list. Click Add Projects, if you want to add more projects. You can also specify regular expressions (regex) to specify the scope. Ensure to select a scope and then only define the regex.

    For the Time option, you must specify if it is always or daily. If you select the daily option, then you must specify the timezone, start time, and end time during the day.

    Actions

    Click CyanWhitePlus.png and specify the action to be taken when the conditions defined in the AutoAction policy are met and an AutoAction is triggered. You can choose from any or all of the following actions:

    • Email

      • Enter the email ID of at least one recipient. Click Add Email to add more recipients. Select Include Owner to include the app's owner as a recipient of the email.

    • HTTP Post

      • Enter at least one URL. For example: http://localhost:8000/?notify=xyz@unraveldata.com&notify_owners=false

        Ensure to open the corresponding port.

        You can add more URLs by clicking Add URL.

    • Post to Slack

      Unravel provides integration with SlackApp, which lets you post information to one more Slack channel and user. Specify the following:

      • Webhook URL: The incoming webhook URL configured in Slack for the channel or user. You can post to multiple webhooks.

      • Token: The OAuth access token for the SlackApp.

      See Slack's Incoming Webhooks for more information.

  6. Click Create.

Also, refer to the Templates section for more information. Templates

Viewing AutoActions

You can view all the AutoActions created from the AutoActions tab. To filter and view the AutoActions, do the following:

  1. On the Unravel UI, click the AutoActions tab.

  2. From the following, select an option to filter the AutoActions:

    • Created Date

    • Cluster

    • Created By

    The AutoActions are listed in the following tabs:

    • All, where all the created AutoActions, irrespective of whether they are active or inactive, are listed. A green green.png bar adjacent to an AutoAction indicates that it is active. A red red.png bar adjacent to an AutoAction indicates that it is inactive.

    • Active, where only the active AutoActions are listed.

    • Inactive, where only the inactive Autoactions are listed.

    • Click the name link of the AutoAction to view the history of runs and the details of the applications that violated the policy.

      autoactions-history-runs.png
    • Click the threedots.png icon corresponding to the AutoAction you want to edit or delete. Select Edit to edit the AutoAction details and Delete to delete the AutoAction.

    • Click the corresponding Deactivate/Activate link to activate or inactivate an AutoAction.

Also, refer to the Limitations section to view the limitations of the AutoActions feature.

Activating or Deactivating an AutoAction
  1. On the Unravel UI, click the AutoActions tab.

  2. From the left panels, select an option to filter the AutoActions:

    • Created Date: Select a date range to view the list of AutoActions. You can either specify a custom range or choose one of the date range options (Today, 7 days, 14 days, 1 month, 3 months, 6 months, 1 year)

    • Cluster: Select a cluster to view the AutoActions created for that specific cluster. You can only choose a single cluster at a time.

    • Created By: Type a user's name to view the list of AutoActions created by that user.

    The AutoActions are listed in the following tabs:

    • All, where all the created AutoActions, irrespective of whether they are active or inactive, are listed.

    • Active, where only the active AutoActions are listed.

    • Inactive, where only the inactive AutoActions are listed.

  3. Do any of the following to activate or deactivate an AutoAction.

    • In the AutoAction tabs > AutoAction list, click the Deactivate/Activate link corresponding to an AutoAction.

    • On the AutoAction tabs > AutoAction list, click the AutoAction link. The corresponding AutoAction details page is displayed. Click the Deactivate/Activate tab.

Editing an AutoAction
  1. On the Unravel UI, click the AutoActions tab.

  2. From the following, select an option to filter the AutoActions:

    • Created Date

    • Cluster

    • Created By

    The AutoActions are listed in the following tabs:

    • All, where all the created AutoActions, irrespective of whether they are active or inactive, are listed.

    • Active, where only the active AutoActions are listed.

    • Inactive, where only the inactive Autoactions are listed.

  3. In the Actions column, click the threedots.png icon corresponding to the AutoAction you want to edit.

  4. Select Edit to edit the AutoAction details.

Deleting an AutoAction
  1. On the Unravel UI, click the AutoActions tab.

  2. From the following, select an option to filter the AutoActions:

    • Created Date

    • Cluster

    • Created By

    The AutoActions are listed in the following tabs:

    • All, where all the created AutoActions, irrespective of whether they are active or inactive, are listed.

    • Active, where only the active AutoActions are listed.

    • Inactive, where only the inactive AutoActions are listed.

  3. In the Actions column, click the threedots.png icon corresponding to the AutoAction you want to delete.

  4. Select Delete to delete the AutoAction details.

Regular Expressions (regex)

You can add regular expressions (regex) to AutoActions templates when you narrow the scope of User and Project with Only or Except. You must first select the scope (UserProject, Time ) and then only specify the regular expressions in the Transform box. Ensure always to select a scope before you specify the regex.

In the following example, regex is used to define the user scope.

{"enabled":true,"admin":true,"policy_name":"BigQuery Job","policy_id":1121,"instance_id":"6252699361701385875","name_by_user":"Long running BigQuery Job","description_by_user":"AutoAction policy for BigQuery jobs that exceed duration or slot MS thresholds.","created_by":"admin","last_edited_by":"admin","created_at":1667411797141,"updated_at":1667411797141,"entity_type":"bigquery","rules":[{"scope":"bigqueryjobs","metric":"bigqueryDuration","compare":">=","value":1}],"actions":[],"cluster_mode":0,"cluster_list":[],"cluster_transform":"","queue_mode":0,"queue_list":[],"queue_transform":"","user_mode":2,"user_list":["vlad","sandip"],"user_transform":"regex/^.*[.](.+)[.](.+)$/$2/","app_mode":1,"app_list":[],"app_transform":"","sustain_mode":0,"sustain_time":0,"time_mode":0}

Note

Keyword regex is mandatory. regex keyword is similar to substitute s in regular expressions.