Skip to main content

Home

Configuring CloudWatch agent

You can configure the CloudWatch agent for Unravel monitoring of your EMR clusters.

  • If you already have CloudWatch agent setup in your environment, then you must add metrics and dimensions to the CloudWatch agent configuration file.

  • If you do not have the CloudWatch agent setup, then you can run the Unravel bootstrap script, which will set up the CloudWatch agent in your environment.

  • If CloudWatch agent setup is available in your environment

    Add the following metrics and dimensions to the CloudWatch agent configuration file:

    {
      "agent": {
        "metrics_collection_interval": 60
      },
      "metrics": {
        "append_dimensions": {
          "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
          "ImageId": "${aws:ImageId}",
          "InstanceId": "${aws:InstanceId}",
          "InstanceType": "${aws:InstanceType}",
          "InstanceName": "${aws:InstanceName}"
        },
        "metrics_collected": {
          "mem": {
            "measurement": [
              "mem_used_percent"
            ],
            "metrics_collection_interval": 60
          },
          "cpu": {
            "measurement": [
              "cpu_usage_idle",
              "cpu_usage_iowait",
              "cpu_usage_user",
              "cpu_usage_system"
            ],
            "metrics_collection_interval": 60,
            "resources": [
              "*"
            ],
            "totalcpu": False
          }
        }
      }
    }
    
  • If CloudWatch agent setup is NOT available in your environment

    CloudWatch agent is packaged as part of the Unravel bootstrap script so to enable the setup of the CloudWatch agent, you must add the following optional argument within the bootstrap action:

    1. Edit the bootstrap action and provide the following location to the bootstrap script:

      s3://unraveldatarepo/unravel_emr_bootstrap.py

    2. In the Optional arguments text box, add the following arguments as shown in the image:

      -cwa or --cloud-watch-agent
      emr-cloudwatch-newusers-bootstrap.png

      Unravel bootstrap script will automatically deploy the CloudWatch Agent to all the EMR cluster nodes.