Amazon EMR cluster setup guide

This section provides instructions to connect an Amazon EMR cluster to Unravel.

Log in to AWS and navigate to Identity and Access Management (IAM).
Click Policies and select Create Policy.

Click JSON and paste the following code:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "pricing:GetProducts",
                "elasticmapreduce:ListClusters",
                "elasticmapreduce:DescribeCluster",
                "elasticmapreduce:ListInstanceFleets",
                "elasticmapreduce:ListInstanceGroups",
                "elasticmapreduce:ListInstances",
                "ec2:DescribeSpotPriceHistory",
                "ec2:DescribeInstances",
                "cloudwatch:GetMetricData",
                "cloudwatch:GetMetricStatistics"
            ],
            "Resource": "*"
        }
    ]
}

Note

You can find the JSON code in this location: Login to an Unravel EMR instance and navigate to Unravel > Settings > AWS Account Settings > New Account.

Give a name for the policy. The new policy is created.
Assign the created policy to a specific user or create a new user and attach the policy.

To create a new user, follow these steps:

In the IAM dashboard, click Users in the left panel.
Select Create a New User.
In the Set Permissions section, choose Attach existing policies directly.
Click Add Permissions to attach the policy to the user.

To set the security credentials, follow these steps:

In the Security credentials tab for the IAM user, click Create access key.
In the Access key best practices & alternatives section, select the Application Running on AWS Compute Service option.
Under Set Description Tag, enter a name for the tag value (optional).
Click Create access key to generate the access key. Ensure you save the Access Key ID and Secret Access Key securely. This is needed for further configuration on the Unravel interface.

Next, configure Unravel to connect to your AWS account settings by following these steps.

Access the Unravel Home Page and navigate to AWS Account Settings.
In the AWS Account details section, enter the user name for your AWS account.
Choose the AWS region you want Unravel to monitor. This is the region where your Amazon EMR cluster and other resources are located.
In the Account Key field, paste the Access Key ID that you obtained from the AWS Security Credentials section.
In the Secret Key field, paste the Secret Access Key associated with the Access Key ID.
Click Save. You have now configured unravel to connect to your AWS settings.

To connect your Amazon EMR clusters to Unravel for monitoring and management, follow the steps listed here:

Connect a new or existing EMR cluster to Unravel

This topic explains how to set up and configure your EMR cluster, so Unravel can begin monitoring jobs running on the cluster.

Assumptions

The EC2 instance for Unravel is created.
Unravel services are running.
The nodes in the EMR cluster allow all traffic from the Unravel EC2 instance. This implies either of the following configurations:
- The EMR cluster and Unravel EC2 instance are on the same VPC and same subnet, and their security group allows all traffic from the same subnet.
- The EMR cluster is on a different VPC, and you have configured VPC peering, route table creation, and updated your security policy.Setting up VPC peering (optional)
Network ACL on VPC allows all traffic.

Connect to a new EMR cluster

Follow the steps below to run Unravel's bootstrap script, unravel_emr_bootstrap.py, on all nodes in the cluster. The bootstrap script makes the following changes:

On the master node:
- On Hive clusters, it updates /lib/hive/conf/hive-site.xml
- On Spark clusters, it updates /lib/spark/conf/spark-defaults.conf
- It updates /lib/hadoop/etc/hadoop/mapred-site.xml
- It updates /lib/hadoop/etc/hadoop/yarn-site.xml
- If Tez is installed, it updates /etc/tez/conf/tez-site.xml
- It installs and starts the unravel_es daemon in /usr/local/unravel_es
- It installs the Spark and MapReduce sensors in /usr/local/unravel-agent/jars
- It installs the Hive Hook sensor in /usr/lib/hive/lib/.
On all other nodes:
- It installs the Spark and MapReduce sensors in /usr/local/unravel-agent/jars.

Run the following steps to connect Unravel to a new EMR cluster. Simultaneously, you can watch the following video tutorial:

Download Unravel's bootstrap script, unravel_emr_bootstrap.py.

curl https://s3.amazonaws.com/unraveldatarepo/unravel_emr_bootstrap.py -o /tmp/unravel_emr_bootstrap.py

Upload the bootstrap script to an S3 bucket.
Permissions needed
You need to write access to the S3 bucket if you want to upload the bootstrap script to. Also, the AWS account you use to create the EMR cluster must have read access to the bootstrap script to execute its directives.
To upload the bootstrap script to the default EMR logging bucket, s3://aws-logs-account_number-region/elasticmapreduce, execute the following command:
```
aws s3 cp unravel_emr_bootstrap.py s3://aws-logs-account_number-region/elasticmapreduce
```
On the AWS console, select the EMR service and click Create cluster.
On the Create Cluster - Quick Options page, click Go to advanced options.
In Step 1: Software and Steps, select emr-6.2 release.

In Step 2: Hardware, enter the following configuration for your EMR cluster and click Next.

Settings	Action
Instance group configuration	By default, the Uniform instance groups option is selected.
Network EC2 Subnet	The default configuration works for Network and EC2 Subnet if you have a virtual machine hosted in AWS in the same subnet. If your virtual machine is hosted on a different subnet in AWS or on GCP or Azure, then the virtual machine and the cloud platform must have access to the public IP.

Settings

Action

Instance group configuration

By default, the Uniform instance groups option is selected.

Network

EC2 Subnet

The default configuration works for Network and EC2 Subnet if you have a virtual machine hosted in AWS in the same subnet.

If your virtual machine is hosted on a different subnet in AWS or on GCP or Azure, then the virtual machine and the cloud platform must have access to the public IP.

In Step 3: General Cluster Settings, specify the following settings in Bootstrap Actions > Add Bootstrap Action, click Custom Action Add and then click Configure and add.

In the Add Bootstrap Action window, enter the following details and click Add:

Settings	Action
Name	Select Custom action.
Script location	Specify the following bootstrap location: `s3://unraveldatarepo/unravel_emr_bootstrap.py` Note If you want to monitor MR jobs, then you must pass an additional optional argument --all
Optional arguments	Enter the following: --unravel-server `UNRAVEL-INSTANCE-IP` --all --bootstrap

Settings

Action

Name

Select Custom action.

Script location

Specify the following bootstrap location:

s3://unraveldatarepo/unravel_emr_bootstrap.py

Note

If you want to monitor MR jobs, then you must pass an additional optional argument --all

Optional arguments

Enter the following:

--unravel-server UNRAVEL-INSTANCE-IP --all --bootstrap

The Amazon EMR cluster starts with this bootstrap action.

In Step 4: Security, edit the configuration for the cluster as required. For example:
- Choose the EC2 key pair.
- Select the EC2 security groups. AWS EMR service automatically applies additional rules that are required for EMR nodes.
Click Create cluster. Your new EMR cluster finishes the bootstrap process and is in the Waiting state.

Connect to an existing EMR cluster

To connect the Unravel EC2 instance to an existing EMR cluster, follow the steps below to run the Unravel EMR Ansible playbook on either the EMR master node or your Mac/Linux workstation.

Important

The following process is for existing clusters created without Unravel bootstrap. Only those clusters of this type that do not have auto-scaling enabled are currently supported.

Repeat the steps below to upgrade Unravel Sensors whenever you upgrade Unravel Server.

Note

In case you have to run unravel_emr_bootstrap.py manually, you must run it with the full path of the system default python, that is /usr/bin/python, as follows:

sudo /usr/bin/python unravel_emr_bootstrap.py

Before you begin

Copy the private key to the EMR master node and all the core nodes to access all the EMR nodes and change the key's permissions to read-only (chmod 400 key).
Identify the Ansible run user such as hadoop with the SSG key pair attached to the EMR cluster, or if the admin does not have access to the default key pair, then create a separate SSH key, either added to hadoop or root user.

Download unravel-emr-ansible.zip:

curl https://s3.amazonaws.com/unraveldatarepo/unravel-emr-ansible.zip --output unravel-emr-ansible.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11708  100 11708    0     0  66541      0 --:--:-- --:--:-- --:--:-- 66902

Unzip unravel-emr-ansible.zip:

Archive:  unravel-emr-ansible.zip
  inflating: unravel_emr_ansible/emr_ansible_inventory
  inflating: unravel_emr_ansible/unravel_emr_bootstrap.py
  inflating: unravel_emr_ansible/prepare_inventory.py
  inflating: unravel_emr_ansible/README.md
  inflating: unravel_emr_ansible/emr_ansible_playbook.yaml

Run prepare_inventory.py:
Enter the following values either interactively at the prompts or through their command-line options:
- --ssh-key path: The full pathname of the SSH private key
- --unravel-host hostname: The Unravel EC2 host's internal IP address.
- --cluster-id j-XXXXXX: The EMR cluster ID.
For example,
```
python prepare_inventory.py 
Please Enter Unravel host IP: 172.31.62.27
Please Enter the ssh key path: /home/hadoop/id_rsa

Ansible Inventory updated
```
Note
Ensure to upgrade pip before running this step.
```
sudo python3 -m pip install --upgrade pip
```
Install Ansible on the EMR master node:
```
sudo pip install ansible
```
(Optional) Determine what directory Ansible was installed in, and add that directory to the $PATH variable in ~/.bashrc, if it isn't there already.
```
which ansible
/usr/local/bin/ansible
```
In ~/.bashrc, update this line:
```
export PATH=/usr/local/bin/:$PATH
```

Run the Unravel Ansible playbook:

$ cd unravel_emr_ansible
$ export ANSIBLE_HOST_KEY_CHECKING=false
$ ansible-playbook -i emr_ansible_inventory emr_ansible_playbook.yaml
 
PLAY [nodes] *******************************************************************
 
TASK [Gathering Facts] *********************************************************
ok: [172.31.109.7]
ok: [172.31.109.251]
ok: [172.31.97.203]
 
TASK [Run emr bootstrap script] ************************************************
changed: [172.31.109.7]
changed: [172.31.109.251]
changed: [172.31.97.203]
 
TASK [Check Unravel sensor version] ********************************************
changed: [172.31.109.7]
changed: [172.31.109.251]
changed: [172.31.97.203]
 
TASK [Print sensor version] ****************************************************
ok: [172.31.109.7] => {
    "msg": "Unravel Version: 4.4.3.0b0005"
}
ok: [172.31.109.251] => {
    "msg": "Unravel Version: 4.4.3.0b0005"
}
ok: [172.31.97.203] => {
    "msg": "Unravel Version: 4.4.3.0b0005"
}
 
PLAY RECAP *********************************************************************
172.31.109.251             : ok=4    changed=2    unreachable=0    failed=0  
172.31.109.7               : ok=4    changed=2    unreachable=0    failed=0  
172.31.97.203              : ok=4    changed=2    unreachable=0    failed=0

Before you begin

Copy the private key to the EMR master node and all the core nodes to access all the EMR nodes and change the key's permissions to read-only (chmod 400 key).

Set up AWS CLI.
Make sure AWS CLI has permission to list EMR clusters.
```
aws emr list-instances --cluster-id cluster id
```
Also, ensure that the region stored in ~/.aws/config must match the region of the EMR cluster.

Download unravel-emr-ansible.zip:

wget https://s3.amazonaws.com/unraveldatarepo/unravel-emr-ansible.zip

Unzip unravel-emr-ansible.zip:

unzip unravel-emr-ansible.zip 
Archive:  unravel-emr-ansible.zip
  inflating: unravel-emr-ansible/README.md  
  inflating: unravel-emr-ansible/emr_ansible_inventory  
  inflating: unravel-emr-ansible/emr_ansible_playbook.yaml  
  inflating: unravel-emr-ansible/prepare_inventory.py  
  inflating: unravel-emr-ansible/unravel_emr_bootstrap.py

Run prepare_inventory.py:
Enter the following values either interactively at the prompts or through their command-line options:
- --cluster-id string: Cluster ID. Optional on EMR master, required if run outside of EMR cluster.
- --region string: AWS region. Optional on EMR master, required if run outside of EMR master node
- --inventory path: Directory containing the Ansible inventory file, emr_ansible_inventory. Default is the same directory as the playbook.
- --ssh-key path: The full pathname of the SSH private key
- --ssh-user string: SSH username with sudo privilege; default is hadoop
- --unravel-host hostname: The Unravel EC2 host's internal IP address.
- --use-public: Use the public IP address instead of the private IP address in Ansible inventory. Include this option if you need to connect to the EMR cluster solely through its public IP address.
For example,
```
python prepare_inventory.py --use-public
Please Enter Unravel host IP: 172.31.62.27
Please Enter ssh key path: /home/hadoop/id_rsa

Ansible Inventory updated
```
Install Ansible:
```
sudo pip install ansible
```

Run the Unravel Ansible playbook:

$ cd unravel-emr-ansible
$ export ANSIBLE_HOST_KEY_CHECKING=false
$ ansible-playbook -i emr_ansible_inventory emr_ansible_playbook.yaml
 
PLAY [nodes] *******************************************************************
 
TASK [Gathering Facts] *********************************************************
ok: [18.61.10.17]
ok: [18.61.10.21]
ok: [18.61.10.20]
 
TASK [Run emr bootstrap script] ************************************************
changed: [18.61.10.17]
changed: [18.61.10.21]
changed: [18.61.10.20]
 
TASK [Check Unravel sensor version] ********************************************
changed: [18.61.10.17]
changed: [18.61.10.21]
changed: [18.61.10.20]
 
TASK [Print sensor version] ****************************************************
ok: [18.61.10.17] => {
    "msg": "Unravel Version: 4.4.3.0b0005"
}
ok: [18.61.10.21] => {
    "msg": "Unravel Version: 4.4.3.0b0005"
}
ok: [18.61.10.20] => {
    "msg": "Unravel Version: 4.4.3.0b0005"
}
 
PLAY RECAP *********************************************************************
18.61.10.21             : ok=4    changed=2    unreachable=0    failed=0  
18.61.10.17             : ok=4    changed=2    unreachable=0    failed=0  
18.61.10.20             : ok=4    changed=2    unreachable=0    failed=0

After you connect the Unravel EC2 instance to your EMR cluster, run some jobs on the EMR cluster and monitor the information displayed in Unravel UI (http://unravel_ec2_node_public_IP:3000).Testing and troubleshooting
Set additional configurations if required.
Optionally, you can run healthcheck, at this point, to verify that all the configurations and services are running successfully.
```
<unravel_installation_directory>/unravel/manager healthcheck
```
Healthcheck is run automatically on an hourly basis in the backend. You can set the healthcheck intervals and email alerts to receive the healthcheck reports.

Refer to the following video:

In this section:

Home