Skip to main content

Home

Unravel for Amazon EMR
Overview

This topic describes how to launch an Unravel Server from the AWS Marketplace and connect EMR clusters to it.

Note

Use these instructions for test or development environments only. For production environments, see Amazon Elastic MapReduce (EMR).

Unravel for Amazon EMR provides full-stack monitoring, tuning, troubleshooting, and resource optimization for big data running on Amazon EMR. Unravel goes beyond passive monitoring to highly automated and intelligent management and optimizing data pipelines and applications.

This topic helps you set up the following configuration:

emr-highlevel-diagram-part2.png

Run the following steps to set up Unravel for Amazon EMR from the AWS Marketplace:

emr-part2-flowchart.png
Background

Ensure to fulfill the following before getting started:

  • For Unravel Server to be able to monitor your EMR cluster(s), you need to:

    • Allow Unravel server to access EMR cluster nodes' TCP ports 8020, 50010, 50020.

    • Open Unravel server port 4043 for receiving traffic from the EMR clusters.

  • To access Unravel UI, you need to open port 3000 on Unravel server for HTTP access.

  • Also, check other Prerequisites for Amazon EMR.

Step 2: Launch and set up Unravel server

Use the following steps to launch and set up Unravel server.

Step 2 (a): Find Unravel for Amazon EMR from the Marketplace

  1. Search for Unravel for Amazon EMR in the AWS Marketplace.

    aws-_marketplace-_step1a-_find-_app-4730.png

  2. On the listing page, click Continue to Subscribe.

  3. On the subscription page, click Continue to Configuration.

  4. Choose a version and location:

    • On the Configure this software page, select the latest software version and the same region where you plan to create EMR clusters (Unravel Server can reside in a different region. However, you must set up VPC peering).

      aws-_marketplace-_step1a-_configure-this-software.png

    • Click Continue to Launch to continue with the setup.

  5. Launch the software:

    On the Launch this software page, select Launch from Website from the dropdown and then click Launch.

    aws-marketplace-step1a-launch-app.png

    Note

    There are some more steps to be completed before the instance is actually launched.

Step 2 (b): Choose an instance type

The instance type you choose depends on the number of applications to be monitored daily and other factors. You can leave the default value as is.

aws-marketplace-step1b-choose-instance.png
Step 2 (c): Configure VPC and subnet setting

Choose the VPC and Subnet you created or a suitable one as discussed in the Background section above.

aws-marketplace-step1c-vpc.png
Step 2 (d): Configure security group settings

  1. Click Create New Based On Seller Settings.

    aws-marketplace-step1d-security-group1.png

  2. Name the security group and provide any description. Ensure to note the name.

    For example, Unravel for Amazon EMR-1.

    aws-marketplace-step1d-security-group2.png

    Note

    Unravel recommends limiting access to only known IP addresses.

  3. Click Save.

Note

You can change the above rules for enhanced security based on your setup and Unravel's requirements.

Step 2 (e): Configure key pair settings

Choose an existing key pair that connects to this instance or create a new one, and then click Launch.

aws-marketplace-step1e-keypair.png

Unravel Installation is completed. The following screen is displayed after a successful installation.

aws-workspace-congratulations.png

Go to the EC2 Dashboard and launch the instance created in the previous step.

Step 2 (f): Log into Unravel UI

Once the AMI has instantiated and Unravel is installed, which can take 10-15 minutes, log into Unravel UI.

  1. Get the hostname or IP of the instance from the AWS Console:

    aws-marketplace-step1f-get-hostname.png

  2. Navigate to http://hostname-or-ip-address-of-the-instance:3000 with a web browser.

  3. Connect to the Unravel instance using SSH and get the admin password for Unravel UI. After this, sign in to Unravel UI with the username as admin and the password that was earlier fetched from Unravel instance.

    emr-_marketplace-_ssh-_password-new.png
    aws-marketplace-dashboard-_demo.png

Next, you must connect a new EMR cluster to Unravel Server.

Step 2: Connect a new EMR cluster to Unravel server

Use the following steps to connect a new EMR cluster to Unravel server

Step 2 (a): Create a new EMR cluster and connect Unravel

  1. Go to AWS EMR Dashboard and click Create Cluster.

    aws-marketplace-step2a-create-cluster.png

  2. Click Go to advanced options.

    aws-_marketplace-_step2-_create-cluster.png

  3. Select the release and the services you want to install and click Next

    aws-marketpplace-step2a-advanced-options-softwareconfig.png

  4. Select the same VPC and subnet as the one chosen for Unravel server and click Next. Unravel server can reside on a different VPC, but you must set up VPC peering.

    aws-marketpplace-step2a-advanced-options-hardwareconfig.png

  5. Add a bootstrap action.

    • In this section, go to the Bootstrap Actions section, select Custom Actions, and click Configure and Add.

      aws-marketpplace-step2a-advanced-options-generaloptions.png

      The Add Bootstrap Action dialog box is displayed.

  6. In the Script location text box, enter s3://unraveldatarepo/unravel_emr_bootstrap.py

    aws-marketpplace-step2a-advanced-options-add-bootstrap.png

  7. In the Optional Arguments text box, do the following:

    • Note the private IP from your Unravel instance:

      aws-marketplace-step2a-create-cluster-add-bootstrap3.png

    • Add --unravel-server unravel-ec2-private-ip-address --bootstrap

      aws-marketplace-step2a-create-cluster-add-bootstrap4.png

  8. Click Save in the dialog box, and then click Next.

  9. In the next screen, as shown below, select a key pair to connect to the EC2 nodes and note the names of the security groups encircled below, as you modify them.

    aws-marketplace-step2a-create-cluster-add-bootstrap5.png

  10. Click Create Cluster.

Step 2 (b): Modify EMR cluster security groups so that Unravel Server has adequate access

  1. To find the security group ID of the security group named Unravel for Amazon EMR-1, navigate to Security Groups from the left panel in your AWS console and search for Unravel for Amazon EMR-1.

    aws-marketplace-step2b-modify-security-group1.png

  2. Note the group ID.

    For example, in the screenshot above, the group ID is sg-0564b1b8902ecf611.

  3. From the EMR cluster’s screen (where it shows its status in the creation process), navigate to each of the two security groups highlighted in the following screenshot.

    aws-marketplace-step2b-modify-security-group2.png

    Note

    You only need to change these security groups once.

  4. Select a group and then click the Inbound tab, and click Edit.

    aws-marketplace-step2b-modify-security-group3.png

  5. Click Add Rule.

    aws-marketplace-step2b-modify-security-group4.png

  6. Add three rules as follows and then Save (as shown in the screenshot below):

    • Type = Custom TCP, Protocol = TCP, Port Range = 8020, Source = security-group-ID-of-Unravel-Server’s-security-group

    • Type = Custom TCP, Protocol = TCP, Port Range = 50010, Source = security-group-ID-of-Unravel-Server’s-security-group

    • Type = Custom TCP, Protocol = TCP, Port Range = 50020, Source = security-group-ID-of-Unravel-Server’s-security-group

    aws-marketplace-step2b-modify-security-group5.png

  7. Make the above changes to both the security groups corresponding to the EMR cluster.

If you are connecting to an existing EMR cluster or for more advanced options, see Connecting Unravel to a new or existing EMR cluster.

Step 3: Check out the Unravel UI with your EMR cluster connected

After creating the cluster, go to Unravel UI (http://hostname-or-ip-address-of-the-instance:3000). Within a few seconds, you will see the cluster ID on the main dashboard dropdown on the right and some data in various charts.

aws-_marketplace-_step3-_unravel-ui1.png

You can now run applications in your cluster and use Unravel to optimize them. Applications you run show up in the Applications tab.

aws-_marketplace-_step3-_unravel-ui.png
Next Steps

Try the full Unravel product free for 30 days. To get help with setting up or using Unravel for AWS EMR, contact Unravel AWS Marketplace help.

In this section: