Home

Deploying Unravel from the AWS Marketplace
Overview

This page describes how to launch an Unravel Server from the AWS Marketplace and connect EMR clusters to it. Use these instructions for test or development environments only; for production environments, see Amazon Elastic MapReduce (EMR).

Unravel for Amazon EMR is the only solution to provide full-stack monitoring, tuning, troubleshooting and resource optimization for big data running on Amazon EMR. Unravel goes beyond passive monitoring to highly automated and intelligent management and optimization of data pipelines and applications.

This topic helps you set up the below configuration.

emr-highlevel-diagram-part2.png

We take you through the following steps, to get Unravel for Amazon EMR up and running via the AWS Marketplace.

emr-part2-flowchart.png
Background

A few things to note before getting started...

  • For Unravel Server to be able to monitor your EMR cluster(s), you need to:

    • Allow Unravel Server to access EMR cluster(s) nodes' TCP ports 8020, 50010, 50020.

    • Open Unravel Server's port 4043 for receiving traffic from the EMR cluster(s).

  • To access Unravel UI, you need to open port 3000 on Unravel Server for HTTP access.

No action is needed right now. We'll walk you through these actions later.

Step 0: Meet the network prerequisites

You can choose your own VPC/subnet settings/security groups as long as you can achieve the requirements mentioned in the Backr section above. In this case, you would also have appropriate VPN connections or NAT Gateways available for this VPC do you can connect to the Unravel Server via the internal/external IP address as applicable.

OR

You can create a new one to quickly check out Unravel for EMR as described below. For ease of use, we are creating a VPC with a Single Public Subnet.

  1. Log into the AWS VPC Dashboard.

  2. Click Launch VPC Wizard.

    aws-marketplace-step0-launch-vpc-wizard.png

  3. Select the first option, Select a VPC Configuration.

    aws-marketplace-step0-select-vpc-config.png

  4. Add the names for the VPC and subnet and click Create VPC to create the VPC and related subcomponents.

    aws-marketplace-step0-vpc-single-subnet.png

  5. On the AWS VPC Dashboard, click Subnets on the left-hand menu and select the subnet just created. Click the Actions | Modify auto-assign IP settings.

    aws-marketplace-step0-modify-auto-assign-ip1.png

  6. Select the Enable auto-assign public IPv4 address checkbox, and click Save.

    aws-marketplace-step0-modify-auto-assign-ip2.png
Step 1: Launch and set up Unravel Server
Step 1 (a): Find Unravel for Amazon EMR from the Marketplace

  1. Search for Unravel for Amazon EMR in the AWS Marketplace.

    aws-marketplace-step1a-find-app.png

  2. On the listing page, Click Continue to Subscribe.

  3. On the subscription page, click Continue to Configuration.

  4. Choose a version and location:

    • On the Configure this software page, select the latest software version and the same region where you plan to create EMR clusters (Unravel Server can reside in a different region, but then you'll need to set up VPC peering.

      aws-marketplace-step1a-config-app.png

    • Click Continue to Launch to continue with the setup.

      Note that there are several steps to be done next before the instance is actually launched.

  5. Launch the software:

    On the Launch this software page, select Launch from Website from the dropdown and then click Launch.

    aws-marketplace-step1a-launch-app.png

Note

Note that some more steps remain before the instance is actually launched.

Step 1 (b): Choose an instance type

The instance type you choose depends on the number of applications to be monitored on a daily basis and other factors. You can leave the default value as is.

aws-marketplace-step1b-choose-instance.png
Step 1 (c): Configure VPC and subnet settings

Choose the VPC and Subnet that you created or a suitable one as discussed in the Background section above.

aws-marketplace-step1c-vpc.png
Step 1 (d): Configure security group settings

  1. Click Create New Based On Seller Settings.

    aws-marketplace-step1d-security-group1.png

  2. Name the security group anything you like and make a note of its name.

    Note: Unravel recommends limiting access to only known IP addresses.

    For example, Unravel for Amazon EMR-1. Enter any description you like.

    aws-marketplace-step1d-security-group2.png

  3. Click Save.

Note

For enhanced security, you can change the rules above based on your setup and Unravel's requirements.Architecture

Step 1 (e): Configure key pair settings

Choose an existing key pair that connects to this instance or create a new one, and then click Launch.

aws-marketplace-step1e-keypair.png
Step 1 (f): Log into Unravel UI

Once the AMI has instantiated and Unravel is installed, which can take 10-15 minutes, log into Unravel UI.

  1. Get the hostname or IP of the instance from the AWS Console:

    aws-marketplace-step1f-get-hostname.png

  2. Navigate to http://hostname-or-ip-address-of-the-instance:3000 with a web browser.

  3. Log in with username admin and admin password. You must SSH into the instance to get the admin password.

    emr-marketplace-ssh-password.png
    dashboard-demo-aws-marketplace

Next, let’s connect a new EMR cluster to Unravel Server.

Step 2: Connect a new EMR cluster to Unravel Server
Step 2 (a): Create a new EMR cluster and connect Unravel

  1. Go to AWS EMR Dashboard and click Create Cluster.

    aws-marketplace-step2a-create-cluster.png

  2. Click Go to advanced options.

    aws-marketplace-step2a-create-cluster-advanced-options.png

  3. Select the release and the services you want to install.

    aws-marketplace-step2a-create-cluster-select-sw.png

  4. Click Next.

  5. Select the same VPC and subnet as the one chosen for Unravel Server (Unravel Server can reside in a different VPC, but then you would need to set up VPC peering).

    aws-marketplace-step2a-create-cluster-select-hw.png

  6. Click Next.

  7. Add a bootstrap action.

    • In this section, go the Bootstrap Actions section, select Custom Actions, and click Configure and Add.

      aws-marketplace-step2a-create-cluster-add-bootstrap1.png

      That brings up a dialog box like this:

      aws-marketplace-step2a-create-cluster-add-bootstrap2.png

  8. In the Script location text box, enter s3://unraveldatarepo/unravel_emr_bootstrap.py

  9. In the Optional Arguments text box, do the following:

    • Make a note of the private IP from your Unravel instance:

      aws-marketplace-step2a-create-cluster-add-bootstrap3.png

    • Add --unravel-server unravel-ec2-private-ip-address --bootstrap

      aws-marketplace-step2a-create-cluster-add-bootstrap4.png

  10. Click Save in the dialog box, and then click Next.

  11. In the next screen, as shown below, select a key pair to be able to connect to the EC2 nodes and also make a note of the names of the security groups encircled below, as you modify them.

    aws-marketplace-step2a-create-cluster-add-bootstrap5.png

  12. Click Create Cluster.

Step 2 (b): Modify EMR cluster security groups so that Unravel Server has adequate access

  1. To find the security group ID of the security group named Unravel for Amazon EMR-1, navigate to Security Groups from the left panel in your AWS console and search for Unravel for Amazon EMR-1.

    aws-marketplace-step2b-modify-security-group1.png

  2. Make a note of the group ID.

    For example, in the screenshot above, the group ID is sg-0564b1b8902ecf611.

  3. From the EMR cluster’s screen (where it shows its status in the creation process), navigate to each of the two security groups highlighted in the screenshot below.

    aws-marketplace-step2b-modify-security-group2.png

    Note

    You only need to change these security groups once.

  4. Click the group, select one, click the Inbound tab, and click Edit.

    aws-marketplace-step2b-modify-security-group3.png

  5. Click Add Rule.

    aws-marketplace-step2b-modify-security-group4.png

  6. Add three rules as follows and then Save (as shown in the screenshot below):

    • Type = Custom TCP, Protocol = TCP, Port Range = 8020, Source = security-group-ID-of-Unravel-Server’s-security-group

    • Type = Custom TCP, Protocol = TCP, Port Range = 50010, Source = security-group-ID-of-Unravel-Server’s-security-group

    • Type = Custom TCP, Protocol = TCP, Port Range = 50020, Source = security-group-ID-of-Unravel-Server’s-security-group

    aws-marketplace-step2b-modify-security-group5.png

  7. Make the above changes to both the security groups corresponding to the EMR cluster.

See ??? If you are connecting to an existing EMR cluster or for more advanced options

Step 3: Check out the Unravel UI with your EMR cluster connected

Once the cluster is created, and you go back to the Unravel UI (http://hostname-or-ip-address-of-the-instance:3000), within a few seconds, you should see the cluster ID in the main dashboard dropdown on the right and some data in various charts.

ops-demo-aws-marketplace.png

You can now run applications in your cluster and use Unravel to optimize them. Applications you run show up in the Applications tab.

apps-demo-aws-marketplace.png
Next Steps

  • See Unravel product documentation to learn more about how to use Unravel.

  • Run Unravel Spark benchmark scripts (specifically Benchmarks 2.0.x) to see the value for Unravel Insights and Recommendations.Running verification scripts and benchmarks

Try the full Unravel product free for 30 days. To get help for setting up or using Unravel for AWS EMR please contact us as at Unravel AWS Markertpcee help.

In this section: