Home

[INTERNAL] Provisioning an EC2 instance from our CloudFormation template

This topic explains how to provision an EC2 instance with Unravel Server preinstalled on it, using our CloudFormation template.

Requirements checklist

Platform compatibility

EC2 instance type

Security group / IAM role

AWS permissions

AWS EMR 5.17, 5.18, 5.19

Minimum: R4.2xlarge (61 GiB RAM)

Maximum: R4.8xlarge (244 GiB RAM)

Recommended: R4.4xlarge (122 GiB RAM)

Allowed ports for inbound access to Unravel EC2 node:

Port 3000

Port 4043

Unravel EC2 node can access EMR cluster all ports

Unravel EC2 node has Read permission on the S3 bucket by EMR clusters.

Allowed port for inbound access to EMR master node:

Port 8020 (Name node access for spark event log)

Allowed port for inbound access to EMR core node:

Port 50010 (Data node access for spark event log)

Port 50020

AmazonEC2FullAccess
IAMFullAccess

Note

You need to create an IAM role that has S3 read, write, and list bucket permission for the specific S3 bucket that the EMR cluster will use for logging.

AmazonS3FullAccess
AmazonVPCFullAccess
AmazonSNSReadOnlyAccess
AWSMarketplaceManageSubscriptions
AWSMarketplaceListBuilds
AWSMarketplaceStartBuild
AWSCloudFormation: Permissions set as follows:
{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Effect": "Allow",
           "Action": [
               "cloudformation:*"
           ],
           "Resource": "*"
       }
   ]
}
EC2 instance settings

Name

Default value

Description

Stack name

The deployed application together with its virtual resources is called a CloudFormation "stack". We recommend you name this stack UnravelEC2number or UnravelEC2date.

InstanceType

r4.4xlarge

Instance type for the Unravel EC2 instance.

Supported values: r4.2xlarge,r4.4xlarge,r4.8xlarge

KeyName

mysshkey

EC2 key name for SSH access to the Unravel EC2 instance.

This name can include uppercase letters, lowercase letters, numbers, dashes, and underscores only and must be 1-64 characters long.

TrustedSshIPBlock

10.10.0.0/16

Trusted IP block for SSH, port 3000, and port 4043 access to the Unravel EC2 instance, in CIDR format (x.x.x.x/x).

UnravelInstanceCount

1

Leave this as 1.

UnravelVPCBlock

10.10.0.0/16

Internal subnet (VPC) IP block for the Unravel EC2 instance, in CIDR format (x.x.x.x/x).

Zone

us-east-1a

Availability zone for the Unravel EC2 instance.

The zone must match any existing EMR clusters which this Unravel EC2 instance will be monitoring.

Create an EC2 instance from our CloudFormation template
  1. Download our CloudFormation template from https://s3.amazonaws.com/unraveldatarepo/unravel_aws_marketspace_01.json to your local /tmp folder.

  2. Use AWS CLI or the AWS Marketplace console to create the Unravel EC2 instance:

    From AWS CLI, run the aws cloudformation command with these parameters.

    Syntax:

     aws cloudformation create-stack --stack-name UnravelEC2 \ 
    --template-body file:///tmp/unravel_aws_marketspace_01.json \ 
    --parameters ParameterKey=Zone,ParameterValue=AWS_ZONE_NAME \ 
    ParameterKey=InstanceType,ParameterValue=AWS_INSTANCE_TYPE \ 
    ParameterKey=KeyName,ParameterValue=my_ssh_key \ 
    ParameterKey=UnravelVPCBlock,ParameterValue=AWS_VPC_BLOCK \ 
    ParameterKey=TrustedSshIPBlock, ParameterValue=AWS_SSH_BLOCK \ 
    --capabilities CAPABILITY_IAM | tee UnravelEC2-stack.json 

    Example:

     aws cloudformation create-stack --stack-name UnravelEC2number \ 
    --template-body file:///tmp/unravel_aws_marketspace_01.json \ 
    --parameters ParameterKey=Zone,ParameterValue=us-east-1d \ 
    ParameterKey=InstanceType,ParameterValue=r4.2xlarge \ 
    ParameterKey=KeyName,ParameterValue=topcat \ 
    ParameterKey=UnravelVPCBlock,ParameterValue=10.12.0.0/16 \ 
    ParameterKey=TrustedSshIPBlock, ParameterValue=0.0.0.0/0 \ 
    --capabilities CAPABILITY_IAM | tee UnravelEC2-stack.json 

    From AWS Marketplace (https://aws.amazon.com/marketplace):

    1. Type unravel in the search box:

      aws-marketplace-5
    2. In the search results, select Unravel APM for EMR and click Continue to Subscribe.

      aws-marketplace-6
    3. Click Continue to Configuration.

    4. On the Configure this software page, select the following:

      • Fulfillment Option: Unravel EC2 instance

      • Software Version: 4.5.0.5

      • Region: Select the region that matches your existing Amazon EMR.

    5. Click Continue to Launch.

      aws-marketplace-1
    6. On the Launch this software page, click Usage Instructions to download the documentation for the Unravel EC2 instance.

      aws-marketplace-2
    7. In the Choose Action pull-down menu, select Launch CloudFormation.

      aws-marketplace-3
    8. Click Launch.

      aws-marketplace-4

      You are now on the CloudFormation portal (https://console.aws.amazon.com/cloudformation/home)

    9. In the Choose a template section, select Specify an Amazon S3 template URL, leave the default URL as is, and click Next.

      aws-ec2-create-3

      You are now on the Specify Details page on the CloudFormation portal.

    10. On the Specify Details page, specify your settings for this EC2 instance, and click Next.

    11. On the Options page, leave all settings as default and click Next.

    12. On the Review page, accept the notice about IAM resources and click Create.

      aws-cloudformation-create-stack

      The deployed application together with its virtual resources is called a CloudFormation "stack" and is named UnravelEC2number.

      Tip

      If you don't see your new stack in the list, wait 30-60 seconds and refresh the page.

  3. When the status of the stack changes to CREATE_COMPLETE, select it and expand its Outputs section: From the AWS CloudFormation portal, click CloudFormation ? UnravelEC2number ? Outputs. Confirm that you see four keys listed in this section:

    • UnravelBackupS3: The S3 bucket used by the Unravel EC2 node for backups

    • UnravelAutoScalingGroupId: The Unravel EC2 node's instance ID

    • VPC: The virtual public cloud (VPC) created for this Unravel EC2 node

    • SecurityGroup: The security group created for this Unravel EC2 node

    For example:

    aws-cloudformation-stacks
  4. Get the public IP address and private IP address of this EC2 instance.

    You can do this through aws CLI or through the EC2 console, (https://console.aws.amazon.com/ec2/).

    To use aws CLI, run the ec2 describe-instances command using the value of the UnravelAutoScalingGroupId key shown in the Outputs tab/section above.

     AUTOSCALING_GROUP=UnravelAutoScalingGroupId_value_from_Outputs_tab 
     aws ec2 describe-instances --filters "Name=tag:aws:autoscaling:groupName,Values=$AUTOSCALING_GROUP" |grep PublicIpAddress |head -1 
     aws ec2 describe-instances --filters "Name=tag:aws:autoscaling:groupName,Values=$AUTOSCALING_GROUP" |grep PrivateIpAddress |head -1 
  5. To use the EC2 console, highlight this EC2 instance and look at its IP addresses in its Description tab.

    For example:

    aws-ec2-running-details
Log into Unravel UI

Using a web browser, navigate to http://unravel-ec2-ip:3000 and log in with username admin with password unraveldata.

Note

For the free trial version, use the Chrome browser.

signin.png

Congratulations! Unravel Server is up and running. Proceed to connect to your existing or new EMR cluster .