Skip to main content

Home

Setting up VPC peering (optional)

Note

Follow these steps only if you have a cluster located in a different VPC than the Unravel EC2 instance.

This topic explains how to resolve connectivity issues when you have an Amazon EMR cluster located in a different VPC than your Unravel EC2 instance or your Unravel EC2 instance connects to an RDS instance created on a different VPC in the same region.

Assumptions
  • The VPC where Unravel EC2 is located is in the same region as the EMR cluster.

  • The subnet used by Unravel EC2 does not overlap the IP block range of the subnet used in the EMR cluster.

  • Network ACLs on both the VPC for Unravel EC2 and the EMR cluster is the default and allow all traffic. The security group is the only security enforcement on network access.

The examples below show both the Unravel EC2 instance and the EMR cluster in the us-east-1 region but configured with different VPC and subnet. There is no network access allowed between Unravel EC2 and EMR cluster by default.

Resources

Internal IP address

Subnet ID

Subnet IP block

VPC ID (name)

IP block in VPC

Security group ID (name)

Unravel EC2 node

10.10.0.7

subnet-03b82c56b2c26dbd1

10.10.0.0/24

vpc-0b0e17b01c4a3b54a (Unravel_VPC)

10.10.0.0/16

sg-0e0a03084398287c9 (Unravel-EC2_SG)

EMR cluster master node

10.11.0.53

subnet-0294cc17a42a9acfd

10.11.0.0/24

vpc-c3d079a4 (VPC_for_VPC Peering)

10.11.0.0/16

sg-0a73c3aea9340ae49 (EMR_VPC_SG)

EMR cluster core nodes

10.11.0.76

10.11.0.130

subnet-0294cc17a42a9acfd

10.11.0.0/24

vpc-c3d079a4 (VPC_for_VPC Peering)

10.11.0.0/16

sg-0a73c3aea9340ae49 (EMR_VPC_SG)

1. Create VPC peering in VPC dashboard
  1. From the AWS console | VPC | Peering Connections, click Create Peering Connection.

  2. Enter the name tag.

    For example, EMR_to_Unravel.

  3. In the VPC (Requester) field, select the VPC of the EMR cluster.

  4. In the VPC (Accepter) field, select the VPC of Unravel Server.

  5. Click Create Peering Connection.

    aws-emr-peer-connect.png

    A success message should appear in the screen.

  6. Click OK.

    aws-emr-create-peer-connect.png
2. Accept the VPC peering request

In the VPC Dashboard, the new VPC peering connection has the status Pending Acceptance.

  1. Select this connection, click Action, and select Accept Request.

    aws-emr-create-peer-accept.png
  2. Click Yes Accept in the prompt screen. You will see a message regarding "Modify my route tables". Click Close.

    aws-emr-peer-accepted.png
3. Create routes between peered VPC

To create the routes between peered VPCs (Unravel Server on Unravel_VPC and the EMR cluster on Test_EMR_VPC):

  1. Go to VPC Dashboard | Route Tables.

  2. Use the VPC name to locate the route tables for the two peered VPCs and then for each route table add another route.

    Repeat Step 3 to Step 13 for each route table. Refer to Updating your Route tables for a VPC peering connection for more details.

  3. Click Edit | Add another route.

  4. Find the Unravel_VPC route table.

  5. In the Destination field, enter the IP block of the EMR VPC.

    For example, 10.11.0.0/16

    aws-emr-routes.png
  6. In the Target field, select or enter the VPC peer connection ID.

    For example, pcx-0a57a978ef9a525e2.

  7. Click Save .

  8. Find the Test_EMR_VPC route table.

  9. Set the Destination to the IP block of Unravel_VPC.

    For example, 10.10.0.0/16.

  10. In the Target field, select the VPC peer connection ID.

    For example, pcx-0a57a978ef9a525e2.

  11. Click Save.

  12. In the Target field, select the connection ID.

    For example, pcx-0a57a978ef9a525e2.

  13. Click Save.

4. Update security groups
  1. Go to VPC Dashboard | Security Group.

  2. Click Add another rule.

    Repeat step-2 to step-7 for each of the security group. Refer to Updating your security groups to reference peer VPC groups for more details.

  3. Set Type to inbound ALL traffic and Protocol to ALL.

  4. Locate the security group used on Unravel EC2 node. Enter the EMR VPC IP block in the Source field.

    For example, 10.11.0.0/16.

    aws-emr-inbound-saved.png
  5. Click Save.

  6. Locate the security group used on EMR cluster node and enter the Unravel VPC IP block.

    For example, 10.10.0.0/16.

  7. Click Save.

5. Verify the connection between Unravel and the EMR master node
  1. Open SSH sessions to both Unravel EC2 node and EMR master node.

    Since the above example allows all traffic from both VPC IP blocks, you should be able to ping the IP address of EMR master node from Unravel Server.

  2. On the Unravel EC2 instance, open a telnet session to the EMR master node port 8082 (the namenode port).

  3. On the EMR master node, open telnet sessions to the Unravel EC2 instance, ports 3000 and 4043.

    aws-emr-cli-telnet-closed.png

If telnet port tests are positive, the VPC peering connection is set up correctly. If not, troubleshoot the configuration on network ACL, security groups, and route tables used on both VPCs.