Sunday, January 4, 2015

Experimenting with AWS EC2 Container Service

Amazon EC2 Container Service ("ecs" for short) is a Docker cluster management service that runs on top of EC2 instances.  There is no additional charge for the service - you pay for the EC2 instances whether you're using them or not.  It's early days but looks like a promising service that should take a lot of the grunt work (networking, security, etc) out of creating your own clusters like Kubernetes and Mesos.

ECS is currently in preview - I needed to wait around two weeks to be granted access after signing up here: 
https://aws.amazon.com/ecs/

This is a transcript of how I fired up a simple Docker container on ECS using Amazon instructions available on 4/Jan/2015.

Introduction

Watch this video: https://www.youtube.com/watch?v=LE5uBqNp2Ds&t=1m58s
Particularly from 1m58s to skip the Amazon propaganda and watch the interesting visualisation.

And this video has some good terminology introduction and a live demo: https://www.youtube.com/watch?v=2vJLS8qfhI0&feature=youtu.be&t=11m12s  (Slides)

Terminology:

  • Tasks: A grouping of related containers (e.g. Nginx, Rails app, MySql, Log collector)
  • Containers
  • Clusters: A grouping of container instances - a pool of resources for Tasks
  • Container Instances: An EC2 instance on which Tasks are scheduled. AMI with ecs agent installed

Setting Up

Follow these instructions:
http://docs.aws.amazon.com/AmazonECS/latest/developerguide/get-set-up-for-amazon-ecs.html

This will walk you through the setup for the following:

  • IAM User
  • IAM Role
  • Key Pair
  • VPC
  • Security Group
  • Special copy of AWS CLI that includes "ecs" commands
    • NOTE: On OS X I needed to:
      • "brew uninstall awscli" (that removed /usr/local/bin/aws from my path)
      • And add "export PATH=$PATH:~/.local/lib/aws/bin" to my .bashrc


Creating The Cluster

Follow these instructions:  http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_GetStarted.html

NOTE: I preferred to create the EC2 instance from the command line (instead of the Launch an Instance with the Amazon ECS AMI instructions): 
aws ec2 run-instances --image-id ami-34ddbe5c --count 1 --instance-type t2.small --subnet-id subnet-xxxxxxxx --key-name ecsdemo-keypair --iam-instance-profile Name=ecsdemo-role

... using the subnet-id for my default VPC and the "ecsdemo" keypair and IAM role name I created during the Setting Up phase above.


Then as per the instructions test it out with:

aws ecs list-container-instances

If you see
{
    "containerInstanceArns": []
}
... then something has gone wrong and you'll need to terminate your instance and try again.

To see more details about your instance:

aws ecs describe-container-instances


Running a Task (Docker process)

As per the instructions, register a Task Definition and start a Task that spins up a single docker container (based on busybox image) that simply sleeps for 6 minutes.

aws ecs register-task-definition --family sleep360 --container-definitions "[{\"environment\":[],\"name\":\"sleep\",\"image\":\"busybox\",\"cpu\":10,\"portMappings\":[],\"entryPoint\":[\"/bin/sh\"],\"memory\":10,\"command\":[\"sleep\",\"360\"],\"essential\":true}]"

aws ecs list-task-definitions
aws ecs run-task --cluster default --task-definition sleep360:1 --count 1
aws ecs list-tasks
aws ecs describe-tasks --tasks 699d5420-1d0d-410e-b105-7e51027b8fd4

Log on to your instance and check the docker container is running:

ssh -i ecsdemo-keypair.pem ec2-user@ec2-instance-public-ip

docker ps
Should see:

[ec2-user@ip-ec2-instance-public-ip-name ~]$ docker ps
CONTAINER ID        IMAGE                            COMMAND             CREATED             STATUS              PORTS                        NAMES
ec8a9fca64b0        busybox:buildroot-2014.02        "sleep 360"         3 minutes ago       Up 3 minutes                                     ecs-sleep360-1-sleep-dc8dd4cdfcf593d07d00
58e68cc5bfc3        amazon/amazon-ecs-agent:latest   "/agent"            34 minutes ago      Up 34 minutes       127.0.0.1:51678->51678/tcp   ecs-agent

See more details about your docker container with:

docker inspect ec8a9fca64b0

After 6 minutes of sleeping, the docker process should disappear from the "docker ps" listing.

More Examples

More interesting examples including Tasks that link together a number of containers are contained in the videos linked to above.