Testing and Development on AWS

Testing and Development on AWS

Joe Ziegler: zieglerj@amazon.com @jiyosub

Attended this Lunch & Learn session on 20/Sept/2012.
Slides are available here: http://www.slideshare.net/AmazonWebServices/development-and-test-on-aws
Similar 3hr workshop presentation here: http://www.slideshare.net/AmazonWebServices/agile-in-the-coud


 

·         Session is about “development on AWS” (not for)

·         VPC (Virtual Private Cloud)

·         Create subnets/dns/routing tables

·         Create mirror of Prod topology (same IPs) for developers

·         Ipsec tunnel VPN connection

·         Cloud Formation

·         Json template

·         Reusable app stack

·         Declarative

·         Resources in dependency-driven order

·         Console support

·         Predefined templates

·         “cloudformer” – uses CF to create CF

·         Stateful, provision and remove

·         Everything is exposed via API – webconsole is built on public API (bit behind)

·         Ruby, .net, php, iOS, android, java, glacier, python (interactive, save as script)

·         Source control

·         Running source in AWS

·         Secure, accessible, scale vertically, durable, reusable

·         Bitbucket: AWS, Australia, Python

·         S3: 99.999999999% (11 9s) availability

·         Self-managed   

·         Self-installed EC2 instance, use community AMIs, AWS Marketplace

·         Asdf

·         Adsf

·         Dev env via CloudFormation

·         VPC – mirror your network topologies

·         Template related resources

·         Integration with CM tools

·         Provide CF templates internally to developrs

·         RDS example

·         VPC example

·         Replicating Prod envs in dev

·         Why

·         Accurate perf testing

·         Empower devs to experiment

·         Prod debugging

·         Improved code quality

·         How

·         Adopt infrastructure as code strategy – prod is automated/repeatable/src control/test

·         Leverage AWS APIs – automatically provision

·         Utilise RDS and point in time snapshots

·         Devs can grab prod snapshot of RDS

·         TBs own hosted data?

·         Create snapshot in S3 and do periodical updates

·         Storage gateway – iSCSI on site, Disaster Recovery, pilot light, failover

·         Test

·         Unit, Smoke, UAT, Integration, Load/Perf, Blue/Green (A/B)

·         Test quadrant

·         Testing in the Cloud priorities:

·         Perf

·         Functional – env matches prod closely

·         Integration – temlating/mocking

·         Unit - VDI sucks for dev – latency

·         Approach

·         Template everything, snapshot image of AMI

·         AMI/CF template matching production

·         Leverage CI server pipeline

·         Automate/repeatable using AWS APIs, source control

·         Load & Perf test

·         Bees with machine guns

·         Bees master, fabric, loads 10 instances (bees)

·         Singapore isntances attacks Japan instances

·         1000 instances for 1hr $= 1 instance for 1000 hrs

·         Blue/Green Testing

·         App ->> blue / green autoscaling group  <<- CloudWatch

·         Auto Scale up/down parts of game being used

·         UAT

·         Quick deployments

·         Secure isolated environment

·         Utilise AWS elastic beanstalk

·         Benefit from elasticity

·         Agile Theory: Continuous Development, Intg & Deployment

·         Optimise for rapid response to change, High tech discipline, Ruthless automation

·         Template everything, repeatable without human

·         Pinterest 12 $1.5bil

·         Concepts: CI, Infra as Dode, CD

·         CI

·         Devs can only integrate with VCS – no way of getting into prod

·         Create a brand new staging env

·         Create full copy of prod env, incl load balancers, route53, point to new prod env, take down old prod env

·         Creating entire prod env (AMIs) every time, completely tested/automated

·         OS tweaker is one of the devs

·         No ops/devops team

·         Roll out code based on customer’s needs

·         Infra as code

·         “Programmatic provisioning by API” – whatever language

·         Toolbox: AMI, puppet/chef, CF, libraries/SDKs

·         Puppet

·         1. Declare config

·         2. Apply config

·         3. (time passes)

·         4. Verify config

·         5. Re-apply config

·         6. If needed

·         Goal: manual -> scripted -> programmatic: Increase Speed/Repeatability/Reliability, Decrese:Risk

·         Testing approach:

·         CI creates AMI

·         Puppet in bucket

·         Put state in SimpleDB, others can query state of machines

·         <Activities>

·         Bootstrapping

·         CI every night creates new AMI image baseline

·         When new image wakes up, gets puppet script from bucket, runs, applies config delta

·         Config

·         Launch

·         CD

·         Prereqs

·         Confidence à codebase

·         Confidence à environment

·         Small batch sizes

·         Ruthless automation

·         Create whole new version of env if it passes through pipeline

·         <CI Pipeline image from CD book>

·         Jenkins

·         Polling infrastructure

·         Polling unit tests

·         Autoscales executors

·         Jenkins EC2 plugin, AMI key, installs agent, Jenkins algorithm scales

·         Promotions, S3 building block

·         Promoted builds plugin

·         Flicker, blog

·         Next steps: Account Mgr, Solution Architects, Webinars and Pod Casts, SlideShare (“Agile in the clourd” 3hr workshop)

Autoscaling of 1 – make sure one is always up

Autoscaling of 2 – 1 in each AZ

Asgard from Netflix

CloudyScripts

Resource Centre: aws.amazon.com/apac/arc-anz

Free training coming up – 2day hands-on lab training

Wide Range of DevOps & Metrics, Metrics Everywhere

Mitchell Hasimoto: The Wide Range of DevOps
http://www.infoq.com/articles/wide-range-devops

Coda Hale: Metrics, Metrics Everywhere
http://www.youtube.com/watch?v=czes-oa0yik

Trying to fix overheating issues in Linux

Have had overheating issues with Dell Vostro 3350 when installing both Linux Mint 13 and Fedora 17  64bit.  Computer just shuts down after 30mins of usage.

Fedora 17

http://forums.fedoraforum.org/showthread.php?s=6be03c2ce5a1dbcee1ab448f5b9c4c09&t=272607&page=2

From Mongo to Riak at Shareaholic

Presentation: http://vimeo.com/49230504 

Slides: http://www.slideshare.net/freerobby/riak-at-shareaholic

Blog: http://blog.shareaholic.com/2012/08/migrating-to-riak-at-shareaholic/

7 Databases - Riak - Setting Up Cluster on EC2

Based on: http://blog.wakatta.jp/blog/2011/12/17/seven-databases-in-seven-weeks-riak-on-ec2/

Log into EC2 -> Instances -> Launch Instance -> Classic Wizard
64bin Amazon Linux
Instance Type: Micro
Defaults for Instance Details screens.
Create new Key Pair: "riak_7databases" (or similar)
Create new Sec Group: "riak-cluster":
    Open up ports: 4369, 8099, 8098, 8087, 6000-6999
Start the instance.
Create two more instances (exactly the same process) and add to the "riak-cluster" security group
Name the instances riak_ec2_1, riak_ec2_2, riak_ec2_3

ssh into each instance (Right-click the instance and Connect for hints) and do the following.

$ wget http://s3.amazonaws.com/downloads.basho.com/riak/CURRENT/rhel/6/riak-1.2.0-1.el6.x86_64.rpm

$ sudo rpm -Uvh riak-1.2.0-1.el6.x86_64.rpm

Safe to ignore following error:
chcon: can't apply partial context to unlabeled file `/usr/lib64/riak/lib/bitcask-1.5.1/priv/bitcask.so'
chcon...


# Get your <IP_ADDRESS>
$ ifconfig  
$ sudo -e /etc/riak/vm.args
# Change name to riak_ec2_1@<IP_ADDRESS>   # or riak_ec2_2/3, etc


$ sudo -e /etc/riak/app.config

# Insert following near the top:

{ kernel, [
            {inet_dist_listen_min, 6000},
            {inet_dist_listen_max, 6999}
            ]},

# Change "http" entry to:

{http, [ {"127.0.0.1", 8098 }, {"<IP_ADDRESS>", 8098} ]},

$ sudo riak start

# To test riak installation: 

$ curl http://localhost:8098/stats | python -mjson.tool

$ sudo riak-admin test
Attempting to restart script through sudo -H -u riak
Successfully completed 1 read/write cycle to 'riak_ec2_1@10.170.93.188'

# On 2nd and 3rd instances, join the cluster: 

$ sudo riak-admin cluster join riak_ec2_1@<IP_ADDRESS>
Attempting to restart script through sudo -H -u riak
Success: staged join request for 'riak_ec2_1@10.170.93.188' to 'riak_ec2_1@10.170.86.93' 

# On any instance: 

$ wget http://wiki.basho.com/attachments/goog.csv

$ wget http://wiki.basho.com/attachments/load_data

# Need Erlang so 'escript' works:

$ sudo yum install erlang

$ . ./load_data goog.csv

$ curl localhost:8098/riak/goog |python -mjson.tool
 - or browse to public address E.g.:
http://ec2-184-169-247-208.us-west-1.compute.amazonaws.com:8098/riak/goog

7 Databases - PostgreSQL

Vector Clocks explained

http://basho.com/blog/technical/2010/01/29/why-vector-clocks-are-easy/

Vector clocks