Testing and Development on AWS

Testing and Development on AWS

Joe Ziegler: zieglerj@amazon.com @jiyosub

Attended this Lunch & Learn session on 20/Sept/2012.
Slides are available here: http://www.slideshare.net/AmazonWebServices/development-and-test-on-aws
Similar 3hr workshop presentation here: http://www.slideshare.net/AmazonWebServices/agile-in-the-coud


 

·         Session is about “development on AWS” (not for)

·         VPC (Virtual Private Cloud)

·         Create subnets/dns/routing tables

·         Create mirror of Prod topology (same IPs) for developers

·         Ipsec tunnel VPN connection

·         Cloud Formation

·         Json template

·         Reusable app stack

·         Declarative

·         Resources in dependency-driven order

·         Console support

·         Predefined templates

·         “cloudformer” – uses CF to create CF

·         Stateful, provision and remove

·         Everything is exposed via API – webconsole is built on public API (bit behind)

·         Ruby, .net, php, iOS, android, java, glacier, python (interactive, save as script)

·         Source control

·         Running source in AWS

·         Secure, accessible, scale vertically, durable, reusable

·         Bitbucket: AWS, Australia, Python

·         S3: 99.999999999% (11 9s) availability

·         Self-managed   

·         Self-installed EC2 instance, use community AMIs, AWS Marketplace

·         Asdf

·         Adsf

·         Dev env via CloudFormation

·         VPC – mirror your network topologies

·         Template related resources

·         Integration with CM tools

·         Provide CF templates internally to developrs

·         RDS example

·         VPC example

·         Replicating Prod envs in dev

·         Why

·         Accurate perf testing

·         Empower devs to experiment

·         Prod debugging

·         Improved code quality

·         How

·         Adopt infrastructure as code strategy – prod is automated/repeatable/src control/test

·         Leverage AWS APIs – automatically provision

·         Utilise RDS and point in time snapshots

·         Devs can grab prod snapshot of RDS

·         TBs own hosted data?

·         Create snapshot in S3 and do periodical updates

·         Storage gateway – iSCSI on site, Disaster Recovery, pilot light, failover

·         Test

·         Unit, Smoke, UAT, Integration, Load/Perf, Blue/Green (A/B)

·         Test quadrant

·         Testing in the Cloud priorities:

·         Perf

·         Functional – env matches prod closely

·         Integration – temlating/mocking

·         Unit - VDI sucks for dev – latency

·         Approach

·         Template everything, snapshot image of AMI

·         AMI/CF template matching production

·         Leverage CI server pipeline

·         Automate/repeatable using AWS APIs, source control

·         Load & Perf test

·         Bees with machine guns

·         Bees master, fabric, loads 10 instances (bees)

·         Singapore isntances attacks Japan instances

·         1000 instances for 1hr $= 1 instance for 1000 hrs

·         Blue/Green Testing

·         App ->> blue / green autoscaling group  <<- CloudWatch

·         Auto Scale up/down parts of game being used

·         UAT

·         Quick deployments

·         Secure isolated environment

·         Utilise AWS elastic beanstalk

·         Benefit from elasticity

·         Agile Theory: Continuous Development, Intg & Deployment

·         Optimise for rapid response to change, High tech discipline, Ruthless automation

·         Template everything, repeatable without human

·         Pinterest 12 $1.5bil

·         Concepts: CI, Infra as Dode, CD

·         CI

·         Devs can only integrate with VCS – no way of getting into prod

·         Create a brand new staging env

·         Create full copy of prod env, incl load balancers, route53, point to new prod env, take down old prod env

·         Creating entire prod env (AMIs) every time, completely tested/automated

·         OS tweaker is one of the devs

·         No ops/devops team

·         Roll out code based on customer’s needs

·         Infra as code

·         “Programmatic provisioning by API” – whatever language

·         Toolbox: AMI, puppet/chef, CF, libraries/SDKs

·         Puppet

·         1. Declare config

·         2. Apply config

·         3. (time passes)

·         4. Verify config

·         5. Re-apply config

·         6. If needed

·         Goal: manual -> scripted -> programmatic: Increase Speed/Repeatability/Reliability, Decrese:Risk

·         Testing approach:

·         CI creates AMI

·         Puppet in bucket

·         Put state in SimpleDB, others can query state of machines

·         <Activities>

·         Bootstrapping

·         CI every night creates new AMI image baseline

·         When new image wakes up, gets puppet script from bucket, runs, applies config delta

·         Config

·         Launch

·         CD

·         Prereqs

·         Confidence à codebase

·         Confidence à environment

·         Small batch sizes

·         Ruthless automation

·         Create whole new version of env if it passes through pipeline

·         <CI Pipeline image from CD book>

·         Jenkins

·         Polling infrastructure

·         Polling unit tests

·         Autoscales executors

·         Jenkins EC2 plugin, AMI key, installs agent, Jenkins algorithm scales

·         Promotions, S3 building block

·         Promoted builds plugin

·         Flicker, blog

·         Next steps: Account Mgr, Solution Architects, Webinars and Pod Casts, SlideShare (“Agile in the clourd” 3hr workshop)

Autoscaling of 1 – make sure one is always up

Autoscaling of 2 – 1 in each AZ

Asgard from Netflix

CloudyScripts

Resource Centre: aws.amazon.com/apac/arc-anz

Free training coming up – 2day hands-on lab training