NoSQL: ACID vs BASE

BASE:
- Basically Available
- Soft-state
- Eventually Consistent

http://www.johndcook.com/blog/2009/07/06/brewer-cap-theorem-base/

Brewer's Keynote 2000: http://www.cs.berkeley.edu/~brewer/cs262b-2004/PODC-keynote.pdf

• ACID: we forfeit "Consistency" and "Isolated" for availability, graceful degradation, and performance.
• It's a spectrum

Rebasing git topic branches

https://github.com/dchelimsky/rspec/wiki/Topic-Branches

 # integrate the latest upstream changes into your "master"

 git checkout master

 git pull 

 # make sure that you're on your topic branch

 git checkout my_topic 

 # do the rebase

 git rebase master

A--B--C--D--E--F

       \

        X--Y--Z

to

 A--B--C--D--E--F

                 \

                  X'--Y'--Z'

Git pull vs fetch

git pull = git fetch + git merge
Great read here: http://longair.net/blog/2009/04/16/git-fetch-and-merge/
Linked from: http://stackoverflow.com/questions/292357/whats-the-difference-between-git-pull-and-git-fetch

Ask HN: How did you become a hardcore back-end developer? | Hacker News

http://news.ycombinator.com/item?id=2290180

Testing and Development on AWS

Testing and Development on AWS

Joe Ziegler: zieglerj@amazon.com @jiyosub

Attended this Lunch & Learn session on 20/Sept/2012.
Slides are available here: http://www.slideshare.net/AmazonWebServices/development-and-test-on-aws
Similar 3hr workshop presentation here: http://www.slideshare.net/AmazonWebServices/agile-in-the-coud


 

·         Session is about “development on AWS” (not for)

·         VPC (Virtual Private Cloud)

·         Create subnets/dns/routing tables

·         Create mirror of Prod topology (same IPs) for developers

·         Ipsec tunnel VPN connection

·         Cloud Formation

·         Json template

·         Reusable app stack

·         Declarative

·         Resources in dependency-driven order

·         Console support

·         Predefined templates

·         “cloudformer” – uses CF to create CF

·         Stateful, provision and remove

·         Everything is exposed via API – webconsole is built on public API (bit behind)

·         Ruby, .net, php, iOS, android, java, glacier, python (interactive, save as script)

·         Source control

·         Running source in AWS

·         Secure, accessible, scale vertically, durable, reusable

·         Bitbucket: AWS, Australia, Python

·         S3: 99.999999999% (11 9s) availability

·         Self-managed   

·         Self-installed EC2 instance, use community AMIs, AWS Marketplace

·         Asdf

·         Adsf

·         Dev env via CloudFormation

·         VPC – mirror your network topologies

·         Template related resources

·         Integration with CM tools

·         Provide CF templates internally to developrs

·         RDS example

·         VPC example

·         Replicating Prod envs in dev

·         Why

·         Accurate perf testing

·         Empower devs to experiment

·         Prod debugging

·         Improved code quality

·         How

·         Adopt infrastructure as code strategy – prod is automated/repeatable/src control/test

·         Leverage AWS APIs – automatically provision

·         Utilise RDS and point in time snapshots

·         Devs can grab prod snapshot of RDS

·         TBs own hosted data?

·         Create snapshot in S3 and do periodical updates

·         Storage gateway – iSCSI on site, Disaster Recovery, pilot light, failover

·         Test

·         Unit, Smoke, UAT, Integration, Load/Perf, Blue/Green (A/B)

·         Test quadrant

·         Testing in the Cloud priorities:

·         Perf

·         Functional – env matches prod closely

·         Integration – temlating/mocking

·         Unit - VDI sucks for dev – latency

·         Approach

·         Template everything, snapshot image of AMI

·         AMI/CF template matching production

·         Leverage CI server pipeline

·         Automate/repeatable using AWS APIs, source control

·         Load & Perf test

·         Bees with machine guns

·         Bees master, fabric, loads 10 instances (bees)

·         Singapore isntances attacks Japan instances

·         1000 instances for 1hr $= 1 instance for 1000 hrs

·         Blue/Green Testing

·         App ->> blue / green autoscaling group  <<- CloudWatch

·         Auto Scale up/down parts of game being used

·         UAT

·         Quick deployments

·         Secure isolated environment

·         Utilise AWS elastic beanstalk

·         Benefit from elasticity

·         Agile Theory: Continuous Development, Intg & Deployment

·         Optimise for rapid response to change, High tech discipline, Ruthless automation

·         Template everything, repeatable without human

·         Pinterest 12 $1.5bil

·         Concepts: CI, Infra as Dode, CD

·         CI

·         Devs can only integrate with VCS – no way of getting into prod

·         Create a brand new staging env

·         Create full copy of prod env, incl load balancers, route53, point to new prod env, take down old prod env

·         Creating entire prod env (AMIs) every time, completely tested/automated

·         OS tweaker is one of the devs

·         No ops/devops team

·         Roll out code based on customer’s needs

·         Infra as code

·         “Programmatic provisioning by API” – whatever language

·         Toolbox: AMI, puppet/chef, CF, libraries/SDKs

·         Puppet

·         1. Declare config

·         2. Apply config

·         3. (time passes)

·         4. Verify config

·         5. Re-apply config

·         6. If needed

·         Goal: manual -> scripted -> programmatic: Increase Speed/Repeatability/Reliability, Decrese:Risk

·         Testing approach:

·         CI creates AMI

·         Puppet in bucket

·         Put state in SimpleDB, others can query state of machines

·         <Activities>

·         Bootstrapping

·         CI every night creates new AMI image baseline

·         When new image wakes up, gets puppet script from bucket, runs, applies config delta

·         Config

·         Launch

·         CD

·         Prereqs

·         Confidence à codebase

·         Confidence à environment

·         Small batch sizes

·         Ruthless automation

·         Create whole new version of env if it passes through pipeline

·         <CI Pipeline image from CD book>

·         Jenkins

·         Polling infrastructure

·         Polling unit tests

·         Autoscales executors

·         Jenkins EC2 plugin, AMI key, installs agent, Jenkins algorithm scales

·         Promotions, S3 building block

·         Promoted builds plugin

·         Flicker, blog

·         Next steps: Account Mgr, Solution Architects, Webinars and Pod Casts, SlideShare (“Agile in the clourd” 3hr workshop)

Autoscaling of 1 – make sure one is always up

Autoscaling of 2 – 1 in each AZ

Asgard from Netflix

CloudyScripts

Resource Centre: aws.amazon.com/apac/arc-anz

Free training coming up – 2day hands-on lab training

Wide Range of DevOps & Metrics, Metrics Everywhere

Mitchell Hasimoto: The Wide Range of DevOps
http://www.infoq.com/articles/wide-range-devops

Coda Hale: Metrics, Metrics Everywhere
http://www.youtube.com/watch?v=czes-oa0yik

Trying to fix overheating issues in Linux

Have had overheating issues with Dell Vostro 3350 when installing both Linux Mint 13 and Fedora 17  64bit.  Computer just shuts down after 30mins of usage.

Fedora 17

http://forums.fedoraforum.org/showthread.php?s=6be03c2ce5a1dbcee1ab448f5b9c4c09&t=272607&page=2

From Mongo to Riak at Shareaholic

Presentation: http://vimeo.com/49230504 

Slides: http://www.slideshare.net/freerobby/riak-at-shareaholic

Blog: http://blog.shareaholic.com/2012/08/migrating-to-riak-at-shareaholic/

7 Databases - Riak - Setting Up Cluster on EC2

Based on: http://blog.wakatta.jp/blog/2011/12/17/seven-databases-in-seven-weeks-riak-on-ec2/

Log into EC2 -> Instances -> Launch Instance -> Classic Wizard
64bin Amazon Linux
Instance Type: Micro
Defaults for Instance Details screens.
Create new Key Pair: "riak_7databases" (or similar)
Create new Sec Group: "riak-cluster":
    Open up ports: 4369, 8099, 8098, 8087, 6000-6999
Start the instance.
Create two more instances (exactly the same process) and add to the "riak-cluster" security group
Name the instances riak_ec2_1, riak_ec2_2, riak_ec2_3

ssh into each instance (Right-click the instance and Connect for hints) and do the following.

$ wget http://s3.amazonaws.com/downloads.basho.com/riak/CURRENT/rhel/6/riak-1.2.0-1.el6.x86_64.rpm

$ sudo rpm -Uvh riak-1.2.0-1.el6.x86_64.rpm

Safe to ignore following error:
chcon: can't apply partial context to unlabeled file `/usr/lib64/riak/lib/bitcask-1.5.1/priv/bitcask.so'
chcon...


# Get your <IP_ADDRESS>
$ ifconfig  
$ sudo -e /etc/riak/vm.args
# Change name to riak_ec2_1@<IP_ADDRESS>   # or riak_ec2_2/3, etc


$ sudo -e /etc/riak/app.config

# Insert following near the top:

{ kernel, [
            {inet_dist_listen_min, 6000},
            {inet_dist_listen_max, 6999}
            ]},

# Change "http" entry to:

{http, [ {"127.0.0.1", 8098 }, {"<IP_ADDRESS>", 8098} ]},

$ sudo riak start

# To test riak installation: 

$ curl http://localhost:8098/stats | python -mjson.tool

$ sudo riak-admin test
Attempting to restart script through sudo -H -u riak
Successfully completed 1 read/write cycle to 'riak_ec2_1@10.170.93.188'

# On 2nd and 3rd instances, join the cluster: 

$ sudo riak-admin cluster join riak_ec2_1@<IP_ADDRESS>
Attempting to restart script through sudo -H -u riak
Success: staged join request for 'riak_ec2_1@10.170.93.188' to 'riak_ec2_1@10.170.86.93' 

# On any instance: 

$ wget http://wiki.basho.com/attachments/goog.csv

$ wget http://wiki.basho.com/attachments/load_data

# Need Erlang so 'escript' works:

$ sudo yum install erlang

$ . ./load_data goog.csv

$ curl localhost:8098/riak/goog |python -mjson.tool
 - or browse to public address E.g.:
http://ec2-184-169-247-208.us-west-1.compute.amazonaws.com:8098/riak/goog

7 Databases - PostgreSQL

Vector Clocks explained

http://basho.com/blog/technical/2010/01/29/why-vector-clocks-are-easy/

Vector clocks

Puppet Hiera

http://projects.puppetlabs.com/projects/hiera/

Heira

A simple pluggable Hierarchical Database.

Why? Hierarchical data is a good fit for the representation of infrastructure information. 

Linux tree command

http://www.centerkey.com/tree/

ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/  /' -e 's/-/|/'


tree.sh

#!/bin/sh
######################################################
#  UNIX TREE                                         #
#  Version: 2.3                                      #
#  File: ~/apps/tree/tree.sh                         #
#                                                    #
#  Displays Structure of Directory Hierarchy         #
#  ------------------------------------------------- #
#  This tiny script uses "ls", "grep", and "sed"     #
#  in a single command to show the nesting of        #
#  sub-directories.  The setup command for PATH      #
#  works with the Bash shell (the Mac OS X default). #
#                                                    #
#  Setup:                                            #
#    $ cd ~/apps/tree                                #
#    $ chmod u+x tree.sh                             #
#    $ ln -s ~/apps/tree/tree.sh ~/bin/tree          #
#    $ echo "PATH=~/bin:\${PATH}" >> ~/.profile      #
#                                                    #
#  Usage:                                            #
#    $ tree [directory]                              #
#                                                    #
#  Examples:                                         #
#    $ tree                                          #
#    $ tree /etc/opt                                 #
#    $ tree ..                                       #
#                                                    #
#  Public Domain Software -- Free to Use as You Like #
#  http://www.centerkey.com/tree  -  By Dem Pilafian #
######################################################

echo
if [ "$1" != "" ]  #if parameter exists, use as base folder
  then cd "$1"
  fi
pwd
ls -R | grep ":$" |  \
  sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/  /' -e 's/-/|/'
# 1st sed: remove colons
# 2nd sed: replace higher level folder names with dashes
# 3rd sed: indent graph three spaces
# 4th sed: replace first dash with a vertical bar
if [ `ls -F -1 | grep "/" | wc -l` = 0 ]  # check if no folders
  then echo "  -> no sub-directories"
  fi
echo
exit

7 Databases in 7 Weeks

Book: http://pragprog.com/book/rwdata/seven-databases-in-seven-weeks
Blog: http://sevenweeks.org/
Code: https://github.com/sevenweeks/databases
Song: http://sevenweeks.org/post/24891440010/lyrics-to-seven-databases-in-song

Contents


1. Introduction
1.1 It Starts with a Question
1.2 The Genres
1.3 Onward and Upward

2. PostgreSQL
2.1 That’s Post-greS-Q-L
2.2 Day 1: Relations, CRUD, and Joins
2.3 Day 2: Advanced Queries, Code, and Rules
2.4 Day 3: Full-Text and Multidimensions
2.5 Wrap-Up

3. Riak
3.1 Riak Loves the Web
3.2 Day 1: CRUD, Links, and MIMEs
3.3 Day 2: Mapreduce and Server Clusters
3.4 Day 3: Resolving Conflicts and Extending Riak
3.5 Wrap-Up
4. HBase
4.1 Introducing HBase
4.2 Day 1: CRUD and Table Administration
4.3 Day 2: Working with Big Data
4.4 Day 3: Taking It to the Cloud
4.5 Wrap-Up 5. MongoDB

5.1 Hu(mongo)us
5.2 Day 1: CRUD and Nesting
5.3 Day 2: Indexing, Grouping, Mapreduce
5.4 Day 3: Replica Sets, Sharding, GeoSpatial, and GridFS
5.5 Wrap-Up

6. CouchDB
6.1 Relaxing on the Couch
6.2 Day 1: CRUD, Futon, and cURL Redux
6.3 Day 2: Creating and Querying Views
6.4 Day 3: Advanced Views, Changes API, and Replicating
Data
6.5 Wrap-Up

7. Neo4J
7.1 Neo4J Is Whiteboard Friendly
7.2 Day 1: Graphs, Groovy, and CRUD
7.3 Day 2: REST, Indexes, and Algorithms
7.4 Day 3: Distributed High Availability
7.5 Wrap-Up

8. Redis
8.1 Data Structure Server Store
8.2 Day 1: CRUD and Datatypes
8.3 Day 2: Advanced Usage, Distribution
8.4 Day 3: Playing with Other Databases
8.5 Wrap-Up

9. Wrapping Up
9.1 Genres Redux
9.2 Making a Choice
9.3 Where Do We Go from Here

A1. Database Overview Tables
A2. The CAP Theorem
A2.1 Eventual Consistency
A2.2 CAP in the Wild
A2.3 The Latency Trade-Off

Vagrant Usage Patterns and Jenkins

Vagrant Usage Patterns

 

https://speakerdeck.com/u/mitchellh/p/vagrant-usage-patterns

 

 

Vagrant and Jenkins

 

https://wiki.jenkins-ci.org/display/JENKINS/Vagrant+Plugin

 

http://stackoverflow.com/questions/6941547/how-to-combine-vagrant-with-jenkins-for-the-perfect-continuous-integration-envir

 

http://codeslife.com/2011/10/21/make-ci-easier-with-jenkins-ci-and-vagrant/

 

http://unethicalblogger.com/2012/03/13/introducing-the-vagrant-plugin.html