7 Databases - Riak - Setting Up Cluster on EC2

Based on: http://blog.wakatta.jp/blog/2011/12/17/seven-databases-in-seven-weeks-riak-on-ec2/

Log into EC2 -> Instances -> Launch Instance -> Classic Wizard
64bin Amazon Linux
Instance Type: Micro
Defaults for Instance Details screens.
Create new Key Pair: "riak_7databases" (or similar)
Create new Sec Group: "riak-cluster":
    Open up ports: 4369, 8099, 8098, 8087, 6000-6999
Start the instance.
Create two more instances (exactly the same process) and add to the "riak-cluster" security group
Name the instances riak_ec2_1, riak_ec2_2, riak_ec2_3

ssh into each instance (Right-click the instance and Connect for hints) and do the following.

$ wget http://s3.amazonaws.com/downloads.basho.com/riak/CURRENT/rhel/6/riak-1.2.0-1.el6.x86_64.rpm

$ sudo rpm -Uvh riak-1.2.0-1.el6.x86_64.rpm

Safe to ignore following error:
chcon: can't apply partial context to unlabeled file `/usr/lib64/riak/lib/bitcask-1.5.1/priv/bitcask.so'
chcon...


# Get your <IP_ADDRESS>
$ ifconfig  
$ sudo -e /etc/riak/vm.args
# Change name to riak_ec2_1@<IP_ADDRESS>   # or riak_ec2_2/3, etc


$ sudo -e /etc/riak/app.config

# Insert following near the top:

{ kernel, [
            {inet_dist_listen_min, 6000},
            {inet_dist_listen_max, 6999}
            ]},

# Change "http" entry to:

{http, [ {"127.0.0.1", 8098 }, {"<IP_ADDRESS>", 8098} ]},

$ sudo riak start

# To test riak installation: 

$ curl http://localhost:8098/stats | python -mjson.tool

$ sudo riak-admin test
Attempting to restart script through sudo -H -u riak
Successfully completed 1 read/write cycle to 'riak_ec2_1@10.170.93.188'

# On 2nd and 3rd instances, join the cluster: 

$ sudo riak-admin cluster join riak_ec2_1@<IP_ADDRESS>
Attempting to restart script through sudo -H -u riak
Success: staged join request for 'riak_ec2_1@10.170.93.188' to 'riak_ec2_1@10.170.86.93' 

# On any instance: 

$ wget http://wiki.basho.com/attachments/goog.csv

$ wget http://wiki.basho.com/attachments/load_data

# Need Erlang so 'escript' works:

$ sudo yum install erlang

$ . ./load_data goog.csv

$ curl localhost:8098/riak/goog |python -mjson.tool
 - or browse to public address E.g.:
http://ec2-184-169-247-208.us-west-1.compute.amazonaws.com:8098/riak/goog

7 Databases - PostgreSQL

Vector Clocks explained

http://basho.com/blog/technical/2010/01/29/why-vector-clocks-are-easy/

Vector clocks

Puppet Hiera

http://projects.puppetlabs.com/projects/hiera/

Heira

A simple pluggable Hierarchical Database.

Why? Hierarchical data is a good fit for the representation of infrastructure information. 

Linux tree command

http://www.centerkey.com/tree/

ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/  /' -e 's/-/|/'


tree.sh

#!/bin/sh
######################################################
#  UNIX TREE                                         #
#  Version: 2.3                                      #
#  File: ~/apps/tree/tree.sh                         #
#                                                    #
#  Displays Structure of Directory Hierarchy         #
#  ------------------------------------------------- #
#  This tiny script uses "ls", "grep", and "sed"     #
#  in a single command to show the nesting of        #
#  sub-directories.  The setup command for PATH      #
#  works with the Bash shell (the Mac OS X default). #
#                                                    #
#  Setup:                                            #
#    $ cd ~/apps/tree                                #
#    $ chmod u+x tree.sh                             #
#    $ ln -s ~/apps/tree/tree.sh ~/bin/tree          #
#    $ echo "PATH=~/bin:\${PATH}" >> ~/.profile      #
#                                                    #
#  Usage:                                            #
#    $ tree [directory]                              #
#                                                    #
#  Examples:                                         #
#    $ tree                                          #
#    $ tree /etc/opt                                 #
#    $ tree ..                                       #
#                                                    #
#  Public Domain Software -- Free to Use as You Like #
#  http://www.centerkey.com/tree  -  By Dem Pilafian #
######################################################

echo
if [ "$1" != "" ]  #if parameter exists, use as base folder
  then cd "$1"
  fi
pwd
ls -R | grep ":$" |  \
  sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/  /' -e 's/-/|/'
# 1st sed: remove colons
# 2nd sed: replace higher level folder names with dashes
# 3rd sed: indent graph three spaces
# 4th sed: replace first dash with a vertical bar
if [ `ls -F -1 | grep "/" | wc -l` = 0 ]  # check if no folders
  then echo "  -> no sub-directories"
  fi
echo
exit

7 Databases in 7 Weeks

Book: http://pragprog.com/book/rwdata/seven-databases-in-seven-weeks
Blog: http://sevenweeks.org/
Code: https://github.com/sevenweeks/databases
Song: http://sevenweeks.org/post/24891440010/lyrics-to-seven-databases-in-song

Contents


1. Introduction
1.1 It Starts with a Question
1.2 The Genres
1.3 Onward and Upward

2. PostgreSQL
2.1 That’s Post-greS-Q-L
2.2 Day 1: Relations, CRUD, and Joins
2.3 Day 2: Advanced Queries, Code, and Rules
2.4 Day 3: Full-Text and Multidimensions
2.5 Wrap-Up

3. Riak
3.1 Riak Loves the Web
3.2 Day 1: CRUD, Links, and MIMEs
3.3 Day 2: Mapreduce and Server Clusters
3.4 Day 3: Resolving Conflicts and Extending Riak
3.5 Wrap-Up
4. HBase
4.1 Introducing HBase
4.2 Day 1: CRUD and Table Administration
4.3 Day 2: Working with Big Data
4.4 Day 3: Taking It to the Cloud
4.5 Wrap-Up 5. MongoDB

5.1 Hu(mongo)us
5.2 Day 1: CRUD and Nesting
5.3 Day 2: Indexing, Grouping, Mapreduce
5.4 Day 3: Replica Sets, Sharding, GeoSpatial, and GridFS
5.5 Wrap-Up

6. CouchDB
6.1 Relaxing on the Couch
6.2 Day 1: CRUD, Futon, and cURL Redux
6.3 Day 2: Creating and Querying Views
6.4 Day 3: Advanced Views, Changes API, and Replicating
Data
6.5 Wrap-Up

7. Neo4J
7.1 Neo4J Is Whiteboard Friendly
7.2 Day 1: Graphs, Groovy, and CRUD
7.3 Day 2: REST, Indexes, and Algorithms
7.4 Day 3: Distributed High Availability
7.5 Wrap-Up

8. Redis
8.1 Data Structure Server Store
8.2 Day 1: CRUD and Datatypes
8.3 Day 2: Advanced Usage, Distribution
8.4 Day 3: Playing with Other Databases
8.5 Wrap-Up

9. Wrapping Up
9.1 Genres Redux
9.2 Making a Choice
9.3 Where Do We Go from Here

A1. Database Overview Tables
A2. The CAP Theorem
A2.1 Eventual Consistency
A2.2 CAP in the Wild
A2.3 The Latency Trade-Off

Vagrant Usage Patterns and Jenkins

Vagrant Usage Patterns

 

https://speakerdeck.com/u/mitchellh/p/vagrant-usage-patterns

 

 

Vagrant and Jenkins

 

https://wiki.jenkins-ci.org/display/JENKINS/Vagrant+Plugin

 

http://stackoverflow.com/questions/6941547/how-to-combine-vagrant-with-jenkins-for-the-perfect-continuous-integration-envir

 

http://codeslife.com/2011/10/21/make-ci-easier-with-jenkins-ci-and-vagrant/

 

http://unethicalblogger.com/2012/03/13/introducing-the-vagrant-plugin.html

Lead Time vs Cycle Time

http://www.isixsigma.com/methodology/lead-time-vs-cycle-time/ 

    ( http://hbswk.hbs.edu/archive/1460.html )

1. Lead Time and Cycle Time don't have the same unit although their names are both "Time." Lead Time is measured by elapsed time (minutes, hours, etc.), whereas Cycle Time is measured by the amount of time per unit (minutes/customer, hours/part, etc.). It does not make any sense to add one to, or subtract one from, another.

2. Cycle Time is actually a measure of Throughput (units per period of time), which is the reciprocal of Cycle Time. This relationship is analogous to Takt Time (amount of time per unit), which is the reciprocal of customer demand rate (units per period of time). Note that by definition, Cycle Time (or Takt Time) is an average value.

3. Lead Time and Cycle Time are related by Work-in-progress (WIP) in the entire process, in a relationship described by the Little's Law:

Lead Time = Cycle Time * WIP

Or,

Lead Time = WIP/Throughput

4. The Cycle Time above must be the process cycle time, which is determined by the bottleneck. Cycle Times of individual steps cannot be used alone to calculate the process Lead Time without knowing the WIP.

Ansible

Ansible: http://ansible.github.com/

• Simple config management
  
• No server
• No daemons
• Uses ssh and simple scripting language for:
  • (1) Configure
  • (2) Deploy
  • (3) Ad-Hoc Tasks

Kanban from the horse's mouth

DevOps Cafe Episode 30

http://devopscafe.org/show/2012/6/13/devops-cafe-episode-30.html

A great podcast interview with David Anderson and Dominica DeGrandis about the roots of Kanban and how to use it to introduce process improvement in an organisation.


Notes:

• Kanban = culture of evolution
• Original Kanban experiement at Microsoft was digital (ticketing system with database triggers)
• Toyota let people see their processes because they know by the time they've copied it Toyota will have evolved further and they know that culture is much harder to copy especially for western cultures at a neuropshycological level
• Guiding principles
• Start where you are now
• Agree to pursue change
• Respect the current situation
• Drum-buffer-rope analogy from "The Goal"
• Blog article: Understanding process of knowledge discovery
• Value Stream Mapping? 
• Metaphor breaks down
• Stream = water flow
• Water can't flow uphill - IT releases involve rework that's hard to map
• Better: "Workflow mapping"
• Go around objections
• Create environment where people can experiment and come up with own unique processes
• Don't create new process for people - create a culture where they can evolve their own process & understand individual changes. 
• Make many small adjustments over the long haul - evolving with each change
• Snowball effect
• Evolution gets you there faster
• Like switching to Mac - can't explain it until you experience it
• Pull
• Toyota don't do a pure pull system
• Users don't always know what they want
• Eg jaguar asked jaguar owners to describe what they want in the new jaguar design - they designed the same car - can't imagine
• "But we're a scrum/agile shop"
• Tribal reaction - emotional response
• Resistance to change
• Can't have logical discussion about devops
• Suggest a "super tribe"
• Lessons agile mgmt

Introduction to Chef

http://learnchef.getharvest.com/introduction.html

Cost of software / Capers Jones links

Tech debt article: http://swreflections.blogspot.com.au/2012/02/technical-debt-how-much-is-it-really.html

Capers Jones PDF: http://sqgne.org/presentations/2011-12/Jones-Sep-2011.pdf

Capers Jones facts and figures: http://allankelly.blogspot.com.au/2011/02/more-facts-and-figures-from-capers.html

Installing RVM on Ubuntu

Install RVM instead of Installing Ruby. 

http://rubysource.com/installing-ruby-with-rvm-on-ubuntu/

bash < <(curl -sk https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer)
echo '[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm"' >> ~/.bashrc
source ~/.bashrc

BDD and REST

Testing REST with BDD

Brian Sletten 

Presentation: http://skillsmatter.com/podcast/ajax-ria/sletten-rest-bdd?goback=%2Egde_4079964_member_112404082

The move to a good REST API yields an explosive combination of options due to content-negotiation and arbitrary workflows. At the same time, the uniform interface you project (representations, how you respond to verbs, response codes, etc.) becomes your contract with your clients. The API itself becomes a completely transportable, reusable bit of organizational knowledge when done well. Testing it sufficiently will give you a safety net but may drive you crazy unless you do it right. We will walk through the use of Behavior-Driven Development (BDD) to test these kinds of APIs.

We will highlight the problems of sufficiently testing REST APIs while providing a rich and relatively straight-forward solution to the problem. We will use a Behavior-Driven Development (BDD) tool like Cucumber to establish reusable steps and comprehensive, but lightweight testing strategies for testing REST APIs.


How to sell BDD to the business

Dan North

Presentation: http://skillsmatter.com/podcast/soa-rest/how-to-sell-bdd-to-the-business
Slides: http://skillsmatter.com/custom/presentations/sellingbddtothebusiness_bdd.pdf

I don't care how you do it - I just want it better, faster and cheaper and I don't want you to keep bugging me

How do you present the benefits case of BDD to a sceptical, time-starved, budget-focused business sponsor? When we talk about Writing software that matters it is from the perspective of the business as much as anybody else.

Software delivery shouldn't be a zero sum game where you have to choose between writing quality software, showing bottom line results and having fun. In this talk, Dan shares his experiences of introducing BDD and Agile principles to senior stakeholders and business executives. Download the slides here: http://skillsmatter.com/custom/presentations/sellingbddtothebusiness_bdd.pdf




Blog: BDD for secure REST APIs using cucumber, pickle, and rack-test
http://blog.tddium.com/2011/04/22/bdd-for-secure-rest-apis-using-cucumber-pickle-and-rack-test/

How to show Maven classpaths

From: http://maven.apache.org/plugins/maven-antrun-plugin/examples/classpaths.html

Add this under <build><plugins> in your pom, then run it with  mvn compile :


            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-antrun-plugin</artifactId>
                <version>1.7</version>
                <executions>
                    <execution>
                        <id>compile</id>
                        <phase>compile</phase>
                        <configuration>
                            <target>
                                <property name="compile_classpath" refid="maven.compile.classpath"/>
                                <property name="runtime_classpath" refid="maven.runtime.classpath"/>
                                <property name="test_classpath" refid="maven.test.classpath"/>
                                <property name="plugin_classpath" refid="maven.plugin.classpath"/>

                                <echo message="compile classpath: ${compile_classpath}"/>
                                <echo message="runtime classpath: ${runtime_classpath}"/>
                                <echo message="test classpath:    ${test_classpath}"/>
                                <echo message="plugin classpath:  ${plugin_classpath}"/>
                            </target>
                        </configuration>
                        <goals>
                            <goal>run</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>