Getting Puppet master and agent running on a single Vagrant box

Ensure you have "lucid32" box:

vagrant box add lucid32 http://files.vagrantup.com/lucid32.box

Add a Vagrantfile in a new directory:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant::Config.run do |config|
  config.vm.box = "lucid32"
end

vagrant up

ssh into new vagrant box (port 2222)

sudo su -
echo -e "deb http://apt.puppetlabs.com/ lucid main\ndeb-src http://apt.puppetlabs.com/ lucid main" >> /etc/apt/sources.list.d/puppet.list
apt-key adv --keyserver keyserver.ubuntu.com --recv 4BD6EC30
apt-get update
apt-get install puppet puppetmaster
apt-cache policy puppet
puppet --version
vi /etc/hosts
      # add 127.0.0.1      puppet.example.com    puppet
vi /etc/puppet/puppet.conf
     # add to [master] section: certname=puppet.example.com
touch /etc/puppet/manifests/site.ppiptables -A INPUT -p tcp -m state --state NEW --dport 8140 -j ACCEPT
puppet master --no-daemonize --verbose --debug

Start another ssh session to same box
sudo su -
puppet agent --verbose --debug


Reference:

• http://projects.puppetlabs.com/projects/1/wiki/Puppet_Ubuntu
• http://serverfault.com/questions/332580/how-can-i-fix-puppet-refusing-to-start-and-asking-for-master-pp

Cryptic Ruby Global Variables

http://www.zenspider.com/Languages/Ruby/QuickRef.html

$!         The exception information message set by 'raise'.
$@         Array of backtrace of the last exception thrown.
$&         The string matched by the last successful match.
$`         The string to the left  of the last successful match.
$'         The string to the right of the last successful match.
$+         The highest group matched by the last successful match.
$1         The Nth group of the last successful match. May be > 1.
$~         The information about the last match in the current scope.
$=         The flag for case insensitive, nil by default.
$/         The input record separator, newline by default.
$\         The output record separator for the print and IO#write. Default is nil.
$,         The output field separator for the print and Array#join.
$;         The default separator for String#split.
$.         The current input line number of the last file that was read.
$<         The virtual concatenation file of the files given on command line (or from $stdin if no files were given).
$>         The default output for print, printf. $stdout by default.
$_         The last input line of string by gets or readline.
$0         Contains the name of the script being executed. May be assignable.
$*         Command line arguments given for the script sans args.
$$         The process number of the Ruby running this script.
$?         The status of the last executed child process.
$:         Load path for scripts and binary modules by load or require.
$"         The array contains the module names loaded by require.
$DEBUG     The status of the -d switch.
$FILENAME  Current input file from $<. Same as $<.filename.
$LOAD_PATH The alias to the $:.
$stderr    The current standard error output.
$stdin     The current standard input.
$stdout    The current standard output.
$VERBOSE   The verbose flag, which is set by the -v switch.
$-0        The alias to $/.
$-a        True if option -a is set. Read-only variable.
$-d        The alias to $DEBUG.
$-F        The alias to $;.
$-i        In in-place-edit mode, this variable holds the extension, otherwise nil.
$-I        The alias to $:.
$-l        True if option -l is set. Read-only variable.
$-p        True if option -p is set. Read-only variable.
$-v        The alias to $VERBOSE.
$-w        True if option -w is set.

http://jimneath.org/2010/01/04/cryptic-ruby-global-variables-and-their-meanings.html

Environmental Global Variables

$: (Dollar Colon)

$: is basically a shorthand version of $LOAD_PATH. $: contains an array of paths that your script will search through when using require.

$0 (Dollar Zero)

$0 contains the name of the ruby program being run. This is typically the script name.

$* (Dollar Splat)

$* is basically shorthand for ARGV. $* contains the command line arguments that were passed to the script.

$? (Dollar Question Mark)

$? returns the exit status of the last child process to finish.

$$ (Dollar Dollar)

$$ returns the process number of the program currently being ran. 

Regular Expression Global Variables

$~ (Dollar Tilde)

$~ contains the MatchData from the previous successful pattern match.

$1, $2, $3, $4 etc

$1-$9 represent the content of the previous successful pattern match.

$& (Dollar Ampersand)

$& contains the matched string from the previous successful pattern match.

$+ (Dollar Plus)

$+ contains the last match from the previous successful pattern match.

$` (Dollar Backtick)

$` contains the string before the actual matched string of the previous successful pattern match.

$’ (Dollar Apostrophe)

$' contains the string after the actual matched string of the previous successful pattern match. 

Exceptional Global Variables

$! (Dollar Bang)

$! contains the Exception that was passed to raise.

$@ (Dollar At Symbol)

$@ contains the backtrace for the last Exception raised. 

Other Global Variables

$_ (Dollar Underscore)

$_ The last input line of string by gets or readline.

$, (Dollar Comma)

$, is the (global) default separator for Array#join and possibly other methods.

Event Sourcing Yow Night with Greg Young

Greg Young
Yow Night 15^th Nov 2012

See also:

• http://codebetter.com/gregyoung/2010/02/20/why-use-event-sourcing/
• http://martinfowler.com/eaaDev/EventSourcing.html

·         Current state:

·         Is awful

·         Requires large amounts of versioning

·         1^st level derivative of facts that have happened

·         Look at systems from perspective of no current state

·         Banking, insurance, gambling, etc

·         We don’t have current state, we have a series of facts

·         Driving point is from business perspective

·         E.g.

·         Purchase order

·         Line items(n)

·         Shipping information

·         Models represent our current state

·         Document stores are awesome - until you need to change your schema

·         Problem is we want to go and change our previous representations of data

·         E.g. Cart created -> 3 items added -> shipping information added

·         At any time can replay 3 events to get data model

·         Events: append only model

·         How do you scale immutable data?  Copy it

·         Immutable data is awesome

·         Once “Cart created” is created it will never change

·         Append-only model, with everything immutable, what about updates/deletes?

·         Update/delete = lost valuable data

·         Code with a magic 8-ball to predict what business is going to want in 2 years?

·         Strategic design with DDD

·         Don’t apply ES globally

·         ES/CQRS is not an architecture

·         Small things you apply within a service/component

·         Not losing information is valuable

·         2 sets of use cases in different orders that end up with same ending state?

·         Lost info

·         Hash collision – non-perfect – lost info coming into system

·         One rule: we don’t lose any data – generating 100Gb per day

·         How do you predict value of data?

·         Humans have history of making bad predictions about future

·         Bigger the expert = worse predictive analysis

·         Can only say: “I cannot price this option”

·         Therefore I should keep it

·         When business ask for unexpected data, can say yes

·         Could be something that makes or breaks company – competitive advantage

·         Accounting is not done with a pencil

·         If make a mistake, do a reversal

·         Partial reversal $10,000 instead of $1,000 = -$9,000

·         Accountants don’t like doing – too complicated across 8 accounts,

·         Do a full reversal instead and then redo

·         E.g. Cart created -> 3 items added -> 1 item removed -> shipping information added

·         Same as 2 items added?

·         As a series of facts, very different from each other

·         Want to know about how many items removed?

·         Most businesses are not just create, read, update, delete…. Many verbs

·         ES gives semantics associated back down to verbs

·         Business value comes from fact that we’re not losing information

·         E.g. Large POS, Amazon

·         Removed items from cart are more likely to purchase in the future – still want them can’t afford them

·         Old model

·         Add RemovedLineItems object or flag & date on line items

·         Query, subquery – time correlation – 3 nested subqueries

·         (Try using a Stream database instead)

·         ES model

·         Write projection with state inside

·         If item found in carts

·         Business person can go back into past and see things at that point in time with a deterministic perception we have today

·         Huge win for business

·         Useful for predicting future  - “Back testing” in finance

·         BI reverse engineer CRUD databases into events (imperfectly)

·         Temporal data model

·         Smoke testing

·         Rerun commands since day 1 every Friday and compare results from last time

·         Won’t protect you from black swans

·         Append-only good for hard drives (even SSDs that burn out rewriting)

·         E.g. Secure system

·         Gambling

·         Chris Harn – edited his bets on hard drive

·         How to prevent a super user attack

·         E.g. Pick 6 tickets

·         CSU/DSU

·         Prevent by putting log on “write-once” media – physically can’t modify data

·         Easier to physically secure a machine than to secure software

·         200 partitions within logs

·         Every aggregate has its own stream

·         Partition

·         Rolling snapshot

·         20,000 requests per sec if all in memory

·         Rents represents functions

·         Current state = left fold

·         Snapshots = memoisation

·         ES = functional way of storing data

·         Pattern match functions to events

·         ES = FP

·         Balance of bank account not a column in db but a function of account history

·         Provable

·         Natural fits for ES

·         Accounting

·         Pubsub

·         Don't have to build your own Event Store

·         Cassandra - stream per colum

·         Scales well

·         Medical system

Questions

·         How to justify cost of storing everything because you don’t know what you will need

·         Cost of data is low - 5gb for can of coke

·         Hard to justify not storing data

·         What is it not used for?

·         Lots of things

·         Things outside of core domain

·         Events represent use cases

·         Some use cases might not be high value

·         E.g claims more valuable than sales

·         Only used for competitive advantage – requires analysis

·         Pitfalls?

·         ES architecture

·         Monolithic - systems of systems instead

·         Expensive to do analysis

·         Does every projection read every event?

·         Projection pattern match, function

·         Only look at events interested in

·         Map reduce

·         I asked which databases other than Cassandra were a good fit for ES?

·         Consistency is important

·         Need CA for writes, AP for reads

·         Hard to find system that can be tuned like that

·         Riak but slow, quorum writes

·         Event Store has BSD license

·         SQL server for small systems

Hiera: Separating config from Puppet code

http://puppetlabs.com/blog/the-problem-with-separating-data-from-puppet-code/

RunDeck and Jenkins

http://rundeck.org/docs/manual/introduction.html
http://rundeck.org/docs/manual/getting-started.html
http://rundeck.org/docs/manual/rundeck-basics.html


RunDeck and Jenkins can be used together to provide a deployment pipeline.

https://github.com/dtolabs/rundeck/wiki/Faq
How is RunDeck different from Jenkins?

• Rundeck not a CI server
• Both are able to:
• provide a self serve job interface to automate routine procedures. 
• execute shell scripts on remote nodes to facilitate deployment tasks. 
• Differentiator: Rundeck's built-in support for pluggable remote command execution
• Comes down to use case. 
• Rundeck == job console for Ops and geared to work with that ecosystem of tools.
• jenkins-rundeck plugin demonstrates how complimentary they are in continuous deployment tool chain. 
• Jenkins handling build end of CI loop and triggering Rundeck to provide distributed orchestration across deployment management tool chain.

How is RunDeck different than Puppet mcollective or Chef knife?

• Some overlap between rundeck and mcollective and knife
• Allow administrators to execute commands in distributed environment, offering a form of real time control
• Use metadata-level searches for targeting remote nodes. 
• Levels of authorization, authentication and auditing
• Rundeck has a few goals of its own though:
• Easy way to define routine sequences as "Job workflows" as a basis for runbook automation solutions.
• Integration of node and environment metadata sources as RunDeck "resource model providers". In this way, Rundeck can use Puppet or Chef node data to drive remote execution.
• Evolve role-based access control definitions into a high level DSL that ties privilege level to resource model and workflow actions
• Plugin system supporting concept of "dispatch providers" to delegate to tools like mcollective, knife, func, fabric, PsExec and others for cross tool execution.
• Ultimate Goal: Simple to use yet flexible enough to complement existing tool chains

Puppet-Rundeck resource provider for Rundeck
https://github.com/jamtur01/puppet-rundeck


Example/Musings on using Rundeck, Puppet, Jenkins, Fabric together
http://www.morethanseven.net/2011/09/11/Rundeck-and-nagios-nrpe-checks/


Bamboo-RunDeck Plugin

• https://github.com/connaryscott/bamboo-rundeck-plugin/wiki
• https://marketplace.atlassian.com/plugins/com.dtolabs.bamboo.plugin.bamboo-rundeck-plugin

Continuous Delivery with Jenkins Webinar (and Deployit)

CloudBees Webinar

• Blog: http://blog.cloudbees.com/2012/09/continuous-integration-to-continuous.html
• Cloudbees link: jumping-continuous-integration-continuous-delivery-jenkins-enterprise
• Vimeo
• Slides
• Q&A: http://blog.cloudbees.com/2012/09/q-continuous-integration-to-continuous.html
• Example Jenkins configurations: 
• https://partnerdemo.ci.cloudbees.com/job/mongo-chess/configure
• https://partnerdemo.ci.cloudbees.com/job/mongo-chess-integration/configure
• Uses Jenkins Copy Artifact plugin
•  

Xebia: http://xebialabs.com/

Deployit for Jenkins: http://www.xebialabs.com/jenkins

Atlassian interview with DeployIt's creator recommending the DeployIt plugin for Bamboo: http://blogs.atlassian.com/2012/05/automated-deployment-getting-started/

Deployit blog: Ultimate Continuous Delivery Deployit Toolkit
Deployt blog: Continuous Delivery with Deployit, Puppet, JBoss and VMware