Tuesday, November 20, 2012

Getting Puppet master and agent running on a single Vagrant box

Ensure you have "lucid32" box:

vagrant box add lucid32 http://files.vagrantup.com/lucid32.box

Add a Vagrantfile in a new directory:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant::Config.run do |config|
  config.vm.box = "lucid32"

vagrant up

ssh into new vagrant box (port 2222)

sudo su -
echo -e "deb http://apt.puppetlabs.com/ lucid main\ndeb-src http://apt.puppetlabs.com/ lucid main" >> /etc/apt/sources.list.d/puppet.list
apt-key adv --keyserver keyserver.ubuntu.com --recv 4BD6EC30
apt-get update
apt-get install puppet puppetmaster
apt-cache policy puppet
puppet --version
vi /etc/hosts

      # add      puppet.example.com    puppet
vi /etc/puppet/puppet.conf

     # add to [master] section: certname=puppet.example.com
touch /etc/puppet/manifests/site.ppiptables -A INPUT -p tcp -m state --state NEW --dport 8140 -j ACCEPT
puppet master --no-daemonize --verbose --debug

Start another ssh session to same box
sudo su -
puppet agent --verbose --debug


Sunday, November 18, 2012

Cryptic Ruby Global Variables


$!         The exception information message set by 'raise'.
$@         Array of backtrace of the last exception thrown.
$&         The string matched by the last successful match.
$`         The string to the left  of the last successful match.
$'         The string to the right of the last successful match.
$+         The highest group matched by the last successful match.
$1         The Nth group of the last successful match. May be > 1.
$~         The information about the last match in the current scope.
$=         The flag for case insensitive, nil by default.
$/         The input record separator, newline by default.
$\         The output record separator for the print and IO#write. Default is nil.
$,         The output field separator for the print and Array#join.
$;         The default separator for String#split.
$.         The current input line number of the last file that was read.
$<         The virtual concatenation file of the files given on command line (or from $stdin if no files were given).
$>         The default output for print, printf. $stdout by default.
$_         The last input line of string by gets or readline.
$0         Contains the name of the script being executed. May be assignable.
$*         Command line arguments given for the script sans args.
$$         The process number of the Ruby running this script.
$?         The status of the last executed child process.
$:         Load path for scripts and binary modules by load or require.
$"         The array contains the module names loaded by require.
$DEBUG     The status of the -d switch.
$FILENAME  Current input file from $<. Same as $<.filename.
$LOAD_PATH The alias to the $:.
$stderr    The current standard error output.
$stdin     The current standard input.
$stdout    The current standard output.
$VERBOSE   The verbose flag, which is set by the -v switch.
$-0        The alias to $/.
$-a        True if option -a is set. Read-only variable.
$-d        The alias to $DEBUG.
$-F        The alias to $;.
$-i        In in-place-edit mode, this variable holds the extension, otherwise nil.
$-I        The alias to $:.
$-l        True if option -l is set. Read-only variable.
$-p        True if option -p is set. Read-only variable.
$-v        The alias to $VERBOSE.
$-w        True if option -w is set.

Environmental Global Variables

$: (Dollar Colon)

$: is basically a shorthand version of $LOAD_PATH. $: contains an array of paths that your script will search through when using require.

$0 (Dollar Zero)

$0 contains the name of the ruby program being run. This is typically the script name.

$* (Dollar Splat)

$* is basically shorthand for ARGV. $* contains the command line arguments that were passed to the script.

$? (Dollar Question Mark)

$? returns the exit status of the last child process to finish.

$$ (Dollar Dollar)

$$ returns the process number of the program currently being ran. 

Regular Expression Global Variables

$~ (Dollar Tilde)

$~ contains the MatchData from the previous successful pattern match.

$1, $2, $3, $4 etc

$1-$9 represent the content of the previous successful pattern match.

$& (Dollar Ampersand)

$& contains the matched string from the previous successful pattern match.

$+ (Dollar Plus)

$+ contains the last match from the previous successful pattern match.

$` (Dollar Backtick)

$` contains the string before the actual matched string of the previous successful pattern match.

$’ (Dollar Apostrophe)

$' contains the string after the actual matched string of the previous successful pattern match. 

Exceptional Global Variables

$! (Dollar Bang)

$! contains the Exception that was passed to raise.

$@ (Dollar At Symbol)

$@ contains the backtrace for the last Exception raised. 

Other Global Variables

$_ (Dollar Underscore)

$_ The last input line of string by gets or readline.

$, (Dollar Comma)

$, is the (global) default separator for Array#join and possibly other methods.

Thursday, November 15, 2012

Event Sourcing Yow Night with Greg Young

·         Current state:
·         Is awful
·         Requires large amounts of versioning
·         1st level derivative of facts that have happened
·         Look at systems from perspective of no current state
·         Banking, insurance, gambling, etc
·         We don’t have current state, we have a series of facts
·         Driving point is from business perspective
·         E.g.
·         Purchase order
·         Line items(n)
·         Shipping information
·         Models represent our current state
·         Document stores are awesome - until you need to change your schema
·         Problem is we want to go and change our previous representations of data
·         E.g. Cart created -> 3 items added -> shipping information added
·         At any time can replay 3 events to get data model
·         Events: append only model
·         How do you scale immutable data?  Copy it
·         Immutable data is awesome
·         Once “Cart created” is created it will never change
·         Append-only model, with everything immutable, what about updates/deletes?
·         Update/delete = lost valuable data
·         Code with a magic 8-ball to predict what business is going to want in 2 years?
·         Strategic design with DDD
·         Don’t apply ES globally
·         ES/CQRS is not an architecture
·         Small things you apply within a service/component
·         Not losing information is valuable
·         2 sets of use cases in different orders that end up with same ending state?
·         Lost info
·         Hash collision – non-perfect – lost info coming into system
·         One rule: we don’t lose any data – generating 100Gb per day
·         How do you predict value of data?
·         Humans have history of making bad predictions about future
·         Bigger the expert = worse predictive analysis
·         Can only say: “I cannot price this option”
·         Therefore I should keep it
·         When business ask for unexpected data, can say yes
·         Could be something that makes or breaks company – competitive advantage
·         Accounting is not done with a pencil
·         If make a mistake, do a reversal
·         Partial reversal $10,000 instead of $1,000 = -$9,000
·         Accountants don’t like doing – too complicated across 8 accounts,
·         Do a full reversal instead and then redo
·         E.g. Cart created -> 3 items added -> 1 item removed -> shipping information added
·         Same as 2 items added?
·         As a series of facts, very different from each other
·         Want to know about how many items removed?
·         Most businesses are not just create, read, update, delete…. Many verbs
·         ES gives semantics associated back down to verbs
·         Business value comes from fact that we’re not losing information
·         E.g. Large POS, Amazon
·         Removed items from cart are more likely to purchase in the future – still want them can’t afford them
·         Old model
·         Add RemovedLineItems object or flag & date on line items
·         Query, subquery – time correlation – 3 nested subqueries
·         (Try using a Stream database instead)
·         ES model
·         Write projection with state inside
·         If item found in carts
·         Business person can go back into past and see things at that point in time with a deterministic perception we have today
·         Huge win for business
·         Useful for predicting future  - “Back testing” in finance
·         BI reverse engineer CRUD databases into events (imperfectly)
·         Temporal data model
·         Smoke testing
·         Rerun commands since day 1 every Friday and compare results from last time
·         Won’t protect you from black swans
·         Append-only good for hard drives (even SSDs that burn out rewriting)
·         E.g. Secure system
·         Gambling
·         Chris Harn – edited his bets on hard drive
·         How to prevent a super user attack
·         E.g. Pick 6 tickets
·         CSU/DSU
·         Prevent by putting log on “write-once” media – physically can’t modify data
·         Easier to physically secure a machine than to secure software
·         200 partitions within logs
·         Every aggregate has its own stream
·         Partition
·         Rolling snapshot
·         20,000 requests per sec if all in memory
·         Rents represents functions
·         Current state = left fold
·         Snapshots = memoisation
·         ES = functional way of storing data
·         Pattern match functions to events
·         ES = FP
·         Balance of bank account not a column in db but a function of account history
·         Provable

·         Natural fits for ES
·         Accounting
·         Pubsub
·         Don't have to build your own Event Store
·         Cassandra - stream per colum
·         Scales well
·         Medical system

·         How to justify cost of storing everything because you don’t know what you will need
·         Cost of data is low - 5gb for can of coke
·         Hard to justify not storing data
·         What is it not used for?
·         Lots of things
·         Things outside of core domain
·         Events represent use cases
·         Some use cases might not be high value
·         E.g claims more valuable than sales
·         Only used for competitive advantage – requires analysis
·         Pitfalls?
·         ES architecture
·         Monolithic - systems of systems instead
·         Expensive to do analysis
·         Does every projection read every event?
·         Projection pattern match, function
·         Only look at events interested in
·         Map reduce
·         I asked which databases other than Cassandra were a good fit for ES?
·         Consistency is important
·         Need CA for writes, AP for reads
·         Hard to find system that can be tuned like that
·         Riak but slow, quorum writes
·         Event Store has BSD license
·         SQL server for small systems

Saturday, November 10, 2012

RunDeck and Jenkins


RunDeck and Jenkins can be used together to provide a deployment pipeline.

How is RunDeck different from Jenkins?
  • Rundeck not a CI server
  • Both are able to:
    • provide a self serve job interface to automate routine procedures. 
    • execute shell scripts on remote nodes to facilitate deployment tasks. 
  • Differentiator: Rundeck's built-in support for pluggable remote command execution
  • Comes down to use case. 
    • Rundeck == job console for Ops and geared to work with that ecosystem of tools.
    • jenkins-rundeck plugin demonstrates how complimentary they are in continuous deployment tool chain. 
    • Jenkins handling build end of CI loop and triggering Rundeck to provide distributed orchestration across deployment management tool chain.

How is RunDeck different than Puppet mcollective or Chef knife?
  • Some overlap between rundeck and mcollective and knife
    • Allow administrators to execute commands in distributed environment, offering a form of real time control
    • Use metadata-level searches for targeting remote nodes. 
    • Levels of authorization, authentication and auditing
  • Rundeck has a few goals of its own though:
    • Easy way to define routine sequences as "Job workflows" as a basis for runbook automation solutions.
    • Integration of node and environment metadata sources as RunDeck "resource model providers". In this way, Rundeck can use Puppet or Chef node data to drive remote execution.
    • Evolve role-based access control definitions into a high level DSL that ties privilege level to resource model and workflow actions
    • Plugin system supporting concept of "dispatch providers" to delegate to tools like mcollective, knife, func, fabric, PsExec and others for cross tool execution.
  • Ultimate Goal: Simple to use yet flexible enough to complement existing tool chains
Puppet-Rundeck resource provider for Rundeck

Example/Musings on using Rundeck, Puppet, Jenkins, Fabric together

Bamboo-RunDeck Plugin