Wednesday, December 28, 2011

Yow 2011 Brisbane


Craig Smith's Wrap Up:

Channel 9 Interviews:

DevOps Reading/Viewing

Velocity 2010: Tom Cook, "A Day in the Life of Facebook Operations" [YouTube]

HipHop for PHP
300+TB live data in RAM
Linux (CentOS 5)
Enigineering + Operations (no QA)
·         Engineers write, test, an deploy their own code
o   Quickly make performance decisions
o   Expose changes to subset of real traffic
·         No ‘commit and quit’
·         Deeply involved in moving services to production
·         Ops ‘embedded’ into engineering teams
o   Help make architecture decisions
o   Better understand needs of product
o   Interface with other ops teams
·         Change logging – log all code changes
Small teams

Facebook has over 60,000 servers:

InfoQ: Infrastructure as Code [InfoQ]
Panel: Theo Schlossnagle – OmniTI; Luke Kanies – Puppet Labs; Adam Jacob – Opscode; Erik Troan – rPath

Infrastructure Automation with Opscode Chef [pdf]
Joshua Timberman (Opscode, Inc.), Adam Jacob (Opscode), Christopher Brown (Opscode),Aaron Peterson (Opscode, Inc.), Matt Ray (Opscode), Seth Chisamore (Opscode, Inc.)

The 7 levels of continuous integration

Friday, December 23, 2011

DevOps: Implications of Infrastructure As Code

Stephen Nelson-Smith

I’m seeing a lot of problems emerging in organisations starting to put these kind of ideas into practice.
Here are a few symptoms:
  • Sprawling masses of Puppet code, duplication, contradiction and a lack of clear understanding of what it all does.
  • A fear of change – a sense that we dare not meddle with the manifests or recipes, because we’re not entirely certain how the system will behave.
  • Bespoke software that started off well-engineered, and thoroughly tested, littered with TODOs, and FIXME and quick hacks.
  • A sense that, despite the lofty goal of capturing the expertise required to understand an infrastructure in the code itself, if one or two key people were to leave, the organisation or team would be in trouble.
As a corrective, I think we need to start thinking about six key areas of our Infrastructure Code:
  • Design
  • Collective Ownership
  • Review
  • Standards
  • Testing
  • Refactoring

DevOps: CAMS

What Devops Means to Me
By John Willis


  • Culture
  • Automation
  • Measurement
  • Sharing

Thursday, December 22, 2011

DevOps: Building a DevOps Team

Brian Henerey: Building a DevOps Team

Eric Minick: Is Having a “DevOps Team” a Problem?

[Patrick Debois] says that if you have a DevOps team you are doing it wrong. He later hedges with the idea that a team to help transition to DevOps is a good thing.

For better or worse, they [DevOps teams] may be to DevOps what Scrum was to Agile - an avenue to enterprise adoption that purists find a little unpalatable.

Wednesday, December 21, 2011

Dynamically Creating a Test Suite in JUnit 4 -- cpsuite

Regex and Lookahead/Lookbehind

E.g. to remove all @Component annotations from non-"Default" classes:

search for (negative lookahead):
@Component\npublic class (?!Default)

replace with:
public class

negative lookahead: q(?!u) matches a "q" not followed by a "u"

positive lookahead:  q(?=u) matches a q that is followed by a u, without making the u part of the match

If you want to store the match of the regex inside a backreference, you have to put capturing parentheses around the regex inside the lookahead, like this:(?=(regex))

negative lookbehind: (?<!a)b matches a "b" that is not preceded by an "a", using negative lookbehind.

positive lookbehind: (?<=a)b matches the b (and only the b) in cab

Thursday, December 15, 2011

Java XML XPath

import org.apache.xml.dtm.ref.DTMNodeList;
import org.w3c.dom.Element;
import org.xml.sax.SAXException;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathExpressionException;
import javax.xml.xpath.XPathFactory;
public class XPathTest extends TestCase {
    private static final String CAR_RECORDS = "<records>       <car name='HSV Maloo' make='Holden' year='2006'>         <country>Australia</country>         <record type='speed'>Production Pickup Truck with speed of 271kph</record>       </car>       <car name='P50' make='Peel' year='1962'>         <country>Isle of Man</country>         <record type='size'>Smallest Street-Legal Car at 99cm wide and 59 kg in weight</record>       </car>       <car name='Royale' make='Bugatti' year='1931'>         <country>France</country>         <record type='price'>Most Valuable Car at $15 million</record>       </car>     </records> ";
    public void test() throws ParserConfigurationException, IOException, SAXException, XPathExpressionException {
        DocumentBuilder builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
        InputStream inputStream = new ByteArrayInputStream(CAR_RECORDS.getBytes());
        Element records = builder.parse(inputStream).getDocumentElement();
        XPath xpath = XPathFactory.newInstance().newXPath();
        DTMNodeList nodes = (DTMNodeList) xpath.evaluate("//car", records, XPathConstants.NODESET);
        for (int i = 0; i < nodes.getLength(); i++) {
            org.w3c.dom.Node node = nodes.item(i);
            String make = xpath.evaluate("@make", node);
            String country = xpath.evaluate("country/text()", node);
            String type = xpath.evaluate("record/@type", node);
            System.out.println(make + " of " + country + " has a " + type + " record");

Wednesday, December 7, 2011

Coffee with Robert Virding

Had a fascinating conversation over coffee with Robert Virding at Yow Conference Tuesday 6th December 2011... jotted down some of the things I remembered:
  • "erlc" produces a BEAM file which is portable byte code
    • Joe's Abstract Machine originally compiled Erlang to C macros which was then JIT translated to machine code on the fly
    • BEAM was a complete rewrite to produce portable byte code that "erl" translates to machine code on the fly
  • Erjang runs on the JVM so it has the advantage of many years and many people of JVM optimisation... Erlang has a team of 5 developers working on the VM.
  • Architecture of Erlang apps
    • Processes that do the work don't contain error handling... error handling is done where and when it matters by supervisor processes further up the tree
    • Supervisor processes are responsible for determining how to handle different types of errors
    • Some people on stack overflow freak out about this idea of letting processes crash but don't realise that approach goes hand-in-hand with the approach of handling errors higher up the process tree
  • Error Handling
    • Modular approach
    • Robert likes to code for all eventualities but often developers take an iterative approach... code the core functionality in processes... run the app, observe how different errors crash the processes... and work out how (and where) they should be captured and dealt with
    • Processes are organised in trees.  A supervisor at the top is responsible for working out what to do when different types of errors occur
    • Typical approach is to let type errors crash the process
  • Erlang movie 
    • Robert was the guy who "fixed the bug"
    • This patching code fixes while the app is running is possible through module loading
    • The app has to be specifically designed with this approach in mind
    • Have to be careful... what if the process is already running?  
    • A module could  (and typically does) contain both the server code and the client code used to access the server.  So a module could contain code that is running in separate processes - a client process, a server process.  Updating it on the fly is tricky.
    • Need to take into account data changes too... a mini data migration might be required on existing state... it's up to the module author what they're going to do
    • Sometimes the best approach is to crash the process and restart it rather than carry all that complexity in the code - depends on the scenario.
    • Also, restarting the entire app is an option... 
    • Or using a load balancer type approach where the router directs the traffic to the new server.
  • Some companies use Erlang as essentially a gateway to processes running on other languages
    • e.g. a fast server server that accepts many concurrent connections and directs each connection to a backend C process.
    • Erlang NIFs (Natively Implemented Functions) are invoked in a similar manner to Java's JNI interface
    • Erlang can call into Java code using the JInterface
  • I remarked I was surprised that I couldn't call my own function from a guard and I understood this was because Erlang didn't want you doing side effects during a guard statement
    • Guards where really just an extension on pattern matching, enabling you to be a little more verbose
  • Can't create your own data types but can use a Binary type which lets you compose data out of other Erlang types.
    • Can reuse the same binary block with other binary types e.g. if your binary type has Int,String,Bool... you could reuse a variable of that binary type as a different type
  • Erjang
    • serious implementation of Erlang on JVM
    • Robert hopes good things come out of it
    • Tricky job though to keep up with Erlang as it is still evolving within Ericsson
  • Ericsson's OTP department maintain the Erlang language
    • Used mainly for testing
    • Surprising that a telecoms hardware company supported the language from infancy and there was initially some tension with others who didn't like it but the OTP team had sympathetic management
  • Was Erlang based on Prolog?
    • It was originally implemented on Prolog (eventually rewritten in Erlang itself)
    • The syntax was influenced a little by Prolog but logic programming wasn't the main thrust - it was more about solving telephony concurrency problems.
  • Where did the functional programming syntax come from?
    • Realised after a bit that the language needed some of the benefits of FP and so they were added
  • Do you find many type errors come out in runtime when a particular piece of code that wasn't exercised during testing is invoked due to the input of different data?
    • Not so much... there's a tool called the Dialyser that you run over your code... it actively tests all your types and tells you where there's a programming error in your code - it's able to catch most of the programming errors that could lead to a type error at runtime.
  • Hipe:
  • Popularity of Erlang
    • It's been pretty steady... there was a bit of a peak in 2008/09

Saturday, December 3, 2011

Sunday, November 20, 2011

Displaying Groovy Classpath
this.class.classLoader.rootLoader.URLs.each{ println it }
def printClassPath(classLoader) {
  println "$classLoader"
  classLoader.getURLs().each {url->
     println "- ${url.toString()}"
  if (classLoader.parent) {
printClassPath this.class.classLoader


Imaginary Boss

"I have an imaginary boss in my head. I'm still scared if I'm late coming in to work. Everybody else has someone to drive them, I had to find that discipline within myself."
Yoganathan Ratheesan of Lebara

git push putty disconnected error

PuTTY Fatal Error: Disconnected: No supported authentication methods available"

set GIT_SSH=c:\progra~2\git\bin\ssh.exe

Friday, November 18, 2011

Snap: Haskell Web Framework

cabal update
cabal install snap  -- installs into C:\Users\<user>\AppData\Roaming\cabal  (or $HOME/.cabal/) (10mins)
mkdir snapblog
cd snapblog
snap init -- creates .ghci, snapblog.cabal, <src>, <log>, <resources>
cabal install  -- installs executable snap (snapblog.exe) into C:\Users\<user>\AppData\Roaming\cabal\bin  (or $HOME/.cabal/bin)
snapblog -p 8000

Needle: Automated tests for your CSS

On the goals of Coderetreat

"The Principles of Product Development Flow: Second Generation Lean Product Development" by Donald G. Reinertsen

"The Principles of Product Development Flow: Second Generation Lean Product Development" by Donald G. Reinertsen

It should be obvious that fast feedback improves the speed of learning. What may be less obvious is that fast feedback also increases the efficiency with which we generate information and learn new things. It does this by compressing the time between cause and effect. When this time is short, there are fewer extraneous signals that can introduce noise into our experiment.

Team New Zealand designed the yacht that won the America's Cup. When they tested improvements in keel designs, they used two virtually identical boats. The unimproved boat would sail against the improved boat to determine the effect of a design change. By sailing one boat against the other, they were able to discriminate small changes in performance very quickly.

In contrast, the competing American design team used a single boat supplemented by computer models and NASA wind tunnels. The Americans could never do comparison runs under truly identical conditions because the runs were separated in time ...

Team New Zealand completed many more cycles of learning, and they generated more information in each cycle. This ultimately enabled them to triumph over a much better funded American team. It is worth noting that Team New Zealand explicitly invested in a second boat to create this superior test environment. It is common that we must invest in creating a superior development environment in order to extract the smaller signals that come with fast feedback.

Thursday, November 17, 2011

Java Date encapsulation
When we design systems, we need to decide what it is (out of all the many different things we could do with dates) that we actually want to do in each instance. Then we should encapsulate just that functionality in our own classes, delegating where appropriate to java.util.Date, java.util.Calendar and java.text.DataFormat?. Importantly, we must decide how we compare, add and subtract intervals, and how we will display each type of date.
See for alternative

Haskell, lazy, dataflow, declarative


Steve Jobs: Life is made up
ONE LAST THING takes an unflinching look at Jobs's difficult, controlling disposition, and offers unique insights into what made him tick. While there has been near-universal agreement that Steve Jobs was a great innovator in business and technology, ONE LAST THING looks into why he was so great. What were the influences that shaped his character? What drove him from such humble beginnings to the heights of success?
In a never-before-broadcast interview from 1994, Jobs expounds on his philosophy of life: "You tend to get told that the world is the way it is, but life can be much broader once you discover one simple fact; and that is that everything around you that you call life was made up by people no smarter than you … Once you learn that, you'll never be the same again."

Software is art

Remote pairing using tmux/Skype with

A simple approach to remote pairing using tmux & Skype:

Saturday, November 12, 2011

Tuesday, September 27, 2011

Mind-map to Presentation

10 Tips to Do Presentations Like Me:
  1. Don’t Use Templates
  2. Use Presenter Notes
  3. Use Presenter View
  4. Learn Your PC
  5. Use Screenshots
  6. Do What You Said You’d Do
  7. Tidy up Those Transitions
  8. Rehearse
  9. Interact with the Audience
  10. It’s a Performance



Monday, September 26, 2011

Groovy Regex

// simple group demo
// You can also match a pattern that includes groups.  First create a matcher object,
// either using the Java API, or more simply with the =~ operator.  Then, you can index
// the matcher object to find the matches.  matcher[0] returns a List representing the
// first match of the regular expression in the string.  The first element is the string
// that matches the entire regular expression, and the remaining elements are the strings
// that match each group.
// Here's how it works:
def m = "foobarfoo" =~ /o(b.*r)f/
assert m[0] == ["obarf", "bar"]
assert m[0][1] == "bar"

XML & HTML Slurping with Groovy

XML Slurping String Content:
GPathResult feed = new XmlSlurper().parseText(source)
def rows = feed.FORM[0].TABLE[0].TR[1..-1]
// ...

Note: element names are case-sensitive

HTML Slurping with TagSoup:
slurper = new XmlSlurper(new org.ccil.cowan.tagsoup.Parser())
url = new URL("http://...")
url.withReader { reader ->
    def html = slurper.parse(reader)
    // ...

HTML Slurping with NekoHtml:
slurper = new XmlSlurper(new org.cyberneko.html.parsers.SAXParser())
new URL(url).withReader { reader ->
    def html = slurper.parse(reader)
    // ...

Friday, September 23, 2011

Cloning GitHub From Behind Proxy – Building Gradle From Source

How to clone a GitHub fork (Gradle in this case) from behind a proxy on a Windows machine without admin privileges…
  1. [Windows] Download Portable Git from:
  2. Follow instructions here:
  3. Using git-bash: export http_proxy=http://proxyuser:passwd@proxy:port
  4. Setup /h/.ssh/config as per:
    ProxyCommand /c/dev/opt/git/bin/connect.exe -H proxy:port %h %p
    User git
    Port 22
    IdentityFile "/<userhome>/.ssh/id_rsa"
    TCPKeepAlive yes
    PreferredAuthentications publickey
    IdentitiesOnly yes
    User git
    Port 443
    IdentityFile "/<userhome>/.ssh/id_rsa"
    TCPKeepAlive yes
    PreferredAuthentications publickey
    IdentitiesOnly yes 

  5. git clone gradle

    1. Provide proxy password
  6. Add the following to <project-dir>/
  7. If you have a hosted Maven repo, change gradle/build.gradle:
    ivy {
        // artifactPattern('[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]')

  8. [OPTIONAL] Proxy settings for For Ivy (not sure if this is necessary):

    • set ANT_OPTS=-Dhttp.proxyHost=proxy -Dhttp.proxyPort=port -Dhttp.proxyUserName=username -Dhttp.proxyPassword=password -Dhttps.proxyHost=proxy -Dhttps.proxyPort=port

  9. It seems minlog is in between com.esotericsoftware and com.googlecode, i.e. it’s moved but there’s still some old dependencies on esotericsoftware.  Download com.esotericsoftware:minlog:1.2 jar and pom, upload to your Maven repo, add repositories
    repositories {
        mavenRepo(url: 'http://myrepo/content/repositories/public')

Still working through some dependency issues attempting to build Gradle…