Matt Callanan's Blog: 2011

Wednesday, December 28, 2011

Yow 2011 Brisbane

Program: http://yowaustralia.com.au/YOW2011/general/programBrisbane.html

Craig Smith's Wrap Up: http://cds43.wordpress.com/2011/12/25/yow-2011-australia-review/

Channel 9 Interviews: http://channel9.msdn.com/Tags/yow+2011

DevOps Reading/Viewing

Velocity 2010: Tom Cook, "A Day in the Life of Facebook Operations" [YouTube]

http://talkminer.com/viewtalk.jsp?videoid=T-Xr_PJdNmQ&q=
Recap (http://www.youtube.com/watch?v=T-Xr_PJdNmQ#t=39m30s)

Version control everything
Optimize early
automate, automate, automate
Use configuration management... no matter how many servers
Plan to fail
Instrument everything

Hardware, software, network, OS.... everything
Discover trends and causes you never knew of

Don't waste time on dumb stuff

ODS
Ganglia
Active users (http://www.youtube.com/watch?v=T-Xr_PJdNmQ#t=36m26s) :

per engineer: 1.1million
per ops: 2.3million

HipHop for PHP

Memcached

300+TB live data in RAM

MySQL

Linux (CentOS 5)

CFEngine

Enigineering + Operations (no QA)

· Engineers write, test, an deploy their own code

o Quickly make performance decisions

o Expose changes to subset of real traffic

· No ‘commit and quit’

· Deeply involved in moving services to production

· Ops ‘embedded’ into engineering teams

o Help make architecture decisions

o Better understand needs of product

o Interface with other ops teams

· Change logging – log all code changes

IRC

BitTorrent

Small teams

Facebook has over 60,000 servers: http://www.infoq.com/news/2011/01/facebook-coding-practices

InfoQ: Infrastructure as Code [InfoQ]
Panel: Theo Schlossnagle – OmniTI; Luke Kanies – Puppet Labs; Adam Jacob – Opscode; Erik Troan – rPath

Infrastructure Automation with Opscode Chef [pdf]
Joshua Timberman (Opscode, Inc.), Adam Jacob (Opscode), Christopher Brown (Opscode),Aaron Peterson (Opscode, Inc.), Matt Ray (Opscode), Seth Chisamore (Opscode, Inc.)

http://www.planetdevops.net/?m=201006

The 7 levels of continuous integration

Using Vagrant as a Team

https://rvm.beginrescueend.com/

http://me.andering.com/2011/02/03/server-login-considered-harmful/

Friday, December 23, 2011

DevOps: Implications of Infrastructure As Code

Stephen Nelson-Smith

http://www.agileweboperations.com/the-implications-of-infrastructure-as-code

I’m seeing a lot of problems emerging in organisations starting to put these kind of ideas into practice.
Here are a few symptoms:

Sprawling masses of Puppet code, duplication, contradiction and a lack of clear understanding of what it all does.
A fear of change – a sense that we dare not meddle with the manifests or recipes, because we’re not entirely certain how the system will behave.
Bespoke software that started off well-engineered, and thoroughly tested, littered with TODOs, and FIXME and quick hacks.
A sense that, despite the lofty goal of capturing the expertise required to understand an infrastructure in the code itself, if one or two key people were to leave, the organisation or team would be in trouble.

As a corrective, I think we need to start thinking about six key areas of our Infrastructure Code:

Design
Collective Ownership
Review
Standards
Testing
Refactoring

DevOps: CAMS

What Devops Means to Me
By John Willis
http://www.opscode.com/blog/2010/07/16/what-devops-means-to-me/

CAMS:

Culture
Automation
Measurement
Sharing

Thursday, December 22, 2011

DevOps: Building a DevOps Team

Brian Henerey: Building a DevOps Team
http://agilesysadmin.net/building-a-devops-team
http://agilesysadmin.net/buiding-a-devops-team-part-2

Eric Minick: Is Having a “DevOps Team” a Problem?
http://devopswire.com/profiles/blogs/is-having-a-devops-team-a-problem

[Patrick Debois] says that if you have a DevOps team you are doing it wrong. He later hedges with the idea that a team to help transition to DevOps is a good thing.

For better or worse, they [DevOps teams] may be to DevOps what Scrum was to Agile - an avenue to enterprise adoption that purists find a little unpalatable.

Wednesday, December 21, 2011

Dynamically Creating a Test Suite in JUnit 4 -- cpsuite

http://stackoverflow.com/questions/3257080/how-do-i-dynamically-create-a-test-suite-in-junit-4

http://johanneslink.net/projects/cpsuite.jsp

Regex and Lookahead/Lookbehind

http://www.regular-expressions.info/lookaround.html

E.g. to remove all @Component annotations from non-"Default" classes:

search for (negative lookahead):
@Component\npublic class (?!Default)

replace with:
public class

negative lookahead: q(?!u) matches a "q" not followed by a "u"

positive lookahead: q(?=u) matches a q that is followed by a u, without making the u part of the match

If you want to store the match of the regex inside a backreference, you have to put capturing parentheses around the regex inside the lookahead, like this:(?=(regex)).

negative lookbehind: (?<!a)b matches a "b" that is not preceded by an "a", using negative lookbehind.

positive lookbehind: (?<=a)b matches the b (and only the b) in cab

Thursday, December 15, 2011

Java XML XPath

Converted from Groovy: http://docs.codehaus.org/display/GROOVY/Reading+XML+with+Groovy+and+XPath

import org.apache.xml.dtm.ref.DTMNodeList;

import org.w3c.dom.Element;

import org.xml.sax.SAXException;

import javax.xml.parsers.DocumentBuilder;

import javax.xml.parsers.DocumentBuilderFactory;

import javax.xml.parsers.ParserConfigurationException;

import javax.xml.xpath.XPath;

import javax.xml.xpath.XPathConstants;

import javax.xml.xpath.XPathExpressionException;

import javax.xml.xpath.XPathFactory;

import java.io.ByteArrayInputStream;

import java.io.IOException;

import java.io.InputStream;

public class XPathTest extends TestCase {

private static final String CAR_RECORDS = "<records> <car name='HSV Maloo' make='Holden' year='2006'> <country>Australia</country> <record type='speed'>Production Pickup Truck with speed of 271kph</record> </car> <car name='P50' make='Peel' year='1962'> <country>Isle of Man</country> <record type='size'>Smallest Street-Legal Car at 99cm wide and 59 kg in weight</record> </car> <car name='Royale' make='Bugatti' year='1931'> <country>France</country> <record type='price'>Most Valuable Car at $15 million</record> </car> </records> ";

public void test() throws ParserConfigurationException, IOException, SAXException, XPathExpressionException {

DocumentBuilder builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();

InputStream inputStream = new ByteArrayInputStream(CAR_RECORDS.getBytes());

Element records = builder.parse(inputStream).getDocumentElement();

XPath xpath = XPathFactory.newInstance().newXPath();

DTMNodeList nodes = (DTMNodeList) xpath.evaluate("//car", records, XPathConstants.NODESET);

for (int i = 0; i < nodes.getLength(); i++) {

org.w3c.dom.Node node = nodes.item(i);

String make = xpath.evaluate("@make", node);

String country = xpath.evaluate("country/text()", node);

String type = xpath.evaluate("record/@type", node);

System.out.println(make + " of " + country + " has a " + type + " record");

}

Wednesday, December 7, 2011

Coffee with Robert Virding

Had a fascinating conversation over coffee with Robert Virding at Yow Conference Tuesday 6th December 2011... jotted down some of the things I remembered:

"erlc" produces a BEAM file which is portable byte code

Joe's Abstract Machine originally compiled Erlang to C macros which was then JIT translated to machine code on the fly
BEAM was a complete rewrite to produce portable byte code that "erl" translates to machine code on the fly

Erjang runs on the JVM so it has the advantage of many years and many people of JVM optimisation... Erlang has a team of 5 developers working on the VM.
Architecture of Erlang apps

Processes that do the work don't contain error handling... error handling is done where and when it matters by supervisor processes further up the tree
Supervisor processes are responsible for determining how to handle different types of errors
Some people on stack overflow freak out about this idea of letting processes crash but don't realise that approach goes hand-in-hand with the approach of handling errors higher up the process tree

Error Handling

Modular approach
Robert likes to code for all eventualities but often developers take an iterative approach... code the core functionality in processes... run the app, observe how different errors crash the processes... and work out how (and where) they should be captured and dealt with
Processes are organised in trees. A supervisor at the top is responsible for working out what to do when different types of errors occur
Typical approach is to let type errors crash the process

Erlang movie

Robert was the guy who "fixed the bug"
This patching code fixes while the app is running is possible through module loading
The app has to be specifically designed with this approach in mind
Have to be careful... what if the process is already running?
A module could (and typically does) contain both the server code and the client code used to access the server. So a module could contain code that is running in separate processes - a client process, a server process. Updating it on the fly is tricky.
Need to take into account data changes too... a mini data migration might be required on existing state... it's up to the module author what they're going to do
Sometimes the best approach is to crash the process and restart it rather than carry all that complexity in the code - depends on the scenario.
Also, restarting the entire app is an option...
Or using a load balancer type approach where the router directs the traffic to the new server.

Some companies use Erlang as essentially a gateway to processes running on other languages

e.g. a fast server server that accepts many concurrent connections and directs each connection to a backend C process.
Erlang NIFs (Natively Implemented Functions) are invoked in a similar manner to Java's JNI interface
Erlang can call into Java code using the JInterface

I remarked I was surprised that I couldn't call my own function from a guard and I understood this was because Erlang didn't want you doing side effects during a guard statement

Guards where really just an extension on pattern matching, enabling you to be a little more verbose

Can't create your own data types but can use a Binary type which lets you compose data out of other Erlang types.

Can reuse the same binary block with other binary types e.g. if your binary type has Int,String,Bool... you could reuse a variable of that binary type as a different type

Erjang

serious implementation of Erlang on JVM
Robert hopes good things come out of it
Tricky job though to keep up with Erlang as it is still evolving within Ericsson

Ericsson's OTP department maintain the Erlang language

Used mainly for testing
Surprising that a telecoms hardware company supported the language from infancy and there was initially some tension with others who didn't like it but the OTP team had sympathetic management

Was Erlang based on Prolog?

It was originally implemented on Prolog (eventually rewritten in Erlang itself)
The syntax was influenced a little by Prolog but logic programming wasn't the main thrust - it was more about solving telephony concurrency problems.

Where did the functional programming syntax come from?

Realised after a bit that the language needed some of the benefits of FP and so they were added

Do you find many type errors come out in runtime when a particular piece of code that wasn't exercised during testing is invoked due to the input of different data?

Not so much... there's a tool called the Dialyser that you run over your code... it actively tests all your types and tells you where there's a programming error in your code - it's able to catch most of the programming errors that could lead to a type error at runtime.

Hipe: http://www.it.uu.se/research/group/hipe/
Popularity of Erlang

It's been pretty steady... there was a bit of a peak in 2008/09

Saturday, December 3, 2011

Erlang setup

Install

Download and install from: the official Erlang website (http://learnyousomeerlang.com/introduction#what-you-need)

Compile & Run

http://learnyousomeerlang.com/modules
c:\erlangcode> erlc program.erl
c:\erlangcode> erl program
1> program:function().

Sunday, November 20, 2011

Displaying Groovy Classpath

http://marxsoftware.blogspot.com/2010/01/viewing-groovy-applications-classpath.html
this.class.classLoader.rootLoader.URLs.each{ println it }

http://blog.blindgaenger.net/print_groovys_classpath_for_debugging.html

def printClassPath(classLoader) {
println "$classLoader"
classLoader.getURLs().each {url->
println "- ${url.toString()}"
}
if (classLoader.parent) {
printClassPath(classLoader.parent)
}
}
printClassPath this.class.classLoader

Imaginary Boss

"I have an imaginary boss in my head. I'm still scared if I'm late coming in to work. Everybody else has someone to drive them, I had to find that discipline within myself."

Yoganathan Ratheesan of Lebara

http://www.smh.com.au/it-pro/business-it/virtual-phone-network-a-real-moneymaker-20111118-1nm28.html

git push putty disconnected error

PuTTY Fatal Error: Disconnected: No supported authentication methods available"

http://stackoverflow.com/questions/3431314/github-no-supported-authentication-methods-available

set GIT_SSH=c:\progra~2\git\bin\ssh.exe

Friday, November 18, 2011

Snap: Haskell Web Framework

http://snapframework.com/docs/quickstart
http://snapframework.com/docs/tutorials/snap-api

cabal update
cabal install snap -- installs into C:\Users\<user>\AppData\Roaming\cabal (or $HOME/.cabal/) (10mins)
mkdir snapblog
cd snapblog
snap init -- creates .ghci, snapblog.cabal, <src>, <log>, <resources>
cabal install -- installs executable snap (snapblog.exe) into C:\Users\<user>\AppData\Roaming\cabal\bin (or $HOME/.cabal/bin)
snapblog -p 8000

Needle: Automated tests for your CSS

http://needle.readthedocs.org/en/latest/index.html

On the goals of Coderetreat

Corey Haines Intro: http://vimeo.com/18955165

http://www.infoq.com/news/2011/11/global_day_of_code_retreat

"The Principles of Product Development Flow: Second Generation Lean Product Development" by Donald G. Reinertsen

http://www.startuplessonslearned.com/2009/07/principles-of-product-development-flow.html

It should be obvious that fast feedback improves the speed of learning. What may be less obvious is that fast feedback also increases the efficiency with which we generate information and learn new things. It does this by compressing the time between cause and effect. When this time is short, there are fewer extraneous signals that can introduce noise into our experiment.

Team New Zealand designed the yacht that won the America's Cup. When they tested improvements in keel designs, they used two virtually identical boats. The unimproved boat would sail against the improved boat to determine the effect of a design change. By sailing one boat against the other, they were able to discriminate small changes in performance very quickly.

In contrast, the competing American design team used a single boat supplemented by computer models and NASA wind tunnels. The Americans could never do comparison runs under truly identical conditions because the runs were separated in time ...

Team New Zealand completed many more cycles of learning, and they generated more information in each cycle. This ultimately enabled them to triumph over a much better funded American team. It is worth noting that Team New Zealand explicitly invested in a second boat to create this superior test environment. It is common that we must invest in creating a superior development environment in order to extract the smaller signals that come with fast feedback.

Thursday, November 17, 2011

Java Date encapsulation

http://c2.com/cgi/wiki?JavaUtilDate

When we design systems, we need to decide what it is (out of all the many different things we could do with dates) that we actually want to do in each instance. Then we should encapsulate just that functionality in our own classes, delegating where appropriate to java.util.Date, java.util.Calendar and java.text.DataFormat?. Importantly, we must decide how we compare, add and subtract intervals, and how we will display each type of date.

See http://joda-time.sourceforge.net for alternative

Haskell, lazy, dataflow, declarative

http://en.wikipedia.org/wiki/Haskell_(programming_language)
http://en.wikipedia.org/wiki/Non-strict_programming_language
http://en.wikipedia.org/wiki/Lazy_evaluation
http://en.wikipedia.org/wiki/Dataflow
https://github.com/larrytheliquid/dataflow
http://en.wikipedia.org/wiki/Declarative_programming
http://en.wikipedia.org/wiki/Comparison_of_programming_paradigms

Steve Jobs: Life is made up

http://www.cultofmac.com/127707/reminder-watch-the-steve-jobs-one-last-thing-documentary-tonight-on-pbs-and-channel-4-in-the-uk/

ONE LAST THING takes an unflinching look at Jobs's difficult, controlling disposition, and offers unique insights into what made him tick. While there has been near-universal agreement that Steve Jobs was a great innovator in business and technology, ONE LAST THING looks into why he was so great. What were the influences that shaped his character? What drove him from such humble beginnings to the heights of success?

In a never-before-broadcast interview from 1994, Jobs expounds on his philosophy of life: "You tend to get told that the world is the way it is, but life can be much broader once you discover one simple fact; and that is that everything around you that you call life was made up by people no smarter than you … Once you learn that, you'll never be the same again."

Software is art

http://java.dzone.com/articles/software-art

Remote pairing using tmux/Skype with rubypair.com

A simple approach to remote pairing using tmux & Skype: http://rubysource.com/loccasions-pair-programming/

Monday, November 14, 2011

How to make your own compiler

language agnostic - Learning to write a compiler - Stack Overflow
http://matt.might.net/articles/compiling-up-to-lambda-calculus/

http://research.microsoft.com/en-us/um/people/simonpj/papers/pj-lester-book/
http://pages.cs.wisc.edu/~cs536-1/notes/1.overview.html

Saturday, November 12, 2011

Testing dynamic typed code can detect much wider range of errors?

http://en.wikipedia.org/wiki/Static_typing#Dynamic_typing

the testing done to ensure correct program operation can detect a much wider range of errors than static type-checking

Saturday, October 22, 2011

Language designer's notebook: First, do no harm

Language designer's notebook: First, do no harm
Use map-reduce to sum size of all the elements rather than modify local sum variable from within a closure

Functional thinking: Coupling and composition

Neal Ford: Functional thinking: Coupling and composition, Part 1
Neal Ford: Functional thinking: Coupling and composition, Part 2

Tuesday, October 11, 2011

Generics, HAML, and LINQ

Generics: Erasure vs Reification - http://beust.com/weblog/2011/07/29/erasure-vs-reification/

HAML Tutorial - http://haml-lang.com/tutorial.html

LINQ for Java - https://github.com/nicholas22/jpropel-light#readme

Saturday, October 8, 2011

"Free Software" doesn't mean "Open Source Software"

http://www.gnu.org/philosophy/open-source-misses-the-point.html

Tuesday, September 27, 2011

Monday, September 26, 2011

Groovy Regex

http://groovy.codehaus.org/Regular+Expressions

// simple group demo
// You can also match a pattern that includes groups.  First create a matcher object,
// either using the Java API, or more simply with the =~ operator.  Then, you can index
// the matcher object to find the matches.  matcher[0] returns a List representing the
// first match of the regular expression in the string.  The first element is the string
// that matches the entire regular expression, and the remaining elements are the strings
// that match each group.
// Here's how it works:
def m = "foobarfoo" =~ /o(b.*r)f/
assert m[0] == ["obarf", "bar"]
assert m[0][1] == "bar"

XML & HTML Slurping with Groovy

XML Slurping String Content:
GPathResult feed = new XmlSlurper().parseText(source)
def rows = feed.FORM[0].TABLE[0].TR[1..-1]
// ...

Note: element names are case-sensitive

HTML Slurping with TagSoup:
slurper = new XmlSlurper(new org.ccil.cowan.tagsoup.Parser())
url = new URL("http://...")
url.withReader { reader ->
    def html = slurper.parse(reader)
    // ...
}

HTML Slurping with NekoHtml:
slurper = new XmlSlurper(new org.cyberneko.html.parsers.SAXParser())
new URL(url).withReader { reader ->
    def html = slurper.parse(reader)
    // ...
}

Friday, September 23, 2011

Cloning GitHub From Behind Proxy – Building Gradle From Source

How to clone a GitHub fork (Gradle in this case) from behind a proxy on a Windows machine without admin privileges…

[Windows] Download Portable Git from: http://code.google.com/p/msysgit/downloads/list?can=3
Follow instructions here: http://help.github.com/win-set-up-git/
Using git-bash: export http_proxy=http://proxyuser:passwd@proxy:port
Setup /h/.ssh/config as per: http://stackoverflow.com/questions/5103083/problem-of-testing-ssh-in-git-behind-proxy-on-window-7
```
ProxyCommand /c/dev/opt/git/bin/connect.exe -H proxy:port %h %p

Host github.com
User git
Port 22
Hostname github.com
IdentityFile "/<userhome>/.ssh/id_rsa"
TCPKeepAlive yes
PreferredAuthentications publickey
IdentitiesOnly yes

Host ssh.github.com
User git
Port 443
Hostname ssh.github.com
IdentityFile "/<userhome>/.ssh/id_rsa"
TCPKeepAlive yes
PreferredAuthentications publickey
IdentitiesOnly yes 
```
git clone https://mattcallanan@github.com/mattcallanan/gradle.git gradle
1. Provide proxy password
Add the following to <project-dir>/gradle.properties:
    systemProp.http.proxyHost=proxy
    systemProp.http.proxyPort=port
    systemProp.http.proxyUser=username
    systemProp.http.proxyPassword=password
    systemProp.http.nonProxyHosts=myrepo|localhost
If you have a hosted Maven repo, change gradle/build.gradle:
```
ivy {
    // artifactPattern('http://repo.jfrog.org/artifactory/gradle-plugins-snapshots/[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]')
    artifactPattern('http://myrepo/content/repositories/public/[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]')
    ...
} 
```
[OPTIONAL] Proxy settings for For Ivy (not sure if this is necessary):

set ANT_OPTS=-Dhttp.proxyHost=proxy -Dhttp.proxyPort=port -Dhttp.proxyUserName=username -Dhttp.proxyPassword=password -Dhttps.proxyHost=proxy -Dhttps.proxyPort=port

It seems minlog is in between com.esotericsoftware and com.googlecode, i.e. it’s moved but there’s still some old dependencies on esotericsoftware. Download com.esotericsoftware:minlog:1.2 jar and pom, upload to your Maven repo, add repositories
```
repositories {
    ...
    mavenCentral()
    mavenRepo(url: 'http://myrepo/content/repositories/public')
    ...
}
```

Still working through some dependency issues attempting to build Gradle…