Wednesday, December 28, 2011

Yow 2011 Brisbane

Program: http://yowaustralia.com.au/YOW2011/general/programBrisbane.html

Craig Smith's Wrap Up: http://cds43.wordpress.com/2011/12/25/yow-2011-australia-review/

Channel 9 Interviews: http://channel9.msdn.com/Tags/yow+2011

Posted by on

DevOps Reading/Viewing

Velocity 2010: Tom Cook, "A Day in the Life of Facebook Operations" [YouTube]


HipHop for PHP
Memcached
300+TB live data in RAM
MySQL
Linux (CentOS 5)
CFEngine
Enigineering + Operations (no QA)
·         Engineers write, test, an deploy their own code
o   Quickly make performance decisions
o   Expose changes to subset of real traffic
·         No ‘commit and quit’
·         Deeply involved in moving services to production
·         Ops ‘embedded’ into engineering teams
o   Help make architecture decisions
o   Better understand needs of product
o   Interface with other ops teams
·         Change logging – log all code changes
IRC
BitTorrent
Small teams

Facebook has over 60,000 servers: http://www.infoq.com/news/2011/01/facebook-coding-practices


InfoQ: Infrastructure as Code [InfoQ]
Panel: Theo Schlossnagle – OmniTI; Luke Kanies – Puppet Labs; Adam Jacob – Opscode; Erik Troan – rPath

Infrastructure Automation with Opscode Chef [pdf]
Joshua Timberman (Opscode, Inc.), Adam Jacob (Opscode), Christopher Brown (Opscode),Aaron Peterson (Opscode, Inc.), Matt Ray (Opscode), Seth Chisamore (Opscode, Inc.)


http://www.planetdevops.net/?m=201006


The 7 levels of continuous integration






Posted by on

Friday, December 23, 2011

DevOps: Implications of Infrastructure As Code

Stephen Nelson-Smith

http://www.agileweboperations.com/the-implications-of-infrastructure-as-code


I’m seeing a lot of problems emerging in organisations starting to put these kind of ideas into practice.
Here are a few symptoms:
  • Sprawling masses of Puppet code, duplication, contradiction and a lack of clear understanding of what it all does.
  • A fear of change – a sense that we dare not meddle with the manifests or recipes, because we’re not entirely certain how the system will behave.
  • Bespoke software that started off well-engineered, and thoroughly tested, littered with TODOs, and FIXME and quick hacks.
  • A sense that, despite the lofty goal of capturing the expertise required to understand an infrastructure in the code itself, if one or two key people were to leave, the organisation or team would be in trouble.
As a corrective, I think we need to start thinking about six key areas of our Infrastructure Code:
  • Design
  • Collective Ownership
  • Review
  • Standards
  • Testing
  • Refactoring


Posted by on

DevOps: CAMS

What Devops Means to Me
By John Willis
http://www.opscode.com/blog/2010/07/16/what-devops-means-to-me/

CAMS:

  • Culture
  • Automation
  • Measurement
  • Sharing


Posted by on

Thursday, December 22, 2011

DevOps: Building a DevOps Team

Brian Henerey: Building a DevOps Team
http://agilesysadmin.net/building-a-devops-team
http://agilesysadmin.net/buiding-a-devops-team-part-2


Eric Minick: Is Having a “DevOps Team” a Problem?
http://devopswire.com/profiles/blogs/is-having-a-devops-team-a-problem


[Patrick Debois] says that if you have a DevOps team you are doing it wrong. He later hedges with the idea that a team to help transition to DevOps is a good thing.


For better or worse, they [DevOps teams] may be to DevOps what Scrum was to Agile - an avenue to enterprise adoption that purists find a little unpalatable.
Posted by on

Wednesday, December 21, 2011

Dynamically Creating a Test Suite in JUnit 4 -- cpsuite


 

http://stackoverflow.com/questions/3257080/how-do-i-dynamically-create-a-test-suite-in-junit-4

 

http://johanneslink.net/projects/cpsuite.jsp

 

Posted by on

Regex and Lookahead/Lookbehind

http://www.regular-expressions.info/lookaround.html

E.g. to remove all @Component annotations from non-"Default" classes:

search for (negative lookahead):
@Component\npublic class (?!Default)

replace with:
public class


negative lookahead: q(?!u) matches a "q" not followed by a "u"


positive lookahead:  q(?=u) matches a q that is followed by a u, without making the u part of the match


If you want to store the match of the regex inside a backreference, you have to put capturing parentheses around the regex inside the lookahead, like this:(?=(regex)). 


negative lookbehind: (?<!a)b matches a "b" that is not preceded by an "a", using negative lookbehind.


positive lookbehind: (?<=a)b matches the b (and only the b) in cab
Posted by on

Thursday, December 15, 2011

Java XML XPath


Converted from Groovy: http://docs.codehaus.org/display/GROOVY/Reading+XML+with+Groovy+and+XPath
 
import org.apache.xml.dtm.ref.DTMNodeList;
import org.w3c.dom.Element;
import org.xml.sax.SAXException;
 
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathExpressionException;
import javax.xml.xpath.XPathFactory;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
 
public class XPathTest extends TestCase {
    private static final String CAR_RECORDS = "<records>       <car name='HSV Maloo' make='Holden' year='2006'>         <country>Australia</country>         <record type='speed'>Production Pickup Truck with speed of 271kph</record>       </car>       <car name='P50' make='Peel' year='1962'>         <country>Isle of Man</country>         <record type='size'>Smallest Street-Legal Car at 99cm wide and 59 kg in weight</record>       </car>       <car name='Royale' make='Bugatti' year='1931'>         <country>France</country>         <record type='price'>Most Valuable Car at $15 million</record>       </car>     </records> ";
 
    public void test() throws ParserConfigurationException, IOException, SAXException, XPathExpressionException {
        DocumentBuilder builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
        InputStream inputStream = new ByteArrayInputStream(CAR_RECORDS.getBytes());
        Element records = builder.parse(inputStream).getDocumentElement();
        XPath xpath = XPathFactory.newInstance().newXPath();
        DTMNodeList nodes = (DTMNodeList) xpath.evaluate("//car", records, XPathConstants.NODESET);
        for (int i = 0; i < nodes.getLength(); i++) {
            org.w3c.dom.Node node = nodes.item(i);
            String make = xpath.evaluate("@make", node);
            String country = xpath.evaluate("country/text()", node);
            String type = xpath.evaluate("record/@type", node);
            System.out.println(make + " of " + country + " has a " + type + " record");
        }
    }
 
}

Posted by on

Wednesday, December 7, 2011

Coffee with Robert Virding

Had a fascinating conversation over coffee with Robert Virding at Yow Conference Tuesday 6th December 2011... jotted down some of the things I remembered:
  • "erlc" produces a BEAM file which is portable byte code
    • Joe's Abstract Machine originally compiled Erlang to C macros which was then JIT translated to machine code on the fly
    • BEAM was a complete rewrite to produce portable byte code that "erl" translates to machine code on the fly
  • Erjang runs on the JVM so it has the advantage of many years and many people of JVM optimisation... Erlang has a team of 5 developers working on the VM.
  • Architecture of Erlang apps
    • Processes that do the work don't contain error handling... error handling is done where and when it matters by supervisor processes further up the tree
    • Supervisor processes are responsible for determining how to handle different types of errors
    • Some people on stack overflow freak out about this idea of letting processes crash but don't realise that approach goes hand-in-hand with the approach of handling errors higher up the process tree
  • Error Handling
    • Modular approach
    • Robert likes to code for all eventualities but often developers take an iterative approach... code the core functionality in processes... run the app, observe how different errors crash the processes... and work out how (and where) they should be captured and dealt with
    • Processes are organised in trees.  A supervisor at the top is responsible for working out what to do when different types of errors occur
    • Typical approach is to let type errors crash the process
  • Erlang movie 
    • Robert was the guy who "fixed the bug"
    • This patching code fixes while the app is running is possible through module loading
    • The app has to be specifically designed with this approach in mind
    • Have to be careful... what if the process is already running?  
    • A module could  (and typically does) contain both the server code and the client code used to access the server.  So a module could contain code that is running in separate processes - a client process, a server process.  Updating it on the fly is tricky.
    • Need to take into account data changes too... a mini data migration might be required on existing state... it's up to the module author what they're going to do
    • Sometimes the best approach is to crash the process and restart it rather than carry all that complexity in the code - depends on the scenario.
    • Also, restarting the entire app is an option... 
    • Or using a load balancer type approach where the router directs the traffic to the new server.
  • Some companies use Erlang as essentially a gateway to processes running on other languages
    • e.g. a fast server server that accepts many concurrent connections and directs each connection to a backend C process.
    • Erlang NIFs (Natively Implemented Functions) are invoked in a similar manner to Java's JNI interface
    • Erlang can call into Java code using the JInterface
  • I remarked I was surprised that I couldn't call my own function from a guard and I understood this was because Erlang didn't want you doing side effects during a guard statement
    • Guards where really just an extension on pattern matching, enabling you to be a little more verbose
  • Can't create your own data types but can use a Binary type which lets you compose data out of other Erlang types.
    • Can reuse the same binary block with other binary types e.g. if your binary type has Int,String,Bool... you could reuse a variable of that binary type as a different type
  • Erjang
    • serious implementation of Erlang on JVM
    • Robert hopes good things come out of it
    • Tricky job though to keep up with Erlang as it is still evolving within Ericsson
  • Ericsson's OTP department maintain the Erlang language
    • Used mainly for testing
    • Surprising that a telecoms hardware company supported the language from infancy and there was initially some tension with others who didn't like it but the OTP team had sympathetic management
  • Was Erlang based on Prolog?
    • It was originally implemented on Prolog (eventually rewritten in Erlang itself)
    • The syntax was influenced a little by Prolog but logic programming wasn't the main thrust - it was more about solving telephony concurrency problems.
  • Where did the functional programming syntax come from?
    • Realised after a bit that the language needed some of the benefits of FP and so they were added
  • Do you find many type errors come out in runtime when a particular piece of code that wasn't exercised during testing is invoked due to the input of different data?
    • Not so much... there's a tool called the Dialyser that you run over your code... it actively tests all your types and tells you where there's a programming error in your code - it's able to catch most of the programming errors that could lead to a type error at runtime.
  • Hipe:  http://www.it.uu.se/research/group/hipe/
  • Popularity of Erlang
    • It's been pretty steady... there was a bit of a peak in 2008/09


Posted by on

Saturday, December 3, 2011

Erlang setup

Install



Compile & Run


Posted by on
Subscribe to: Posts (Atom)