Wednesday, May 30, 2012

Geppetto: Puppet IDE (old but good example screenshots)

"As usual, the team ignored Management and did whatever they damn well pleased, working on Geppetto in secret through 15 long months of nights, weekends and vacations"

Thursday, May 24, 2012

Lifehacker: Focussing goals

Level 1: The Primary Goals

  • Base of all other goals - one or two things you aspire to do before you die - that will truly matter to you in 20 years. 
  • Everything above this bottom level should help you to one day complete these goals. 
  • Should only have two or three high stakes goals

Level 2: Long Term Goals

  • Major goals that are required to get to the primary goals. 
  • E.g. Sustainable habits you need to form over the years 
  • E.g. Achievements you want to reach

Level 3: Short Term Goals

  • Weeks or months out. 
  • Short term, not short-sighted. 
  • What you can do now to make primary goal happen?

Level 4: Recurring Goals

  • What you want to do daily/weekly/monthly regardless of what else is going on. 
  • These aren't quite the same as short-term goals because they're to form a habit. 
  • Ask yourself what you need to do on a daily basis to make that happen in the long run.

Level 5: Immediate Goals

  • Goals and to-dos that you can and want to accomplish right this second. 
  • Ever-changing but necessary part of pyramid - allows you to measure your daily duties to see how they have an affect on overall life goals. 
  • Could be as simple as making a phone call. 
  • How do they affect other aspects of the pyramid both positively and negatively?

Weed Out Junk and Accomplish Your Goals

  • "Life consists of what you pay attention to" - you can structure your goals the same way.
    • When you have too many goals conflicting with each other your attention is shifted too often. 
    • Trim away junk goals to get things done and find an actionable path.
  • The goal of the pyramid is to ensure that every aspect of your goals work together:
    • Can see where your ideas fail and aren't coalescing.

  1. Start at the bottom of your pyramid and draw lines up through goals that match each other. The line should move through each level and hit one or two different goals along the way.
  2. Do this with all your goals moving upwards through the pyramid.
  3. When you're done you'll probably have a few outliers scattered about. 
    1. Ask yourself: Why do I want this? 
    2. Does this relate to anything else I want? 
    3. If you don't have a good answer, cut them from the list. 
    4. If you want to keep goals then focus them to help you with another goal.
  4. Go back through your levels and see what goals you can outsource to other people.
Focus only on the goals that matter, break them into smaller steps, and start work immediately

Tuesday, May 22, 2012

HTTP Status Decision Graph

Just saw a demo of webmachine's decision graph by @TheColonial:

Inspired by:

Sunday, May 6, 2012

The Shirky Principle

"Institutions will try to preserve the problem to which they are the solution"

Thursday, May 3, 2012

Klint Finley: How an Established Company Can Start Doing DevOps

finding an discrete project that members of both the development and operations teams collaborate on
He suggests that developers who want to get started with DevOps start by reaching out to ops and getting themselves on the call list so they can see what sort of issues ops has to deal with every day. If you're in infrastructure, he suggests reaching out to development and getting in on some build meetings and learn about how that process works and the pressures they are under.
Starting with small, pilot programs and taking small specific steps are a great starting point. Even if a "pure" agile/DevOps methodology can never been adopted at a company, certain proven technologies like configuration management and practices like putting developers on the call rotation can improve productivity and collaboration.

Notes from Cloudera Hadoop talk

·         Scalable
·         Fault tolerant
·         OS
HDFS + MapReduce
HDFS: Like a normal Filesystem but Distributed (highly scalable)
MapReduce: Compute framework

Can process the data where it resides
Direct attached storage at individual nodes

·         Apps are written in high-level code
·         Share nothing architecture (between nodes)
·         Data is spread among machines in advance

Schema on-read
SerDe (Serialiser/Deserialiser)

DB: reads are fast, standards
Hadoop: loads are fast, flexibility

Enterprise DataWarehouse Arch
Sqoop & Flume --> Hadoop
Hive & ODBC
Oozie (work flow coordinator)
Sqoop out to BI reports

·         NameNode – holds all metadata for HDFS
·         Needs to be highly reliable machine (Raid 10, dual power supplies, dual network cards bonded)
·         More memory the better 48Gb-96Gb (depending on # of files/blocks)
·         v1.0 If NameNode disappears cluster is down
·         Secondary NameNode – check pointing for NameNode (same hardware)
·         DataNodes
·         Hardware will depend on specific
·         JBOD (just a bunch of disks), no raid, no SAN, no virtualisation
·         Maximise for IO throughput
·         Direct attached disk
·         Data on S3, EMR instances (maxed out by how quickly S3 can access EMR)
·         Pay a price for virtualisation

E.g. DataNode
·         12 3TB drives
·         2 4-6core CPUs
·         24-96Gb RAM

Job tracker takes job submitted to cluster
Task Tracker
·         take job and tries to run on same machine where data resides (90% possible)
·         works on multiple files
·         map task per block
·         split into multiple blocks
·         don’t want massive block - 2GB file split into smaller pieces
·         128kb chunks
Input is split and fed in parallel to the map tasks which process the data given to them and write immediate data o their local storage (intermediate data)
Shuffle and sort (network intensive)
Reduce tasks pick up data from the local storage of the map tasks relevant to the key range they have to perform the reduce task on They perform the reduce op and wwrite the output to HDFS

WriteOnceReadMany, e.g. log data, snapshot DB, not for system of record multiple writes
Can only append to file

Final Result
The, 1
Aardvark, 1
Cat, 1
Mat, 1
Aardvark, 1
Cat, 1
Mat, 1
On, 2
Sat, 2
Sofa, 1
The, 4
Aardvark, 1
Cat, 1
Mat, 1
On, 2
Sat, 2
Sofa, 1
The 4
Cat, 1
Sat, 1
On, 1
The, 1
Mat, 1
The, 1
Aardvark, 1
Sat, 1
On, 1
The, 1
Sofa, 1

SQL like language for wiring MapReduce Jobs
Supports SELECT, JOIN, GROUP BY, etc.
Can support very large datasets by allowing
        Partitioning sampling

Sqoop Sql to Hadoop
·         Pull data in, push data out
·         Connectors
·         Generates MR job to generate parallel load import or export to RDBMS

Bring in log (or other) data
Multi-agents talk to other agents
Log, twitter, avro, netcat, exec

Workflow/coordination service to magan data processing jobs for Hadoop
Chain jobs together (e.g. every 15mins)

Pipes and Streaming
Write native-code MR in C++ / arbitrary scripting languages
Stdin – stdout map à reduce à final result

Fuse – DFS
Allows mounting HDFS volumes via Linux FUSE FS

·         RealTime (not batch) column-oriented datastore (Modeled on BigTable)
·         Handles billions rows, petabytes
·         Facebook 1mil writes/s likes, 1.6mil/s insights, log data
·         Profile data in MySQL with memcache on top, moving to HBase
·         Gave up on developing Cassandra (eventual consistency better for static/write-only) for HBase
·         Mobile devices
·         Strong consistency
·         Not RDBMS: No joins, no indexes, no SQL
·         Columns can be added on the fly and store any kind of data
·         Keeps 3 versions of column cells, write-ahead log, sequential writes/reads -> append-type model
·         Johnathon Grey HadoopWorld videos
·         Row key only (not index) otherwise scan whole dataset

Cloudera’s Distribution enterprise-ready dist of Hadoop

Dataflow language for MapReduce

OpenTSDB (time series DB)
Interesting example - metrics for 1000s machines

YCSB (Yahoo Cloud Serving Benchmark)
·         Test cloud and bigdata technologies with real loads to demonstrate how they will perform before making expensive decisions with little data.

Real-time processing
·         Trigger-like “coprocessors” attached to data nodes

Wednesday, May 2, 2012

Elasticsearch vs SOLR

  • Want to have Compass distributed. 
  • Kimchy started working on a distributed Compass, by integrating with data grid solutions like GigaSpaces, Coherence and Terracotta. At its core, a distributed Lucene solution needs to be sharded. 
  • Also, with the advancement of HTTP and JSON as ubiquitous APIs, it means that a solution that many different systems with different languages can easily be used.
  • It has a very advance distributed model, speaks natively JSON, and exposes many advance search features, all seamlessly expressed through JSON DSL.
  • Solr is also a solution for exposing an indexing/search server over HTTP, but I would argue that ElasticSearch provides a much superior distributed model and ease of use.

Cloudera Hadoop

Download CDH3 VM:

Once you launch the VM, you are automatically logged in as the cloudera user.
The account details are:
  • username: cloudera
  • password: cloudera
The cloudera account has sudo privileges in the VM.
There is a Hadoop Tutorial available from the Cloudera website.
You can access status through the browser at the following URLs:
  • NameNode status (localhost:50070)
  • JobTracker status (localhost:50030)
  • The Hue user interface (localhost:8088)
  • The HBase web UI (localhost:60010)

Try some commands:

MongoDB write lock concurrency