tag:blogger.com,1999:blog-62878137037602217282024-03-13T04:56:13.505-07:00Matt Callanan's BlogSnippets and thoughts from a passionate software developer interested in all things DevOps, Continuous Delivery, Functional Programming, Distributed Systems. Also on Twitter as @mcallana.Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comBlogger363125tag:blogger.com,1999:blog-6287813703760221728.post-71481164619922928782015-10-03T22:41:00.001-07:002015-10-12T15:06:08.244-07:00c3vis: Finding Your Docker Containers on Amazon ECS Clusters<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">I've just created my first open source project: <a href="https://github.com/ExpediaDotCom/c3vis">c3vis</a> (Cloud Container Cluster Visualizer); and released it under the auspices of Expedia's OSS contribution program. It aims to give administrators and developers a simple interface to visualise how Docker containers have been deployed to Amazon's ECS service (for more information on ECS see </span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><a href="http://www.infoq.com/articles/intro-aws-ecs">Intro to AWS ECS</a> and </span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><a href="http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html">Under the Hood of the Amazon EC2 Container Service</a>).</span><br />
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span>
<br />
<h3>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">The Problem</span></h3>
<div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Deploying software as containers promises to solve many problems with regards to interoperability of environments, speed to deploy, and cost reduction. But understanding where our software lives now becomes more difficult both for development and operations teams. This is due to the fact that it is quite laborious to find the information indicating where the software is now located and the quantity of resources still available for more software. Several ECS console screens must be viewed, and the amount of time required to process this information grows with the amount of software deployed. </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<h3>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">c3vis</span></h3>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">c3vis aims to give administrators and teams one place to gain rapid insight into the state of where the containers are running and the capacity available for more containers.</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://raw.githubusercontent.com/ExpediaDotCom/c3vis/master/docs/graph.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="148" src="https://raw.githubusercontent.com/ExpediaDotCom/c3vis/master/docs/graph.png" width="320" /></a></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<h3>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">How it Works</span></h3>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">c3vis is a NodeJS server that retrieves information about the Instances and Tasks for the selected cluster from the ECS API (using the AWS JavaScript SDK). The client processes this information using D3.js to display the EC2 instances in the selected cluster as vertical bars. The Tasks allocated to the instances are represented as stacked boxes indicating their reserved memory. Each unique Task Definition is represented as a different colour, with the legend showing the Task Family name and revision number. Each Task will contain one or more containers, the size of the Task box represents accumulated reserved memory for all containers in the Task.</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<h3>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Why I Created it</span></h3>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">At Expedia we're using Amazon ECS to serve highly available clusters of applications. ECS helps to save costs and speed up deploys (traditionally done with blue-green deploys to clusters of dedicated immutable instances, requiring over-allocation of resources and time-consuming AMI creation). <a href="https://www.iocaine.org/posts/amazon-ecs-wishlist.html">My colleague</a> and I have been automating deploys of microservices to ECS. Diagnosing problems post-deploy is time-consuming: where was the container installed</span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">? why don't I have enough resources? </span></div>
<div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Having a single console that summarises ECS clusters visually has been helpful for 1) quickly identifying where to perform further diagnosis, 2) </span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">quickly estimating whether the current instance types will suffice as the cluster grows</span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">, and 2) explaining concepts to development teams that are on-boarding to ECS.</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<h3>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Future Directions</span></h3>
<div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Here are some enhancement ideas for c3vis:</span></div>
<div>
<ul>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Multi-region, multi-account support</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Represent reserved CPU in addition to memory</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Show an exploded view of task with more details:</span></li>
<ul>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Show containers within tasks</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Show memory breakdown across containers</span></li>
</ul>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Sliding time-bar to see historical data for comparison of cluster state</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Show container actual memory utilisation vs reserved memory utilisation</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Show Service names (toggle-able with Task names)</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Support more than 100 instances</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Write a plugin system that lets adopters plugin visualisations of their own statistics from favourite monitoring tool</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Cache responses server-side to reduce AWS API calls</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Make the data transfer between client and server more efficient - Separate requests for task and instance data and populate graph asynchronously</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Pluggable backend system that could support other public or private cloud providers</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Provide access to more troubleshooting information (such as docker logs, ECS logs)</span></li>
</ul>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
</div>
</div>
<h3>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Amazon ECS Data Model Limitations</span></h3>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The ECS data model has made getting the information needed for c3vis difficult. Looking forward to hopefully seeing more two-way relationships between ECS data types:</span></div>
<div>
<ul>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Mapping from ContainerInstance to Tasks - currently have to traverse all Tasks and group by InstanceId.</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Mapping from Task to Service - currently have to traverse all Services and group by Task ARN.</span></li>
</ul>
</div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<h3>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Helping Out</span></h3>
<div style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
Feedback and pull requests are welcome, c3vis is licensed under Apache 2.0.</div>
</div>
<div style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
<br /></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="https://github.com/ExpediaDotCom/c3vis">https://github.com/ExpediaDotCom/c3vis</a></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-701975110428117562015-06-03T19:33:00.001-07:002015-06-03T19:36:35.567-07:00Updating Definition of Done for Continuous Delivery and DevOps<br />
Agile brought us the idea of the "definition of done" - basically team members must have a shared understanding of what it means for work to be complete (see <a href="http://www.scrumguides.org/scrum-guide.html#artifact-transparency-done">scrumguides</a> for a good explanation).<br />
<br />
But how does Continuous Delivery and DevOps change this? Now that we've brought the scope of agile out to the last mile, now that we're including operations in the picture?<br />
<br />
The "Continuous Delivery" book says: "done" meaning released into production is the ideal situation for a software development project" (page 27)<br />
<br />
<a href="http://blogs.versionone.com/agile_management/2015/04/30/what-done-looks-like-in-devops/">Steve Ropa</a> on April 30, 2015 says that "no story is done until the code has been written, and the peer review has happened, the tests are all passing, and the hooks for monitoring, performance evaluation and health checks are in as well", "you have to be prepared for when the code meets the real world", and "you have to find out before the customer does if something is not working right".<br />
<br />
He suggests expanding stories to incorporate DevOps in their definition of DONE:<br />
<br />
<ol>
<li>Requirements that enable you to understand how the code is doing after deployment. </li>
<li>Estimations based on what it means to have the code in a state that will be successful post-deployment. </li>
<li>Everyone is a developer/tester/operations.</li>
</ol>
<br />
<br />
<a href="http://www.datical.com/updating-the-definition-of-done-for-devops/">Rex Morrow</a> on May 1, 2015 says we need to update it to also include database changes.<br />
<br />
<a href="http://www.krisbuytaert.be/blog/what-done-really-looks-devops">Kris Buytaert</a> on May 2, 2015 says he already updated it in 2013 to mean it's not done until the app is decommissioned (or all end-users are in their graves).<br />
<br />
<a href="http://www.agileweboperations.com/devops-anti-patterns">Matthias Marschall</a> has a good definition of what "done" is <i>not:</i><br />
<br />
<ul>
<li>For developers, <b>committing a piece of code is far from being done</b>. It needs to work in all kinds of weird use cases. And it’s not only QA’s job to find all the bugs. Good developers want to ensure that the new features are not only coded, but tested and ultimately released to their users. Only then the task is really Done.</li>
<li>For sysadmins, <b>having a nice script on your own box is not enough</b>. Every sysadmin needs to make sure it’s possible to re-create each part of the infrastructure at any time. When that slick, new script is under version control, written in a way others can understand and modify it, is their task really Done.</li>
</ul>
<br />
<br />
Essentially, it's up to each agile team to come up with the definition that makes sense for them. And we need to include Operations in the picture. Our Definition of Done needs to incorporate a feedback loop from production.<br />
<br />
<br />Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-8682363513461514452015-02-22T14:42:00.002-08:002015-02-22T14:59:16.167-08:00Resources for David Logan's "Tribal Leadership"<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">I stumbled on David Logan's <b>"Tribal Leadership: Leveraging Natural Groups to Build a Thriving Organization" </b>recently (a</span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; font-size: x-small;">vailable on </span><a href="http://www.amazon.com/Tribal-Leadership-Leveraging-Thriving-Organization/dp/0061251321" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; font-size: small;">amazon.com</a><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; font-size: x-small;">). It's an interesting way to categorise stages of culture within teams/organisations and identify how to improve the culture for the mutual benefit of the tribe members. </span><br />
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; font-size: x-small;"><br /></span>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; font-size: x-small;">My quick understanding is that moving team members to a place where they invest in each other instead of looking out for themselves improves the culture of the whole tribe.</span><br />
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; font-size: x-small;"><br /></span>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; font-size: x-small;">Below are some useful resources.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://emergentbydesign.com/2012/06/28/a-step-by-step-guide-to-tribal-leadership-part-1-the-five-stages-of-tribal-culture/"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><img border="0" height="237" src="https://technologybubbles.files.wordpress.com/2012/06/screen-shot-2012-06-28-at-12-21-22-pm.png" width="320" /></span></a></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><br /></span>
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">Quick Overview</span></b><br />
<a href="http://emergentbydesign.com/2012/06/28/a-step-by-step-guide-to-tribal-leadership-part-1-the-five-stages-of-tribal-culture/"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">http://emergentbydesign.com/2012/06/28/a-step-by-step-guide-to-tribal-leadership-part-1-the-five-stages-of-tribal-culture/</span></a><br />
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><br /></span></div>
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">In-depth Book Summary</span></b><br />
<a href="http://wiki.dsoglobal.org/t/library/tribal_leadership/summary"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">http://wiki.dsoglobal.org/t/library/tribal_leadership/summary</span></a><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><br /></span>
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">Cheat Sheet on how to "level up"</span></b><br />
<a href="http://finding-marbles.com/2012/02/29/tribal-leadership-the-1-page-cheat-sheet/"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">http://finding-marbles.com/2012/02/29/tribal-leadership-the-1-page-cheat-sheet/</span></a><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><b>TED Talk</b> from 2009 (16min)</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><a href="http://www.ted.com/talks/david_logan_on_tribal_leadership">http://www.ted.com/talks/david_logan_on_tribal_leadership</a></span><br />
<br />
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">Stage 3 is common among really smart people</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">Hardest thing is moving from Stage 3 to Stage 4</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">Stage 4 Example: Zappos</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">Stage 5 Example: South Africa Reconciliation Council (Desmond Tutu)</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">Distribution of Employed Tribes:</span></li>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">Stage 1: 2%</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">Stage 2: 25%</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">Stage 3: 48%</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">Stage 4: 22%</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">Stage 5: 2%</span></li>
</ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">You need to talk all 5 tribe stages because people are at all 5 stages</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">Tribes can only hear one level above or below where they are - leaders need to nudge them up</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">Stage 3 to 4: Find someone you don't know and someone else you don't know and connect them - connect them to something greater than themselves</span></li>
</ul>
Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-23836774866738904262015-02-09T18:18:00.001-08:002015-02-09T18:18:23.794-08:00Nathan Marz: Suffering-Oriented Programming<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">Just read "<a href="http://nathanmarz.com/blog/suffering-oriented-programming.html">Suffering-Oriented Programming</a>" - a blog post from Nathan Marz (creator of <a href="https://storm.apache.org/">Storm</a>). The central idea is that you need to understand a problem well before you try to solve it properly - especially before you build generic abstractions to solve multiple use cases. Beyond the ideas of <a href="https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it">YAGNI</a> and <a href="https://en.wikipedia.org/wiki/Minimum_viable_product">MVP</a>, the post outlines 3 distinct phases that are helpful to think about when setting out to tackle a tricky problem: 1) make it possible, 2) make it beautiful, 3) make it fast.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">It has some great takeaway quotes:</span><br />
<br />
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><span style="color: #323229; line-height: 20px;">don't build technology unless you <b>feel the pain of not having it</b></span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><span style="color: #323229; line-height: 20px;">reduce risk by ensuring that you're <b>always working on something important </b></span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><span style="color: #323229; line-height: 20px;"><b><span style="font-weight: normal;">ensure that you are </span>well-versed in a problem space<span style="font-weight: normal;"> before attempting a large investment.</span></b></span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><span style="color: #323229; line-height: 20px;">When encountering a problem domain with which you're unfamiliar, it's a <b>mistake</b> to try to build a "<b>general</b>" or "<b>extensible</b>" <b>solution</b> right off the bat</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><span style="color: #323229; line-height: 20px;"><b>develop a "map" of the problem space</b> as you explore it by <b>hacking things out</b></span></span></li>
<li><span style="color: #323229; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><span style="line-height: 20px;">The key to developing the "beautiful" solution is figuring out the <strong>simplest</strong> <b>set of abstractions</b> that solve the <b>concrete use cases</b> you already have. </span></span></li>
<li><span style="color: #323229; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><span style="line-height: 20px;">Don't try to <b>anticipate use cases</b> you don't actually have or else you'll end up <b>overengineering</b> your solution.</span></span></li>
<li><span style="color: #323229; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><span style="line-height: 20px;">development of beautiful abstractions is similar to <b>statistical regression</b>: you have a set of points on a graph (your use cases) and you're <b>looking for the simplest curve</b> that fits those points (a set of abstractions).</span></span></li>
<li><span style="color: #323229; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><span style="line-height: 20px;">most important... is a <b>relentless focus on refactoring... </b></span></span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><span style="color: #323229; line-height: 20px;">to <b>prevent accidental complexity</b></span><span style="color: #323229; line-height: 20px;"> from <b>sabotaging</b> the codebase</span></span></li>
<li><span style="color: #323229; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><span style="line-height: 20px;"><b>Use cases are ... worth their weight in gold</b>. The only way to acquire use cases is through <b>gaining experience</b> through hacking.</span></span></li>
<li><span style="color: #323229; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;"><span style="line-height: 20px;">attempts to make things <b>generic without a deep understanding of the problem domain</b> will lead to <b>complexity and waste</b>. Designs must always be <b>driven by real, tangible use cases</b>.</span></span></li>
</ul>
<div>
<span style="color: #323229; font-family: Verdana, Lucida Grande, Arial, sans-serif;"><span style="font-size: 14px; line-height: 20px;"><br /></span></span></div>
Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-5251896062813976962015-01-18T02:22:00.000-08:002015-01-18T02:22:11.249-08:00Great AWS Tips<div class="MsoNormal" style="margin-bottom: 12.0pt; mso-layout-grid-align: none; mso-pagination: none; text-autospace: none;">
</div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span style="font-size: 13.1428575515747px;">Came across some great AWS tips (care of <a href="http://www.devopsweekly.com/">Devops Weekly</a>)</span></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span style="font-size: 13.1428575515747px;"><br /></span></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span style="font-size: 13.1428575515747px;">The list is below - go to this website for the full details: </span></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span style="font-size: 13.1428575515747px;"><a href="https://wblinks.com/notes/aws-tips-i-wish-id-known-before-i-started/"><b>https://wblinks.com/notes/aws-tips-i-wish-id-known-before-i-started/</b></a></span></span></div>
<div>
<span style="color: #134089; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span style="font-size: 13.1428575515747px;"><br /></span></span></div>
<ul>
<li><span style="color: #134089; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Application Development</span></li>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Store no application state on your servers.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Store extra information in your logs.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">If you need to interact with AWS, use the SDK for your language.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Have tools to view application logs.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
</ul>
<li><span style="color: #134089; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Operations</span></li>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Disable SSH access to all servers.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Servers are ephemeral, you don't care about them. You only care about the
service as a whole.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Don't give servers static/elastic IPs.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Automate everything.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Everyone gets an IAM account. Never login to the master.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Get your alerts to become notifications.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
</ul>
<li><span style="color: #134089; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Billing</span></li>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Set up granular billing alerts.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
</ul>
<li><span style="color: #134089; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Security</span></li>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Use EC2 roles, do not give applications an IAM account.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Assign permissions to groups, not users.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Set up automated security auditing.</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Use CloudTrail to keep an audit log.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
</ul>
<li><span style="color: #134089; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">S3</span></li>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Use "-" instead of "." in bucket names for SSL.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Avoid filesystem mounts (FUSE, etc).</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">You don't have to use CloudFront in front of S3 (but it can help).</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Use random strings at the start of your keys.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
</ul>
<li><span style="color: #134089; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">EC2/VPC</span></li>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Use tags!</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Use termination protection for non-auto-scaling instances. Thank me
later.</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Use a VPC.</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Use reserved instances to save big $$$.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Lock down your security groups.</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Don't keep unassociated Elastic IPs.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
</ul>
<li><span style="color: #134089; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">ELB</span></li>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Terminate SSL on the load balancer.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Pre-warm your ELBs if you're expecting heavy traffic.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
</ul>
<li><span style="color: #134089; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">ElastiCache</span></li>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Use the configuration endpoints, instead of individual node endpoints.</span></li>
</ul>
<li><span style="color: #134089; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">RDS</span></li>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Set up event subscriptions for failover.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
</ul>
<li><span style="color: #134089; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">CloudWatch</span></li>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Use the CLI tools.</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Use the free metrics.</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Use custom metrics.</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Use detailed monitoring.</span></li>
</ul>
<li><span style="color: #134089; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Auto-Scaling</span></li>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Scale down on INSUFFICIENT_DATA as well as ALARM.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Use ELB health check instead of EC2 health checks.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Only use the availability zones (AZs) your ELB is configured for.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Don't use multiple scaling triggers on the same group.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
</ul>
<li><span style="color: #134089; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">IAM</span></li>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Use IAM roles.</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Users can have multiple API keys.</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">IAM users can have multi-factor authentication, use it!</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
</ul>
<li><span style="color: #134089; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Route53</span></li>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Use ALIAS records.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
</ul>
<li><span style="color: #134089; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Elastic MapReduce</span></li>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Specify a directory on S3 for Hive results.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
</ul>
<li><span style="color: #134089; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Miscellaneous Tips</span></li>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: 10pt;">Scale horizontally.</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Your application may require changes to work on AWS.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Always be redundant across availability zones (AZs).</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Be aware of AWS service limits before you deploy.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Decide on a naming convention early, and stick to it.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Decide on a key-management strategy from the start.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">Make sure AWS is right for your workload.</span><span lang="EN-US" style="font-size: 10pt;">
</span></span></li>
</ul>
</ul>
<br />
<!--[if gte mso 9]><xml>
<o:OfficeDocumentSettings>
<o:AllowPNG/>
</o:OfficeDocumentSettings>
</xml><![endif]-->
<!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves/>
<w:TrackFormatting/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-AU</w:LidThemeOther>
<w:LidThemeAsian>JA</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:EnableOpenTypeKerning/>
<w:DontFlipMirrorIndents/>
<w:OverrideTableStyleHps/>
<w:UseFELayout/>
</w:Compatibility>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="true"
DefSemiHidden="true" DefQFormat="false" DefPriority="99"
LatentStyleCount="276">
<w:LsdException Locked="false" Priority="0" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" Priority="39" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" Name="toc 9"/>
<w:LsdException Locked="false" Priority="35" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" Priority="10" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" Priority="1" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" Priority="11" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" Priority="22" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" Priority="59" SemiHidden="false"
UnhideWhenUsed="false" Name="Table Grid"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" QFormat="true" Name="TOC Heading"/>
</w:LatentStyles>
</xml><![endif]-->
<!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin:0cm;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:Cambria;
mso-ascii-font-family:Cambria;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Cambria;
mso-hansi-theme-font:minor-latin;}
</style>
<![endif]-->
<!--StartFragment-->
<!--EndFragment-->Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-46178628754037054242015-01-11T16:56:00.000-08:002015-01-11T16:58:24.862-08:00Henrik Kniberg's "Scaling Agile at Spotify"Henrik Kniberg's done it again. He's taken a working example of complex process implementation and compressed it into a highly informative, easily-watchable and easily-readable format that makes it simple to grab the concepts and, crucially, to help show others to inspire those around you with what's possible in your IT department.<br />
<br />
If you've read Henrik's <a href="http://www.infoq.com/minibooks/scrum-xp-from-the-trenches">Scrum and XP from the Trenches</a> or <a href="https://www.crisp.se/file-uploads/futureofagile-slides-henrikkniberg.pdf">Kanban vs Scrum</a> or <a href="https://www.crisp.se/file-uploads/Lean-from-the-trenches.pdf">Lean from the Trenches</a>, you need to see what he's been helping Spotify do with their engineering culture with scaling agile, lean, devops, culture, A/B experiments.<br />
<br />
<h4>
Two-part animated series depicting Spotify's Engineering culture</h4>
Part 1:<br />
<br />
<ul>
<li>Video: <a href="http://vimeo.com/85490944">Vimeo</a>, <span class="s1"><a href="https://www.youtube.com/watch?v=g42MRJpohFc">YouTube</a></span><span class="s2"> </span></li>
<li><a href="https://spotifylabscom.files.wordpress.com/2014/03/spotify-engineering-culture-part1.jpeg">Full picture (JPEG)</a></li>
<li><a href="https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part">Spotify's blog</a></li>
</ul>
<br />
<br />
Part 2:<br />
<br />
<ul>
<li>Video: <a href="http://vimeo.com/94950270">Vimeo</a>, <a href="https://www.youtube.com/watch?v=nKVK43oiOEo">YouTube</a></li>
<li><a href="https://spotifylabscom.files.wordpress.com/2014/09/spotify-engineering-culture-part2.jpeg">Full picture (JPEG)</a></li>
<li><a href="https://labs.spotify.com/2014/09/20/spotify-engineering-culture-part-2/">Spotify's blog</a></li>
</ul>
<br />
<br />
<br />
<h4>
<b>Scaling Agile at Spotify with Tribes, Squads, Chapters and Guilds</b></h4>
<br />
<ul>
<li><a href="http://blog.crisp.se/2012/11/14/henrikkniberg/scaling-agile-at-spotify">Henrik's blog</a></li>
<li>PDF: <a href="https://dl.dropbox.com/u/1018963/Articles/SpotifyScaling.pdf">dropbox</a>, <a href="http://ucvox.files.wordpress.com/2012/11/113617905-scaling-agile-spotify-11.pdf">non-dropbox</a></li>
</ul>
<br />
<br />
<br />Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-66932496413571239372015-01-06T16:02:00.001-08:002015-01-06T16:02:21.069-08:00Little's Law<b>Little's Law:</b><br />
<br />
Avg queue time = WIP / Throughput<br />
Delivery rate = WIP / Lead Time<br />
MeanResponseTime = MeanNumberInSystem / MeanThroughput<br />
<br />
<i> or</i><br />
<div>
<i><br /></i></div>
Avg Inventory = Throughput * Avg queue time<br />
Avg Customers in system = Avg arrival rate * avg customer time in system<br />
<br />
<br />
<div class="p1">
</div>
<ul>
<li><b>Chris Young explains how he is using Little’s Law to estimate WIP a project’s delivery rate (5min video): </b></li>
<ul>
<li><a href="http://www.infoq.com/presentations/little-law-estimate">http://www.infoq.com/presentations/little-law-estimate</a></li>
</ul>
<li><b>Little's Law - The ONE thing you can do to improve process performance (6min video): </b></li>
<ul>
<li><a href="https://www.youtube.com/watch?v=lHQZcMRr2n0">https://www.youtube.com/watch?v=lHQZcMRr2n0</a></li>
</ul>
<li><b>Little's Law worked problem</b><b> (6min video):</b></li>
<ul>
<li><a href="https://www.youtube.com/watch?v=h-1Q-uuuQkQ">https://www.youtube.com/watch?v=h-1Q-uuuQkQ</a></li>
</ul>
<li><b>Wikipedia:</b> </li>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Little%27s_law">https://en.wikipedia.org/wiki/Little%27s_law</a></li>
<li><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 17.6000003814697px; line-height: 1.6;">The long-term average number of customers in a stable system </span><i style="background-color: white; color: #252525; font-family: sans-serif; font-size: 17.6000003814697px; line-height: 1.6;">L</i><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 17.6000003814697px; line-height: 1.6;"> </span><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 17.6000003814697px; line-height: 1.6;">is equal to the long-term average effective arrival rate,</span><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 17.6000003814697px; line-height: 1.6;"> </span><i style="background-color: white; color: #252525; font-family: sans-serif; font-size: 17.6000003814697px; line-height: 1.6;">λ</i><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 17.6000003814697px; line-height: 1.6;">, multiplied by the (</span><a href="https://en.wikipedia.org/wiki/Palm_calculus" style="background-attachment: initial; background-clip: initial; background-image: none; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #0b0080; font-family: sans-serif; font-size: 17.6000003814697px; line-height: 1.6; text-decoration: none;" title="Palm calculus">Palm</a><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 17.6000003814697px; line-height: 1.6;">‑)average time a customer spends in the system,</span><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 17.6000003814697px; line-height: 1.6;"> </span><i style="background-color: white; color: #252525; font-family: sans-serif; font-size: 17.6000003814697px; line-height: 1.6;">W</i><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 17.6000003814697px; line-height: 1.6;">; or expressed algebraically: </span><i style="background-color: white; color: #252525; font-family: sans-serif; font-size: 17.6000003814697px; line-height: 1.6;">L</i><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 17.6000003814697px; line-height: 1.6;"> = </span><i style="background-color: white; color: #252525; font-family: sans-serif; font-size: 17.6000003814697px; line-height: 1.6;">λW</i><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 17.6000003814697px; line-height: 1.6;">.</span></li>
<li><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 17.6000003814697px; line-height: 22.3999996185303px;">Although it looks intuitively reasonable, it is quite a remarkable result, as the relationship is "not influenced by the arrival process distribution, the service distribution, the service order, or practically anything else."</span></li>
</ul>
</ul>
<br />
<div style="background-color: white; color: #252525; font-family: sans-serif; font-size: 17.6000003814697px; line-height: 22.3999996185303px; margin-bottom: 0.5em; margin-top: 0.5em;">
<br /></div>
Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-69674384207182358092015-01-04T19:09:00.001-08:002015-01-04T19:09:41.733-08:00Experimenting with AWS EC2 Container Service<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Amazon EC2 Container Service ("ecs" for short) is a Docker cluster management service that runs on top of EC2 instances. There is no additional charge for the service - you pay for the EC2 instances whether you're using them or not. It's early days but looks like a promising service that should take a lot of the grunt work (networking, security, etc) out of creating your own clusters like Kubernetes and </span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Mesos.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">ECS is currently in preview - I needed to wait around two weeks to be granted access after signing up here: </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="https://aws.amazon.com/ecs/">https://aws.amazon.com/ecs/</a></span><br />
<br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">This is a transcript of how I fired up a simple Docker container on ECS using Amazon instructions available on 4/Jan/2015.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<h2>
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Introduction</span></b></h2>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Watch this video: <span style="color: #0000ee;"><u><a href="https://www.youtube.com/watch?v=LE5uBqNp2Ds&t=1m58s">https://www.youtube.com/watch?v=LE5uBqNp2Ds&t=1m58s</a></u></span></span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Particularly from 1m58s to skip the Amazon propaganda and watch the interesting visualisation.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">And this video has some good terminology introduction and a live demo: <a href="https://www.youtube.com/watch?v=2vJLS8qfhI0&feature=youtu.be&t=11m12s">https://www.youtube.com/watch?v=2vJLS8qfhI0&feature=youtu.be&t=11m12s</a> </span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">(</span><a href="http://www.slideshare.net/AmazonWebServices/app313-new-launch-amazon-ec2-container-service-in-action-aws-reinvent-2014" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Slides</a><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">)</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Terminology:</span><br />
<br />
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Tasks:</b> A grouping of related containers (e.g. Nginx, Rails app, MySql, Log collector)</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Containers</b></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Clusters:</b> A grouping of container instances - a pool of resources for Tasks</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Container Instances:</b> An EC2 instance on which Tasks are scheduled. AMI with ecs agent installed</span></li>
</ul>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<h2>
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Setting Up</span></b></h2>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Follow these instructions:</span><br />
<a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/get-set-up-for-amazon-ecs.html"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">http://docs.aws.amazon.com/AmazonECS/latest/developerguide/get-set-up-for-amazon-ecs.html</span></a><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">This will walk you through the setup for the following:</span><br />
<br />
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">IAM User</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">IAM Role</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Key Pair</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">VPC</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Security Group</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Special copy of AWS CLI that includes "ecs" commands</span></li>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">NOTE: On OS X I needed to:</span></li>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">"brew uninstall awscli" (that removed /usr/local/bin/aws from my path)</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">And add "export PATH=$PATH:~/.local/lib/aws/bin" to my .bashrc</span></li>
</ul>
</ul>
</ul>
<br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<h2>
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Creating The Cluster</span></b></h2>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Follow these instructions: <a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_GetStarted.html">http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_GetStarted.html</a></span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">NOTE: I preferred to create the EC2 instance from the command line (instead of the <a class="xref" href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_GetStarted.html#getting_started_launch_container_instance" style="color: #996633; font-size: 12px;" title="Launch an Instance with the Amazon ECS AMI">Launch an Instance with the Amazon ECS AMI</a> instructions): </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">aws ec2 run-instances --image-id ami-34ddbe5c --count 1 --instance-type t2.small --subnet-id subnet-xxxxxxxx --key-name ecsdemo-keypair --iam-instance-profile Name=ecsdemo-role</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">... using the <b>subnet-id</b> for my default VPC and the "ecsdemo" <b>keypair</b> and <b>IAM role name</b> I created during the Setting Up phase above.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Then as per the instructions test it out with:</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<b><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">aws ecs list-container-instances</span></b><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">If you see</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">{</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> "containerInstanceArns": []</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">}</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">... then something has gone wrong and you'll need to terminate your instance and try again.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">To see more details about your instance:</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<b><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">aws ecs describe-container-instances</span></b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span><br />
<b><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span></b>
<h2>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Running a Task (Docker process)</span></h2>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">As per the instructions, register a Task Definition and start a Task that spins up a single docker container (based on busybox image) that simply sleeps for 6 minutes.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>aws ecs register-task-definition --family sleep360 --container-definitions "[{\"environment\":[],\"name\":\"sleep\",\"image\":\"busybox\",\"cpu\":10,\"portMappings\":[],\"entryPoint\":[\"/bin/sh\"],\"memory\":10,\"command\":[\"sleep\",\"360\"],\"essential\":true}]"</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>aws ecs list-task-definitions</b></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>aws ecs run-task --cluster default --task-definition sleep360:1 --count 1</b></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>aws ecs list-tasks</b></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>aws ecs describe-tasks --tasks 699d5420-1d0d-410e-b105-7e51027b8fd4</b></span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Log on to your instance and check the docker container is running:</span><br />
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>ssh -i ecsdemo-keypair.pem ec2-user@ec2-instance-public-ip</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<b><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">docker ps</span></b><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Should see:</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">[ec2-user@ip-ec2-instance-public-ip-name ~]$ docker ps</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">ec8a9fca64b0 busybox:buildroot-2014.02 "sleep 360" 3 minutes ago Up 3 minutes ecs-sleep360-1-sleep-dc8dd4cdfcf593d07d00</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">58e68cc5bfc3 amazon/amazon-ecs-agent:latest "/agent" 34 minutes ago Up 34 minutes 127.0.0.1:51678->51678/tcp ecs-agent</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">See more details about your docker container with:</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<b><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">docker inspect ec8a9fca64b0</span></b><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">After 6 minutes of sleeping, the docker process should disappear from the "</span><span style="font-family: Courier New, Courier, monospace;">docker ps</span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">" listing.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<h2>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">More Examples</span></h2>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">More interesting examples including Tasks that link together a number of containers are contained in the videos linked to above.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-68988201446269095392014-06-28T03:35:00.001-07:002014-06-29T18:40:33.071-07:00What is DevOps Days Brisbane 2014?<a href="http://www.devopsdays.org/events/2014-brisbane/logos/dodbne2014.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><span style="font-family: Arial, Helvetica, sans-serif;"><img border="0" src="http://www.devopsdays.org/events/2014-brisbane/logos/dodbne2014.png" /></span></a><br />
<span style="font-family: Arial, Helvetica, sans-serif;">I'm one of the organisers for an awesome conference coming up next month July 25-26. It's called <a href="http://www.devopsdays.org/events/2014-brisbane">DevOps Days Brisbane 2014</a>. If you've never heard of "DevOps Days", it's an international grassroots conference run for the benefit of the IT community in cities all over the world. For the Australian branch, formerly known as "DevOps Downunder", this is the first year it has been held in Brisbane.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">We're really keen to get as many people along as we can and help put Brisbane on the international map. We're not-for-profit - run solely for the benefit of the community so tickets are really affordable at $185 (+fees) each for a 2-day conference in the Brisbane Convention & Exhibition Centre with full food & beverage + reception drinks - <b>crazy</b>! It's held over a Friday & Saturday so you or your employees need take only one day off work. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">The <a href="http://www.devopsdays.org/events/2014-brisbane/program">program</a> has just been announced with great local, national, international speakers from companies such as <b>Chef, ThoughtWorks, RedHat, IOOF, REA & Wotif Group</b> (including me - my first devops days talk!). And the keynote address will be delivered by none other than world-renowned human factors and safety expert <a href="http://sidneydekker.com/"><b>Sidney Dekker</b></a>.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">The format for both days is a mix of great presentations in the mornings and collaborative tech talk and problem solving in the afternoons:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">In the mornings we have <b>high quality single-track presentations</b> and rapid-fire <a href="http://www.devopsdays.org/pages/ignite-talks-format">ignite</a> talks with well respected international speakers as well as national and local thought leaders.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">In the afternoons we have multi-tracked "<a href="http://www.devopsdays.org/pages/open-space-format">openspaces</a>" - where the content is driven <b>by attendees for attendees</b> - we provide advice for each other's technical and cultural problems, talk about the latest technology, and help each other think outside the square to where we want to be as an industry. It's a great format where attendees can not only receive advice from thought-leaders and peers but also bring the benefits of their perspectives to the wider community in an interactive manner.</span></li>
</ul>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">DevOps Days conferences are inspiring, challenging, thought-provoking, and lots of fun. Attendees walk away with new friends and a sense of camaraderie. As well as new perspectives, not just on the latest technology, but importantly also on culture and how to collaborate, communicate and empathise with other disciplines within their own company. DevOps is essentially an extension of Agile - bringing all the goodness of fast feedback loops (with added emphasis on collaboration and automation) to the "last mile" of deployment to production. It's not only open to developers and operations, we find that managers, architects, testers and BAs all get value out of it and contribute too.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Website with registration and full program details is here: <a href="http://www.devopsdays.org/events/2014-brisbane/">http://www.devopsdays.org/events/2014-brisbane/</a></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">And there's still a chance to <a href="http://www.devopsdays.org/events/2014-brisbane/sponsor">sponsor</a> the event and get your logo alongside great international companies like PuppetLabs and AppDynamics. Gold sponsors get 4 free tickets and Silver sponsors get 2 free tickets to send their employees or mates along.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">I'm happy to come and talk in person with teams, groups, individuals around Brisbane and answer questions anyone may have. Let me know if there's anything I can do to help you help us <b>put Brisbane on the map</b>!</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Twitter:</b> @mcallana</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Google+:</b> matthew.callanan</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Email:</b> matt at mattcallanan.net</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<div class="separator" style="clear: both; text-align: center;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<br />Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-19091047056818237412014-05-06T19:30:00.001-07:002014-05-08T15:15:54.754-07:00Is TDD Dead?<div class="moz-text-html" lang="x-western">
An interesting read: <a href="http://david.heinemeierhansson.com/2014/tdd-is-dead-long-live-testing.html">http://david.heinemeierhansson.com/2014/tdd-is-dead-long-live-testing.html</a><br />
A purposely inflammatory post that had the desired effect of bringing out some TDD heavyweight advocates on Twitter including Bob Martin and Martin Fowler....<br />
<br />
There are some interesting points to draw from David's <a href="https://twitter.com/dhh">Twitter stream</a>:<br />
<ul>
<li>He learnt the ins-and-outs of TDD first before moving on (and he learnt a lot from it).</li>
<li>He says his view of the testing world only applies to Rails apps where tight Web + DB integration/coupling is a design choice.</li>
<li>He has very opinionated beliefs about how Rails apps should be built including</li>
<ul>
<li>Rails apps should remain "small-scale"</li>
<li>ActiveRecord usage should be carefully maintained</li>
<li>... both of which are often not the case in reality.</li>
</ul>
<li>Patterns of building/testing Rails apps like Hexagon are overkill if you're building a small Rails app with business logic intended only for web audience - but can be useful if that Rails app is a peer among other interfaces.</li>
</ul>
<br />
My summary:<br />
<ul>
<li>Learn TDD inside-out, learn where it fits rather than getting caught up in dogma. </li>
<li>Know what type of app you're building</li>
<li>Then carefully design and architect your application to be testable through careful and appropriate use of unit/integration/system testing</li>
<li>Ensure your architecture is carefully and purposefully maintained.</li>
</ul>
I personally use a "Test Oriented Development" approach:<br />
<br />
<ul>
<li>Exploratory coding should be done as part of a spike - normally on a "spike branch"</li>
<ul>
<li>There's often not much point writing up-front tests when you need to try something you've never done before - this can be a form of waste</li>
<li>Kent Beck (the inventor of TDD) himself doesn't write tests before exploratory code - <a href="http://www.se-radio.net/2010/09/episode-167-the-history-of-junit-and-the-future-of-testing-with-kent-beck/">podcast</a></li>
</ul>
<li>When you've solved the problem, jump back onto master and rewrite the solution this time with testing in mind - carefully plan Unit Tests, Component/Integration Tests, & Acceptance Tests</li>
<li>If you can't pair commit to master, push to a temporary review branch and submit a pull/merge request to a reviewer</li>
</ul>
<br />
<br />
<br />
Interesting response: <a href="https://www.destroyallsoftware.com/blog/2014/tdd-straw-men-and-rhetoric" moz-do-not-send="true">https://www.destroyallsoftware.com/blog/2014/tdd-straw-men-and-rhetoric</a></div>
<div class="moz-text-html" lang="x-western">
<br /></div>
And another one from Martin Fowler about self-testing code: <a href="http://martinfowler.com/bliki/SelfTestingCode.html">http://martinfowler.com/bliki/SelfTestingCode.html</a><br />
<blockquote class="tr_bq">
<span style="font-family: inherit;"><i><span style="background-color: white; color: #333333; font-size: 16px; font-style: normal; line-height: 14.933333396911621px;">These kinds of benefits are often talked about with respect to</span></i><i><a href="http://martinfowler.com/bliki/TestDrivenDevelopment.html" style="background-color: white; border: 0px; color: #94388e; font-size: 16px; font-style: normal; line-height: 14.933333396911621px; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">TestDrivenDevelopment</a></i><span style="background-color: white; color: #333333; font-size: 16px; line-height: 14.933333396911621px;"><i> </i><i>(TDD), but it's useful to separate the concepts of TDD and self-testing code. I think of TDD as a particular practice whose benefits include producing self-testing code. It's a great way to do it, and TDD is a technique I'm a big fan of. But you can also produce self-testing code by writing tests after writing code - although you can't consider your work to be done until you have the tests (and they pass). The important point of self-testing code is that you have the tests, not how you got to them.</i></span></span></blockquote>
Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-43976527531728321732014-04-14T04:32:00.002-07:002014-04-14T04:32:16.152-07:00Ideas for Running Cucumber-JVM Selenium Tests in Parallel<h2>
Ideas on how to run Selenium-backed Cucumber tests in parallel</h2>
<br />
Use Selenium Grid 2 to coordinate the Selenium tests across a server farm of Selenium RC servers<br />
<br />
<ul>
<li><a href="http://docs.seleniumhq.org/docs/07_selenium_grid.jsp">http://docs.seleniumhq.org/docs/07_selenium_grid.jsp</a></li>
<li><a href="https://code.google.com/p/selenium/wiki/Grid2">https://code.google.com/p/selenium/wiki/Grid2</a></li>
</ul>
<br />
<br />
<div class="p1">
Example Parallel Cucumber JVM execution with Maven surefire plugin forkCount</div>
<div class="p2">
</div>
<ul>
<li><a href="http://www.opencredo.com/2013/07/02/running-cucumber-jvm-tests-in-parallel/">http://www.opencredo.com/2013/07/02/running-cucumber-jvm-tests-in-parallel/</a></li>
<li><a href="https://github.com/tristanmccarthy/Cucumber-JVM-Parallel">https://github.com/tristanmccarthy/Cucumber-JVM-Parallel</a></li>
<li>based on tag name</li>
<li>could also exclude @wip tag using Maven profiles (see comments)</li>
</ul>
<br />
<div class="p1">
Alternatively,</div>
<div class="p1">
<br /></div>
<div class="p1">
Test Load Balancer</div>
<br />
<div class="p2">
</div>
<ul>
<li><a href="http://test-load-balancer.github.io/">http://test-load-balancer.github.io</a></li>
<li><a href="https://github.com/test-load-balancer/tlb">https://github.com/test-load-balancer/tlb</a></li>
<li>great idea</li>
<li>project seems abandoned</li>
<li>only supports Ant</li>
</ul>
<br />
<div class="p2">
<br /></div>
<div class="p2">
<br /></div>
<div class="p2">
<br /></div>
<div class="p2">
<br /></div>
Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-39874374475247520322014-02-27T17:44:00.001-08:002014-03-01T04:29:31.625-08:00How Etsy do Code Reviews with Continuous Delivery (incl. branching, culture)<div id="main-header" style="color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; line-height: 16px; margin: 0px 0px 20px; padding: 0px;">
<div class="pagetitle with-breadcrumbs" id="title-heading" style="color: #0065b0; margin: 0px; padding: 0px;">
<h1 class="with-breadcrumbs" id="title-text" style="font-size: 24px; font-weight: normal; line-height: 1.25; margin: 0px; padding: 0px;">
<span style="color: #333333; font-size: 14.399999618530273px; line-height: 16px;">Continuous Delivery strongly advocates developing on mainline/master/trunk. It also advocates releasing as frequently as possible, pointing out that Continuous Deployment, where all commits that pass all the stages of the build pipeline are automatically released, is the ultimate logical extension. But how do teams have a chance to perform code reviews if all commits to master are potential release candidates? This question was posed on the </span><span style="color: #333333; font-size: 14.399999618530273px; line-height: 16px;">Continuous Delivery </span><span style="color: #333333; font-size: 14.399999618530273px; line-height: 16px;">Google Group recently, with a response from Etsy's Mike Brittain about how they do it. TL;DR: It involves temporary short-lived branches for reviews and a culture: of keeping changes small, not deploying without review & being good at assessing risk of releasing untested code.</span></h1>
</div>
</div>
<div class="page view" id="content" style="box-sizing: border-box; clear: none !important; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; line-height: 16px; margin: 0px; padding: 0px; position: static !important;">
<div class="wiki-content" id="main-content" style="margin: 0px; padding: 0px;">
<div style="padding: 0px;">
Below is a summary of <a class="external-link" href="https://groups.google.com/forum/#!topic/continuousdelivery/9_5xpiHJZUc" rel="nofollow" style="color: #0065b0; text-decoration: none;">https://groups.google.com/forum/#!topic/continuousdelivery/9_5xpiHJZUc</a></div>
<div style="margin-top: 10px; padding: 0px;">
<br /></div>
<div class="GD-PFW4BDFB GD-PFW4BM1B GD-PFW4BFFB GD-PFW4BHIC GD-PFW4BEFB" style="margin: 0px; padding: 0px;">
<div class="GD-PFW4BDFB _mce_tagged_br" style="margin: 0px; padding: 0px;">
<div class="GD-PFW4BCEB" style="margin: 0px; padding: 0px;">
<div class="GD-PFW4BKDB" style="margin: 0px; padding: 0px;">
<div class="GD-PFW4BAUD" style="margin: 0px; padding: 0px;">
<div class="GD-PFW4BMEB" style="margin: 0px; padding: 0px;">
<div style="margin: 0px; padding: 0px;">
<ul style="margin: 0px;">
<li>Are <strong>code reviews mandatory?</strong> <span style="line-height: 1.4285715;">Any formal process or tool that "<strong>gates</strong>" check-ins to trunk, based on peer review?</span><ul style="list-style-type: disc; margin: 0px;">
<li><span style="line-height: 1.4285715;">Code review is not mandatory, nor does it gate check-ins to trunk or deployments to production. </span></li>
<li>Some of the hangups we have about gating deploys in this fashion is that gating artificially slows us down emergencies/outages, and as Jez put it in the topic you linked to, gating assumes stupidity that has to be corrected for. </li>
<li>We have built the <strong>culture</strong> in our Engineering team to assume everyone is going to do the right thing, and we should trust them. When that trust falls apart, I think of it as a <strong>bug in our hiring and on boarding processes</strong>.</li>
<li><span style="line-height: 1.4285715;">It is <strong>socially unacceptable at Etsy to deploy without some form of review</strong>, and that's true for nearly every sub-team. </span></li>
<li><span style="line-height: 1.4285715;">Many of our <strong>reviews happen in GitHub (Enterprise),</strong> and so have an <strong>audit trail</strong> attached to them of who reviewed and what was discussed. </span></li>
<li><span style="line-height: 1.4285715;">Other code reviews might be a simple copy-and-paste of a <strong>patch file</strong>, or an <strong>over the shoulder review</strong> by a peer. </span></li>
<li><span style="line-height: 1.4285715;">Most changes are relatively <strong>small</strong> and can be easily grokked</span></li>
</ul>
</li>
<li><span style="line-height: 1.4285715;">With respect to <strong>config flags</strong> (<b>feature switches</b>) and other *<strong>new* code</strong>:</span><ul style="list-style-type: disc; margin: 0px;">
<li><span style="line-height: 1.4285715;">It's possible that we'll deploy code to our production servers that is not live (e.g. "dark code"), but that a more formal code <strong>review will happen later</strong> when all of the changes for some new feature or product are complete. </span></li>
<li><span style="line-height: 1.4285715;">Maybe this is just a side effect that we don't use feature branches, and that <strong>we think of committing to git as deploying to web servers</strong> (they happen nearly in sync). </span></li>
<li><span style="line-height: 1.4285715;">That <strong>doesn't mean that the code will be live to a public user</strong>. (Think: new class files, or new and unlinked JS or CSS, features that are gated off so that they are only available to staff members).</span></li>
</ul>
</li>
</ul>
</div>
<div style="margin: 0px; padding: 0px;">
<div style="margin: 0px; padding: 0px;">
<ul style="margin: 0px;">
<li><span style="line-height: 1.4285715;">Likely that any change which would negatively impact customers in the short-term will be caught by <strong>automated tests</strong>?</span><ul style="list-style-type: disc; margin: 0px;">
<li>I'd argue that there are <strong>a number of changes that could negatively impact consumers</strong> or, more to the point, their behavior, that would <strong>never be caught by automated tests</strong>. </li>
<li>These are the types of things you might think of for classic A/B tests—size, shape, position, color of UI elements, feature discoverability, fall-off in multi-page flows. </li>
<li>And that is why <strong>we move quickly to get these changes to the (A/B) testing phase</strong>.</li>
<li>This statement seems to imply that automated tests are the only way to catch bugs. <strong>We find a lot of bugs through production monitoring</strong> that we would have never conceived of writing tests for.</li>
</ul>
</li>
<li>We have the benefit of operating our own products - not dealing with contractual obligations to deliver "bug-free" code to clients. <ul style="list-style-type: disc; margin: 0px;">
<li>We're <strong>deeply immersed in the trade-offs of testing vs. speed</strong>, and I think that's well understood by many engineers on the team who are making decisions around those <strong>trade-offs every day</strong>. </li>
<li>We <strong>regularly make mistakes</strong> at this, identify the mistakes through <strong>open discussion</strong>, and those engineers who were involved <strong>come away more experienced and wiser for it</strong>.</li>
</ul>
</li>
<li>Assumption of "<strong>under-reviewed code</strong>"?<ul style="list-style-type: disc; margin: 0px;">
<li>Many cases where <strong>we deploy code</strong>, even publicly, that you would consider to be "<strong>under-reviewed</strong>"</li>
<li>The relative degree of <strong>testing</strong> that we apply is <strong>proportional</strong> to the amount of <strong>risk</strong> involved. </li>
<li>We are <strong>decent at assessing and socializing risk</strong> :) </li>
<li>Our Payments team, for example, is much more rigid about their review process than teams that, say, are running experiments at the UI level. </li>
<li>It's <strong>not one-size-fits-all</strong>.</li>
</ul>
</li>
<li>You mentioned that you don't use <strong>feature branches</strong>, but also mentioned that most of your <strong>reviews</strong> are done in Github Enterprise?<ul style="list-style-type: disc; margin: 0px;">
<li><span style="line-height: 1.4285715;">What you're pointing out is some subtlety around how we refer to branches and reviews.</span></li>
<li><span style="line-height: 1.4285715;">In terms of development process, we don't use feature branches in the traditional sense—especially if you think of branches existing for more than a day. </span></li>
<li><span style="line-height: 1.4285715;">In practice, there are certainly cases where <strong>individuals will use a short-lived branch to isolate</strong> a couple of things they're working on at a time, much like you might use git stash—such as to switch context to work on an immediate bug fix that needs to go to production.</span></li>
<li><span style="line-height: 1.4285715;">Our review process does<strong> utilize branches </strong>b</span><span style="line-height: 1.4285715;">ut <strong>only to support the review process</strong>. </span></li>
<li><span style="line-height: 1.4285715;">As an engineer prepares to commit a change set, they create a (<strong>short-lived</strong>) review branch in git. </span></li>
<li><span style="line-height: 1.4285715;">This allows us to look at the review in <strong>GitHub Enterprise</strong> and use their <strong>commenting</strong> and <strong>visual diff'ing</strong>. </span></li>
<li><span style="line-height: 1.4285715;">It also allows other engineers to <strong>checkout the branch</strong> and put it through its paces, when necessary.</span></li>
<li><span style="line-height: 1.4285715;">Once the code review is complete, the branch is typically <strong>auto-merged in GitHub</strong> when the engineer is prepared to deploy it.</span></li>
<li><span style="line-height: 1.4285715;">Yes, it's a subtle (and perhaps stubborn) distinction. :)</span></li>
<li><em><span style="line-height: 1.4285715;">It seems that "time to review" would be a fairly key metric in that case</span></em><ul style="list-style-type: disc; margin: 0px;">
<li><span style="line-height: 1.4285715;">This is true, but it's not really a problem. </span></li>
<li><strong><span style="line-height: 1.4285715;">Daily stand-ups generally work for socializing if you're blocked on someone's feedback on a code review</span></strong><em><span style="line-height: 1.4285715;"><br /></span></em></li>
</ul>
</li>
</ul>
</li>
<li><span style="line-height: 1.4285715;">For your review process, would you say you're using a variant of what GitHub calls "<strong>GitHub Flow</strong>"? </span><ul style="list-style-type: disc; margin: 0px;">
<li><em><a class="external-link" href="http://scottchacon.com/2011/08/31/github-flow.html" rel="nofollow" style="color: #0065b0; line-height: 1.4285715; text-decoration: none;">http://scottchacon.com/2011/08/31/github-flow.html</a><span style="line-height: 1.4285715;"> - (as distinct from what Scott calls the overly "complicated" </span><span style="line-height: 1.4285715;">gitflow model). </span></em></li>
<li><em><span style="line-height: 1.4285715;">Perhaps tweaking step #2 to happen after making local changes to 'master' on an as-needed basis</span></em><ul style="list-style-type: disc; margin: 0px;">
<li><span style="line-height: 1.4285715;">That's pretty much right. We <strong>only go onto a branch for the review stage</strong>. </span></li>
</ul>
</li>
</ul>
</li>
<li><span style="line-height: 1.4285715;">How do you ensure branches (whether for bugfixes or reviews) remain <strong>short-lived</strong>? </span><ul style="list-style-type: disc; margin: 0px;">
<li><span style="line-height: 1.4285715;">It's a <strong>cultural</strong> thing. </span></li>
<li><span style="line-height: 1.4285715;">You <strong>learn that practice in the first days</strong> of work on the engineering team. </span></li>
<li><span style="line-height: 1.4285715;">If you get into a review where you've got a <strong>boatload of code that's difficult to understand or to assess the risk</strong>, your <strong>peers will point that out</strong>.</span></li>
<li><span style="line-height: 1.4285715;">This is also the reason we <strong>don't advocate new work to start immediately on a feature branch</strong> - t</span><span style="line-height: 1.4285715;">hose have a tendency to <strong>allow code to build up</strong>, especially in local branches.</span></li>
<li><span style="line-height: 1.4285715;"><strong>Small, frequent changes to master</strong> by our engineers probably cause a <strong>greater frequency of bumping into each other</strong>, but at the same time help <strong>avoid large-scale merge conflicts</strong>.</span></li>
<li><span style="line-height: 1.4285715;">Scott lays out that <strong>remote branch tracking</strong> (in GitHub and "git pull") is excellent for <strong>monitoring</strong> what people are working on. </span><ul style="list-style-type: disc; margin: 0px;">
<li><span style="line-height: 1.4285715;">We get the same through <strong>commit logs to master</strong> and <strong>high visibility into the state of master</strong> (i.e. what's in production). </span></li>
<li><span style="line-height: 1.4285715;">I'm not going to suggest that one is inherently better than the other. </span></li>
<li><span style="line-height: 1.4285715;">Both of us have models that have worked well within our own teams.</span></li>
</ul>
</li>
</ul>
</li>
</ul>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-59236354022392816372013-12-31T08:19:00.001-08:002014-01-14T04:34:39.816-08:00How to build Netflix RSS Reader Example on Centos 6 with Amazon AWS<h2>
Instructions for installing Netflix RSS Reader on Amazon EC2</h2>
In the last week of December 2013, I built and installed the Netflix example RSS Reader application through following these instructions on the <a href="https://github.com/Netflix/recipes-rss/wiki">Netflix recipes-rss wiki</a>. See also the <a href="http://techblog.netflix.com/2013/03/introducing-first-netflixoss-recipe-rss.html">Netflix Tech Blog</a> for an overview.<br />
<br />
There were a lot of dead ends and rabbit warrens involved in this process as there are a lot of components to get up and running, including: infrastructure, networking, JDK, Gradle, source code, Tomcat, Jetty.<br />
<br />
Hopefully these step-by-step instructions make it easier for someone.<br />
<br />
It's worth noting a couple of default assumptions that aren't immediately clear that do actually make your life easier:<br />
<ol>
<li>All 3 services (Eureka, RSS Middletier, RSS Edge) are installed on the same host - you don't need to create 3 separate machines and network them all together</li>
<li>The RSS Reader example does not require Cassandra but uses in-memory data storage by default (connecting to Cassandra is optional)</li>
</ol>
<div>
The first set of instructions below are how to get it running on a single instance. The follow-on instructions describe how to scale it out to separate clusters of nodes - which is where Turbine/Hystrix gets interesting.</div>
<br />
<div>
On to the instructions...</div>
<div>
<h2>
Basic Single-Instance Installation</h2>
</div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #f0f0f0; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #60a0b0; font-style: italic;"># Instructions prefixed with "###" are dead-ends I went down - may save you time to skip them.</span>
<span style="color: #60a0b0; font-style: italic;"># Create the machine</span>
<span style="color: #60a0b0; font-style: italic;"># Create an EC2 Instance in the AWS console with the following:</span>
<span style="color: #60a0b0; font-style: italic;"># AMI base image: Centos 6 x86_64 with updates</span>
<span style="color: #60a0b0; font-style: italic;"># Instance: m1.small (WARNING: t1.micro's 600M RAM is insufficient)</span>
<span style="color: #60a0b0; font-style: italic;"># Create a security group and save the key. Login to your new instance as root with the downloaded key.</span>
<span style="color: #60a0b0; font-style: italic;"># Setup the networking configuration to allow the services to talk to each other and allow you to browse to them:</span>
<span style="color: #60a0b0; font-style: italic;"># Configure AWS security Group: </span>
<span style="color: #60a0b0; font-style: italic;"># Open TCP Input ports: 22, 80, 9090, 9092, 9191, 9192</span>
<span style="color: #60a0b0; font-style: italic;"># Ideally but optionally expose them only to your IP address instead of the whole world (0.0.0.0/0)</span>
<span style="color: #60a0b0; font-style: italic;"># Configure iptables:</span>
<span style="color: #60a0b0; font-style: italic;"># Flush all existing rules</span>
iptables -F
<span style="color: #60a0b0; font-style: italic;"># Block null recon packets</span>
iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP
<span style="color: #60a0b0; font-style: italic;"># Reject syn-flood attack</span>
iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP
<span style="color: #60a0b0; font-style: italic;"># Allow loopback for internal services</span>
iptables -A INPUT -i lo -j ACCEPT
<span style="color: #60a0b0; font-style: italic;"># Open ports 22 (ssh) & 80 (http)</span>
iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
<span style="color: #60a0b0; font-style: italic;"># Open ports 9090 & 9092 for RSS Reader Edge webserver</span>
iptables -A INPUT -p tcp -m tcp --dport 9090 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 9092 -j ACCEPT
<span style="color: #60a0b0; font-style: italic;"># Open ports 9191 for RSS Reader Middletier webserver</span>
iptables -A INPUT -p tcp -m tcp --dport 9191 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 9192 -j ACCEPT
<span style="color: #60a0b0; font-style: italic;"># Allow outgoing connections</span>
iptables -I INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
<span style="color: #60a0b0; font-style: italic;"># Allow all outgoing connections</span>
iptables -P OUTPUT ACCEPT
<span style="color: #60a0b0; font-style: italic;"># Drop everything else</span>
iptables -P INPUT DROP
iptables -L -n
service iptables save
service iptables restart
<span style="color: #60a0b0; font-style: italic;"># For better security, consider leaving port 80 closed and forwarding requests on port 80 to port 8080. Then, in the Tomcat instructions below, we could leave Tomcat on the default 8080 port without requiring "root" user</span>
<span style="color: #60a0b0; font-style: italic;"># iptables -A PREROUTING -t nat -i eth1 -p tcp --dport 80 -j REDIRECT --to-port 8080</span>
<span style="color: #60a0b0; font-style: italic;"># iptables -A INPUT -p tcp -m tcp --dport 8080 -j ACCEPT</span>
<span style="color: #60a0b0; font-style: italic;"># Install JDK</span>
<span style="color: #60a0b0; font-style: italic;">### I shouldn't have done this first step... it only installs the JRE... Gradle needs the JDK</span>
<span style="color: #60a0b0; font-style: italic;">### yum install -y java-1.7.0-openjdk.x86_64</span>
<span style="color: #60a0b0; font-style: italic;"># Install Oracle JDK as per: http://parijatmishra.wordpress.com/2013/03/09/oraclesun-jdk-on-ec2-amazon-linux/</span>
<span style="color: #60a0b0; font-style: italic;">### Remove OpenJDK. Hopefully not required if you didn't do the above "yum install java-1.7.0-openjdk.x86_64"</span>
<span style="color: #60a0b0; font-style: italic;">### rpm --erase --nodeps java-1.7.0-openjdk java-1.7.0-openjdk-devel</span>
yum install -y wget
<span style="color: #60a0b0; font-style: italic;"># There's probably a better way to get the most recent JDK - but these instructions made it easy</span>
wget --no-check-certificate --no-cookies --header <span style="color: #4070a0;">"Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2Ftechnetwork%2Fjava%2Fjavase%2Fdownloads%2Fjdk-7u3-download-1501626.html;"</span> http://download.oracle.com/otn-pub/java/jdk/7u25-b15/jdk-7u25-linux-x64.rpm
mv jdk-7u25-linux-x64.rpm<span style="color: #4070a0; font-weight: bold;">\?</span>AuthParam<span style="color: #4070a0; font-weight: bold;">\=</span>1388300280_9fd087722658cfbb8e571f2d0449beea jdk-7u25-linux-x64.rpm
yum install -y jdk-7u25-linux-x64.rpm
<span style="color: #007020; font-weight: bold;">for </span>i in /usr/java/jdk1.7.0_25/bin/* ; <span style="color: #007020; font-weight: bold;">do</span> <span style="color: #4070a0; font-weight: bold;">\</span>
<span style="color: #bb60d5;">f</span><span style="color: #666666;">=</span><span style="color: #007020; font-weight: bold;">$(</span>basename <span style="color: #bb60d5;">$i</span><span style="color: #007020; font-weight: bold;">)</span>; <span style="color: #007020;">echo</span> <span style="color: #bb60d5;">$f</span>; <span style="color: #4070a0; font-weight: bold;">\</span>
sudo alternatives --install /usr/bin/<span style="color: #bb60d5;">$f</span> <span style="color: #bb60d5;">$f</span> <span style="color: #bb60d5;">$i</span> 20000 ; <span style="color: #4070a0; font-weight: bold;">\</span>
sudo update-alternatives --config <span style="color: #bb60d5;">$f</span> ; <span style="color: #4070a0; font-weight: bold;">\</span>
<span style="color: #007020; font-weight: bold;">done</span>
<span style="color: #007020;">cd</span> /etc/alternatives
ln -sfn /usr/java/jdk1.7.0_25 java_sdk
<span style="color: #007020;">cd</span> /usr/lib/jvm
ln -sfn /usr/java/jdk1.7.0_25/jre jre
<span style="color: #60a0b0; font-style: italic;"># JAVA_HOME must be set for Gradle to work</span>
<span style="color: #007020;">echo</span> <span style="color: #4070a0;">"export JAVA_HOME=/usr/java/jdk1.7.0_25"</span> >> .bashrc
. ~/.bashrc
<span style="color: #60a0b0; font-style: italic;">### Install Gradle - may not be required as the Netflix build steps below use self-contained "gradlew" script (which downloads Gradle)</span>
<span style="color: #60a0b0; font-style: italic;">### curl -O "http://downloads.gradle.org/distributions/gradle-1.10-all.zip"</span>
<span style="color: #60a0b0; font-style: italic;">### yum install -y unzip</span>
<span style="color: #60a0b0; font-style: italic;">### cd /opt</span>
<span style="color: #60a0b0; font-style: italic;">### unzip ~/gradle-1.10-bin.zip </span>
<span style="color: #60a0b0; font-style: italic;">### cd</span>
<span style="color: #60a0b0; font-style: italic;">### echo "export PATH=$PATH:/opt/gradle-1.10/bin" >> .bashrc</span>
<span style="color: #60a0b0; font-style: italic;">### . ~/.bashrc </span>
<span style="color: #60a0b0; font-style: italic;"># Build RSS Reader Middletier and Edge webapps</span>
yum install -y git
git clone https://github.com/Netflix/recipes-rss.git
<span style="color: #007020;">cd </span>recipes-rss
./gradlew clean build
<span style="color: #60a0b0; font-style: italic;">### This was required to fix error "Error compiling file: /tmp//org/apache/jsp/jsp/rss_jsp.java org.apache.jasper.JasperException: PWC6033: Unable to compile class for JSP"</span>
<span style="color: #60a0b0; font-style: italic;">### javac /tmp/org/apache/jsp/jsp/rss_jsp.java -cp /root/recipes-rss/rss-edge/build/libs/rss-edge-0.1.0-SNAPSHOT.jar:/tmp -source 1.5 -target 1.5</span>
<span style="color: #60a0b0; font-style: italic;"># Install Tomcat 6 (didn't bother with Tomcat 7 as it wasn't available in defuault Centos 6 yum repo, as per "yum search tomcat")</span>
yum install -y tomcat6
sed -i <span style="color: #4070a0;">'s/port=\"8080\"/port=\"80\"/g'</span> /etc/tomcat6/server.xml
<span style="color: #60a0b0; font-style: italic;"># Set TOMCAT_USER to "root"</span>
vim /etc/tomcat6/tomcat6.conf
<span style="color: #60a0b0; font-style: italic;"># Replace TOMCAT_USER setting with:</span>
<span style="color: #bb60d5;">TOMCAT_USER</span><span style="color: #666666;">=</span><span style="color: #4070a0;">"root"</span>
<span style="color: #60a0b0; font-style: italic;"># End of edit</span>
<span style="color: #007020;">cd</span>
<span style="color: #007020;">echo</span> <span style="color: #4070a0;">"export TOMCAT_HOME=/usr/share/tomcat6"</span> >> .bashrc
. ~/.bashrc
<span style="color: #60a0b0; font-style: italic;"># Build and deploy Eureka</span>
git clone https://github.com/Netflix/eureka.git
<span style="color: #007020;">cd </span>eureka/
./gradlew clean build
cp ./eureka-server/build/libs/eureka-server-XXX-SNAPSHOT.war <span style="color: #bb60d5;">$TOMCAT_HOME</span>/webapps/eureka.war
service tomcat6 start
<span style="color: #60a0b0; font-style: italic;"># Make sure there are no errors</span>
grep <span style="color: #4070a0;">"ERROR"</span> /usr/share/tomcat6/logs/catalina.out | less -S
<span style="color: #60a0b0; font-style: italic;"># (takes around 2mins to startup, expect some startup errors due to Eureka not running in an established cluster)</span>
<span style="color: #60a0b0; font-style: italic;"># Browse to http://[IP ADDRESS]/eureka/ <-- trailing slash required</span>
<span style="color: #60a0b0; font-style: italic;"># In another terminal session</span>
<span style="color: #60a0b0; font-style: italic;"># Start RSS Middletier Webserver</span>
<span style="color: #007020;">export </span><span style="color: #bb60d5;">APP_ENV</span><span style="color: #666666;">=</span>dev
<span style="color: #007020;">cd </span>recipes-rss
java -Xmx128m -XX:MaxPermSize<span style="color: #666666;">=</span>32m -jar rss-middletier/build/libs/rss-middletier-*SNAPSHOT.jar
<span style="color: #60a0b0; font-style: italic;"># Test via Admin port: Browse to: http://[IP ADDRESS]:9192</span>
<span style="color: #60a0b0; font-style: italic;"># In another terminal session</span>
<span style="color: #60a0b0; font-style: italic;"># Start RSS Edge Webserver</span>
<span style="color: #007020;">export </span><span style="color: #bb60d5;">APP_ENV</span><span style="color: #666666;">=</span>dev
<span style="color: #007020;">cd </span>recipes-rss
java -Xmx128m -XX:MaxPermSize<span style="color: #666666;">=</span>32m -jar rss-edge/build/libs/rss-edge-*SNAPSHOT.jar
<span style="color: #60a0b0; font-style: italic;"># Test via Admin port: Browse to: http://[IP ADDRESS]:9092</span>
<span style="color: #60a0b0; font-style: italic;"># Browse to http://[IP ADDRESS]:9090/jsp/rss.jsp</span>
<span style="color: #60a0b0; font-style: italic;"># Add the following RSS feeds:</span>
<span style="color: #60a0b0; font-style: italic;"># http://rss.cnn.com/rss/edition.rss</span>
<span style="color: #60a0b0; font-style: italic;"># http://feeds.washingtonpost.com/rss/politics</span>
<span style="color: #60a0b0; font-style: italic;"># http://news.yahoo.com/rss/us</span>
<span style="color: #60a0b0; font-style: italic;"># http://rss.cnn.com/rss/money_autos.rss</span>
<span style="color: #60a0b0; font-style: italic;"># Optional Extras...</span>
<span style="color: #60a0b0; font-style: italic;"># Install Hystrix: https://github.com/Netflix/recipes-rss/wiki/Hystrix-Metrics-%28Optional%29</span>
<span style="color: #60a0b0; font-style: italic;"># Open port 7979 in AWS Security Group</span>
iptables -A INPUT -p tcp -m tcp --dport 7979 -j ACCEPT
iptables -L -n
service iptables save
service iptables restart
git clone https://github.com/Netflix/Hystrix.git
<span style="color: #007020;">cd </span>Hystrix/hystrix-dashboard
../gradlew jettyRun
<span style="color: #60a0b0; font-style: italic;"># Browse to: http://[IP ADDRESS]:7979/hystrix-dashboard</span>
<span style="color: #60a0b0; font-style: italic;"># Enter http://[IP ADDRESS]:9090/hystrix.stream to see the Hystrix metrics show up in the dashboard. You will have to send a few transactions from the Edge service to have the Hystrix metrics loaded.</span>
<span style="color: #60a0b0; font-style: italic;"># Stress the Edge webserver and watch the circuit trip into "Open" state:</span>
<span style="color: #007020; font-weight: bold;">for </span>i in <span style="color: #666666;">{</span>1..100<span style="color: #666666;">}</span>; <span style="color: #007020; font-weight: bold;">do </span>curl -s -o /dev/null -w <span style="color: #4070a0;">"%{http_code} %{url_effective}\\n"</span> <span style="color: #4070a0;">"http://[IP ADDRESS]:9090/jsp/rss.jsp"</span> & <span style="color: #007020; font-weight: bold;">done</span>
<span style="color: #60a0b0; font-style: italic;"># Hystrix Example application: https://github.com/Netflix/Hystrix/tree/master/hystrix-examples-webapp</span>
<span style="color: #60a0b0; font-style: italic;"># Open port 8989 in AWS Security Group</span>
iptables -A INPUT -p tcp -m tcp --dport 8989 -j ACCEPT
iptables -L -n
service iptables save
service iptables restart
<span style="color: #007020;">cd </span>Hystrix/hystrix-examples-webapp
../gradlew jettyRun
<span style="color: #60a0b0; font-style: italic;"># Browse to: http://[IP ADDRESS]:8989/hystrix-examples-webapp</span>
<span style="color: #60a0b0; font-style: italic;"># View it on Hystrix dashboard</span>
<span style="color: #60a0b0; font-style: italic;"># Browse to: http://[IP_ADDRESS]:7979/hystrix-dashboard/</span>
<span style="color: #60a0b0; font-style: italic;"># Enter: http://[IP_ADDRESS]:8989/hystrix-examples-webapp/hystrix.stream</span>
<span style="color: #60a0b0; font-style: italic;"># Click "Monitor Stream"</span>
<span style="color: #60a0b0; font-style: italic;"># See the metrics change with this in one window:</span>
curl <span style="color: #666666;">[</span>IP_ADDRESS<span style="color: #666666;">]</span>:8989/hystrix-examples-webapp/hystrix.stream
<span style="color: #60a0b0; font-style: italic;"># And this in another window:</span>
<span style="color: #007020; font-weight: bold;">while </span><span style="color: #007020;">true</span> ; <span style="color: #007020; font-weight: bold;">do </span>curl <span style="color: #4070a0;">"[IP_ADDRESS]:8989/hystrix-examples-webapp/"</span>; <span style="color: #007020; font-weight: bold;">done</span> <span style="color: #60a0b0; font-style: italic;"># <-- The trailing "/" is required</span>
<span style="color: #60a0b0; font-style: italic;"># Install Turbine: https://github.com/Netflix/Hystrix/wiki/Dashboard</span>
curl -L -O https://github.com/downloads/Netflix/Turbine/turbine-web-1.0.0.war
cp turbine-web-1.0.0.war <span style="color: #bb60d5;">$TOMCAT_HOME</span>/webapps/turbine.war
<span style="color: #60a0b0; font-style: italic;"># Configure Turbine (using Archaius)</span>
vi /root/rss-edge-turbine.properties
<span style="color: #60a0b0; font-style: italic;"># From https://github.com/Netflix/Hystrix/wiki/Dashboard</span>
<span style="color: #60a0b0; font-style: italic;"># Hystrix stream for RSS Edge webapp</span>
turbine.ConfigPropertyBasedDiscovery.default.instances<span style="color: #666666;">=</span>localhost
turbine.instanceUrlSuffix<span style="color: #666666;">=</span>:9090/hystrix.stream
<span style="color: #60a0b0; font-style: italic;"># End edit</span>
<span style="color: #60a0b0; font-style: italic;"># Add Archaius config to Tomcat "archaius.configurationSource.additionalUrls"</span>
vi /etc/tomcat6/tomcat6.conf
<span style="color: #60a0b0; font-style: italic;"># Archaius properties for Turbine and Netflix</span>
<span style="color: #bb60d5;">JAVA_OPTS</span><span style="color: #666666;">=</span><span style="color: #4070a0;">"${JAVA_OPTS} -Darchaius.configurationSource.additionalUrls=file:///root/rss-edge-turbine.properties"</span>
<span style="color: #60a0b0; font-style: italic;"># End edit</span>
service tomcat6 restart
<span style="color: #60a0b0; font-style: italic;"># Should see this in the logs: "URLs to be used as dynamic configuration source: [file:/root/rss-edge-turbine.properties]"</span>
grep rss-edge /usr/share/tomcat6/logs/catalina.out
<span style="color: #60a0b0; font-style: italic;"># Browse to: http://[IP_ADDRESS]:7979/hystrix-dashboard/</span>
<span style="color: #60a0b0; font-style: italic;"># Enter: http://54.206.19.137/turbine/turbine.stream</span>
<span style="color: #60a0b0; font-style: italic;"># Click "Monitor Stream"</span>
<span style="color: #60a0b0; font-style: italic;"># Install hystrix-dashboard in Tomcat</span>
curl -O http://search.maven.org/remotecontent?filepath<span style="color: #666666;">=</span>com/netflix/hystrix/hystrix-dashboard/1.3.8/hystrix-dashboard-1.3.8.war
<span style="color: #60a0b0; font-style: italic;"># Install hystrix-examples-webapp in Tomcat</span>
<span style="color: #007020;">cd </span>Hystrix/hystrix-examples-webapp
../gradlew build
cp build/libs/hystrix-examples-webapp-1.3.9-SNAPSHOT.war /usr/share/tomcat6/webapps/hystrix-examples-webapp.war
</pre>
</div>
<div>
</div>
<!-- HTML generated using hilite.me --><br />
<h2>
Cluster Install</h2>
Turbine really only starts to shine once there are clusters involved.<br />
<br />
Let's set up a cluster of dedicated Edge and Middletier instances, a dedicated Eureka instance (on Tomcat together with Hystrix Dashboard + Turbine) and with ELB in front of the Edge cluster. Something like this:<br />
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> Internet Internet</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> | |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> ------------|----------------------------------|---------------------------------------------</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> | v</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> | AWS ELB</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> | |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> /----------|------------------------\ /|\</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> | v | v v v</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> | Hystrix Dashboard <-- Turbine <----- RSS Edge (x3) </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> | | ^ ^ ^ ^</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> | | / \|/</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> | Eureka <-------- |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> | ^ | |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> \--------------------------|--------/ /|\</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> | v v v</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> ---------- RSS Middletier (x3)</span></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div style="background: #f0f0f0; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #60a0b0; font-style: italic;"># To make things easier, before we scale out we'll set up some convenience </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: #60a0b0; font-style: italic;"># hostnames and scripts - in reality we'd come back and automate this </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: #60a0b0; font-style: italic;"># properly later on with Puppet/Chef/Ansible/Salt/Baked-into-image</span>
<span style="color: #60a0b0; font-style: italic;"># Add an entry with the Private IP address of the server in /etc/hosts e.g.:</span>
127.0.0.1 eureka
vim <span style="color: #4070a0;">"recipes-rss/rss-edge/src/main/resources/edge.properties"</span>
eureka.serviceUrl.default<span style="color: #666666;">=</span>http://eureka/eureka/v2/
<span style="color: #60a0b0; font-style: italic;"># End edit</span>
vim <span style="color: #4070a0;">"recipes-rss/rss-middletier/src/main/resources/middletier.properties"</span>
eureka.serviceUrl.default<span style="color: #666666;">=</span>http://eureka/eureka/v2/
<span style="color: #60a0b0; font-style: italic;"># End edit</span>
<span style="color: #60a0b0; font-style: italic;"># Rebuild both jars</span>
<span style="color: #007020;">cd</span> /root/recipes-rss <span style="color: #666666;">&&</span> gradlew build
<span style="color: #60a0b0; font-style: italic;"># Create the following scripts in root's homedir:</span>
<span style="color: #60a0b0; font-style: italic;"># change_eureka_host.sh</span>
<span style="color: #60a0b0; font-style: italic;">#!/bin/bash</span>
<span style="color: #007020; font-weight: bold;">if</span> <span style="color: #666666;">[</span> <span style="color: #bb60d5;">$# </span>-lt 1 <span style="color: #666666;">]</span>; <span style="color: #007020; font-weight: bold;">then</span>
<span style="color: #007020; font-weight: bold;"> </span><span style="color: #007020;">echo</span> <span style="color: #4070a0;">"Usage: $0 IP_ADDRESS"</span>
<span style="color: #007020; font-weight: bold;">fi</span>
sed -i <span style="color: #4070a0;">"s/.*eureka/$1 eureka/g"</span> /etc/hosts
<span style="color: #60a0b0; font-style: italic;"># start_rss-edge.sh</span>
<span style="color: #60a0b0; font-style: italic;">#!/bin/bash</span>
<span style="color: #007020;">cd</span> /root/recipes-rss
nohup java -Xmx128m -XX:MaxPermSize<span style="color: #666666;">=</span>32m -jar rss-edge/build/libs/rss-edge-*SNAPSHOT.jar &
<span style="color: #007020;">echo</span> <span style="color: #4070a0;">"Output is logged to /root/recipes-rss/logs/rss-edge.log"</span>
<span style="color: #60a0b0; font-style: italic;"># start_rss-middletier.sh</span>
<span style="color: #60a0b0; font-style: italic;">#!/bin/bash</span>
<span style="color: #007020;">cd</span> /root/recipes-rss
nohup java -Xmx128m -XX:MaxPermSize<span style="color: #666666;">=</span>32m -jar rss-middletier/build/libs/rss-middletier-*SNAPSHOT.jar &
<span style="color: #007020;">echo</span> <span style="color: #4070a0;">"Output is logged to /root/recipes-rss/logs/rss-middletier.log"</span>
<span style="color: #60a0b0; font-style: italic;"># tail_rss-edge.sh</span>
<span style="color: #60a0b0; font-style: italic;">#!/bin/bash</span>
tail -100f /root/recipes-rss/logs/rss-edge.log
<span style="color: #60a0b0; font-style: italic;"># tail_rss-middletier.sh</span>
<span style="color: #60a0b0; font-style: italic;">#!/bin/bash</span>
tail -100f /root/recipes-rss/logs/rss-middletier.log
<span style="color: #60a0b0; font-style: italic;"># Save the instance as an AMI</span>
<span style="color: #60a0b0; font-style: italic;"># Now create 6 more instances based off this AMI - they can all be t1.micro's (in fact we can reprovision our main node as a t1.micro and only run Tomcat on it for Eureka, Hystrix Dashboard, & Turbine)</span>
<span style="color: #60a0b0; font-style: italic;"># Optional: Name them in the EC2 console as rss-edge-N, rss-middletier-N, rss-eureka</span>
<span style="color: #60a0b0; font-style: italic;"># Create 2 more security groups: rss-edge & rss-middletier</span>
<span style="color: #60a0b0; font-style: italic;"># Ensure 'rss-edge' security group has ports 9090 & 9092 open</span>
<span style="color: #60a0b0; font-style: italic;"># Ensure 'rss-middletier' security group has ports 9191 & 9192 open</span>
<span style="color: #60a0b0; font-style: italic;"># Create a Load Balancer named 'RssEdgeLoadBalancer' containing all 3 </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: #60a0b0; font-style: italic;"># rss-edge-* nodes. Attach ports 9090 & 9092 to the same ports.</span>
<span style="color: #60a0b0; font-style: italic;"># Optional: add a health check on:</span>
<span style="color: #60a0b0; font-style: italic;"># Protocol: HTTP</span>
<span style="color: #60a0b0; font-style: italic;"># Port: 9092</span>
<span style="color: #60a0b0; font-style: italic;"># Path: /adminres/webadmin/index.html</span>
<span style="color: #60a0b0; font-style: italic;"># Timeout: 5s</span>
<span style="color: #60a0b0; font-style: italic;"># Interval: 0.5min</span>
<span style="color: #60a0b0; font-style: italic;"># Unhealthy Threshold: 2</span>
<span style="color: #60a0b0; font-style: italic;"># Optional: Add cloudwatch alarm in Monitoring tab</span>
<span style="color: #60a0b0; font-style: italic;"># UnHealthyHostCount >= 1 for 1 minute</span>
<span style="color: #60a0b0; font-style: italic;"># Send message to topic "NotifyMe"</span>
<span style="color: #60a0b0; font-style: italic;"># On the Eureka node add the Private IP addresses of the nodes to /etc/hosts </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: #60a0b0; font-style: italic;"># e.g.:</span>
127.0.0.1 eureka
172.31.111.333 rss-edge-1
172.31.111.444 rss-edge-2
172.31.111.555 rss-edge-3
172.31.111.666 rss-middletier-1
172.31.111.777 rss-middletier-2
172.31.111.888 rss-middletier-3
<span style="color: #60a0b0; font-style: italic;"># On the Eureka node - modify the all-turbine.properties</span>
vi /root/all-turbine.properties
turbine.ConfigPropertyBasedDiscovery.rss-edge.instances<span style="color: #666666;">=</span>rss-edge-1,rss-edge-2,rss-edge-3
<span style="color: #60a0b0; font-style: italic;"># End edit</span>
<span style="color: #60a0b0; font-style: italic;"># Create a script to load test the Edge nodes</span>
<span style="color: #60a0b0; font-style: italic;"># hammer_rss_vip.sh</span>
<span style="color: #60a0b0; font-style: italic;">#!/bin/bash</span>
<span style="color: #60a0b0; font-style: italic;"># Usage: hammer_rss_vip.sh [pause_seconds] </span>
<span style="color: #60a0b0; font-style: italic;"># e.g. hammer_rss_vip.sh 0.1 - will pause for 100ms between requests</span>
<span style="color: #60a0b0; font-style: italic;"># e.g. hammer_rss_vip.sh - no pause: fire requests as fast as we can fork processes in the background</span>
<span style="color: #bb60d5;">RSS_EDGE_VIP</span><span style="color: #666666;">=</span>rssedgeloadbalancer-NNNNNNNNNN.xx-region-N.elb.amazonaws.com
<span style="color: #bb60d5;">DELAY</span><span style="color: #666666;">=</span><span style="color: #007020; font-weight: bold;">${</span><span style="color: #bb60d5;">1</span>-0<span style="color: #007020; font-weight: bold;">}</span>
<span style="color: #007020; font-weight: bold;">while </span><span style="color: #007020;">true</span> ; <span style="color: #007020; font-weight: bold;">do</span>
<span style="color: #007020; font-weight: bold;"> </span>curl -s -o /dev/null http://<span style="color: #bb60d5;">$RSS_EDGE_VIP</span>:9090/jsp/rss.jsp &
sleep <span style="color: #bb60d5;">$DELAY</span>
<span style="color: #007020; font-weight: bold;">done</span>
<span style="color: #60a0b0; font-style: italic;"># Now watch Turbine via the Hystrix Dashboard while you load test the Edge via the ELB and see the difference for different timings</span>
hammer_rss_vip 1
hammer_rss_vip 0.1
</pre>
</div>
Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-12537253612995514942013-12-14T17:10:00.001-08:002014-01-14T04:30:03.129-08:00Important concepts in software<div>
The following is a collection of important concepts all software professionals should understand. For each of them, the moment of understanding is often an ephiphany - an "aha" moment when what has up until then either felt good or was second nature now has a label, a vocabulary.<br />
<br />
Jeff Atwood lists the first 3: <a href="http://www.codinghorror.com/blog/2009/03/five-dollar-programming-words.html">http://www.codinghorror.com/blog/2009/03/five-dollar-programming-words.html</a><br />
Brandon Byers lists #4 and #5 when explaining #3: <a href="http://brandonbyars.com/2008/07/21/orthogonality/">http://brandonbyars.com/2008/07/21/orthogonality/</a><br />
Some good definitions in this <a href="http://stackoverflow.com/a/1030631">SO answer</a>.</div>
<div>
<br /></div>
<div>
<b>Idempotence</b><br />
The result of doing something more than once is the exact same result as doing it once<br />
<br /></div>
<div>
<b>Immutability</b><br />
<br /></div>
<div>
<b>Orthogonality</b></div>
<div>
Nonorthogonal systems are hard to manipulate because it's hard to tweak isolated parts<br />
<br />
<b>Composability</b><br />
Features can be combined. Unix philosophy.<br />
<br /></div>
<div>
<b>Consistency</b><br />
<a href="http://en.wikipedia.org/wiki/Principle_of_least_astonishment">Principle of Least Surprise</a><br />
<br />
<b>Determinism</b><br />
There's a single, well-defined correct value. Important for reasoning about what a program is doing at any point in time. In contrast, imperative concurrency can give different answers, depending on a scheduler or whether you're looking or not, and they can even deadlock.<br />
<b><br /></b>
<b>Complexity: Essential vs Accidental</b></div>
<div>
<b><br /></b>
<b>Temporality</b><br />
State changes over time. The temporally discrete nature of imperative computation is an accommodation to a particular style of machine, rather than a natural description of behavior itself<br />
<br /></div>
Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-8396719218226862582013-12-14T06:03:00.003-08:002013-12-14T06:04:20.723-08:00Netflix audit Prod rather than controlling releases through central approval processI asked Adrian Cockcroft & Ben Christensen from Netflix how they control the release cycle of production configuration items (pointing out that each of the levels in the <a href="http://3.bp.blogspot.com/-LvvCo-wWjlo/T9-QCxI60KI/AAAAAAAAAAg/CMOsvfuhn5k/s640/Archaius%2BComposite%2BConfiguration%2BDiagram.png">Archaius hierarchy</a> potentially have a separate release cycle). In short, they don't. Rather, they constantly audit production (using Chronos) and keep track of real-time events (what, when, where) to use for ad-hoc search queries when diagnosing problems. To read between the lines, there's more value in getting changes out quickly and having them fail than not releasing changes while waiting for approval and causing bottlenecks in the approval process. Put more effort into testing in prod (monitoring/alerting/compliance) and less effort into release ceremony.<br />
<br />
This snippet seems to cover it:<br />
<br />
<a href="http://techblog.netflix.com/2013/03/python-at-netflix.html">http://techblog.netflix.com/2013/03/python-at-netflix.html</a><br />
<b>Chronos</b><br />
We push hard to always increase our speed of innovation, and at the same time reduce the cost of making changes in the environment. In the datacenter days, we forced every production change to be logged in a change control system because the first question everyone asks when looking at an issue is “What changed recently?”. We found a formal change control system didn’t work well for with our culture of freedom and responsibility, so we deprecated a formal change control process for the vast majority of changes in favor of Chronos. Chronos accepts events via a REST interface and allows humans and machines to ask questions like “what happened in the last hour?” or “what software did we deploy in the last day?”. It integrates with our monkeys and Asgard so the vast majority of changes in our environment are automatically reported to it, including event types such as deployments, AB tests, security events, and other automated actions.<br />
<br />Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-78309276705227174902013-12-11T16:36:00.000-08:002014-09-24T13:26:13.443-07:00Notes from Netflix OSS Yow Workshop (11/Dec/2013)<h1>
<div class="wiki-content" id="main-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div style="margin: 0px; padding: 0px;">
<span style="font-weight: normal;">Notes taken from this workshop: </span><a class="external-link" href="http://brisbane.yowconference.com.au/content/?url=http%3A%2F%2Fa.confui.com%2Fpublic%2Fconferences%2F517fce8207933939cd000001%2Flocations%2F517fcea7854219379f000001%2Fworkshops%2F5296b0152c6030506700004b%3Fframehost%3Dhttp%3A%2F%2Fbrisbane.yowconference.com.au" rel="nofollow" style="color: #0065b0; font-size: 14px; line-height: 1.4285715; text-decoration: none;">Patterns for Continuous Delivery, Reactive, High Availability, DevOps & Cloud Native Open Source with NetflixOSS</a></div>
<div style="font-weight: normal; margin: 10px 0px 0px; padding: 0px;">
The workshop was essentially a series of presentations by Adrian Cockcroft and Ben Christensen</div>
</div>
</h1>
<h2 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Slides" style="border-bottom-style: solid; border-bottom-width: 1px; color: #0065b0; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px; padding: 0px;">
Slides</h2>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Adrian's Slides:<ul style="list-style-type: disc; margin: 0px;">
<li><a class="external-link" href="https://speakerdeck.com/adrianco/patterns-for-continuous-delivery-reactive-high-availability-devops-and-cloud-native-open-source-with-netflixoss" rel="nofollow" style="color: #0065b0; text-decoration: none;">SpeakerDeck</a></li>
</ul>
</li>
<li>Ben's Slides:<ul style="list-style-type: disc; margin: 0px;">
<li><a class="external-link" href="http://cdn.oreillystatic.com/en/assets/1/event/94/Application%20Resilience%20Engineering%20and%20Operations%20at%20Netflix%20Presentation.pdf" rel="nofollow" style="color: #0065b0; font-size: 14px; line-height: 20px; text-decoration: none;">Application Resilience Engineering and Operations at Netflix</a> [pdf] <-- this is the closest set of slides to the workshop content</li>
<li><a class="external-link" href="https://speakerdeck.com/benjchristensen" rel="nofollow" style="color: #0065b0; text-decoration: none;">Other slides on SpeakerDeck</a></li>
</ul>
</li>
</ul>
</div>
</h1>
<h2 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Diagrams" style="border-bottom-style: solid; border-bottom-width: 1px; color: #0065b0; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px; padding: 0px;">
Diagrams</h2>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div style="margin: 10px 0px 0px; padding: 0px;">
</div>
<div class="gallery" style="margin: 10px; overflow-x: auto; overflow-y: hidden; padding: 0px;">
<table style="border-collapse: collapse;"></table>
</div>
</div>
</h1>
<h4 class="gallery-title" style="color: black; font-size: 14px; font-weight: bold; line-height: 20px; margin: 0px; padding: 0px;">
NetFlix Architecture Diagrams</h4>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-UxUcv6lPHvA/UrDSDyihlOI/AAAAAAAAAKs/xLtVA7QDhZI/s1600/ApiToService.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-UxUcv6lPHvA/UrDSDyihlOI/AAAAAAAAAKs/xLtVA7QDhZI/s320/ApiToService.png" height="240" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/--klSAdw0tbA/UrDSERdXtFI/AAAAAAAAAK0/WRMCO5E1gRg/s1600/Archaius.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/--klSAdw0tbA/UrDSERdXtFI/AAAAAAAAAK0/WRMCO5E1gRg/s320/Archaius.png" height="234" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-QJCMzN_F9mQ/UrDSEZUoM5I/AAAAAAAAAKw/ChETk4uVTwQ/s1600/CloudAccessControl-Bastion.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-QJCMzN_F9mQ/UrDSEZUoM5I/AAAAAAAAAKw/ChETk4uVTwQ/s320/CloudAccessControl-Bastion.png" height="192" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-Jra-hdvRBOE/UrDSErP34KI/AAAAAAAAALA/3C9hZiwCLaw/s1600/Directing&ShapingUserRequests.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-Jra-hdvRBOE/UrDSErP34KI/AAAAAAAAALA/3C9hZiwCLaw/s320/Directing&ShapingUserRequests.png" height="180" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-sBjftz_-H0s/UrDSFdwuzeI/AAAAAAAAALI/wzVpc1qQRgs/s1600/DirectingUserRequests.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-sBjftz_-H0s/UrDSFdwuzeI/AAAAAAAAALI/wzVpc1qQRgs/s320/DirectingUserRequests.png" height="208" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-YFun8HoqN4c/UrDSFh6oWPI/AAAAAAAAALQ/rEEQZh2TslA/s1600/DynamicEndpoints.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-YFun8HoqN4c/UrDSFh6oWPI/AAAAAAAAALQ/rEEQZh2TslA/s320/DynamicEndpoints.png" height="316" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-AJR0767Hc3c/UrDSGUU_xJI/AAAAAAAAALc/-asFGMAik-E/s1600/Edda.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-AJR0767Hc3c/UrDSGUU_xJI/AAAAAAAAALc/-asFGMAik-E/s320/Edda.png" height="190" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-mzZvQhihkOg/UrDSGg-zESI/AAAAAAAAALg/D8BnlsOVkz4/s1600/Eureka.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-mzZvQhihkOg/UrDSGg-zESI/AAAAAAAAALg/D8BnlsOVkz4/s320/Eureka.png" height="198" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-OSdVCNr2-wc/UrDSGNHYFpI/AAAAAAAAALY/r1Afu0hPSq4/s1600/EVCache.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-OSdVCNr2-wc/UrDSGNHYFpI/AAAAAAAAALY/r1Afu0hPSq4/s320/EVCache.png" height="231" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-IDK0whbkE5M/UrDSHCiG5cI/AAAAAAAAAL0/W9XArLKsQa8/s1600/FaultToleranceTypes.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-IDK0whbkE5M/UrDSHCiG5cI/AAAAAAAAAL0/W9XArLKsQa8/s320/FaultToleranceTypes.png" height="244" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-XBLGFZ5BZqA/UrDSLMRk7lI/AAAAAAAAAM0/TmiiUuEONFo/s1600/fluxcapacitor-netflixoss-overview.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-XBLGFZ5BZqA/UrDSLMRk7lI/AAAAAAAAAM0/TmiiUuEONFo/s320/fluxcapacitor-netflixoss-overview.jpg" height="320" width="247" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-KlX3DVmbVnw/UrDSJlrNveI/AAAAAAAAAMU/0Lli0VoE_VY/s1600/Hystrix-CircuitBreaker-Dashboard.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-KlX3DVmbVnw/UrDSJlrNveI/AAAAAAAAAMU/0Lli0VoE_VY/s320/Hystrix-CircuitBreaker-Dashboard.png" height="182" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-h3xa-oPDGi4/UrDSHm6XeqI/AAAAAAAAAL4/ATG30F7HxSk/s1600/Hystrix-CircuitBreaker-StateFlow.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-h3xa-oPDGi4/UrDSHm6XeqI/AAAAAAAAAL4/ATG30F7HxSk/s320/Hystrix-CircuitBreaker-StateFlow.png" height="128" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-GGkbLWPYtxg/UrDSIpbrkyI/AAAAAAAAAMI/iR5S8cX6t38/s1600/Hystrix-CircuitBreaker.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-GGkbLWPYtxg/UrDSIpbrkyI/AAAAAAAAAMI/iR5S8cX6t38/s320/Hystrix-CircuitBreaker.png" height="320" width="229" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-rCbgREXnQW4/UrDSIhYBiqI/AAAAAAAAAME/qTkI-_HtrEs/s1600/Hystrix-ZoomIn.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-rCbgREXnQW4/UrDSIhYBiqI/AAAAAAAAAME/qTkI-_HtrEs/s320/Hystrix-ZoomIn.png" height="164" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-7x1O_EzjIbY/UrDSJ_qdGXI/AAAAAAAAAMY/06hxGpPKN7Y/s1600/NetflixOverview_RSS_arch.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-7x1O_EzjIbY/UrDSJ_qdGXI/AAAAAAAAAMY/06hxGpPKN7Y/s320/NetflixOverview_RSS_arch.png" height="190" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-BNqiOoFgTuA/UrDSKRVaQwI/AAAAAAAAAMc/OV2y63bLBCw/s1600/Zuul-FromSlides.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-BNqiOoFgTuA/UrDSKRVaQwI/AAAAAAAAAMc/OV2y63bLBCw/s320/Zuul-FromSlides.png" height="247" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-3ONCe7vlSpM/UrDSKuh1lwI/AAAAAAAAAMg/oW8gN2C_t8Q/s1600/Zuul.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-3ONCe7vlSpM/UrDSKuh1lwI/AAAAAAAAAMg/oW8gN2C_t8Q/s320/Zuul.png" height="240" width="320" /></a></div>
<div>
<br /></div>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div class="gallery" style="margin: 10px; overflow-x: auto; overflow-y: hidden; padding: 0px;">
<table style="border-collapse: collapse;"><tbody>
<tr><td style="padding: 0px; text-align: center; vertical-align: top;"></td><td style="padding: 0px; text-align: center; vertical-align: top;"></td><td style="padding: 0px; text-align: center; vertical-align: top;"></td><td style="padding: 0px; text-align: center; vertical-align: top;"></td></tr>
</tbody></table>
</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
</div>
</div>
</h1>
<h2 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-CloudatScale" style="border-bottom-style: solid; border-bottom-width: 1px; color: #0065b0; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px; padding: 0px;">
Cloud at Scale</h2>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div style="margin: 10px 0px 0px; padding: 0px;">
<em>Adrian Cockcroft</em></div>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-TimetomarketvsQuality" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Time to market vs Quality</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Aggressively go out and assume that things are broken</li>
<li>e.g. Land grab, market disruption, web services</li>
<li>Default assumptions<ul style="list-style-type: disc; margin: 0px;">
<li>Always shipping code that is broken</li>
<li>Hardware is broken</li>
<li>Operationally defensive</li>
</ul>
</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Needabilitytoseeinrealtime" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Need ability to see in realtime</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Small change, quick to fix</li>
<li>Able to revert</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-CloudNative–anewengineeringchallenge" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Cloud Native – a new engineering challenge</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Construct highly agile and hilgly available service from ephemeral and assumjed broken components</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Inspiration" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Inspiration</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li class="MsoListBulletCxSpFirst"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Release it<ul style="list-style-type: disc; margin: 0px;">
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Bulkhead & circuit-breaker</li>
</ul>
</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Thinking in Systems – Donella H Meadows<ul style="list-style-type: disc; margin: 0px;">
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Chaotic system</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Order from Chaos</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Emergent behaviour is that it shows movies</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Right feedback loops</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Not about software – about building feedback loops & rules around things so that they’re stable & predictable</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Looks like an ants nest – chaotic but order is an emergent property</li>
</ul>
</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Anti-fragile</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Drift Into Failure – Sidney Decker<ul style="list-style-type: disc; margin: 0px;">
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Aircraft industry lessons</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Ways to avoid</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Latent failures</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Netflix outages are unique</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Byzantine</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Have enough margin</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Gradually take fat out of the system – drop dead if you miss a meal</li>
</ul>
</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Everything is obvious – Duncan J. Watts<ul style="list-style-type: disc; margin: 0px;">
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Avoid untrained people rushing in a pushing buttons</li>
</ul>
</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>The REST API Design Handbook<ul style="list-style-type: disc; margin: 0px;">
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>100s of microservices</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Short book ranting on bad apis</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Dell Cloud something</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>$3</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>“REST in Practice” also good</li>
</ul>
</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Continuous Delivery</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Cloudonomics<ul style="list-style-type: disc; margin: 0px;">
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Cost model</li>
<li class="MsoListBulletCxSpMiddle"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Detailed analysis of cloud costs and how it fits together</li>
</ul>
</li>
<li class="MsoListBulletCxSpLast"><span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Phoenix Project</li>
</ul>
</div>
</h1>
<h3 class="MsoListBulletCxSpFirst" id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-HowtogettoCloudNative" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
How to get to Cloud Native</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li class="MsoListBulletCxSpMiddle">Freedom and Responsibility for Developers</li>
<li class="MsoListBulletCxSpMiddle">Decentralize and Automate Ops Activities</li>
<li class="MsoListBulletCxSpMiddle">Integrate DevOps into the Business Organization</li>
<li class="MsoNormal"><span style="font-size: 14px; line-height: 1.4285715;">Re-org!</span></li>
</ul>
</div>
</h1>
<h3 class="MsoListBulletCxSpFirst" id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Fourtransitions" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Four transitions</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li class="MsoListBulletCxSpMiddle">Management: Integrated Roles in a Single Organization<ul style="list-style-type: disc; margin: 0px;">
<li class="MsoListBulletCxSpMiddle">Business, Development, Operations -> BusDevOps</li>
</ul>
</li>
<li class="MsoListBulletCxSpMiddle">Developers: Denormalized Data – NoSQL<ul style="list-style-type: disc; margin: 0px;">
<li class="MsoListBulletCxSpMiddle">Decentralized, scalable, available, polyglot</li>
<li class="MsoListBulletCxSpMiddle">Hardest thing to get everyone’s head around</li>
<li class="MsoListBulletCxSpMiddle">Don't really need transactions anyway</li>
<li class="MsoListBulletCxSpMiddle">Data checkers run around checking bits of data have correct ids and “foreign keys”</li>
<li class="MsoListBulletCxSpMiddle">No such thing as consistency</li>
<li class="MsoListBulletCxSpMiddle">Paranoia covered by backups</li>
</ul>
</li>
<li class="MsoListBulletCxSpMiddle">Responsibility from Ops to Dev: Continuous Delivery<ul style="list-style-type: disc; margin: 0px;">
<li class="MsoListBulletCxSpMiddle">Decentralized small daily production updates</li>
<li class="MsoListBulletCxSpMiddle">Push to prod end of every day</li>
</ul>
</li>
<li class="MsoListBulletCxSpMiddle">Responsibility from Ops to Dev: Agile Infrastructure - Cloud<ul style="list-style-type: disc; margin: 0px;">
<li class="MsoNormal"><span style="font-size: 14px; line-height: 1.4285715;">Hardware in minutes, provisioned directly by developers</span></li>
</ul>
</li>
</ul>
</div>
</h1>
<h3 class="MsoNormal" id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Fittingintopublicscale" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Fitting into public scale</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div class="MsoListBulletCxSpFirst" style="margin: 10px 0px 0px; padding: 0px;">
<span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>1,000 – 100,000 instances is ideal for AWS</div>
<div class="MsoListBulletCxSpMiddle" style="margin: 10px 0px 0px; padding: 0px;">
<span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>500k instances 3 years ago, 5m today</div>
<div class="MsoNormal" style="margin: 10px 0px 0px; padding: 0px;">
<span style="font-size: 14px; line-height: 1.4285715;"> </span><span style="font-family: Symbol; font-size: 14px; line-height: 1.4285715;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span><a class="external-link" href="http://bit.ly/awsiprange" rel="nofollow" style="color: #0065b0; font-size: 14px; line-height: 1.4285715; text-decoration: none;">http://bit.ly/awsiprange</a></div>
</div>
</h1>
<h3 class="MsoListBulletCxSpFirst" id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Netflixdon’tuseAWSfor" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Netflix don’t use AWS for</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div class="MsoListBulletCxSpMiddle" style="margin: 10px 0px 0px; padding: 0px;">
<span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>SaaS Applications – Pagerduty, Onelogin etc.</div>
<div class="MsoListBulletCxSpMiddle" style="margin: 10px 0px 0px; padding: 0px;">
<span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Content Delivery Service</div>
<div class="MsoNormal" style="margin: 10px 0px 0px; padding: 0px;">
<a class="external-link" href="http://bit.ly/awsiprange" rel="nofollow" style="color: #0065b0; font-size: 14px; line-height: 1.4285715; text-decoration: none;"><span style="font-family: Symbol; font-size: 14px; line-height: 1.4285715;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span><span style="font-size: 14px; line-height: 1.4285715;">DNS Service</span> </a></div>
</div>
</h1>
<h3 class="MsoListBulletCxSpFirst" id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-OpenConnectApplianceHardware-NetflixOpenSourceContentDeliveryService" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Open Connect Appliance Hardware - Netflix Open Source Content Delivery Service</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div class="MsoListBulletCxSpMiddle" style="margin: 10px 0px 0px; padding: 0px;">
<span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>5 engineers got together for 4 months and built their own hardware</div>
<div class="MsoListBulletCxSpMiddle" style="margin: 10px 0px 0px; padding: 0px;">
<span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Build it themselves</div>
<div class="MsoListBulletCxSpMiddle" style="margin: 10px 0px 0px; padding: 0px;">
<span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Give them away</div>
<div class="MsoListBulletCxSpMiddle" style="margin: 10px 0px 0px; padding: 0px;">
<span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>$15k</div>
<div class="MsoListBulletCxSpMiddle" style="margin: 10px 0px 0px; padding: 0px;">
<span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Pre-loaded static content</div>
<div class="MsoListBulletCxSpMiddle" style="margin: 10px 0px 0px; padding: 0px;">
<span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Nginx, Bind, Bird bind library</div>
<div class="MsoListBulletCxSpMiddle" style="margin: 10px 0px 0px; padding: 0px;">
<span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>BSD</div>
<div class="MsoListBulletCxSpMiddle" style="margin: 10px 0px 0px; padding: 0px;">
<span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>UFS+ filesystem</div>
<div class="MsoListBulletCxSpMiddle" style="margin: 10px 0px 0px; padding: 0px;">
<span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Unmount disk if it fails, lower capacity – no striping</div>
<div class="MsoListBulletCxSpLast" style="margin: 10px 0px 0px; padding: 0px;">
<span style="font-family: Symbol;">·<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span>Hot content on SSD</div>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-DNSService" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
DNS Service</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Route53 missing too many features</li>
<li>Amazon will clean up in the DNS market when they finish Route53</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-EscapingtheDeathSpiral" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Escaping the Death Spiral</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Get out of the way of innovation<ul style="list-style-type: disc; margin: 0px;">
<li>Process reduction - aggressively</li>
</ul>
</li>
<li>Hardware: Best of breed, by the hour<ul style="list-style-type: disc; margin: 0px;">
<li>If don’t like it – get rid of it</li>
</ul>
</li>
<li>Choices based on scale<ul style="list-style-type: disc; margin: 0px;">
<li>E.g. Big scale = US East Region</li>
<li>Build your own DNS</li>
</ul>
</li>
</ul>
<div style="margin: 10px 0px 0px; padding: 0px;">
<br /></div>
</div>
</h1>
<h2 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-GettingtoCloudNative" style="border-bottom-style: solid; border-bottom-width: 1px; color: #0065b0; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px; padding: 0px;">
Getting to Cloud Native</h2>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-GettingstartedwithNetflixOSSStepbyStep" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 10px 0px 0px; padding: 0px;">
Getting started with NetflixOSS Step by Step</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ol style="list-style-type: decimal; margin: 10px 0px 0px;">
<li>Set up AWS Accounts to get the foundation in place</li>
<li>Security and access management setup</li>
<li>Account Management: Asgard to deploy & Ice for cost monitoring</li>
<li>Build Tools: Aminator to automate baking AMIs</li>
<li>Service Registry and Searchable Account History: Eureka & Edda</li>
<li>Configuration Management: Archaius dynamic property system</li>
<li>Data storage: Cassandra, Astyanax, Priam, EVCache</li>
<li>Dynamic traffic routing: Denominator, Zuul, Ribbon, Karyon</li>
<li>Availability: Simian Army (Chaos Monkey), Hystrix, Turbine</li>
<li>Developer productivity: Blitz4J, GCViz, Pytheas, RxJava</li>
<li>Big Data: Genie for Hadoop PaaS, Lipstick visualizer for Pig, Suro (logging pipeline)</li>
<li>Sample Apps to get started: RSS Reader, ACME Air (IBM), FluxCapacitor </li>
</ol>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-FlowofCode&DatabetweenAWSAccounts" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Flow of Code & Data between AWS Accounts</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Auditable Account: Code with dollar signs goes into this account</li>
<li>Archive Account<ul style="list-style-type: disc; margin: 0px;">
<li>Test account gets refreshed every weekend of all schemas/data from Prod</li>
<li>Trashes test data</li>
<li>Confidential data is encrypted at rest</li>
<li>Tokenise sensitive data</li>
</ul>
</li>
<li>Cloud LDAP gives you access to Production Account</li>
<li>Auditable Account is different LDAP group, need a reason to access it</li>
<li>Vault account needs a background check</li>
<li>Higher security apps<ul style="list-style-type: disc; margin: 0px;">
<li>Monitoring systems inside</li>
<li>Smaller and smaller for higher security</li>
<li>Hyperguard & cloudpassage</li>
</ul>
</li>
<li>Cloud security architect used to be a PCI auditor – could talk to auditors at their level<ul style="list-style-type: disc; margin: 0px;">
<li><span style="font-size: 14px; line-height: 1.4285715;">Had to educate auditors</span></li>
</ul>
</li>
<li>Archive Account<ul style="list-style-type: disc; margin: 0px;">
<li>Versioned</li>
<li>Can’t delete anything from archive account</li>
<li>Delete old copies</li>
<li>PGP encrypted backup copies to Google, last resort DR copy</li>
<li>Immutable logs from Cassandra for full history</li>
</ul>
</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-AccountSecurity" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Account Security</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Protect Accounts<ul style="list-style-type: disc; margin: 0px;">
<li>Two factor authentication for primary login</li>
</ul>
</li>
<li>Delegated Minimum Privilege<ul style="list-style-type: disc; margin: 0px;">
<li>Create IAM roles for everything</li>
<li>Fine-grained – as-needs basis</li>
</ul>
</li>
<li>Security Groups<ul style="list-style-type: disc; margin: 0px;">
<li>Control who can call your services</li>
<li>Every service has a security group with the same name</li>
<li>Have to be in the group to be able to call the service</li>
<li>Managing service ingress permission = customer base</li>
<li>Can ignore services not in my security group</li>
<li>Superset of interactions</li>
<li style="background-image: none; list-style-type: none;"><ul style="list-style-type: disc; margin: 0px;">
<li>Not all customers call the service</li>
<li>Call tree is monitored through other means</li>
</ul>
</li>
</ul>
</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-CloudAccessControl" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Cloud Access Control</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>SSH Bastion</li>
<li>Sumo Logic</li>
<li>Ssh sudo bastion</li>
<li>Can’t ssh between instances – have to go via bastion which wraps sudo with audit logs<ul style="list-style-type: disc; margin: 0px;">
<li>Login is yourself</li>
<li>oq ssh wrapper into machine root or login as regular e.g. default is “dal-prod”</li>
<li>Your user doesn’t exist on machines</li>
<li>E.g. “dal-prod” is 105, “www-prod” is 106</li>
<li>Dont’ run anything as root</li>
<li>Register of accounts – Asgard</li>
</ul>
</li>
<li>Failure modes<ul style="list-style-type: disc; margin: 0px;">
<li>Datacenter dependency</li>
<li>2 copies of bastion host<ul style="list-style-type: disc; margin: 0px;">
<li>Homedir is the same</li>
</ul>
</li>
<li>Scripts that trample in through bastion – bad idea<ul style="list-style-type: disc; margin: 0px;">
<li>NFS server died lost all shares</li>
</ul>
</li>
<li>Datacenters keep breaking your cloud</li>
</ul>
</li>
<li>1 service per host</li>
<li>AWS firewall layer is dodgy – creates variance in the network</li>
</ul>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-QJCMzN_F9mQ/UrDSEZUoM5I/AAAAAAAAAK8/QEvnu_S_6r0/s1600/CloudAccessControl-Bastion.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-QJCMzN_F9mQ/UrDSEZUoM5I/AAAAAAAAAK8/QEvnu_S_6r0/s320/CloudAccessControl-Bastion.png" height="192" width="320" /></a></div>
<div>
<br /></div>
<div style="margin: 10px 0px 0px; padding: 0px;">
</div>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-FastStartAMIs" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Fast Start AMIs</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>AWS Answers</li>
<li>1 Asgard copy per account</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Statelessservicestalktomemcache/Cassandra/RDS" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Stateless services talk to memcache/Cassandra/RDS</h3>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-NoSQLqueries–allRESTcallstowebservices" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 10px 0px 0px; padding: 0px;">
No SQL queries – all REST calls to webservices</h3>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Asgard" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 10px 0px 0px; padding: 0px;">
Asgard</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Grails app</li>
<li>All UI endpoints can be read by adding .json</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Eureka" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Eureka</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div style="margin: 10px 0px 0px; padding: 0px;">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-mzZvQhihkOg/UrDSGg-zESI/AAAAAAAAALs/X0BD-uB52WY/s1600/Eureka.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-mzZvQhihkOg/UrDSGg-zESI/AAAAAAAAALs/X0BD-uB52WY/s320/Eureka.png" height="198" width="320" /></a></div>
<br /></div>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Edda" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Edda</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Timestamped delta cache of service status mongodb/ES back-end</li>
<li>Searchable history of AWS instance, deployment version, etc changes</li>
<li>Every 1min</li>
<li>Janitor monkey cleans up</li>
<li>Eucalyptus = AWS-compatible private cloud – lets you see more underlying infra data e.g. switches</li>
<li>Cloud Trail give you record of calls made to configure cloud and who made them</li>
<li>E.g. Machines that blew up last week no longer exist</li>
<li>Very powerfull for security / auditing</li>
<li>CMDBs in data center don’t work – this actually works – strong assertions</li>
</ul>
<div style="margin: 10px 0px 0px; padding: 0px;">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-AJR0767Hc3c/UrDSGUU_xJI/AAAAAAAAALw/w3DEZV9jIe0/s1600/Edda.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-AJR0767Hc3c/UrDSGUU_xJI/AAAAAAAAALw/w3DEZV9jIe0/s320/Edda.png" height="190" width="320" /></a></div>
<br /></div>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Archaius" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Archaius</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Property Console<ul style="list-style-type: disc; margin: 0px;">
<li>Not OS yet</li>
<li>Based on Pytheas</li>
</ul>
</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-ArchaiusLibraryConfigMgmt" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Archaius Library Config Mgmt</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Hierarchy of properties</li>
<li>Changes are logged with Chronos</li>
</ul>
<div style="margin: 10px 0px 0px; padding: 0px;">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/--klSAdw0tbA/UrDSERdXtFI/AAAAAAAAALE/xEyhXS28r5o/s1600/Archaius.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/--klSAdw0tbA/UrDSERdXtFI/AAAAAAAAALE/xEyhXS28r5o/s320/Archaius.png" height="234" width="320" /></a></div>
<br /></div>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Astyanax" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Astyanax</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>A6x</li>
<li>Son of Hector (brother of Cassandra)</li>
<li>Recipes<ul style="list-style-type: disc; margin: 0px;">
<li>Patterns to solve common problems</li>
</ul>
</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-EVCache" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
EVCache</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Eccentric (Ephemeral) Volatile Cache</li>
<li>Memcache in each zone</li>
<li>“Dynomite” Cassandra-like layer above memcache</li>
<li>Prium-like sidecar exposes metrics over JMX</li>
</ul>
<div style="margin: 10px 0px 0px; padding: 0px;">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-OSdVCNr2-wc/UrDSGNHYFpI/AAAAAAAAALk/-aOitlxolU8/s1600/EVCache.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-OSdVCNr2-wc/UrDSGNHYFpI/AAAAAAAAALk/-aOitlxolU8/s320/EVCache.png" height="231" width="320" /></a></div>
<br /></div>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-RoutingCustomerstoCode" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Routing Customers to Code</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Denominator: DNS for multi-region availability<ul style="list-style-type: disc; margin: 0px;">
<li>Manage traffic via multiple DNS providers with Java code (or command-line)</li>
<li>Talks to Ultra, Dyn, Route53, OpenStack</li>
<li>Pluggable</li>
<li>Ultra does Geo split (partitioning)<ul style="list-style-type: disc; margin: 0px;">
<li>Switch to Dyn if Ultra breaks</li>
</ul>
</li>
</ul>
</li>
<li>Route53 does switching</li>
<li>If a Region goes down denominator switches LB at Route53 layer<ul style="list-style-type: disc; margin: 0px;">
<li>50 endpoints</li>
</ul>
</li>
<li>Talks to Zuul API Router</li>
<li>Zuul – Smart routing<ul style="list-style-type: disc; margin: 0px;">
<li>Groovy filters update every 30s</li>
<li>E.g. block Russian addresses</li>
<li>Similar to masher, apigee</li>
</ul>
</li>
<li>Ribbon<ul style="list-style-type: disc; margin: 0px;">
<li>Internal LB</li>
<li>Wrapper around HTTP Client</li>
<li>Round robin connections</li>
<li>Backed by Eureka</li>
</ul>
</li>
<li><a href="http://techblog.netflix.com/2013/03/karyon-nucleus-of-composable-web-service.html">Karyon</a> – common server container</li>
<ul style="list-style-type: disc; margin: 0px;">
<li>Hello world</li>
<li>Embedded status page console</li>
<li style="background-image: none; list-style-type: none;"></li>
<ul style="list-style-type: disc; margin: 0px;">
<li>Machine readable</li>
<li>Enables conformity monkey</li>
<li>E.g. Reject if it has versions of libraries</li>
</ul>
</ul>
</ul>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-BNqiOoFgTuA/UrDSKRVaQwI/AAAAAAAAAMw/FagEXQrShNg/s1600/Zuul-FromSlides.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-BNqiOoFgTuA/UrDSKRVaQwI/AAAAAAAAAMw/FagEXQrShNg/s320/Zuul-FromSlides.png" height="247" width="320" /></a></div>
<div>
<br /></div>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Availability" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Availability</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Torture Monkeys – barrel fill of monkeys<ul style="list-style-type: disc; margin: 0px;">
<li>Block DNS</li>
<li>Fill up root disk</li>
<li>Unmount ebs</li>
<li>Block access to ec2 apis</li>
<li>CPU busy</li>
<li>Killing all Java processes</li>
</ul>
</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Developerproductivity" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Developer productivity</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Blitz4J – non-blocking logging</li>
<li>GCViz<ul style="list-style-type: disc; margin: 0px;">
<li>Runs off log files</li>
</ul>
</li>
<li>Pytheas – OSS based tooling framework<ul style="list-style-type: disc; margin: 0px;">
<li>Powerful - Just a little code in the right place</li>
<li>Scaffolding</li>
<li>Guice, Jersey, FreeMarker, JQuery, DataTabler, D3, Bootstrap</li>
</ul>
</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-BigData&Analytics" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
BigData & Analytics</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Genie - Hadoop jobs<ul style="list-style-type: disc; margin: 0px;">
<li>Complex Processing of S3 data</li>
</ul>
</li>
<li>Lipstick – visualisation for Pig queries</li>
<li>Suro – event logging pipeline<ul style="list-style-type: disc; margin: 0px;">
<li>Feeds Kafka, Storm, Druid</li>
<li>Alerting</li>
<li>80-100bn events/day</li>
</ul>
</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-SampleApp–RSSReader" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Sample App – RSS Reader</h3>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Glisten–WorkflowDSL–AmazonSimpleWorkflow" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 10px 0px 0px; padding: 0px;">
Glisten – Workflow DSL – Amazon Simple Workflow</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div style="margin: 10px 0px 0px; padding: 0px;">
<br /></div>
</div>
</h1>
<h2 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Scale&Resilience(ResilientAPIPatterns)" style="border-bottom-style: solid; border-bottom-width: 1px; color: #0065b0; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px; padding: 0px;">
Scale & Resilience (Resilient API Patterns)</h2>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div style="margin: 10px 0px 0px; padding: 0px;">
<em>Ben Christensen</em></div>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Constraints" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Constraints</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Client libraries<ul style="list-style-type: disc; margin: 0px;">
<li>Service provides a client library</li>
<li>To enable speed of iteration</li>
<li>While we want resources integration, procedure integration works out faster</li>
</ul>
</li>
<li>Mixed Environment<ul style="list-style-type: disc; margin: 0px;">
<li>Polyglot</li>
</ul>
</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Clientlibraries" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Client libraries</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Deal with Logic, Serialisation, Network Request, Deserialisation, Logic</li>
<li>Bulkheading to prevent socket timeouts<ul style="list-style-type: disc; margin: 0px;">
<li>Limit the blast radius</li>
</ul>
</li>
<li>Hystrix</li>
<li>Tryable Semaphore<ul style="list-style-type: disc; margin: 0px;">
<li>E.g. 3-4 threads will do 50rps</li>
<li>Cap at 10 – don’t reject until hit 10 threads</li>
<li>Reject in non-blocking way – queue</li>
<li>Fast-fail shared load or fall-back</li>
</ul>
</li>
<li>Thread pool<ul style="list-style-type: disc; margin: 0px;">
<li>Size of thread pool + queue size (typically 0)</li>
<li>Slight overhead for extra threads</li>
<li>Gives extra safety of being able to release the blocking user instead of waiting on connection</li>
<li>Enables interrupting blocking threadHystrix Command Object pattern</li>
<li style="background-image: none; list-style-type: none;"><ul style="list-style-type: disc; margin: 0px;">
<li>Synchronous execute</li>
<li>Asynchronously</li>
<li>Circuit open? Rate limit?</li>
</ul>
</li>
</ul>
</li>
<li>Failure options<ul style="list-style-type: disc; margin: 0px;">
<li><span style="font-size: 14px; line-height: 1.4285715;">Fail fast instead of backing up – backed up systems do not recover quickly</span><ul style="list-style-type: disc; margin: 0px;">
<li>Shed load so can start processing immediately</li>
</ul>
</li>
<li>Fail silent<ul style="list-style-type: disc; margin: 0px;">
<li>Netflix shouldn’t fail for customer just because Netflix can’t talk to Facebook</li>
</ul>
</li>
<li>Static fallback<ul style="list-style-type: disc; margin: 0px;">
<li>Instead of turning feature off... turn it to default state (true, DEFAULT_OBJECT)</li>
<li>Fail open instead of failing closed</li>
</ul>
</li>
<li>Stubbed fallback<ul style="list-style-type: disc; margin: 0px;">
<li>Stub parameters with defaults if don’t know</li>
</ul>
</li>
<li>Fallback via network<ul style="list-style-type: disc; margin: 0px;">
<li>Try something else based on similar data</li>
</ul>
</li>
</ul>
</li>
<li>Hystrix<ul style="list-style-type: disc; margin: 0px;">
<li>Each app cluster has a page of Circuit Breakers</li>
<li>Shows last 10s</li>
<li>Links to historical data</li>
<li>Different decisions about</li>
</ul>
</li>
</ul>
<div style="margin: 10px 0px 0px; padding: 0px;">
<br /></div>
<div style="margin: 10px 0px 0px; padding: 0px;">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-GGkbLWPYtxg/UrDSIpbrkyI/AAAAAAAAAMQ/TB1wIxKWAp4/s1600/Hystrix-CircuitBreaker.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-GGkbLWPYtxg/UrDSIpbrkyI/AAAAAAAAAMQ/TB1wIxKWAp4/s320/Hystrix-CircuitBreaker.png" height="320" width="229" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-h3xa-oPDGi4/UrDSHm6XeqI/AAAAAAAAAMA/KAR7XLxt1KQ/s1600/Hystrix-CircuitBreaker-StateFlow.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-h3xa-oPDGi4/UrDSHm6XeqI/AAAAAAAAAMA/KAR7XLxt1KQ/s320/Hystrix-CircuitBreaker-StateFlow.png" height="128" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-KlX3DVmbVnw/UrDSJlrNveI/AAAAAAAAAMk/OyUV8HpSb3A/s1600/Hystrix-CircuitBreaker-Dashboard.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-KlX3DVmbVnw/UrDSJlrNveI/AAAAAAAAAMk/OyUV8HpSb3A/s320/Hystrix-CircuitBreaker-Dashboard.png" height="182" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-rCbgREXnQW4/UrDSIhYBiqI/AAAAAAAAAMM/6-LhZ3lhxCc/s1600/Hystrix-ZoomIn.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-rCbgREXnQW4/UrDSIhYBiqI/AAAAAAAAAMM/6-LhZ3lhxCc/s320/Hystrix-ZoomIn.png" height="164" width="320" /></a></div>
<br /></div>
<div style="margin: 10px 0px 0px; padding: 0px;">
<br /></div>
<div style="margin: 10px 0px 0px; padding: 0px;">
<br /></div>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Deployment" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Deployment</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Zuul Routing Layer<ul style="list-style-type: disc; margin: 0px;">
<li>Replaces ELBs + Commercial proxy layer<ul style="list-style-type: disc; margin: 0px;">
<li>Hated it: 1-2 days to make simple rule change</li>
</ul>
</li>
</ul>
</li>
<li>Simple Pre and Post filters on HTTP Request/Response</li>
<li>Can add & remove filters at runtime</li>
<li>Routing changes</li>
<li>Use Cases<ul style="list-style-type: disc; margin: 0px;">
<li>Want to know all logs for particular User-id across entire fleet – via Turbine</li>
<li>Canary vs Baseline<ul style="list-style-type: disc; margin: 0px;">
<li>Launch 2 clusters</li>
<li>Run through 1 peak cycle</li>
</ul>
</li>
<li>Squeeze Testing<ul style="list-style-type: disc; margin: 0px;">
<li>Impossible with out Zuul</li>
<li>RPS on a particular instance (Math in a filter)</li>
<li>Increment in 5rps increments</li>
<li>Test every binary to see what it’s breaking point is</li>
<li>How many machines will we need in prod?</li>
<li>Is this change inefficient?</li>
<li>Auto-scaling parameters?</li>
<li>Test can’t come close to prod load – different load</li>
<li>Really hard to simulate true load, cache hits</li>
<li>Don’t bother with Load Tests</li>
<li>Squeeze tests as part of prod</li>
<li>Acceptable break?<ul style="list-style-type: disc; margin: 0px;">
<li>Client will typically retry (or get a “Try again” prompt)</li>
<li>Retry will probably hit a different box (out of 100s)</li>
</ul>
</li>
<li>Rules are tested in a Zuul canary<ul style="list-style-type: disc; margin: 0px;">
<li>Prod Zuul cluster – small Zuul cluster</li>
<li>Activate on main cluster after tested</li>
</ul>
</li>
</ul>
</li>
<li>Coalmine<ul style="list-style-type: disc; margin: 0px;">
<li>Long-term canary cluster</li>
<li>Java agents with byte code manipulation</li>
<li>Intercept network traffic</li>
<li>Watch a particular binary</li>
<li>Raise alarm if see network traffic not isolated by Hystrix<ul style="list-style-type: disc; margin: 0px;">
<li>E.g. Someone Flips code Not correctly isolated</li>
</ul>
</li>
<li>Look in Chronous to see what was changed</li>
</ul>
</li>
<li>Production</li>
</ul>
</li>
<li>Scryer – predicitive auto-scaling<ul style="list-style-type: disc; margin: 0px;">
<li>Uses last 4 weeks for any particular day of week</li>
<li>Creates an auto-scaling plan for ASG</li>
<li>Move min-floor up and down, leave max high</li>
<li>5% buffer better than 10-20% buffer for reactive</li>
<li>Still have reactive plan to kick in for a safety net (e.g. Snow day)</li>
<li>Will keep scaling up even if not receiving predicted traffic – avoids outage when traffic suddenly comes back online</li>
</ul>
</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Testing" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Testing</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Testers do manual testing on UI</li>
<li>Engineers have to make their stuff work</li>
<li>Code reviews are up to each team culture</li>
<li>Risky request feedback on pull request</li>
<li>Canary<ul style="list-style-type: disc; margin: 0px;">
<li>If significant degradation – canary test fails</li>
</ul>
</li>
<li>Integration testing?<ul style="list-style-type: disc; margin: 0px;">
<li>Smoke testing does a lot of the service testing</li>
<li>Expected data testing “prod” branch is latest dependency integration<ul style="list-style-type: disc; margin: 0px;">
<li>Nightly build</li>
<li>If fails, integration guys won’t promote it</li>
</ul>
</li>
<li>Not scientific</li>
</ul>
</li>
<li>A/B Testing – See Jason Brown presentation</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-MigratingDataChanges" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Migrating Data Changes</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Don’t make breaking changes</li>
<li>Never attempt to synchronise releases</li>
<li>Wait until all consumers are up-to-date</li>
<li>Client library facade</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-OtherNotes" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Other Notes</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Build compatibility into client library</li>
<li>Client library boils down to just fallback decisions in Hystrix circuit breakers</li>
<li>Working on static analysis to check</li>
<li>Keep track of which clients call which endpoints – know who’s affected by changes</li>
</ul>
<div style="margin: 10px 0px 0px; padding: 0px;">
<br /></div>
</div>
</h1>
<h2 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Performance&Innovation" style="border-bottom-style: solid; border-bottom-width: 1px; color: #0065b0; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px; padding: 0px;">
Performance & Innovation</h2>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div style="margin: 10px 0px 0px; padding: 0px;">
<em style="color: #333333; font-size: 14px; line-height: 1.4285715;">Ben Christensen</em></div>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-SuroEventPipeline" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
<span style="color: black;">Suro Event Pipeline</span></h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Cloud native, dynamic, configurable offline and realtime data sinks</li>
<li>Open-sourced yesterday</li>
<li>S3 demultiplexing -> Hadoop -> BI</li>
<li>Kafka -> Druid and ES</li>
<li style="list-style-type: none;">Druid = realtime data cube<ul style="list-style-type: disc; margin: 0px;">
<li>Not event processing</li>
<li>Counting on multiple dimensions</li>
</ul>
</li>
<li>Can plug Storm into Kafka for event processing</li>
<li>ES for searching events</li>
</ul>
<div style="margin: 10px 0px 0px; padding: 0px;">
<br /></div>
</div>
</h1>
<h2 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Availability.1" style="border-bottom-style: solid; border-bottom-width: 1px; color: #0065b0; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px; padding: 0px;">
Availability</h2>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div style="margin: 10px 0px 0px; padding: 0px;">
<em style="color: #333333; font-size: 14px; line-height: 1.4285715;">Adrian Cockcroft</em><span style="color: #333333;"> </span></div>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Incidentmanagement" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
<span style="color: #333333;"> </span><span style="color: black;">Incident management</span></h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>PagerDuty alerts are automated</li>
<li>Incident creation is not automated</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-CassandraatScale" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Cassandra at Scale</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Boundary.com – network flow analysis<ul style="list-style-type: disc; margin: 0px;">
<li>Small number of nodes for free</li>
</ul>
</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-FailureModesandEffects" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Failure Modes and Effects</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div style="margin: 10px 0px 0px; padding: 0px;">
<img class="confluence-embedded-image" data-image-src="/download/attachments/65801877/image2013-12-17%2010%3A46%3A21.png?version=1&modificationDate=1387241181814&api=v2" src="https://confluence.wotifgroup.com/download/attachments/65801877/image2013-12-17%2010%3A46%3A21.png?version=1&modificationDate=1387241181814&api=v2" style="border: 0px; margin: 0px; max-width: 100%; padding: 0px;" /></div>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Auto-scalingsavesupto70%" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Auto-scaling saves up to 70%</h3>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Janitormonkey" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 10px 0px 0px; padding: 0px;">
Janitor monkey</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Uses Edda to find things not used</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-CompareTCO" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Compare TCO</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Place</li>
<li>Power</li>
<li>Pipes</li>
<li>People</li>
<li style="list-style-type: none;">Patterns<ul style="list-style-type: disc; margin: 0px;">
<li>Managing, overhead, tooling</li>
</ul>
</li>
<li>Jevons Paradox<ul style="list-style-type: disc; margin: 0px;">
<li><span style="font-size: 14px; line-height: 1.4285715;">If you make something more efficient people will consume more of it more people will use it – more than the efficiency gains</span></li>
<li><span style="font-size: 14px; line-height: 1.4285715;">Amazon incent their sales reps to help you save money</span></li>
</ul>
</li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-SizefortheamountofRAMyouneedthenscalehorizontally" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Size for the amount of RAM you need then scale horizontally</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div style="margin: 10px 0px 0px; padding: 0px;">
<br /></div>
<div style="margin: 10px 0px 0px 30px; padding: 0px;">
<br /></div>
<div style="margin: 10px 0px 0px; padding: 0px;">
<br /></div>
</div>
</h1>
<h2 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-Bonusnotesfromsidechatabout"Consumer-DrivenContracts"withSamNewman" style="border-bottom-style: solid; border-bottom-width: 1px; color: #0065b0; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px; padding: 0px;">
Bonus notes from sidechat about "Consumer-Driven Contracts" with Sam Newman</h2>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-RecommendedResourcesfortestingConsumer-DrivenContracts?" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 10px 0px 0px; padding: 0px;">
Recommended Resources for testing Consumer-Driven Contracts?</h3>
<h1>
<div class="wiki-content" style="-webkit-text-stroke-width: 0px; background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 16px; margin: 0px; orphans: auto; padding: 0px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<ul style="list-style-type: disc; margin: 10px 0px 0px;">
<li>Anything by Ian Robinson (“Rest in practice” author)</li>
<li><a class="external-link" href="http://www.infoq.com/articles/consumer-driven-contracts" rel="nofollow" style="color: #0065b0; text-decoration: none;">http://www.infoq.com/articles/consumer-driven-contracts</a></li>
</ul>
</div>
</h1>
<h3 id="NotesfromNetflixOSSYowWorkshop(11/Dec/2013)-OtherTips" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625; margin: 30px 0px 0px; padding: 0px;">
Other Tips</h3>
<h1>
<div class="wiki-content" style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14.399999618530273px; font-weight: normal; line-height: 16px; margin: 0px; padding: 0px;">
<ul style="margin: 10px 0px 0px;">
<li>Rely more on monitoring in prod<ul style="list-style-type: disc; margin: 0px;">
<li>Synthetic Transactions</li>
<li>A/B Testing</li>
</ul>
</li>
<li>Releasing apps together is the wrong way to go<ul style="list-style-type: disc; margin: 0px;">
<li>It’s a smell that your apps are becoming tightly coupled</li>
<li>Ends up tying you back to a monolithic structure</li>
</ul>
</li>
</ul>
<div style="margin-top: 10px; padding: 0px;">
<br /></div>
</div>
<div id="likes-and-labels-container" style="background-color: white; clear: both; color: #333333; font-family: Arial, sans-serif; font-size: 13px; font-weight: normal; margin: 10px 0px; overflow: hidden; padding: 10px 0px;">
<div id="likes-section" style="float: left; margin: 0px; padding: 0px; width: 500.82501220703125px;">
</div>
</div>
</h1>
Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-86543431712739741672013-12-10T15:29:00.001-08:002013-12-10T15:35:49.053-08:00Notes from Yow 2013 - December 9-10 2013<div>
My notes from Yow Brisbane 2013<br />
<a href="http://brisbane.yowconference.com.au/">http://brisbane.yowconference.com.au/</a><br />
<br />
<br />
<b>Jeff Patten</b></div>
<div>
<br /></div>
<div>
Scrum is based on rugby as mentioned in Takeuchi paper. </div>
<div>
To win the game takes more than just playing your position </div>
<div>
Finishing on time != winning </div>
<div>
<br /></div>
<div>
Individuals and pegs prioritise safety over success</div>
<div>
<br /></div>
<div>
Scrum is an agile strategy not the rules of the game</div>
<div>
<br /></div>
<div>
Even if ur in time ur prob wrong anyway</div>
<div>
Ur guesses about the future are prob wrong</div>
<div>
Even if really good = right about 1/3 of the time</div>
<div>
50-80% software fails to meet outcomes</div>
<div>
It's only after delivery that we really understand value</div>
<div>
<br /></div>
<div>
"Design thinking" focuses whole teams in solving customer & user problems</div>
<div>
Edmunds.com, IDEO Tim Brown Change by Design</div>
<div>
<br /></div>
<div>
Empathy, define, ideate, prototype, test</div>
<div>
<br /></div>
<div>
Whole team</div>
<div>
Not always sequential</div>
<div>
<br /></div>
<div>
Lightweight personas</div>
<div>
Sketching independently</div>
<div>
Sketch studio</div>
<div>
Lean us</div>
<div>
Share results</div>
<div>
Internal "trade show"</div>
<div>
<br /></div>
<div>
Stop worrying about velocity & start focusing on winning the game</div>
<div>
<br /></div>
<div>
Adding lean Startup to fix agile</div>
<div>
<br /></div>
<div>
Working software over comprehensive doco</div>
<div>
Kent Beck:</div>
<div>
Validated learning over working software (or comprehensive doco)</div>
<div>
<br /></div>
<div>
Explicit release, measure</div>
<div>
Goals in Rpv: revenue per visit</div>
<div>
<br /></div>
<div>
Adding lean startup to fix design thinking</div>
<div>
<br /></div>
<div>
Spend time learning rather than arguing</div>
<div>
<br /></div>
<div>
Process is a fame strategy and au to win</div>
<div>
Invest in understanding problem</div>
<div>
Invest in validating outcomes</div>
<div>
Maximise ? Minimise ?</div>
<div>
<br /></div>
<div>
Do it anyway they don't know what you're doing anyway</div>
<div>
Just do it award</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b>Charles Nutter @headius</b></div>
<div>
Drip</div>
<div>
Indy</div>
<div>
Graal Interpreter/optimiser</div>
<div>
Truffle AST</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b>Michael Nygard</b></div>
<div>
<br /></div>
<div>
Culture</div>
<div>
Enablement</div>
<div>
- make ops responsible for fast delivery</div>
<div>
- make dev responsible for uptime</div>
<div>
<br /></div>
<div>
Share</div>
<div>
- share power</div>
<div>
-- tools from ops side that let devs self-deploy</div>
<div>
- Share knowledge </div>
<div>
-- Donella Meadows</div>
<div>
--- "structure of info flows"</div>
<div>
-- big visible displays - first way to introduce devops</div>
<div>
<br /></div>
<div>
Dev is prod</div>
<div>
- Development infrastructure is mission critical</div>
<div>
<br /></div>
<div>
Statistical Sophistication</div>
<div>
- Dan McKinley Homo Narrens</div>
<div>
<br /></div>
<div>
Cautions</div>
<div>
- hype cycle</div>
<div>
- the paradox of automation</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
System comprehension tools?</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b>Patrick Debois</b></div>
<div>
<br /></div>
<div>
Monitoring Stream Mapping</div>
<div>
- reverse vsm</div>
<div>
1. Seeing the devops system</div>
<div>
2. Focus on flow</div>
<div>
- remove friction</div>
<div>
- Tech debt vs social debt</div>
<div>
- codifying-devops-area-practices</div>
<div>
Example practices</div>
<div>
- infra as code</div>
<div>
-- same tech versioning/testing</div>
<div>
-- executable doco</div>
<div>
-- common language</div>
<div>
- reproducible envs </div>
<div>
- monitoring & metrics</div>
<div>
-- enable devs to see prod usage/problems </div>
<div>
-- make it self-servicing </div>
<div>
-- devs can add metric to their code (statsd)</div>
<div>
--- namespaces</div>
<div>
-- make non-subjective decisions at start of project</div>
<div>
- Devs wearing pagers</div>
<div>
-- devs learn ops</div>
<div>
-- ops learn dev</div>
<div>
-- write better tests</div>
<div>
- chaos monkey</div>
<div>
- Zoome Atlassian </div>
<div>
-- visualise prod errors frequency per group </div>
<div>
-- enable extraction if context</div>
<div>
- ChatOps</div>
<div>
-- shared view</div>
<div>
- Public post-mortems</div>
<div>
-- chat makes it visible</div>
<div>
- Game days</div>
<div>
-- Find bottlenecks</div>
<div>
- SadOps</div>
<div>
- "You build it you run it"</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b>Lyssa Adkins & Michael Spyd</b></div>
<div>
<i>Hiring Agile Coach</i></div>
<div>
Silent release plan</div>
<div>
ICAgile</div>
<div>
Learning Objectives</div>
<div>
Agile coach</div>
<div>
Agile team facilitator</div>
<div>
Be good at practitioner </div>
<div>
Be good at teaching or mentoring</div>
<div>
Be good at coaching or facilitating</div>
<div>
Only one of tech/busn/transformation</div>
<div>
<br /></div>
<div>
Coaching lessons </div>
<div>
Software ed</div>
<div>
ICF</div>
<div>
Email Lyssa</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b>Sam Newman</b></div>
<div>
Hexagonal Architecture - Alistair Cockburn</div>
<div>
***Tip: Standardise gaps between services but be flexible about internals</div>
<div>
Integration style evolution</div>
<div>
- Data oriented -> procedure oriented -> doc oriented -> resource oriented</div>
<div>
- further right = less coupled</div>
<div>
Tip: Avoid RPC or shared serialisation</div>
<div>
Have at most 3 ways of integrating - not 20</div>
<div>
Books:</div>
<div>
- "Web API Design" Apigee</div>
<div>
-- cross some sections out</div>
<div>
- "REST in Practice"</div>
<div>
Tip: Avoid Distributed transactions/locking at all costs (CAP theorem)</div>
<div>
Monitoring</div>
<div>
- have to get *much* better at monitoring</div>
<div>
- not badass if u use ssh multiplexer</div>
<div>
- use synthetic transaction to test prod</div>
<div>
- use correlation ids to help track down nasty bugs</div>
<div>
Deployments</div>
<div>
- abstract underlying deployment</div>
<div>
- Tip: have a single way of deploying to any env</div>
<div>
Testing</div>
<div>
- integration test</div>
<div>
-- lots of services = while to run, false negatives, poor isolation, large scope to blame</div>
<div>
-- keep to a minimum, how to test other service without intg test?</div>
<div>
- A -> B(v2)</div>
<div>
- diff apps should know what resources they need</div>
<div>
- Upstream (A) writes test that tests dependencies</div>
<div>
- downstream (B) runs those tests</div>
<div>
- "Consumer driven contracts"</div>
<div>
- growing pile of pending stuff</div>
<div>
-- big release = higher risk</div>
<div>
*** - Tip: don't let releases build up - release as soon as possible, preferably one at a time</div>
<div>
- if u can't fix that before building new service</div>
<div>
Architectural safety</div>
<div>
- cascading failure</div>
<div>
- Tip: use timeouts, circuit breakers & bulkheads to avoid cascading failures</div>
<div>
-- fail fast, don't pass up the chain</div>
<div>
-- read Release It</div>
<div>
Tip: Service templates make it easy to do the right thing for each deployment platform - implement conventions yourself</div>
<div>
Delay the need to intro breaking change</div>
<div>
- consumer driven contracts </div>
<div>
- prefer ver # in URI</div>
<div>
- coexist in same app as endpoints </div>
<div>
<br /></div>
<div>
Other Talks</div>
<div>
- "Rapid release"</div>
<div>
- "Macro to micro"</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b>Aaron Bedra</b></div>
<div>
Braintree</div>
<div>
ModSecurity</div>
<div>
"Carded", "carding rings" = criminals test stolen credit cards against your website to then sell on black market</div>
<div>
- abuse of your infra</div>
<div>
HTTP method Verb distribution is important</div>
<div>
Getrepsheet.com</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b>Adrian Cockcroft</b></div>
<div>
bit.ly/netflix-workshop</div>
<div>
Anti-fragility = enough stress </div>
<div>
- Not to break or go flabby</div>
<div>
<br /></div>
<div>
Speed at scale = everything is broken</div>
<div>
Karyon status page is machine and human readable e.g. Chaos monkey</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b>Joel Pobar</b></div>
<div>
32core, 144Gb, 3tb </div>
<div>
Entire code base for all teams</div>
<div>
Fabricator</div>
<div>
- Image macros</div>
<div>
ODS</div>
<div>
Perf lab</div>
<div>
Scuba</div>
<div>
Cubism</div>
<div>
<br /></div>
<div>
<br /></div>
Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-16710933224959243772013-09-24T04:22:00.000-07:002013-09-25T14:38:36.766-07:00PuppetConf 2013 LiveStreaming Notes<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px;">
<span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">PuppetConf 2013 was held on 22nd & 23rd August. I tuned in to the live streaming and took the notes below.</span></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px;">
<span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">Full list of talks with links to pages with video and slides: </span><a data-mce-href="http://puppetlabs.com/resources/puppetconf-2013" data-mce-style="line-height: 1.4285715;" href="http://puppetlabs.com/resources/puppetconf-2013" style="color: rgb(59, 115, 175) !important; line-height: 1.4285715; text-decoration: none;">http://puppetlabs.com/resources/puppetconf-2013</a><br />
<span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">Full playlist of videos: </span><a data-mce-href="http://www.youtube.com/playlist?list=PLV86BgbREluU02Ytlz80seDSKAbkx5pRg" data-mce-style="line-height: 1.4285715;" href="http://www.youtube.com/playlist?list=PLV86BgbREluU02Ytlz80seDSKAbkx5pRg" style="color: rgb(59, 115, 175) !important; line-height: 1.4285715; text-decoration: none;">http://www.youtube.com/playlist?list=PLV86BgbREluU02Ytlz80seDSKAbkx5pRg</a><span data-mce-style="color: #333333; font-size: 20.0px; line-height: 1.5;" style="font-size: 20px; line-height: 1.5;"> </span></div>
<h1 style="border-bottom-style: solid; border-bottom-width: 2px; color: #003366; font-family: Arial, sans-serif; font-size: 24px; font-weight: normal; line-height: 1.25; margin: 30px 0px 0px;">
<span data-mce-style="color: #333333; font-size: 20.0px; line-height: 1.5;" style="color: #333333; font-size: 20px; line-height: 1.5;">Thursday 2013-08-23 [Friday BNE time]</span></h1>
<ul style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin: 10px 0px 0px;">
<li><a data-mce-href="http://puppetconf2013b.sched.org/event/7aaf3305604487f5ffd2c0dd6bc7ff0c" href="http://puppetconf2013b.sched.org/event/7aaf3305604487f5ffd2c0dd6bc7ff0c" style="color: rgb(59, 115, 175) !important; text-decoration: none;"><strong>9:40am [2:40am] • </strong>Keynote: Why Did We Think Large Scale Distributed Systems Would be Easy?</a> – Gordon Rowell - Google</li>
<li><a data-mce-href="http://puppetconf2013b.sched.org/event/5e29b527170fa108e33a0273ab001879" href="http://puppetconf2013b.sched.org/event/5e29b527170fa108e33a0273ab001879" style="color: rgb(59, 115, 175) !important; text-decoration: none;"><strong>10:15am [3:15am] • </strong>Keynote: Open Sourcing the Cloud</a> – Brian Stevens - RedHat</li>
<li><a data-mce-href="http://puppetconf2013b.sched.org/event/fb5f02870c7be30bd6e9cf39b4470139" href="http://puppetconf2013b.sched.org/event/fb5f02870c7be30bd6e9cf39b4470139" style="color: rgb(59, 115, 175) !important; text-decoration: none;"><strong>11:10am [4:10am] • </strong>How Do We Better Sell DevOps?</a> – Gene Kim</li>
<li><a data-mce-href="http://puppetconf2013b.sched.org/event/eb4724b7aadce4f7d837b7a161ee3b51" href="http://puppetconf2013b.sched.org/event/eb4724b7aadce4f7d837b7a161ee3b51" style="color: rgb(59, 115, 175) !important; text-decoration: none;"><strong>1:30pm [6:30am] • </strong>Nobody Has To Die Today: Keeping The Peace With The Other Meat Sacks</a> – Mykel Alvis</li>
<li><a data-mce-href="http://puppetconf2013b.sched.org/event/f6abdc4a8ca931b8a5a781ba260663e5" href="http://puppetconf2013b.sched.org/event/f6abdc4a8ca931b8a5a781ba260663e5" style="color: rgb(59, 115, 175) !important; text-decoration: none;"><strong>2:20pm [7:20am] • </strong>Vampires vs Werewolves: Ending the War Between Developers and Sysadmins with Puppet</a> – Bess Sadler - Stanford</li>
<li><a data-mce-href="http://puppetconf2013b.sched.org/event/9dbc4eb984c4648b1690475f88d532f8" href="http://puppetconf2013b.sched.org/event/9dbc4eb984c4648b1690475f88d532f8" style="color: rgb(59, 115, 175) !important; text-decoration: none;"><strong>3:10pm [8:10am] • </strong>So You've Got Scalability. Now What?</a> – Carla Souza – Reliant</li>
<li><a data-mce-href="http://puppetconf2013b.sched.org/event/38fb0bf04975e9e421ef58f28f341abf" href="http://puppetconf2013b.sched.org/event/38fb0bf04975e9e421ef58f28f341abf" style="color: rgb(59, 115, 175) !important; text-decoration: none;"><strong>4:20pm [9:20am] • </strong>Multi-Provider Vagrant: AWS, VMware, and More</a> – Mitchell Hashimoto</li>
<li><a data-mce-href="http://puppetconf2013b.sched.org/event/9cc8346bbafa29f77633699c3f8e5a6d" href="http://puppetconf2013b.sched.org/event/9cc8346bbafa29f77633699c3f8e5a6d" style="color: rgb(59, 115, 175) !important; text-decoration: none;"><strong>5:10pm [10:10am] • </strong>Building Data-Driven Infrastructure with Puppet</a> – James Fryman - GitHub</li>
</ul>
<h2 style="border-bottom-style: solid; border-bottom-width: 1px; color: #333333; font-family: Arial, sans-serif; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px;">
<a data-mce-href="http://puppetconf2013b.sched.org/event/7aaf3305604487f5ffd2c0dd6bc7ff0c?iframe=yes&w=990&sidebar=yes&bg=no" href="http://puppetconf2013b.sched.org/event/7aaf3305604487f5ffd2c0dd6bc7ff0c?iframe=yes&w=990&sidebar=yes&bg=no" style="color: rgb(59, 115, 175) !important; text-decoration: none;">Keynote: Why Did We Think Large Scale Distributed Systems Would be Easy?</a></h2>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetlabs.com/presentations/keynote-why-did-we-think-large-scale-distributed-systems-would-be-easy" href="http://puppetlabs.com/presentations/keynote-why-did-we-think-large-scale-distributed-systems-would-be-easy" style="color: rgb(59, 115, 175) !important; text-decoration: none;">http://puppetlabs.com/presentations/keynote-why-did-we-think-large-scale-distributed-systems-would-be-easy</a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Google's Corporate Engineering SRE team provides infrastructure services used by many of Google's desktops, laptops and servers. This talk gives an overview of the design philosophy, challenges, technologies and some interesting failures seen while implementing infrastructure at scale.</div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetconf2013b.sched.org/speaker/gordon_rowell.1r7gqhos?iframe=yes&w=990&sidebar=yes&bg=no" href="http://puppetconf2013b.sched.org/speaker/gordon_rowell.1r7gqhos?iframe=yes&w=990&sidebar=yes&bg=no" style="color: rgb(59, 115, 175) !important; text-decoration: none;" title="Gordon Rowell"><strong>Gordon Rowell</strong></a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Site Reliability Manager, Google</div>
<h3 style="font-family: Arial, sans-serif; font-size: 16px; font-variant: small-caps; line-height: 1.5625; margin: 30px 0px 0px;">
<strong>Notes:</strong></h3>
<ul style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin: 10px 0px 0px;">
<li>Post-mortems not about blame – lots to leran from medical community</li>
<li>Thundering herds: Randomise cron jobs</li>
<li>Anycast<ul style="list-style-type: disc; margin: 0px;">
<li>Helps you be consistent</li>
<li>Traffic could go anywhere</li>
<li>Can I handle any type of query?</li>
<li>IP address injection in route table</li>
<li>TCP or UDP</li>
<li>Won’t maintain state</li>
</ul>
</li>
<li>Cascading failures far worse than dropping traffic</li>
<li>Diversity for people – not for platforms</li>
<li>If you can’t automate it, you shouldn’t do it</li>
</ul>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Q: DNS Round-Robin depend on client?</div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
SNMP</div>
<ul style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin: 10px 0px 0px;">
<li>Canary<ul style="list-style-type: disc; margin: 0px;">
<li>Dev, Test, Stage envs for devs</li>
<li>Turn some machines into canary machines</li>
<li>Full cluster of canaries</li>
<li>Then Rollout</li>
</ul>
</li>
<li>How to round-robin DB backends?<ul style="list-style-type: disc; margin: 0px;">
<li>Typically local LB</li>
<li>Not anycast</li>
<li>Tricky</li>
<li>Depends how persistent</li>
<li>Inter-transaction consistency</li>
<li>No good/complete answer</li>
</ul>
</li>
</ul>
<h2 style="border-bottom-style: solid; border-bottom-width: 1px; color: #333333; font-family: Arial, sans-serif; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px;">
<a data-mce-href="http://puppetconf2013b.sched.org/event/5e29b527170fa108e33a0273ab001879?iframe=yes&w=990&sidebar=yes&bg=no" href="http://puppetconf2013b.sched.org/event/5e29b527170fa108e33a0273ab001879?iframe=yes&w=990&sidebar=yes&bg=no" style="color: rgb(59, 115, 175) !important; text-decoration: none;">Keynote: Open Sourcing the Cloud</a></h2>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetlabs.com/presentations/keynote-open-sourcing-cloud" href="http://puppetlabs.com/presentations/keynote-open-sourcing-cloud" style="color: rgb(59, 115, 175) !important; text-decoration: none;">http://puppetlabs.com/presentations/keynote-open-sourcing-cloud</a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Red Hat is putting serious emphasis on cloud computing – with the goal of building agile infrastructure and platform clouds, which can be used to free developers and IT to do great things, faster. Brian will talk about how Red Hat’s “all-in” technology investments will help make this happen; including the external, upstream open source development model for RHEL, the Red Hat OpenStack community and the elasticity of deploying Openshift on top of Openstack.</div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetconf2013b.sched.org/speaker/brian_stevens.1r7gqhmu?iframe=yes&w=990&sidebar=yes&bg=no" href="http://puppetconf2013b.sched.org/speaker/brian_stevens.1r7gqhmu?iframe=yes&w=990&sidebar=yes&bg=no" style="color: rgb(59, 115, 175) !important; text-decoration: none;" title="Brian Stevens"><strong>Brian Stevens</strong></a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
CTO and VP, Worldwide Engineering, Red Hat</div>
<h3 style="font-family: Arial, sans-serif; font-size: 16px; font-variant: small-caps; line-height: 1.5625; margin: 30px 0px 0px;">
<strong>Notes:</strong></h3>
<ul style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin: 10px 0px 0px;">
<li>Mantra: Upstream First</li>
<li>Open Daylight: Open-source network-controlled network<ul style="list-style-type: disc; margin: 0px;">
<li>Decouple application workflows from infrastructure workflows</li>
<li>Network virtualisation – similar to server viritualisation</li>
<li>Neutron ties together OpenStack and OpenDaylight</li>
</ul>
</li>
<li>CI in OpenShift<ul style="list-style-type: disc; margin: 0px;">
<li>Clone production service</li>
<li>Jenkins cartridge</li>
<li>Easy</li>
<li>Don’t have to modify app, but will change the way you work</li>
</ul>
</li>
</ul>
<h2 style="border-bottom-style: solid; border-bottom-width: 1px; color: #333333; font-family: Arial, sans-serif; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px;">
<a data-mce-href="http://puppetconf2013b.sched.org/event/fb5f02870c7be30bd6e9cf39b4470139?iframe=yes&w=990&sidebar=yes&bg=no" href="http://puppetconf2013b.sched.org/event/fb5f02870c7be30bd6e9cf39b4470139?iframe=yes&w=990&sidebar=yes&bg=no" style="color: rgb(59, 115, 175) !important; text-decoration: none;">How Do We Better Sell DevOps?</a></h2>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
In this talk, I will share my top lessons learned over my years studying high performing IT organizations on how to sell the value of DevOps, and help other stakeholders and executives have their own a-ha moments. I will talk about specific stories about the circumstances that led to these a-ha moments, how they created DevOps champions in surprising places (e.g., Development, CTOs, Product Management, UX, Infosec) in organizations you'll recognize, and how they enabled implementing DevOps patterns that had awesome results.</div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetconf2013b.sched.org/speaker/gene_kim.6bjo8yz?iframe=yes&w=990&sidebar=yes&bg=no" href="http://puppetconf2013b.sched.org/speaker/gene_kim.6bjo8yz?iframe=yes&w=990&sidebar=yes&bg=no" style="color: rgb(59, 115, 175) !important; text-decoration: none;" title="Gene Kim"><strong>Gene Kim</strong></a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Author "The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win", IT Revolution Press</div>
<h2 style="border-bottom-style: solid; border-bottom-width: 1px; color: #333333; font-family: Arial, sans-serif; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px;">
<a data-mce-href="http://puppetconf2013b.sched.org/event/eb4724b7aadce4f7d837b7a161ee3b51?iframe=yes&w=990&sidebar=yes&bg=no" href="http://puppetconf2013b.sched.org/event/eb4724b7aadce4f7d837b7a161ee3b51?iframe=yes&w=990&sidebar=yes&bg=no" style="color: rgb(59, 115, 175) !important; text-decoration: none;">Nobody Has To Die Today: Keeping The Peace With The Other Meat Sacks</a></h2>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<br /></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
A frank (and, frankly, loud) discussion about the kinds of miscommunication that arise between developers and operations, how it leads to trouble and possible ways we can avoid (figurative) violence in the workplace using both social techniques as well as tooling.</div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetconf2013b.sched.org/speaker/mykel_alvis.1r7gqhsg?iframe=yes&w=990&sidebar=yes&bg=no" href="http://puppetconf2013b.sched.org/speaker/mykel_alvis.1r7gqhsg?iframe=yes&w=990&sidebar=yes&bg=no" style="color: rgb(59, 115, 175) !important; text-decoration: none;" title="Mykel Alvis"><strong>Mykel Alvis</strong></a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Sr. DevOps Consultant, MomentumSI</div>
<h2 style="border-bottom-style: solid; border-bottom-width: 1px; color: #333333; font-family: Arial, sans-serif; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px;">
<a data-mce-href="http://puppetconf2013b.sched.org/event/f6abdc4a8ca931b8a5a781ba260663e5?iframe=yes&w=990&sidebar=yes&bg=no" href="http://puppetconf2013b.sched.org/event/f6abdc4a8ca931b8a5a781ba260663e5?iframe=yes&w=990&sidebar=yes&bg=no" style="color: rgb(59, 115, 175) !important; text-decoration: none;">Vampires vs Werewolves: Ending the War Between Developers and Sysadmins with Puppet</a></h2>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetlabs.com/presentations/vampires-vs-werewolves-ending-war-between-developers-and-sysadmins-puppet" href="http://puppetlabs.com/presentations/vampires-vs-werewolves-ending-war-between-developers-and-sysadmins-puppet" style="color: rgb(59, 115, 175) !important; text-decoration: none;">http://puppetlabs.com/presentations/vampires-vs-werewolves-ending-war-between-developers-and-sysadmins-puppet</a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Developers need to be able to write software and deploy it, and often require cutting edge software tools and system libraries. Sysadmins are charged with maintaining stability in the production environment, and so are often resistant to rapid upgrade cycles. This has traditionally pitted us against each other, but it doesn't have to be that way. Using tools like puppet for maintaining and testing server configuration, nagios for monitoring, and jenkins for continuous code integration, Stanford University Library has brokered a peace that has given us the ability to maintain a stable production environment with a rapid upgrade cycle. I'll discuss the individual tools, our server configuration, and the social engineering that got us here.</div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetconf2013b.sched.org/speaker/bess_sadler.1r7gqhnf?iframe=yes&w=990&sidebar=yes&bg=no" href="http://puppetconf2013b.sched.org/speaker/bess_sadler.1r7gqhnf?iframe=yes&w=990&sidebar=yes&bg=no" style="color: rgb(59, 115, 175) !important; text-decoration: none;" title="Bess Sadler"><strong>Bess Sadler</strong></a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Manager for Application Development, Stanford University Library</div>
<h3 style="font-family: Arial, sans-serif; font-size: 16px; font-variant: small-caps; line-height: 1.5625; margin: 30px 0px 0px;">
<strong>Notes:</strong></h3>
<ul style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin: 10px 0px 0px;">
<li>You don’t take risks with ppl you don’t trust<ul style="list-style-type: disc; margin: 0px;">
<li>Need to start with building interpersonal relationships, building trust</li>
<li>Let go of the anger</li>
<li>Recognise common goals</li>
</ul>
</li>
<li>Tactics<ul style="list-style-type: disc; margin: 0px;">
<li>Get to know the people on the other side, take them out for coffee</li>
<li>Show of good faith, show them your test coverage</li>
<li>Monitoring with e.g. Nagios</li>
<li>Goal: Are all of our projects functioning correctly right now?</li>
<li>Friendly Manual</li>
<li>Puppet: the ultimate challenge in vampire-werewolf relationships</li>
<li> </li>
</ul>
</li>
<li><a data-mce-href="http://www.codinghorror.com/blog/2010/08/vampires-programmers-versus-werewolves-sysadmins.html" href="http://www.codinghorror.com/blog/2010/08/vampires-programmers-versus-werewolves-sysadmins.html" style="color: rgb(59, 115, 175) !important; text-decoration: none;">http://www.codinghorror.com/blog/2010/08/vampires-programmers-versus-werewolves-sysadmins.html</a><ul style="list-style-type: disc; margin: 0px;">
<li>The art of managing vampires and werewolves, I think, is to ensure that they spend their time not fighting amongst themselves, but instead, <strong>using those supernatural powers together to achieve a common goal they could not otherwise</strong>. In my experience, when programmers and system administrators fight, it's because they're bored. You haven't given them a sufficiently daunting task, one that requires the full combined use of their unique skills to achieve.</li>
<li>Remember, it's not vampires versus werewolves. It's vampires <em>and </em>werewolves.<span data-mce-style="font-size: 14.0px; line-height: 1.4285715;" style="line-height: 1.4285715;"> </span></li>
</ul>
</li>
</ul>
<h2 style="border-bottom-style: solid; border-bottom-width: 1px; color: #333333; font-family: Arial, sans-serif; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px;">
<a data-mce-href="http://puppetconf2013b.sched.org/event/9dbc4eb984c4648b1690475f88d532f8?iframe=yes&w=990&sidebar=yes&bg=no" href="http://puppetconf2013b.sched.org/event/9dbc4eb984c4648b1690475f88d532f8?iframe=yes&w=990&sidebar=yes&bg=no" style="color: rgb(59, 115, 175) !important; text-decoration: none;">So You've Got Scalability. Now What?</a></h2>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetlabs.com/presentations/so-youve-got-scalability-now-what" href="http://puppetlabs.com/presentations/so-youve-got-scalability-now-what" style="color: rgb(59, 115, 175) !important; text-decoration: none;">http://puppetlabs.com/presentations/so-youve-got-scalability-now-what</a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Managing over 10k nodes brings unique challenges, one of them is managing all data in a scalable way, but solving the scalability issue isn't enough. The data must be available and manageable in a user friendly way. This talk is about how we successfully implemented a solution using Hiera, Redis, Sensu, Rails and Grape that made us capable of providing our customers with the ability to not only manage their own data but also build their own applications to manage their infrastructure using our API.</div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetconf2013b.sched.org/speaker/carla_souza.1r7gqhlr?iframe=yes&w=990&sidebar=yes&bg=no" href="http://puppetconf2013b.sched.org/speaker/carla_souza.1r7gqhlr?iframe=yes&w=990&sidebar=yes&bg=no" style="color: rgb(59, 115, 175) !important; text-decoration: none;" title="Carla Souza"><strong>Carla Souza</strong></a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Senior Engineer, Reliant Security</div>
<h3 style="font-family: Arial, sans-serif; font-size: 16px; font-variant: small-caps; line-height: 1.5625; margin: 30px 0px 0px;">
<strong>Notes:</strong></h3>
<ul style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin: 10px 0px 0px;">
<li><a data-mce-href="https://github.com/reliantsecurity/hiera-resources" href="https://github.com/reliantsecurity/hiera-resources" style="color: rgb(59, 115, 175) !important; text-decoration: none;">https://github.com/reliantsecurity/hiera-resources</a></li>
</ul>
<h2 style="border-bottom-style: solid; border-bottom-width: 1px; color: #333333; font-family: Arial, sans-serif; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px;">
<a data-mce-href="http://puppetconf2013b.sched.org/event/38fb0bf04975e9e421ef58f28f341abf?iframe=yes&w=990&sidebar=yes&bg=no" href="http://puppetconf2013b.sched.org/event/38fb0bf04975e9e421ef58f28f341abf?iframe=yes&w=990&sidebar=yes&bg=no" style="color: rgb(59, 115, 175) !important; text-decoration: none;">Multi-Provider Vagrant: AWS, VMware, and More</a></h2>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetlabs.com/presentations/multi-provider-vagrant-aws-vmware-and-more" href="http://puppetlabs.com/presentations/multi-provider-vagrant-aws-vmware-and-more" style="color: rgb(59, 115, 175) !important; text-decoration: none;">http://puppetlabs.com/presentations/multi-provider-vagrant-aws-vmware-and-more</a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
With Vagrant 1.1+, you can use the same configuration and workflow to spin up and provision machines in VirtualBox, VMware, AWS, RackSpace, and more. You get all the benefits of Vagrant with the power of working in whatever environment you need to. This capability unlocks entirely new use cases for Vagrant that can help better optimize the entire process of developing and testing Puppet code. In this talk, you'll learn how about the new multi-provider features, why they exist, and how they can be used. Your life will never be the same again.</div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetconf2013b.sched.org/speaker/mitchell_hashimoto.1r7gqhsf?iframe=yes&w=990&sidebar=yes&bg=no" href="http://puppetconf2013b.sched.org/speaker/mitchell_hashimoto.1r7gqhsf?iframe=yes&w=990&sidebar=yes&bg=no" style="color: rgb(59, 115, 175) !important; text-decoration: none;" title="Mitchell Hashimoto"><strong>Mitchell Hashimoto</strong></a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Founder, HashiCorp</div>
<h3 style="font-family: Arial, sans-serif; font-size: 16px; font-variant: small-caps; line-height: 1.5625; margin: 30px 0px 0px;">
Notes:</h3>
<ul style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin: 10px 0px 0px;">
<li>Packer<ul style="list-style-type: disc; margin: 0px;">
<li>If vagrant up becomes a blocker due to length of time Packer becomes important</li>
<li>Multi-cloud portability e.g. Dev->Test->Prod</li>
<li>Stability and portability:<ul style="list-style-type: disc; margin: 0px;">
<li>E.g. Provision image with Puppet, don’t run Puppet live.. works after 12months</li>
</ul>
</li>
<li>Handles reboots</li>
</ul>
</li>
</ul>
<h2 style="border-bottom-style: solid; border-bottom-width: 1px; color: #333333; font-family: Arial, sans-serif; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px;">
<a data-mce-href="http://puppetconf2013b.sched.org/event/9cc8346bbafa29f77633699c3f8e5a6d?iframe=yes&w=990&sidebar=yes&bg=no" href="http://puppetconf2013b.sched.org/event/9cc8346bbafa29f77633699c3f8e5a6d?iframe=yes&w=990&sidebar=yes&bg=no" style="color: rgb(59, 115, 175) !important; text-decoration: none;">Building Data-Driven Infrastructure with Puppet</a></h2>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetlabs.com/presentations/building-data-driven-infrastructure-puppet" href="http://puppetlabs.com/presentations/building-data-driven-infrastructure-puppet" style="color: rgb(59, 115, 175) !important; text-decoration: none;">http://puppetlabs.com/presentations/building-data-driven-infrastructure-puppet</a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
As your Puppet Infrastructure grows, so does the complexity of the Puppet codebase. The complexity of the codebase often creates a scenario where it becomes more time consuming to modify/add to the codebase. Likewise, any new addition or node still may require modifications to the Puppet database, which could include the management of many edge cases. Fortunately, the software industry has been working on developing techniques with code abstraction, refactoring, and software maturity. This talk will focus on how to write scalable modules within Puppet to be used to create Data Driven Infrastructures. In addition, this talk will demonstrate how to structure process/procedure/code to quickly and rapidly scale operations with minimal modifications to Puppet code.</div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetconf2013b.sched.org/speaker/james_fryman.1r7gqhoh?iframe=yes&w=990&sidebar=yes&bg=no" href="http://puppetconf2013b.sched.org/speaker/james_fryman.1r7gqhoh?iframe=yes&w=990&sidebar=yes&bg=no" style="color: rgb(59, 115, 175) !important; text-decoration: none;" title="James Fryman"><strong>James Fryman</strong></a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Operations Hacker, GitHub, Inc.</div>
<h3 style="font-family: Arial, sans-serif; font-size: 16px; font-variant: small-caps; line-height: 1.5625; margin: 30px 0px 0px;">
Notes:</h3>
<ul style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin: 10px 0px 0px;">
<li><a data-mce-href="https://gist.github.com/jfryman/6310477" href="https://gist.github.com/jfryman/6310477" style="color: rgb(59, 115, 175) !important; text-decoration: none;">https://gist.github.com/jfryman/6310477</a><ul style="list-style-type: disc; margin: 0px;">
<li><a data-mce-href="http://blog.websages.com/2010/12/10/jameswhite-manifesto/" href="http://blog.websages.com/2010/12/10/jameswhite-manifesto/" style="color: rgb(59, 115, 175) !important; text-decoration: none;">James White Manifesto</a></li>
<li><a data-mce-href="https://www.youtube.com/watch?v=NST3u-GjjFw" href="https://www.youtube.com/watch?v=NST3u-GjjFw" style="color: rgb(59, 115, 175) !important; text-decoration: none;">ChatOps</a></li>
<li><a data-mce-href="https://gist.github.com/jfryman/6299238" href="https://gist.github.com/jfryman/6299238" style="color: rgb(59, 115, 175) !important; text-decoration: none;">check-graphite</a></li>
<li><a data-mce-href="https://gist.github.com/jfryman/6290603" href="https://gist.github.com/jfryman/6290603" style="color: rgb(59, 115, 175) !important; text-decoration: none;">HAProxy Snippets</a></li>
<li><a data-mce-href="https://speakerdeck.com/jfryman/refactoring-puppet" href="https://speakerdeck.com/jfryman/refactoring-puppet" style="color: rgb(59, 115, 175) !important; text-decoration: none;">Refactoring Puppet</a></li>
<li><a data-mce-href="https://gist.github.com/jfryman/6290389" href="https://gist.github.com/jfryman/6290389" style="color: rgb(59, 115, 175) !important; text-decoration: none;">CloudFormation</a></li>
<li><a data-mce-href="https://gist.github.com/jfryman/6310732" href="https://gist.github.com/jfryman/6310732" style="color: rgb(59, 115, 175) !important; text-decoration: none;">Autoloader</a></li>
<li><a data-mce-href="https://gist.github.com/jfryman/6310957" href="https://gist.github.com/jfryman/6310957" style="color: rgb(59, 115, 175) !important; text-decoration: none;">Syslog-NG Model</a></li>
<li><a data-mce-href="https://gist.github.com/jfryman/6290985" href="https://gist.github.com/jfryman/6290985" style="color: rgb(59, 115, 175) !important; text-decoration: none;">NagiosDB</a></li>
<li><a data-mce-href="http://about.me/james.d.fryman" href="http://about.me/james.d.fryman" style="color: rgb(59, 115, 175) !important; text-decoration: none;">Bio</a></li>
</ul>
</li>
<li>Beer-Ops</li>
<li><a data-mce-href="http://blog.websages.com/2010/12/10/jameswhite-manifesto/" href="http://blog.websages.com/2010/12/10/jameswhite-manifesto/" style="color: rgb(59, 115, 175) !important; text-decoration: none;">James White Manifesto</a></li>
<li>Machine parseable: every input and output should be readable by computer</li>
<li>There is only one system</li>
<li>Systems-thinking</li>
<li>gPanel -> PuppetDB</li>
<li>gPanel self-driven portal CMDB</li>
<li>Machine readable metrics can go right back into the system</li>
<li>Unix Philosophy – apps do one thing really well</li>
<li>Controller<ul style="list-style-type: disc; margin: 0px;">
<li>Think about Puppet as a “controller”</li>
<li><a data-mce-href="https://speakerdeck.com/jfryman/refactoring-puppet" href="https://speakerdeck.com/jfryman/refactoring-puppet" style="color: rgb(59, 115, 175) !important; text-decoration: none;">https://speakerdeck.com/jfryman/refactoring-puppet</a></li>
<li>Facter is meant to be info introspection but can cheat and use it as a state engine when you have a controller</li>
</ul>
</li>
<li>Orchestrator<ul style="list-style-type: disc; margin: 0px;">
<li><a data-mce-href="https://www.youtube.com/watch?v=NST3u-GjjFw" href="https://www.youtube.com/watch?v=NST3u-GjjFw" style="color: rgb(59, 115, 175) !important; text-decoration: none;">Chat Ops</a> – Jesse Newland talk</li>
<li>mcollective</li>
<li>Puppet is a state machine</li>
<li>Don’t want to wait for entire Puppet convergence</li>
</ul>
</li>
<li>Metrics</li>
<li><strong>System must be able to self-correct</strong><ul style="list-style-type: disc; margin: 0px;">
<li><a data-mce-href="https://gist.github.com/jfryman/6299238" href="https://gist.github.com/jfryman/6299238" style="color: rgb(59, 115, 175) !important; text-decoration: none;">check-graphite</a></li>
</ul>
</li>
<li>Deployable using text files</li>
<li>Modularity</li>
<li>To stop config drift: Level up from Templates to Data Driven</li>
<li><a data-mce-href="https://gist.github.com/jfryman/6290985" href="https://gist.github.com/jfryman/6290985" style="color: rgb(59, 115, 175) !important; text-decoration: none;">NagiosDB</a><ul style="list-style-type: disc; margin: 0px;">
<li>Dynamically generate Nagios setups outside of Puppet</li>
</ul>
</li>
<li>System should introspect itself</li>
<li>Refactoring Puppet – least to most specific, modules become systems</li>
<li><a data-mce-href="https://gist.github.com/jfryman/6290389" href="https://gist.github.com/jfryman/6290389" style="color: rgb(59, 115, 175) !important; text-decoration: none;">CloudFormation</a></li>
<li>Autoloading<ul style="list-style-type: disc; margin: 0px;">
<li>Nagios-autoload: <a data-mce-href="https://gist.github.com/jfryman/6310732" href="https://gist.github.com/jfryman/6310732" style="color: rgb(59, 115, 175) !important; text-decoration: none;">https://gist.github.com/jfryman/6310732</a></li>
</ul>
</li>
<li>Modelling<ul style="list-style-type: disc; margin: 0px;">
<li>Puppet has to know about app intimately</li>
<li>Augeas allows you to build infra dynamically</li>
<li>Pass structured data into module</li>
<li>“Turntup” slide</li>
</ul>
</li>
<li>Fencing resources<ul style="list-style-type: disc; margin: 0px;">
<li>Being built but not ready to serve</li>
</ul>
</li>
<li>Constant introspection, feedback loop into the system</li>
<li>What’s missing?<ul style="list-style-type: disc; margin: 0px;">
<li>We don’t have a good language for complex actions</li>
<li>Need a way to model complex interactions if-then-else and ngaios event handlers only gets you so far</li>
<li>Predictive Analysis</li>
</ul>
</li>
<li>Data is coming from itself, think about system/feedback loop</li>
<li>Only one system<ul style="list-style-type: disc; margin: 0px;">
<li>Everything machine parsable</li>
<li>Must be thinking “how do we make computers do my job?”</li>
</ul>
</li>
</ul>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<br /></div>
<h1 style="border-bottom-style: solid; border-bottom-width: 2px; color: #003366; font-family: Arial, sans-serif; font-size: 24px; font-weight: normal; line-height: 1.25; margin: 30px 0px 0px;">
<span data-mce-style="color: #333333; font-size: 20.0px; line-height: 1.5;" style="color: #333333; font-size: 20px; line-height: 1.5;">Friday 2013-08-24 [Saturday BNE time]</span></h1>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px;">
<br /></div>
<ul style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin: 10px 0px 0px;">
<li><a data-mce-href="http://puppetconf2013b.sched.org/event/9623245b8d8d9a5fe04415f1f2e2cc54" href="http://puppetconf2013b.sched.org/event/9623245b8d8d9a5fe04415f1f2e2cc54" style="color: rgb(59, 115, 175) !important; text-decoration: none;"><em><strong>9:00am •</strong></em><strong> </strong>Keynote: Stop Hiring Devops Experts (And Start Growing Them)</a></li>
<li><a data-mce-href="http://puppetconf2013b.sched.org/event/f1e807928c444ba6d463ce8cb1fb1929" href="http://puppetconf2013b.sched.org/event/f1e807928c444ba6d463ce8cb1fb1929" style="color: rgb(59, 115, 175) !important; text-decoration: none;"><em><strong>9:30am •</strong></em><strong> </strong>Keynote: Puppet for Production in WebEx</a></li>
<li><a data-mce-href="http://puppetconf2013b.sched.org/event/4c917c018bcb1912cde35b6b0bf8e8f2" href="http://puppetconf2013b.sched.org/event/4c917c018bcb1912cde35b6b0bf8e8f2" style="color: rgb(59, 115, 175) !important; text-decoration: none;"><em><strong>10:00am •</strong></em><strong> </strong>Keynote: Puppet at Scale – Case Study of PayPal's Learnings</a></li>
<li><a data-mce-href="http://puppetconf2013b.sched.org/event/69737e047bb25563258c1ddbc15fed2c" href="http://puppetconf2013b.sched.org/event/69737e047bb25563258c1ddbc15fed2c" style="color: rgb(59, 115, 175) !important; text-decoration: none;"><em><strong>10:25am •</strong></em><strong> </strong>Keynote: VMware vCHS, Puppet, and Project Zombie</a></li>
<li><a data-mce-href="http://puppetconf2013b.sched.org/event/e04740c470d121155dde26c75e50ade5" href="http://puppetconf2013b.sched.org/event/e04740c470d121155dde26c75e50ade5" style="color: rgb(59, 115, 175) !important; text-decoration: none;"><em><strong>11:10am •</strong></em><strong> </strong>More Logstash Awesome</a></li>
<li><a data-mce-href="http://puppetconf2013b.sched.org/event/4771c4db32af382cafa6464cc58e0604" href="http://puppetconf2013b.sched.org/event/4771c4db32af382cafa6464cc58e0604" style="color: rgb(59, 115, 175) !important; text-decoration: none;"><em><strong>1:30pm •</strong></em><strong> </strong>Puppet Module Reusability - What I Learned from Shipping to the Forge</a></li>
<li><a data-mce-href="http://puppetconf2013b.sched.org/event/08b3e7b92b530048174c2c8f47f358c9" href="http://puppetconf2013b.sched.org/event/08b3e7b92b530048174c2c8f47f358c9" style="color: rgb(59, 115, 175) !important; text-decoration: none;"><em><strong>2:20pm •</strong></em><strong> </strong>DevOps Isn’t Just for WebOps: The Guerrilla’s Guide to Cultural Change</a></li>
<li><a data-mce-href="http://puppetconf2013b.sched.org/event/6a9f44343c33b167b19b0117024c1b7c" href="http://puppetconf2013b.sched.org/event/6a9f44343c33b167b19b0117024c1b7c" style="color: rgb(59, 115, 175) !important; text-decoration: none;"><em><strong>3:10pm •</strong></em><strong> </strong>The Road to the White House with Puppet & AWS</a></li>
<li><a data-mce-href="http://puppetconf2013b.sched.org/event/e8e46314992849c94075592197965f38" href="http://puppetconf2013b.sched.org/event/e8e46314992849c94075592197965f38" style="color: rgb(59, 115, 175) !important; text-decoration: none;"><em><strong>4:20pm •</strong></em><strong> </strong>Continuously Integrated Puppet in a Dynamic Environment</a></li>
<li><a data-mce-href="http://puppetconf2013b.sched.org/event/cd4d92399ef38fc193c738f6e3299497" href="http://puppetconf2013b.sched.org/event/cd4d92399ef38fc193c738f6e3299497" style="color: rgb(59, 115, 175) !important; text-decoration: none;"><em><strong>5:10pm •</strong></em><strong> </strong>Monitoring in an Infrastructure as Code Age</a></li>
</ul>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px;">
<br /></div>
<h2 style="border-bottom-style: solid; border-bottom-width: 1px; color: #333333; font-family: Arial, sans-serif; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px;">
<a data-mce-href="http://puppetconf2013b.sched.org/event/9623245b8d8d9a5fe04415f1f2e2cc54" href="http://puppetconf2013b.sched.org/event/9623245b8d8d9a5fe04415f1f2e2cc54" style="color: rgb(59, 115, 175) !important; text-decoration: none;">Keynote: Stop Hiring Devops Experts (And Start Growing Them)</a></h2>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetlabs.com/presentations/keynote-stop-hiring-devops-experts-and-start-growing-them" href="http://puppetlabs.com/presentations/keynote-stop-hiring-devops-experts-and-start-growing-them" style="color: rgb(59, 115, 175) !important; text-decoration: none;">http://puppetlabs.com/presentations/keynote-stop-hiring-devops-experts-and-start-growing-them</a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Everyone is putting "devops" on their LinkedIn profile, and everyone is trying to hire them. In this talk, Jez will argue this is not a recruitment problem but an organizations failure. This talk discusses how to grow great people and great organizations, and how the two problems are connected.</div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetconf2013b.sched.org/speaker/jez_humble.1r7gqhoy" href="http://puppetconf2013b.sched.org/speaker/jez_humble.1r7gqhoy" style="color: rgb(59, 115, 175) !important; text-decoration: none;" title="Jez Humble">Jez Humble</a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Principal, ThoughtWorks</div>
<h3 style="font-family: Arial, sans-serif; font-size: 16px; font-variant: small-caps; line-height: 1.5625; margin: 30px 0px 0px;">
Notes:</h3>
<ul style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin: 10px 0px 0px;">
<li><a data-mce-href="http://es.slideshare.net/michael.sahota/agile-culture-and-adoption-survival-guide" href="http://es.slideshare.net/michael.sahota/agile-culture-and-adoption-survival-guide" style="color: rgb(59, 115, 175) !important; text-decoration: none;">http://es.slideshare.net/michael.sahota/agile-culture-and-adoption-survival-guide</a></li>
<li>Ops is in top right – control</li>
<li>It’s about the bottom right – cultivation culture</li>
<li>You can’t hire in cultural change<ul style="list-style-type: disc; margin: 0px;">
<li>It’s very hard to change organisations</li>
<li>Organisations reject change in general</li>
</ul>
</li>
<li>Cultivation culture<ul style="list-style-type: disc; margin: 0px;">
<li>Have to build it into your organisation</li>
<li>20% time (3m postit notes, Google)</li>
<li>Can’t just experiment – need to be able to act on it – Kodak</li>
<li>Pairing, only by doing it for 6months, have to work with people – not alongside them, work with them</li>
<li>Focus on experimentation/innovation<ul style="list-style-type: disc; margin: 0px;">
<li>Okay to fail</li>
</ul>
</li>
<li>Role of mgmt<ul style="list-style-type: disc; margin: 0px;">
<li>Create env in which it’s safe to learn</li>
<li>How well do we cultivate knowledge?</li>
</ul>
</li>
<li>Measure ppl on how good they are<ul style="list-style-type: disc; margin: 0px;">
<li>Toyota Kata</li>
<li>Can only become manager when you’ve worked on factory floor for 10-15yrs</li>
<li>Focus on experimentation</li>
<li>Job of mgr = facilitate ppl doing the work to get better at doing it</li>
</ul>
</li>
<li>Not effective<ul style="list-style-type: disc; margin: 0px;">
<li>Training<ul style="list-style-type: disc; margin: 0px;">
<li>Bring the people with you –make it natural</li>
</ul>
</li>
<li>Buying tools</li>
<li>DevOps team<ul style="list-style-type: disc; margin: 0px;">
<li>Creating a silo to fix a silo</li>
<li>If you create an org that values learning & ppl enjoy being able to learn and innovte = maybe won’t have a hiring problem</li>
</ul>
</li>
</ul>
</li>
<li>Taleb 3<ul style="list-style-type: disc; margin: 0px;">
<li>Resilience is not the opposite of fragile</li>
<li>Antifragile is the opposite of fragile</li>
<li>People – e.g. Arnie, applying stress makes you stronger</li>
<li>Game days e.g. Jesse Robbins Amazon<ul style="list-style-type: disc; margin: 0px;">
<li>Firedrills</li>
<li>Turn off the power and find out what actually happens</li>
<li>Google dirt exercises, earthquake simulation</li>
<li>Expose assumptions e.g. Mountain View failing over to Mountain View</li>
</ul>
</li>
<li>Netflix<ul style="list-style-type: disc; margin: 0px;">
<li>Simian army</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
<h2 style="border-bottom-style: solid; border-bottom-width: 1px; color: #333333; font-family: Arial, sans-serif; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px;">
<a data-mce-href="http://puppetconf2013b.sched.org/event/f1e807928c444ba6d463ce8cb1fb1929" href="http://puppetconf2013b.sched.org/event/f1e807928c444ba6d463ce8cb1fb1929" style="color: rgb(59, 115, 175) !important; text-decoration: none;">Keynote: Puppet for Production in WebEx</a></h2>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetlabs.com/presentations/keynote-puppet-production-webex" href="http://puppetlabs.com/presentations/keynote-puppet-production-webex" style="color: rgb(59, 115, 175) !important; text-decoration: none;">http://puppetlabs.com/presentations/keynote-puppet-production-webex</a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Getting started with Puppet configuring an individual machine is straightforward. Managing a cluster of machines across multiple data centers, supporting upgrades while running a 7x24 service, and building for collaboration is significantly more challenging. The WebEx team will discuss the problems and some strategies they are using to manage this complexity</div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetconf2013b.sched.org/speaker/reinhardt_quelle.1r7gqhs4" href="http://puppetconf2013b.sched.org/speaker/reinhardt_quelle.1r7gqhs4" style="color: rgb(59, 115, 175) !important; text-decoration: none;" title="Reinhardt Quelle">Reinhardt Quelle</a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Cloud Services Architect, Cisco/WebEx</div>
<h3 style="font-family: Arial, sans-serif; font-size: 16px; font-variant: small-caps; line-height: 1.5625; margin: 30px 0px 0px;">
Notes:</h3>
<ul style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin: 10px 0px 0px;">
<li>Sequencing: technical + business</li>
<li>Never run just one of anything<ul style="list-style-type: disc; margin: 0px;">
<li>DC, nodes</li>
<li>Why cluster: Technical, business (commercial retail vs federal, privacy etc)</li>
</ul>
</li>
<li>Version migration<ul style="list-style-type: disc; margin: 0px;">
<li>By DC</li>
<li>By Node</li>
</ul>
</li>
<li>Blueprints -> Orchestration</li>
<li>Unix toolchain philosophy<ul style="list-style-type: disc; margin: 0px;">
<li>Fabric/Salt/Ansible is the leatherman pocketknife multitool</li>
<li>Puppet is the real knife</li>
</ul>
</li>
<li>Masterless Puppet<ul style="list-style-type: disc; margin: 0px;">
<li>E.g. Google</li>
<li>distribute modules/manifests to nodes</li>
<li>Copy /etc/puppet/* to each node</li>
<li>Complete resiliency per node</li>
<li>No single point of control</li>
<li>puppet apply –modulepath “...” --execute “include ...”</li>
</ul>
</li>
<li>See also: Sam Bashton “<a data-mce-href="http://puppetconf2013b.sched.org/event/e8e46314992849c94075592197965f38" href="http://puppetconf2013b.sched.org/event/e8e46314992849c94075592197965f38" style="color: rgb(59, 115, 175) !important; text-decoration: none;">Continuously Integrated Puppet in a Dynamic Environment</a>”</li>
<li><span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">Keep problems small</span></li>
<li><span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">Push dependencies into Puppet instead of RPMs</span><ul style="list-style-type: disc; margin: 0px;">
<li>“4am-proofing” – Puppet = transparency</li>
<li>Favour transparency over DRY<span data-mce-style="line-height: 0.0px;" style="line-height: 0px;"> </span></li>
</ul>
</li>
</ul>
<h2 style="border-bottom-style: solid; border-bottom-width: 1px; color: #333333; font-family: Arial, sans-serif; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px;">
<a data-mce-href="http://puppetconf2013b.sched.org/event/4c917c018bcb1912cde35b6b0bf8e8f2" href="http://puppetconf2013b.sched.org/event/4c917c018bcb1912cde35b6b0bf8e8f2" style="color: rgb(59, 115, 175) !important; text-decoration: none;">Keynote: Puppet at Scale – Case Study of PayPal's Learnings</a></h2>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetlabs.com/presentations/keynote-puppet-scale-%E2%80%93-case-study-paypals-learnings" href="http://puppetlabs.com/presentations/keynote-puppet-scale-%E2%80%93-case-study-paypals-learnings" style="color: rgb(59, 115, 175) !important; text-decoration: none;">http://puppetlabs.com/presentations/keynote-puppet-scale-%E2%80%93-case-study-paypals-learnings</a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Large scale and app level management pose challenges to any implementation of puppet. Come and learn some of the challenges PayPal Deployment Systems team faced and the how these were overcome.</div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">Sr Dev Manager, PayPal</span><span style="line-height: 1.4285715;"> </span></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<span data-mce-style="color: #000000; font-size: 16.0px; font-variant: small-caps; font-weight: bold; line-height: 1.5625;" style="color: black; font-size: 16px; font-variant: small-caps; font-weight: bold; line-height: 1.5625;">Notes:</span></div>
<ul style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin: 10px 0px 0px;">
<li>Staging = mini paypal.com</li>
<li>Commits happening every few seconds<ul style="list-style-type: disc; margin: 0px;">
<li>How do you mange dependencies?</li>
</ul>
</li>
<li><span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">Web API for deployment</span></li>
<li><span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">OpenStack</span></li>
<li><span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">“Project Velocity”</span></li>
<li><span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">Who looks after Puppet in the middle of the night?</span></li>
</ul>
<ul style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin: 10px 0px 0px;">
<li data-mce-style="list-style-type: none; background-image: none;" style="background-image: none; list-style-type: none;"><ul style="list-style-type: disc; margin: 0px;">
<li>3,000 developers?</li>
</ul>
</li>
<li>Deployment system where puppet coding is not required to deploy new apps</li>
<li><span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">Ninja engine</span><ul style="list-style-type: disc; margin: 0px;">
<li>Takes list of apps you want to install</li>
<li>Assemble list of</li>
<li>Discovery dependencies</li>
<li>Generate puppet resources from dependency graph</li>
<li>Then execute them</li>
</ul>
</li>
<li>Caches dynamic resources for next run</li>
<li>What packages to install?<ul style="list-style-type: disc; margin: 0px;">
<li><span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">Roles & Labels</span></li>
</ul>
</li>
<li><span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">System Hierarchy</span><ul style="list-style-type: disc; margin: 0px;">
<li>ENC -> Hiera</li>
<li>Web tool to visualise Hiera data</li>
<li><span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">REST API for CRUD over Hiera data</span></li>
</ul>
</li>
<li><span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">Scaling ActiveMQ</span><ul style="list-style-type: disc; margin: 0px;">
<li>Problem with mcollective beyond dev/test where multiple puppetmasters/MQ</li>
<li>Mcollective gave inconsistent results</li>
<li>Were using MQ cluster through LB – connections would time out</li>
<li>Removing LB fixed</li>
<li><span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">When ActiveMQ host dies have to reconfigure clients – Use Puppet</span></li>
</ul>
</li>
<li><span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">Mcollective at Scale</span><ul style="list-style-type: disc; margin: 0px;">
<li>Mcollective is equally useful as Puppet</li>
<li>Paypal heavily depends on mcollective</li>
<li><span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">Replaces all SSH scriptsUse it to:</span><ul style="list-style-type: disc; margin: 0px;">
<li>Query systems</li>
<li>Verify package versions</li>
<li>Kick off on-demand puppet runs</li>
<li>Ssh script replacement</li>
</ul>
</li>
<li>REST API enables Mcollective to web and other tools<ul style="list-style-type: disc; margin: 0px;">
<li>Powerful = careful approval required</li>
</ul>
</li>
</ul>
</li>
<li>Worked with PuppetLabs to create “progress” module to work out how long a deploy will take</li>
</ul>
<h2 style="border-bottom-style: solid; border-bottom-width: 1px; color: #333333; font-family: Arial, sans-serif; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px;">
<a data-mce-href="http://puppetconf2013b.sched.org/event/69737e047bb25563258c1ddbc15fed2c" href="http://puppetconf2013b.sched.org/event/69737e047bb25563258c1ddbc15fed2c" style="color: rgb(59, 115, 175) !important; text-decoration: none;">Keynote: VMware vCHS, Puppet, and Project Zombie</a></h2>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetlabs.com/presentations/keynote-vmware-vchs-puppet-and-project-zombie" href="http://puppetlabs.com/presentations/keynote-vmware-vchs-puppet-and-project-zombie" style="color: rgb(59, 115, 175) !important; text-decoration: none;">http://puppetlabs.com/presentations/keynote-vmware-vchs-puppet-and-project-zombie</a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetconf2013b.sched.org/speaker/nicholas_weaver.1r7gqhrj" href="http://puppetconf2013b.sched.org/speaker/nicholas_weaver.1r7gqhrj" style="color: rgb(59, 115, 175) !important; text-decoration: none;" title="Nicholas Weaver">Nicholas Weaver</a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Cloud Automation Architect, Hybrid Cloud Service, VMware</div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Nicholas Weaver is the Cloud Automation Architect for VMware's vCloud Hybrid Service (vCHS) platform and the primary architect behind the vCHS automation framework (Project Zombie). He is also a co-creator of the PuppetLabs Razor project and many VMware-specific free tools. He previously worked in the CTO office for EMC, in the EMC field as a vSpecialist, and as a infrastructure engineer in financial, media, and retail companies. Nick loves software-driven control, hacking prototypes together...</div>
<h3 style="font-family: Arial, sans-serif; font-size: 16px; font-variant: small-caps; line-height: 1.5625; margin: 30px 0px 0px;">
Notes:</h3>
<ul style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin: 10px 0px 0px;">
<li><strong>Automation = Effort Evolution</strong></li>
<li data-mce-style="list-style-type: none;" style="list-style-type: none;">Why is it important?<ul style="list-style-type: disc; margin: 0px;">
<li>Warehouse</li>
</ul>
</li>
<li data-mce-style="list-style-type: none;" style="list-style-type: none;">Resiliency = we expect things to fail<ul style="list-style-type: disc; margin: 0px;">
<li>Can never assume anything is going to stay up</li>
</ul>
</li>
<li>Project Zombie</li>
<li>Puppet has critical things for VMWare to choose</li>
<li data-mce-style="list-style-type: none;" style="list-style-type: none;"><ul style="list-style-type: disc; margin: 0px;">
<li>Mcollective</li>
<li>VM support</li>
<li>Cassandra</li>
<li>Netflix Astayanax</li>
<li>JRuby – good middle ground for dev and ops</li>
</ul>
</li>
<li>Rabbit MQ</li>
<li><span data-mce-style="font-size: 14.0px; line-height: 1.4285715;" style="line-height: 1.4285715;">Modules</span></li>
<li><span data-mce-style="font-size: 14.0px; line-height: 1.4285715;" style="line-height: 1.4285715;">Rez = globally distributed “refrigerator”</span><ul style="list-style-type: disc; margin: 0px;">
<li>Difficult to manage resources at global distributed scale</li>
<li>Automation = baking</li>
<li>Millions of resources</li>
<li>Razor feed into Rez to manage state</li>
<li><span data-mce-style="font-size: 14.0px; line-height: 1.4285715;" style="line-height: 1.4285715;">REST API</span></li>
</ul>
</li>
<li><span data-mce-style="line-height: 1.4285715;" style="line-height: 1.4285715;">Engine = “the chef”</span><ul style="list-style-type: disc; margin: 0px;">
<li>Orchestration</li>
<li>Wrote their own Operational-based language</li>
<li>Controls flow and concurrency</li>
<li>ZED (Zombie Engine DSL)</li>
<li>Distributed and location-awareness</li>
<li>B = Broker</li>
<li>P = Processor</li>
</ul>
</li>
</ul>
<h2 style="border-bottom-style: solid; border-bottom-width: 1px; color: #333333; font-family: Arial, sans-serif; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 30px 0px 0px;">
<a data-mce-href="http://puppetconf2013b.sched.org/event/e8e46314992849c94075592197965f38" data-mce-style="line-height: 1.5;" href="http://puppetconf2013b.sched.org/event/e8e46314992849c94075592197965f38" style="color: rgb(59, 115, 175) !important; line-height: 1.5; text-decoration: none;">Continuously Integrated Puppet in a Dynamic Environment</a></h2>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetlabs.com/presentations/continuously-integrated-puppet-dynamic-environment" href="http://puppetlabs.com/presentations/continuously-integrated-puppet-dynamic-environment" style="color: rgb(59, 115, 175) !important; text-decoration: none;">http://puppetlabs.com/presentations/continuously-integrated-puppet-dynamic-environment</a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
This talk will show how we deploy Puppet without a Puppetmaster on an autoscaling Amazon Web Services infrastructure. Key points of interest: - Masterless Puppet - Use of Jenkins for Puppet manifest testing and environment promotion (test->staging->production) - Puppet integration with Amazon CloudFormation</div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
<a data-mce-href="http://puppetconf2013b.sched.org/speaker/sam_bashton.1r7gqhr8" href="http://puppetconf2013b.sched.org/speaker/sam_bashton.1r7gqhr8" style="color: rgb(59, 115, 175) !important; text-decoration: none;" title="Sam Bashton">Sam Bashton</a></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Director, Bashton Ltd<br />
<br />
<br />
<h2>
<b>Other Notes</b></h2>
<br />
<ul>
<li><a href="https://github.com/djbkd/puppetconf_2013">https://github.com/djbkd/puppetconf_2013</a></li>
<li><a href="http://www.virtualizationpractice.com/recap-puppetconf-2013-san-francisco-22849/">http://www.virtualizationpractice.com/recap-puppetconf-2013-san-francisco-22849/</a></li>
<li><a href="http://www.kavistechnology.com/blog/top-10-quotes-from-day-1-of-puppetconf-2013/">http://www.kavistechnology.com/blog/top-10-quotes-from-day-1-of-puppetconf-2013/</a></li>
</ul>
<br />
<br /></div>
Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-37361877255202763052013-09-16T13:29:00.001-07:002013-09-16T13:35:09.595-07:00DevOps Cafe: DevOps At SalesForce and Measuring Service<div class="y-module message-header " data-action="toggle-msg-header" id="yui_3_7_2_1_1379194870947_1897" role="complementary">
<br /></div>
<div class="msg-body inner undoreset" id="yui_3_7_2_1_1379194870947_1899" role="main" style="font-size-adjust: none; font-stretch: normal; font: 12pt/normal verdana, helvetica, sans-serif;">
<div style="position: absolute; z-index: -5;">
</div>
<a href="http://devopscafe.org/show/2013/9/9/devops-cafe-episode-41.html">DevOps Cafe Ep 41</a></div>
<div class="msg-body inner undoreset" id="yui_3_7_2_1_1379194870947_1899" role="main" style="font-size-adjust: none; font-stretch: normal; font: 12pt/normal verdana, helvetica, sans-serif;">
<span style="font-size: 12pt;">Dave Mangot -</span><span style="font-size: 12pt;"> </span>SalesForce Orchestration Engine / Release Tools Team<br />
<div>
<br /></div>
</div>
<div class="msg-body inner undoreset" id="yui_3_7_2_1_1379194870947_1899" role="main" style="font-size-adjust: none; font-stretch: normal; font: 12pt/normal verdana, helvetica, sans-serif;">
Great to see DevOps Cafe Podcast finally back online after a six month break and with a corker of an episode. Dave Mangot from the "Orchestration Engine" team at SalesForce talks through culture and tools and Damon and John as always bring insightful comments on devops and cultural change.</div>
<div class="msg-body inner undoreset" id="yui_3_7_2_1_1379194870947_1899" role="main" style="font-size-adjust: none; font-stretch: normal; font: 12pt/normal verdana, helvetica, sans-serif;">
<br />
<span style="-webkit-composition-fill-color: rgba(175, 192, 227, 0.230469); -webkit-composition-frame-color: rgba(77, 128, 180, 0.230469); -webkit-tap-highlight-color: rgba(26, 26, 26, 0.296875);">SalesForce teams are organised in "Clouds" which is the equivalent of what other organisations would call "Tribes". Made up of ops, dev and QA, Clouds are r</span><span style="-webkit-composition-fill-color: rgba(175, 192, 227, 0.230469); -webkit-composition-frame-color: rgba(77, 128, 180, 0.230469); -webkit-tap-highlight-color: rgba(26, 26, 26, 0.292969); font-size: 12pt;">esponsible for ensuring the apps they build are kept running in prod.</span></div>
<div class="msg-body inner undoreset" id="yui_3_7_2_1_1379194870947_1899" role="main" style="font-size-adjust: none; font-stretch: normal; font: 12pt/normal verdana, helvetica, sans-serif;">
<br /></div>
<div class="msg-body inner undoreset" id="yui_3_7_2_1_1379194870947_1899" role="main" style="font-size-adjust: none; font-stretch: normal; font: 12pt/normal verdana, helvetica, sans-serif;">
Something that has stuck with Dave and influences his philosophy is the priorities given to him by a CEO at a former company. <span style="font-size: 12pt;">CEO gave Dave two priorities:</span></div>
<div class="msg-body inner undoreset" id="yui_3_7_2_1_1379194870947_1899" role="main" style="font-size-adjust: none; font-stretch: normal; font: 12pt/normal verdana, helvetica, sans-serif;">
<ol>
<li>Keep devs moving as fast as possible</li>
<li>Keep the site up</li>
</ol>
In that order. Getting software to customers faster is more critical than preventing outages.</div>
<div class="msg-body inner undoreset" id="yui_3_7_2_1_1379194870947_1899" role="main" style="font-size-adjust: none; font-stretch: normal; font: 12pt/normal verdana, helvetica, sans-serif;">
</div>
<div class="msg-body inner undoreset" id="yui_3_7_2_1_1379194870947_1899" role="main" style="font-family: verdana, helvetica, sans-serif; font-size: 12pt; font-style: normal; font-variant: normal; line-height: normal;">
<br />
Ops is a service to dev.<br />
IT department is a service to customers.<br />
<br />
For a great example of an a<span style="font-size: 12pt;">ligned organisation, w</span><span style="font-size: 12pt;">atch "The Carrier" series on Netflix. Every person onboard the aircraft carrier articulated their role in terms of "winning the war on terror".</span><br />
<br />
Jez Humble's "DevOps kata" - always trying to get better.<br />
<br />
Salesforce "Chatter" webapp for online communication<br />
<br />
Their IM groups have "organisers". E.g. Dave is the organiser of the devops group.<br />
<br />
They record demos and make them available on demand to those who couldn't be there.<br />
<br />
Do you set goals for improvement?<br />
Do you have common visibility? Training?<br />
<br />
<span style="-webkit-composition-fill-color: rgba(175, 192, 227, 0.230469); -webkit-composition-frame-color: rgba(77, 128, 180, 0.230469); -webkit-tap-highlight-color: rgba(26, 26, 26, 0.292969);">When they identify an interruption to flow, they w</span>rite ideas for metrics that would help improve feedback on a piece of paper<br />
"Feel the pain"<br />
Track metrics, get better<br />
<br />
Damon: Measuring Service<br />
Bringing visibility to service side of organisation (not just development visibility e.g. velocity)<br />
<br />
Damon's 4 things to measure service:<br />
<ol style="font-weight: normal;">
<li>Cycle time: From backlog to customer</li>
<li>Mean Time To Detect problems</li>
<ul>
<li>How quickly can we get to detection of problem to know something is wrong</li>
<li>Enough visibility/testing/understanding of system? </li>
</ul>
<li>Mean Time To Repair</li>
<ul>
<li>How good is control of system?</li>
<li>Right Automation/testing/packaging/structures in place</li>
</ul>
<li>Quality at the source</li>
<ul>
<li>Scrap rate</li>
<li>How often does a problem get out of the realm of person who created the problem?</li>
<li>E.g. </li>
<ul>
<li>Dev - does testing framework, ci, continuous deployment catch problem?</li>
<li>Ops - DNS change</li>
<li>Dev - changed config param</li>
</ul>
<li>How far does the problem get down the stream?</li>
</ul>
<li>How good are you at not having the problem happen again?</li>
<ul>
<li>How well do you learn</li>
<li>Executing the feedback loop</li>
<li>Lowering the recidivism rate </li>
</ul>
</ol>
#5 added by John<br />
<br /></div>
<div class="msg-body inner undoreset" id="yui_3_7_2_1_1379194870947_1899" role="main" style="font-size-adjust: none; font-stretch: normal; font: 12pt/normal verdana, helvetica, sans-serif;">
Bring prod as far upstream as possible.<br />
E.g. Salesforce run chaos monkey in test envs<br />
<br />
"Rouster" for testing puppet functionally with Vagrant:</div>
<div class="msg-body inner undoreset" id="yui_3_7_2_1_1379194870947_1899" role="main" style="font-size-adjust: none; font-stretch: normal; font: 12pt/normal verdana, helvetica, sans-serif;">
<br />
<ul>
<li><a href="http://puppetlabs.com/presentations/vagrant-rouster-salesforcecom" style="font-size: 12pt;">http://puppetlabs.com/presentations/vagrant-rouster-salesforcecom</a></li>
<li><a href="https://github.com/chorankates/rouster" style="font-size: 12pt;">https://github.com/chorankates/rouster</a></li>
</ul>
</div>
<div class="msg-body inner undoreset" id="yui_3_7_2_1_1379194870947_1899" role="main" style="font-size-adjust: none; font-stretch: normal; font: 12pt/normal verdana, helvetica, sans-serif;">
<br /></div>
<div class="msg-body inner undoreset" id="yui_3_7_2_1_1379194870947_1899" role="main" style="font-size-adjust: none; font-stretch: normal; font: 12pt/normal verdana, helvetica, sans-serif;">
SalesForce toolchain:<br />
Homebrew tooling -> Rundeck for Orchestration -> SaltStack -> Puppet<br />
<br />
<span style="-webkit-composition-fill-color: rgba(175, 192, 227, 0.230469); -webkit-composition-frame-color: rgba(77, 128, 180, 0.230469); -webkit-tap-highlight-color: rgba(26, 26, 26, 0.296875);">VMware's </span>Project Zombie (PuppetConf 2013) toolchain:<br />
<ul>
<li>Cassandra for CMDB</li>
<li>Mcollective</li>
<li>Puppet</li>
</ul>
<br />
"Trigger" for network automation:<br />
<br />
<ul>
<li><a href="https://github.com/trigger/trigger" style="font-size: 12pt;">https://github.com/trigger/trigger</a></li>
<li><a href="https://twitter.com/pytrigger" style="font-size: 12pt;">https://twitter.com/pytrigger</a></li>
</ul>
</div>
<div class="msg-body inner undoreset" id="yui_3_7_2_1_1379194870947_1899" role="main" style="font-size-adjust: none; font-stretch: normal; font: 12pt/normal verdana, helvetica, sans-serif;">
<br /></div>
<div class="msg-body inner undoreset" id="yui_3_7_2_1_1379194870947_1899" role="main" style="font-size-adjust: none; font-stretch: normal; font: 12pt/normal verdana, helvetica, sans-serif;">
A few years ago it was common to find managers that couldn't see the value of Puppet/Chef - nowadays it's an acknowledged best practice. The same resistance is now happening with network automation.<br />
<br /></div>
Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-23673642038581923892013-08-05T06:22:00.001-07:002013-08-05T06:22:18.869-07:00Puppet testing, cookbook, security<div>Modern Module Development</div><div><a href="http://www.slideshare.net/PuppetLabs/modern-module-development-ken-barber-2012-edinburgh-puppet-camp">http://www.slideshare.net/PuppetLabs/modern-module-development-ken-barber-2012-edinburgh-puppet-camp</a></div><div><br></div><div>Testing</div><div><a href="http://ruby.dzone.com/articles/puppet-unit-testing-pro">http://ruby.dzone.com/articles/puppet-unit-testing-pro</a></div><div><a href="https://puppetlabs.com/blog/test-driven-development-with-puppet/">https://puppetlabs.com/blog/test-driven-development-with-puppet/</a></div><div><div class="separator" style="clear: both;"><a href="https://lh3.googleusercontent.com/-VgWTUKGIZnk/Uf-nCO43rKI/AAAAAAAAAJA/jjUOFRWuj2Q/s640/blogger-image--1980059258.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://lh3.googleusercontent.com/-VgWTUKGIZnk/Uf-nCO43rKI/AAAAAAAAAJA/jjUOFRWuj2Q/s640/blogger-image--1980059258.jpg"></a></div><br></div><div><br></div><div>Cookbook</div><div><a href="http://www.puppetcookbook.com/">http://www.puppetcookbook.com/</a></div><div><br></div><div>Security</div><div><a href="http://www.masterzen.fr/2010/11/14/puppet-ssl-explained/">http://www.masterzen.fr/2010/11/14/puppet-ssl-explained/</a></div><div><a href="http://www.amazon.com/dp/0201615983">http://www.amazon.com/dp/0201615983</a></div><div><br></div>Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-59310496423682181692013-03-25T06:45:00.000-07:002013-08-22T13:25:16.457-07:00Devops Toolchains - Rundeck and ChefThis is a write-up of a great presentation by Anthony Shortland at ChefCon 2012 called "Using Rundeck And Chef To Build DevOps ToolChain":<br />
<div id="yui_3_7_2_39_1363806269524_58">
<ul>
<li>Blog: <a href="http://dev2ops.org/2012/05/using-rundeck-and-chef-to-build-devops-toolchains-at-chefcon/" id="yui_3_7_2_39_1363806269524_88" style="color: blue; cursor: text !important;">http://dev2ops.org/2012/05/using-rundeck-and-chef-to-build-devops-toolchains-at-chefcon/</a></li>
<li>Video: <a href="https://www.youtube.com/watch?feature=player_embedded&v=9v1Hnl3ZyBg">https://www.youtube.com/watch?feature=player_embedded&v=9v1Hnl3ZyBg</a></li>
<li>Slides: <a href="http://www.slideshare.net/AnthonyShortland/dto-chefconf2012">http://www.slideshare.net/AnthonyShortland/dto-chefconf2012</a></li>
</ul>
</div>
<div>
<b>Design points</b></div>
<div>
<ol>
<li><b>Everything is code</b></li>
<li><b>Everything is packaged</b></li>
<li><b>Separate code and config</b></li>
<ol>
<li>Code and configuration flow at different rates</li>
</ol>
<li><b>Separate env-dependent attributes</b></li>
<ol>
<li>More volatile - flows much faster</li>
<li>Separate packages</li>
</ol>
<li><b>Balance distributed vs local orchestration</b></li>
<ol>
<li>On the deploy side - decision to be made about how to approach orchestration</li>
<li><b>Distributed</b></li>
<ol>
<li>Network level coordinatinon</li>
<li>Multi-box, off-box</li>
<li>Coordinating rolling upgrades</li>
<li>Pool of boxes</li>
<li>Workflow captured separately, captured/encoded independent of environment, e.g. Rundeck, specify business/technical process to orechestrate in distributed manner divorced of environment</li>
<li>Same template, same workflows meet the env-specific data/nodeset & get instantiated at runtime</li>
</ol>
<li><b>Local</b></li>
<ol>
<li>On-box, within given node</li>
<li>Set of tasks to bring system to target state</li>
<li>Modular automation - capture implementation/code in modules, move them around the network, e.g. chef cookbooks</li>
</ol>
<li>Separate concerns - met with separate tools fitting together in tool chain</li>
</ol>
<li><b>Resolve directed vs convergent orchestration</b></li>
<ol>
<li><b>Directed</b> orchestration: </li>
<ol>
<li>single node in network</li>
<li>fire out commands in authoritative manner</li>
<li>Continuous deploy / rolling upgrade - best met with directed orchestration</li>
</ol>
<li><b>Convergent</b> orchestration:</li>
<ol>
<li>Rise of cloud/different design (e.g. chef)</li>
<li>Fuzziness - room for environment to converge</li>
<li>Good for scale / compliance</li>
</ol>
<li>Not either-or... need a solution that accommodates both </li>
</ol>
<li><b>Integrate application and infrastructure provisioning</b></li>
<ol>
<li>"integrate build and deploy"</li>
<li>Want a systemic solution</li>
<li>Single orchestrated provisioning process</li>
<li>Moving away from static infra - we need infra and app up quickly</li>
</ol>
<li><b>Design for flow not the organisation</b></li>
<ol>
<li>Subordinate organisational differences/divisions - forget those requirements/constraints</li>
<li>Build for business/process flow across the system</li>
</ol>
</ol>
</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-yhEoBBWnJpE/UVBA77XaBxI/AAAAAAAAAHA/E9CJpAxxjGc/s1600/2013-03-25_22-12-25_image.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="241" src="http://4.bp.blogspot.com/-yhEoBBWnJpE/UVBA77XaBxI/AAAAAAAAAHA/E9CJpAxxjGc/s320/2013-03-25_22-12-25_image.png" width="320" /></a></div>
<br />
<a href="https://www.youtube.com/watch?feature=player_embedded&v=9v1Hnl3ZyBg#t=1289s">1289s</a><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-Nr88LYY_W5o/UVBA9Dyw4oI/AAAAAAAAAHI/B7MZpKoqhQE/s1600/2013-03-25_22-17-17_image.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="244" src="http://4.bp.blogspot.com/-Nr88LYY_W5o/UVBA9Dyw4oI/AAAAAAAAAHI/B7MZpKoqhQE/s320/2013-03-25_22-17-17_image.png" width="320" /></a></div>
<div>
<br /></div>
<div>
<a href="https://www.youtube.com/watch?feature=player_embedded&v=9v1Hnl3ZyBg#t=1466s">1466s</a> - <b>The tool for the job</b></div>
<div>
<ul>
<li>There's often a number of tools to choose to do the job</li>
<li>Choose the job to fit the people in the org</li>
<li>Java guy = Ant/Maven</li>
<li>Ruby = Rake</li>
<li>Systems = Make</li>
<li>All languages can build RPMs: Ant/Mave/Rake/Make</li>
<li>Tool you choose may vary depending on who's consuming them</li>
</ul>
<div>
<br /></div>
<div>
<a href="https://www.youtube.com/watch?feature=player_detailpage&v=9v1Hnl3ZyBg#t=2294s">2294s</a> - <b>Environment-Specific Application-Level Attributes</b></div>
<ul>
<li>Mapping databag into context</li>
<li>Package version and application state defined per environment within package's databag</li>
</ul>
<div>
<br /></div>
<div>
<div>
<a href="https://www.youtube.com/watch?feature=player_detailpage&v=9v1Hnl3ZyBg#t=2548s">2548s</a> - <b>Demo</b><br />
<br /></div>
<div>
Build console (Jenkins) does build orchestration, deploy console (Rundeck) does deploy orchestration, Chef does local orchestration.</div>
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-YUGLfDejGxk/UVBQqDbdWvI/AAAAAAAAAHY/6rJwYjxFcuY/s1600/2013-03-25_23-25-00_image.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="222" src="http://3.bp.blogspot.com/-YUGLfDejGxk/UVBQqDbdWvI/AAAAAAAAAHY/6rJwYjxFcuY/s320/2013-03-25_23-25-00_image.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-lbessOoTHXI/UVBT9LLNzmI/AAAAAAAAAHo/lVsAVtmYXJE/s1600/2013-03-25_23-25-00_image.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="222" src="http://2.bp.blogspot.com/-lbessOoTHXI/UVBT9LLNzmI/AAAAAAAAAHo/lVsAVtmYXJE/s320/2013-03-25_23-25-00_image.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-6aYlCuG4RdI/UVBT-7nlBoI/AAAAAAAAAHw/78SS95qn2s0/s1600/2013-03-25_23-28-22_image.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="222" src="http://1.bp.blogspot.com/-6aYlCuG4RdI/UVBT-7nlBoI/AAAAAAAAAHw/78SS95qn2s0/s320/2013-03-25_23-28-22_image.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-fH8nOr1Tkd0/UVBT_8GiZjI/AAAAAAAAAH4/2zB2tDZuDNY/s1600/2013-03-25_23-28-41_image.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://2.bp.blogspot.com/-fH8nOr1Tkd0/UVBT_8GiZjI/AAAAAAAAAH4/2zB2tDZuDNY/s320/2013-03-25_23-28-41_image.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-va4zfsQ4paw/UVBUDefK3gI/AAAAAAAAAIA/zzh4Vhc8dQk/s1600/2013-03-25_23-29-22_image.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="218" src="http://1.bp.blogspot.com/-va4zfsQ4paw/UVBUDefK3gI/AAAAAAAAAIA/zzh4Vhc8dQk/s320/2013-03-25_23-29-22_image.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-VGv6fvPMvPQ/UVBUDpgSmjI/AAAAAAAAAIE/aKxAJYlBp48/s1600/2013-03-25_23-31-04_image.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="220" src="http://3.bp.blogspot.com/-VGv6fvPMvPQ/UVBUDpgSmjI/AAAAAAAAAIE/aKxAJYlBp48/s320/2013-03-25_23-31-04_image.png" width="320" /></a></div>
<br /></div>
<div>
<br /></div>
Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-56846176574315675392013-03-25T04:58:00.001-07:002015-01-27T08:27:53.130-08:00Puppet vs Fabric for deploys<div style="background-color: white;">
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<br /></div>
<div style="background-color: transparent; color: black; font-family: verdana, helvetica, sans-serif; font-size: 13.600000381469727px; font-style: normal;">
<br /></div>
<div style="background-color: transparent; color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<span style="font-size: x-small;"><a href="http://agiletesting.blogspot.com.au/2009/11/automated-deployments-with-puppet-and.html">http://agiletesting.blogspot.com.au/2009/11/automated-deployments-with-puppet-and.html</a></span></div>
<div style="background-color: transparent; color: black; font-family: verdana, helvetica, sans-serif; font-size: 16px; font-style: normal;">
<span style="font-size: x-small;"><br /></span></div>
<div class="yui_3_7_2_38_1363806269524_85" style="background-color: transparent; color: black; font-family: verdana, helvetica, sans-serif; font-size: 12.800000190734863px; font-style: normal;">
<span style="font-size: x-small;"><span style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px;">I've been looking into various configuration management/automated deployment tools lately. At OpenX we used </span><a href="http://code.google.com/p/slack/" style="color: #888888; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px; text-decoration: none;">slack</a><span style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px;">, but I wanted something with a bit more functionality than that (although I'm not badmouthing slack by any means -- it can definitely be bent to your will to do pretty much whatever you need in terms of automating your deployments).</span><br style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px;" /><br style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px;" /><span style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px;">From what I see, there are 2 types of configuration management tools:</span></span><br />
<ol style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px;">
<li style="margin: 0px 0px 0.25em; padding: 0px;"><span style="font-size: x-small;">The first type I call '<b>pull</b>', which means that the servers pull their configurations and their marching orders in terms of applying those configurations from a centralized location -- both slack and Puppet are in this category. I think this is great for initial configuration of a server. As I described in another post, you can have a server bootstrap itself by installing Puppet (or slack) and then 'call home' to the central Puppet master (or slack repository) and get all the information it needs to configure itself</span></li>
<li style="margin: 0px 0px 0.25em; padding: 0px;"><span style="font-size: x-small;">The second type I call '<b>push</b>', which means that you send configurations and commands to a list of servers from a centralized location -- Fabric is in this category. I think this is a more appropriate mode for application-specific deployments, where you might want to deploy first to a subset of servers, then push it to all servers.</span></li>
</ol>
<span style="font-size: x-small;"><span style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px;">So, as a rule of thumb, I think it makes sense to use a tool like Puppet for the initial configuration of the OS and of the packages required by your application (things like MySQL, Apache, Tomcat, Tornado, Nginx, or whatever your application relies on). When it comes time to deploy your application, I think a tool like Fabric is more appropriate, since it gives you more immediate and finer-grained control over what you want to do.</span><br style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px;" /><br style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px;" /><span style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px;">I also like the categorization of these tools done by the people at ControlTier. Check out their </span><a href="http://blog.controltier.com/2009/04/new-whitepaper-achieving-fully.html" style="color: #888888; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px; text-decoration: none;">blog post on Achieving Fully Automated Provisioning </a><span style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px;">(which also links to a white paper </span><a href="http://mcrinc.com/Documents/Newsletters/201202_WebOps2.pdf" style="color: #888888; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px; text-decoration: none;">PDF</a><span style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px;">) for a nice diagram of hierarchy of deployment tools:</span></span><br />
<ul style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px; margin: 0.5em 0px; padding: 0px 2.5em;"><span style="font-size: x-small;">
<li style="margin: 0px 0px 0.25em; padding: 0px;">at the bottom you have tools that install or launch the initial OS on physical servers (via Kickstart/Jumpstart/Cobbler) or on virtual machines/cloud instances (via various vendor tools, or by rolling your own)</li>
<li style="margin: 0px 0px 0.25em; padding: 0px;">in the middle you have what they call 'system configuration' tools, such as Puppet/Chef/SmartFrog/cfengine/bcfg2</li>
<li style="margin: 0px 0px 0.25em; padding: 0px;">at the top you have what they call 'application service deployment' tools, such as Fabric/Capistrano/Func -- and of course their own ControlTier tool</li>
</span></ul>
<span style="font-size: x-small;">
<span style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px;">In a </span><a href="http://agiletesting.blogspot.com/2009/09/bootstrapping-ec2-images-as-puppet.html#c6668282733983700258" style="color: #888888; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px; text-decoration: none;">comment</a><span style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px;"> on one of my posts, Damon Edwards from ControlTier calls Fabric a "command dispatching tool", as opposed to Puppet, which he calls a "configuration management tool". I think this relates to the 2 types of tools I described above, where you 'push' or 'dispatch' commands with Fabric, and you 'pull' configurations and actions with Puppet.</span></span></div>
<div class="yui_3_7_2_38_1363806269524_85" style="background-color: transparent; color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 12.800000190734863px; font-style: normal;">
<span style="font-size: x-small;"><span style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px;"><br /></span></span>
<span style="font-size: x-small;"><span style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px;"><br /></span></span>
<span style="font-size: x-small;"><span style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.142857551574707px; line-height: 10.285714149475098px;"><br /></span></span></div>
<div class="yui_3_7_2_38_1363806269524_85" style="background-color: transparent; color: black; font-family: verdana, helvetica, sans-serif; font-size: 12.800000190734863px; font-style: normal;">
<span style="font-size: x-small;"><br /></span></div>
<div style="background-color: transparent; color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<span style="font-size: x-small;"><a href="http://blog.controltier.com/2009/04/new-whitepaper-achieving-fully.html">http://blog.controltier.com/2009/04/new-whitepaper-achieving-fully.html</a></span></div>
<div class="yui_3_7_2_38_1363806269524_90" style="background-color: transparent; color: black; font-family: verdana, helvetica, sans-serif; font-size: 12.800000190734863px; font-style: normal;">
<br />
<h3 class="post-title entry-title" itemprop="name" style="color: #333333; font-family: Verdana, Arial, sans-serif; font-size: 16px; line-height: 1.1em; margin: 0px; padding: 0px;">
New Whitepaper: Achieving Fully Automated Provisioning</h3>
</div>
<div class="yui_3_7_2_38_1363806269524_90 yui_3_7_2_38_1363806269524_103" style="background-color: transparent; color: black; font-family: verdana, helvetica, sans-serif; font-size: 12.800000190734863px; font-style: normal;">
<span style="font-size: x-small;"><br /></span></div>
<div class="yui_3_7_2_38_1363806269524_90 yui_3_7_2_38_1363806269524_103" style="background-color: transparent; color: black; font-family: verdana, helvetica, sans-serif; font-size: 12.800000190734863px; font-style: normal;">
<span style="font-size: x-small;"><span style="color: #333333; font-family: Verdana, Arial, sans-serif; font-size: 13.333333015441895px; line-height: 16.89583396911621px;">What we are proposing is an outline of an open source toolchain for fully automated provisioning.</span><br style="color: #333333; font-family: Verdana, Arial, sans-serif; font-size: 13.333333015441895px; line-height: 16.89583396911621px;" /><br style="color: #333333; font-family: Verdana, Arial, sans-serif; font-size: 13.333333015441895px; line-height: 16.89583396911621px;" /><span style="color: #333333; font-family: Verdana, Arial, sans-serif; font-size: 13.333333015441895px; line-height: 16.89583396911621px;">What's the criteria for "fully automated provisioning"?</span><br style="color: #333333; font-family: Verdana, Arial, sans-serif; font-size: 13.333333015441895px; line-height: 16.89583396911621px;" /><br style="color: #333333; font-family: Verdana, Arial, sans-serif; font-size: 13.333333015441895px; line-height: 16.89583396911621px;" /><span style="color: #333333; font-family: Verdana, Arial, sans-serif; font-size: 13.333333015441895px; line-height: 16.89583396911621px;">That details are in the whitepaper but here's the list we came up with:</span></span><br />
<ol style="color: #333333; font-family: Verdana, Arial, sans-serif; font-size: 13.333333015441895px; line-height: 16.89583396911621px;">
<li><span style="font-size: x-small;">Be able to automatically provision an entire environment -- from "bare-metal" to running business services -- completely from specification</span></li>
<li><span style="font-size: x-small;">No direct management of individual boxes</span></li>
<li><span style="font-size: x-small;">Be able to revert to a "previously known good" state at any time</span></li>
<li><span style="font-size: x-small;">It's easier to re-provision than it is to repair</span></li>
<li><span style="font-size: x-small;">Anyone on your team with minimal domain specific knowledge can deploy or update an environment</span></li>
</ol>
<span style="font-size: x-small;"><span style="color: #333333; font-family: Verdana, Arial, sans-serif; font-size: 13.333333015441895px; line-height: 16.89583396911621px;">Here's the proposed open source toolchain (with ControlTier and Puppet highlighted):</span></span><br />
<div class="separator" style="clear: both; color: #333333; font-family: Verdana, Arial, sans-serif; font-size: 13.333333015441895px; line-height: 16.89583396911621px; margin: 0px 0px 0.75em; text-align: center;">
<span style="font-size: x-small;"><a href="http://2.bp.blogspot.com/_B-19AvbslxM/SuD4UBITyrI/AAAAAAAAAN8/zkW3tv_ZpC8/s1600-h/OpenProvisioningToolchain.png" imageanchor="1" style="color: #336699; margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/_B-19AvbslxM/SuD4UBITyrI/AAAAAAAAAN8/zkW3tv_ZpC8/s320/OpenProvisioningToolchain.png" style="border: 1px solid rgb(204, 204, 204); padding: 4px;" /></a></span></div>
<span style="font-size: x-small;">
</span>
<br />
<div>
<span style="font-size: x-small;"><br /></span></div>
<span style="font-size: x-small;">
</span>
<br />
<div>
<span style="font-size: x-small;"><br /></span></div>
<span style="font-size: x-small;">
</span></div>
<div class="yui_3_7_2_38_1363806269524_90 yui_3_7_2_38_1363806269524_103" style="background-color: transparent; color: black; font-family: verdana, helvetica, sans-serif; font-size: 12.800000190734863px; font-style: normal;">
<span style="font-size: x-small;"><br /></span></div>
<div class="yui_3_7_2_38_1363806269524_90" style="background-color: transparent; color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<span style="font-size: x-small;"><a href="http://agiletesting.blogspot.com.au/2009/09/bootstrapping-ec2-images-as-puppet.html#c6668282733983700258">http://agiletesting.blogspot.com.au/2009/09/bootstrapping-ec2-images-as-puppet.html#c6668282733983700258</a></span></div>
<div class="yui_3_7_2_38_1363806269524_90 yui_3_7_2_38_1363806269524_108" style="background-color: transparent; color: black; font-family: verdana, helvetica, sans-serif; font-size: 12.800000190734863px; font-style: normal;">
<span style="font-size: x-small;"><br /></span></div>
<div class="yui_3_7_2_38_1363806269524_90 yui_3_7_2_38_1363806269524_108 yui_3_7_2_38_1363806269524_121" style="background-color: transparent; color: black; font-family: verdana, helvetica, sans-serif; font-size: 12.800000190734863px; font-style: normal;">
<span style="font-size: x-small;"></span><br />
<br />
<br />
<br />
<br />
<dt class="comment-author blogger-comment-icon" id="c6668282733983700258" style="background-image: url(https://www.blogger.com/img/b16-rounded.gif); background-position: 0px 1.5em; background-repeat: no-repeat no-repeat; border-top-color: rgb(238, 238, 238); border-top-style: solid; border-top-width: 1px; color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 12px; font-weight: bold; line-height: 16px; padding-left: 25px; padding-top: 1.5em;"><span style="font-size: x-small;"><a href="http://www.blogger.com/profile/08075446611632722802" rel="nofollow" style="color: #888888; text-decoration: none;">Damon Edwards</a> said...</span></dt>
<dd class="comment-body" id="Blog1_cmt-6668282733983700258" style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 12px; line-height: 9.142857551574707px; margin: 0.5em 25px;"><div>
<span style="font-size: x-small;">Great post.</span><br />
<span style="font-size: x-small;"><br /></span>
<span style="font-size: x-small;">I found it interesting that you would directly compare a command dispatching tool like Fabric to a configuration management tool like Puppet. </span><br />
<span style="font-size: x-small;"><br /></span>
<span style="font-size: x-small;">We did a whitepaper on how these different types of tools can be used as an open source toolchain to achieve fully automated provisioning (going from bare metal all the way to running multitier applications)</span><br />
<span style="font-size: x-small;"><br /></span>
<span style="font-size: x-small;">The whitepaper is written about a specific ControlTier and Puppet implementation but the toolchain and the general concepts hold for tools like Chef, Fabric, Cobbler, Capistrano, etc.</span><br />
<span style="font-size: x-small;"><br /></span>
<span style="font-size: x-small;">Here's the link:</span><br />
<span style="font-size: x-small;">http://blog.controltier.com/2009/04/new-whitepaper-achieving-fully.html</span></div>
</dd><dd class="comment-footer" style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 12px; line-height: 9.142857551574707px; margin: 0.5em 25px 1.5em;"><span style="font-size: x-small;"><span class="comment-timestamp"><a href="http://agiletesting.blogspot.com/2009/09/bootstrapping-ec2-images-as-puppet.html?showComment=1256259157312#c6668282733983700258" style="color: #888888; text-decoration: none;" title="comment permalink">5:52 PM</a></span></span></dd>
<dt class="comment-author blog-author blogger-comment-icon" id="c4818785579503069315" style="background-image: url(https://www.blogger.com/img/b16-rounded.gif); background-position: 0px 1.5em; background-repeat: no-repeat no-repeat; border-top-color: rgb(238, 238, 238); border-top-style: solid; border-top-width: 1px; color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 12px; font-weight: bold; line-height: 16px; padding-left: 25px; padding-top: 1.5em;"><span style="font-size: x-small;"><a href="http://www.blogger.com/blogger.g?blogID=6287813703760221728" name="c4818785579503069315"></a><a href="http://www.blogger.com/profile/17863511617654196370" rel="nofollow" style="color: #888888; text-decoration: none;">Grig Gheorghiu</a> said...</span></dt>
<dd class="comment-body" id="Blog1_cmt-4818785579503069315" style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 12px; line-height: 9.142857551574707px; margin: 0.5em 25px;"><div>
<span style="font-size: x-small;">Damon -- I guess my intention wasn't clear. I didn't mean to compare Fabric to Puppet as an apples-to-apples comparison. I was merely wishing that a Puppet-like configuration management tool existed in Python. I know about bcfg2, I haven't played with it though.</span><br />
<span style="font-size: x-small;"><br /></span>
<span style="font-size: x-small;">Thanks for the pointer to your white paper.</span><br />
<span style="font-size: x-small;"><br /></span>
<span style="font-size: x-small;">Grig</span></div>
</dd><dd class="comment-footer" style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 12px; line-height: 9.142857551574707px; margin: 0.5em 25px 1.5em;"><span style="font-size: x-small;"><span class="comment-timestamp"><a href="http://agiletesting.blogspot.com/2009/09/bootstrapping-ec2-images-as-puppet.html?showComment=1256273772242#c4818785579503069315" style="color: #888888; text-decoration: none;" title="comment permalink">9:56 PM</a></span></span></dd>
<dt class="comment-author blogger-comment-icon" id="c8491129601066561215" style="background-image: url(https://www.blogger.com/img/b16-rounded.gif); background-position: 0px 1.5em; background-repeat: no-repeat no-repeat; border-top-color: rgb(238, 238, 238); border-top-style: solid; border-top-width: 1px; color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 12px; font-weight: bold; line-height: 16px; padding-left: 25px; padding-top: 1.5em;"><span style="font-size: x-small;"><a href="http://www.blogger.com/blogger.g?blogID=6287813703760221728" name="c8491129601066561215"></a><a href="http://www.blogger.com/profile/08075446611632722802" rel="nofollow" style="color: #888888; text-decoration: none;">Damon Edwards</a> said...</span></dt>
<dd class="comment-body" id="Blog1_cmt-8491129601066561215" style="color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 12px; line-height: 9.142857551574707px; margin: 0.5em 25px;"><div>
<span style="font-size: x-small;">Ah got it. You know, that does touch on an excellent subject... when you build a toolchain of management tools, what kind of overhead are you adding because of differing languages or configuration methods? It begs the question, is it easier to bend (or some would say abuse) one tool to do most of everything that you want or should use a toolchain that lets each tool do what its supposed to be good at?</span></div>
</dd></div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
</div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<br /></div>
<div class="yui_3_7_2_38_1363806269524_57" style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<br /></div>
<div class="yui_3_7_2_38_1363806269524_57" style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
______________________________________________________________</div>
<div class="yui_3_7_2_38_1363806269524_57">
<span style="font-family: verdana, helvetica, sans-serif;"><span style="font-size: 13.3333330154419px;"><br /></span></span>
<span style="font-family: verdana, helvetica, sans-serif;"><span style="font-size: 13.3333330154419px;"><a href="http://serverfault.com/questions/372575/application-deployment-with-puppet">http://serverfault.com/questions/372575/application-deployment-with-puppet</a></span></span><br />
<span style="font-family: verdana, helvetica, sans-serif;"><span style="font-size: 13.3333330154419px;"><br /></span></span></div>
<div class="yui_3_7_2_38_1363806269524_57" style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<br /></div>
<div class="yui_3_7_2_38_1363806269524_57" style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<div id="question-header" style="border-bottom-color: rgb(85, 85, 85); border-bottom-style: solid; border-width: 0px 0px 1px; font-family: 'Trebuchet MS', 'Liberation Sans', 'DejaVu Sans', sans-serif; font-size: 13px; line-height: 6px; margin: 0px 0px 20px; padding: 0px; vertical-align: baseline; width: 960px;">
<h1 itemprop="name" style="background-color: transparent; border: 0px; font-size: 23px; line-height: 1.3; margin: 0px 0px 7px; padding: 0px; vertical-align: baseline;">
<a class="question-hyperlink" href="http://serverfault.com/questions/372575/application-deployment-with-puppet" style="background-color: transparent; border: 0px; color: black; cursor: pointer; font-size: 23px; line-height: 15px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Application deployment with Puppet</a></h1>
</div>
<div id="mainbar" style="border: 0px; float: left; font-family: Arial, 'Liberation Sans', 'DejaVu Sans', sans-serif; font-size: 13px; line-height: 6px; margin: 0px 0px 40px; padding: 0px; vertical-align: baseline; width: 735px;">
<div class="question" data-questionid="372575" id="question" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; clear: both; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline; width: 728px;">
<table style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-collapse: collapse; border-spacing: 0px; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><tbody style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<tr style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><td class="votecell" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: top; width: 60px;"><div class="vote" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; text-align: center; vertical-align: baseline;">
<a class="vote-up-off" href="http://www.blogger.com/blogger.g?blogID=6287813703760221728" style="background-color: transparent; background-image: url(https://cdn.sstatic.net/serverfault/img/sprites.png?v=6); background-position: 0px -265px; background-repeat: no-repeat no-repeat; border: 0px; color: #10456a; cursor: pointer; display: block; font-size: 1px; height: 25px; margin: 0px auto; overflow: hidden; padding: 0px; text-decoration: none; text-indent: -9999em; vertical-align: baseline; width: 41px;" title="This question shows research effort; it is useful and clear (click again to undo)">up vote</a><span class="vote-count-post " style="background-color: transparent; border: 0px; color: #555555; display: block; font-size: 24px; margin: 0px; padding: 2px 0px 0px; vertical-align: baseline;">7</span><a class="vote-down-off" href="http://www.blogger.com/blogger.g?blogID=6287813703760221728" style="background-color: transparent; background-image: url(https://cdn.sstatic.net/serverfault/img/sprites.png?v=6); background-position: 0px -300px; background-repeat: no-repeat no-repeat; border: 0px; color: #10456a; cursor: pointer; display: block; font-size: 1px; height: 25px; margin: 0px auto; overflow: hidden; padding: 0px; text-decoration: none; text-indent: -9999em; vertical-align: baseline; width: 41px;" title="This question does not show any research effort; it is unclear or not useful (click again to undo)">down vote</a><a class="star-off" href="http://serverfault.com/questions/372575/application-deployment-with-puppet#" style="background-color: transparent; background-image: url(https://cdn.sstatic.net/serverfault/img/sprites.png?v=6); background-position: 0px -150px; background-repeat: no-repeat no-repeat; border: 0px; color: #456583; cursor: pointer; display: block; font-size: 1px; height: 31px; margin: 0px auto; overflow: hidden; padding: 0px; text-decoration: none; text-indent: -9999em; vertical-align: baseline; width: 33px;" title="This is a favorite question (click again to undo)">favorite</a><br />
<div class="favoritecount" style="background-color: transparent; border: 0px; color: #808185; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<b style="background-color: transparent; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">5</b></div>
</div>
</td><td class="postcell" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><div style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<div class="post-text" itemprop="description" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13.5px; line-height: 17.5px; margin: 0px 5px 5px 0px; padding: 0px; vertical-align: baseline; width: 660px;">
<div style="background-color: transparent; border: 0px; clear: both; font-size: 13.5px; margin-bottom: 1em; padding: 0px; vertical-align: baseline; word-wrap: break-word;">
I'm new to Puppet and while I've been using *nix systems for many years, I've never worked as a sysadmin or in ops.</div>
<div style="background-color: transparent; border: 0px; clear: both; font-size: 13.5px; margin-bottom: 1em; padding: 0px; vertical-align: baseline; word-wrap: break-word;">
I'm currently writing Puppet manifests for hosting a set of (PHP/MySQL/MongoDB, code in git) web applications. Clearly Puppet needs to have some knowledge of the actual applications because I'll set up a virtual host for each one, but I'm not sure whether Puppet should be managing things like code deployment and database creation.</div>
<div style="background-color: transparent; border: 0px; clear: both; font-size: 13.5px; margin-bottom: 1em; padding: 0px; vertical-align: baseline; word-wrap: break-word;">
Is Puppet an appropriate tool for application deployment? If not, can you recommend a more appropriate tool?</div>
</div>
<div class="post-taglist" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; clear: both; font-size: 13px; margin: 0px 0px 10px; padding: 0px; vertical-align: baseline;">
<a class="post-tag" href="http://serverfault.com/questions/tagged/deployment" rel="tag" style="background-color: #f3f1d9; background-position: initial initial; background-repeat: initial initial; border: 1px solid rgb(197, 184, 73); color: #444444; cursor: pointer; font-size: 11.5px; line-height: 2.4; margin: 2px 2px 2px 0px; padding: 3px 4px; text-decoration: none; vertical-align: baseline; white-space: nowrap;" title="show questions tagged 'deployment'">deployment</a> <a class="post-tag" href="http://serverfault.com/questions/tagged/puppet" rel="tag" style="background-color: #f3f1d9; background-position: initial initial; background-repeat: initial initial; border: 1px solid rgb(197, 184, 73); color: #444444; cursor: pointer; font-size: 11.5px; line-height: 2.4; margin: 2px 2px 2px 0px; padding: 3px 4px; text-decoration: none; vertical-align: baseline; white-space: nowrap;" title="show questions tagged 'puppet'">puppet</a></div>
<table class="fw" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-collapse: collapse; border-spacing: 0px; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline; width: 665px;"><tbody style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<tr style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><td class="vt" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: top;"><div class="post-menu" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 2px 0px 0px; vertical-align: baseline;">
<a class="short-link" href="http://serverfault.com/q/372575" id="link-post-372575" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #888888; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px 3px 2px; text-decoration: none; vertical-align: baseline;" title="short permalink to this question">share</a><span class="lsep" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #cccccc; font-size: 14px; margin: 0px; padding: 0px; vertical-align: baseline;">|</span><a class="suggest-edit-post" href="http://serverfault.com/posts/372575/edit" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #888888; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px 3px 2px; text-decoration: none; vertical-align: baseline;" title="">improve this question</a></div>
</td><td class="post-signature owner" style="background-color: #f3f1d9; border: 0px; font-size: 13px; height: 58px; margin: 0px; padding: 2px 0px 0px 5px; vertical-align: top; width: 175px;"><div class="user-info " style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; height: 35px; margin: 0px; padding: 0px; vertical-align: baseline; width: 185px;">
<div class="user-action-time" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 2px 0px 4px; padding: 0px; vertical-align: baseline;">
asked <span class="relativetime" style="background-color: transparent; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" title="2012-03-22 22:03:08Z">Mar 22 '12 at 22:03</span></div>
<div class="user-gravatar32" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; float: left; font-size: 13px; height: 32px; margin: 0px; padding: 0px; vertical-align: baseline; width: 32px;">
<a href="http://serverfault.com/users/87611/michaeltwofish" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"></a><br />
<div class="" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://serverfault.com/users/87611/michaeltwofish" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"><img alt="" src="http://www.gravatar.com/avatar/8f163120341854b69991cff1c39829ad?s=32&d=identicon&r=PG" height="32" style="background-color: transparent; border-width: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" width="32" /></a></div>
</div>
<div class="user-details" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #888888; float: left; font-size: 13px; line-height: 17px; margin: 0px 0px 0px 5px; overflow: hidden; padding: 0px; vertical-align: baseline; white-space: nowrap; width: 145px;">
<a href="http://serverfault.com/users/87611/michaeltwofish" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">michaeltwofish</a><br />
<span class="reputation-score" dir="ltr" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #444444; font-size: 13px; font-weight: bold; margin: 0px 2px 0px 0px; padding: 0px; vertical-align: baseline;" title="reputation score">119</span><span style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" title="8 bronze badges"><span class="badge3" style="background-color: transparent; background-image: url(https://cdn.sstatic.net/serverfault/img/sprites.png?v=6); background-position: 0px -1034px; background-repeat: no-repeat no-repeat; border: 0px; display: inline-block; font-size: 13px; height: 11px; line-height: inherit; margin: 0px 1px 1px 3px; overflow: hidden; padding: 0px; vertical-align: text-bottom; width: 6px;"></span><span class="badgecount" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #808185; font-size: 13px; margin: 0px; padding: 0px 0px 0px 1px; vertical-align: baseline;">8</span></span></div>
</div>
</td></tr>
</tbody></table>
</div>
</td></tr>
<tr style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><td class="votecell" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: top; width: 60px;"></td><td style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><div class="comments" id="comments-372575" style="-webkit-tap-highlight-color: rgba(255, 255, 255, 0); background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-top-color: rgb(170, 170, 170); border-top-style: dotted; border-width: 1px 0px 0px; color: #444444; font-size: 13px; line-height: 17.5px; margin: 10px 0px 0px; padding: 0px 0px 10px; vertical-align: baseline; width: 660px;">
<table style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-collapse: collapse; border-spacing: 0px; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline; width: 660px;"><tbody style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<tr class="comment" id="comment-380876" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><td style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-color: rgb(221, 221, 221); border-bottom-style: dotted; border-width: 0px 0px 1px; font-size: 13px; margin: 0px; padding: 5px 0px; vertical-align: baseline;"></td><td class="comment-text" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-color: rgb(221, 221, 221); border-bottom-style: dotted; border-width: 0px 0px 1px; font-size: 13px; margin: 0px; padding: 5px 6px 5px 7px; vertical-align: text-top;"><div style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span class="comment-copy" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">If you're coming along to this later, all the answers are good, don't just read the one that I chose as the answer.</span> – <a class="comment-user owner" href="http://serverfault.com/users/87611/michaeltwofish" style="background-color: #f3f1d9; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13px; margin: 0px; padding: 2px 5px; text-decoration: none; vertical-align: baseline; white-space: nowrap;" title="119 reputation">michaeltwofish</a> <span class="comment-date" dir="ltr" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-style: none; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; color: #999999; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><a class="comment-link" href="http://serverfault.com/questions/372575/application-deployment-with-puppet#comment380876_372575" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-style: none; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; color: #999999; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"><span class="relativetime-clean" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" title="2012-03-25 22:28:01Z">Mar 25 '12 at 22:28</span></a></span></div>
</td></tr>
</tbody></table>
</div>
</td></tr>
</tbody></table>
</div>
<div id="answers" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; clear: both; font-size: 13px; margin: 0px; padding: 5px 0px 0px; vertical-align: baseline; width: 728px;">
<a href="http://www.blogger.com/blogger.g?blogID=6287813703760221728" name="tab-top" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #10456a; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"></a><br />
<div id="answers-header" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 10px 0px; padding: 0px; vertical-align: baseline; width: 728px;">
<div class="subheader answers-subheader" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-color: rgb(102, 102, 102); border-bottom-style: solid; border-width: 0px 0px 1px; clear: both; font-size: 13px; height: 34px; margin: 0px 0px -10px; padding: 0px; vertical-align: baseline; width: 728px;">
<h2 style="background-color: transparent; border: 0px; float: left; font-family: 'Trebuchet MS', 'Liberation Sans', 'DejaVu Sans', sans-serif; font-size: 18px; line-height: 34px; margin: 0px; padding: 0px; vertical-align: baseline;">
6 Answers</h2>
<div id="tabs" style="background-color: transparent; border: 0px; float: right; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://serverfault.com/questions/372575/application-deployment-with-puppet?answertab=active#tab-top" style="background-attachment: inherit; background-clip: inherit; background-color: inherit; background-image: inherit; background-origin: inherit; background-position: inherit inherit; background-repeat: inherit inherit; background-size: inherit; border: 1px solid rgb(255, 255, 255); color: #777777; cursor: pointer; display: block; float: left; font-size: 11.5px; height: 24px; line-height: 22px; margin: 8px 8px 0px 0px; padding: 0px 11px; text-decoration: none; vertical-align: baseline;" title="Answers with the latest activity first">active</a><a href="http://serverfault.com/questions/372575/application-deployment-with-puppet?answertab=oldest#tab-top" style="background-attachment: inherit; background-clip: inherit; background-color: inherit; background-image: inherit; background-origin: inherit; background-position: inherit inherit; background-repeat: inherit inherit; background-size: inherit; border: 1px solid rgb(255, 255, 255); color: #777777; cursor: pointer; display: block; float: left; font-size: 11.5px; height: 24px; line-height: 22px; margin: 8px 8px 0px 0px; padding: 0px 11px; text-decoration: none; vertical-align: baseline;" title="Answers in the order they were provided">oldest</a><a class="youarehere" href="http://serverfault.com/questions/372575/application-deployment-with-puppet?answertab=votes#tab-top" style="border-color: rgb(204, 204, 204) rgb(204, 204, 204) rgb(255, 252, 247); border-style: solid; border-width: 1px; color: black; cursor: pointer; display: block; float: left; font-size: 15.5px; height: 30px; line-height: 28px; margin: 3px 8px 0px 0px; padding: 0px 11px; text-decoration: none; vertical-align: baseline;" title="Answers with the highest score first">votes</a></div>
</div>
</div>
<a href="http://www.blogger.com/blogger.g?blogID=6287813703760221728" name="372617" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #10456a; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"></a><br />
<div class="answer accepted-answer" data-answerid="372617" id="answer-372617" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-color: rgb(102, 102, 102); border-bottom-style: dotted; border-width: 0px 0px 1px; font-size: 13px; margin: 0px; padding: 20px 0px; vertical-align: baseline; width: 728px;">
<table style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-collapse: collapse; border-spacing: 0px; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><tbody style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<tr style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><td class="votecell" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: top; width: 60px;"><div class="vote" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; text-align: center; vertical-align: baseline;">
<a class="vote-up-off" href="http://www.blogger.com/blogger.g?blogID=6287813703760221728" style="background-color: transparent; background-image: url(https://cdn.sstatic.net/serverfault/img/sprites.png?v=6); background-position: 0px -265px; background-repeat: no-repeat no-repeat; border: 0px; color: #10456a; cursor: pointer; display: block; font-size: 1px; height: 25px; margin: 0px auto; overflow: hidden; padding: 0px; text-decoration: none; text-indent: -9999em; vertical-align: baseline; width: 41px;" title="This answer is useful (click again to undo)">up vote</a><span class="vote-count-post " style="background-color: transparent; border: 0px; color: #555555; display: block; font-size: 24px; margin: 0px; padding: 2px 0px 0px; vertical-align: baseline;">7</span><a class="vote-down-off" href="http://www.blogger.com/blogger.g?blogID=6287813703760221728" style="background-color: transparent; background-image: url(https://cdn.sstatic.net/serverfault/img/sprites.png?v=6); background-position: 0px -300px; background-repeat: no-repeat no-repeat; border: 0px; color: #10456a; cursor: pointer; display: block; font-size: 1px; height: 25px; margin: 0px auto; overflow: hidden; padding: 0px; text-decoration: none; text-indent: -9999em; vertical-align: baseline; width: 41px;" title="This answer is not useful (click again to undo)">down vote</a><span class="vote-accepted-on load-accepted-answer-date" style="background-color: transparent; background-image: url(https://cdn.sstatic.net/serverfault/img/sprites.png?v=6); background-position: 0px -460px; background-repeat: no-repeat no-repeat; border: 0px; color: #808185; display: block; font-size: 1px; height: 40px; margin: 0px auto; overflow: hidden; padding: 0px; text-indent: -9999em; vertical-align: baseline; width: 39px;" title="loading when this answer was accepted...">accepted</span></div>
</td><td class="answercell" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><div class="post-text" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13.5px; line-height: 17.5px; margin: 0px 5px 5px 0px; padding: 0px; vertical-align: baseline; width: 660px;">
<div style="background-color: transparent; border: 0px; clear: both; font-size: 13.5px; margin-bottom: 1em; padding: 0px; vertical-align: baseline; word-wrap: break-word;">
I'd look into either <a href="http://rubyforge.org/projects/capistrano/" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13.5px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Capistrano</a> or <a href="http://docs.fabfile.org/en/1.4.0/index.html" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13.5px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Fabric</a> for deployments..</div>
<div style="background-color: transparent; border: 0px; clear: both; font-size: 13.5px; margin-bottom: 1em; padding: 0px; vertical-align: baseline; word-wrap: break-word;">
You'll have better control over how the deployment happens with these two tools.</div>
</div>
<table class="fw" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-collapse: collapse; border-spacing: 0px; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline; width: 665px;"><tbody style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<tr style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><td class="vt" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: top;"><div class="post-menu" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 2px 0px 0px; vertical-align: baseline;">
<a class="short-link" href="http://serverfault.com/a/372617" id="link-post-372617" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #888888; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px 3px 2px; text-decoration: none; vertical-align: baseline;" title="short permalink to this answer">share</a><span class="lsep" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #cccccc; font-size: 14px; margin: 0px; padding: 0px; vertical-align: baseline;">|</span><a class="suggest-edit-post" href="http://serverfault.com/posts/372617/edit" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #888888; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px 3px 2px; text-decoration: none; vertical-align: baseline;" title="">improve this answer</a></div>
</td><td align="right" class="post-signature" style="background-color: transparent; border: 0px; font-size: 13px; height: 58px; margin: 0px; padding: 2px 0px 0px 5px; vertical-align: top; width: 175px;"><div class="user-info user-hover" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; height: 35px; margin: 0px; padding: 0px; vertical-align: baseline; width: 185px;">
<div class="user-action-time" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 2px 0px 4px; padding: 0px; vertical-align: baseline;">
edited <a href="http://serverfault.com/posts/372617/revisions" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" title="show all edits to this post"><span class="relativetime" style="background-color: transparent; border: 0px; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" title="2012-03-23 08:22:12Z">Mar 23 '12 at 8:22</span></a></div>
<div class="user-gravatar32" style="-webkit-box-shadow: rgba(0, 0, 0, 0.498039) 2px 2px 4px; background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-collapse: separate; border: 0px; box-shadow: rgba(0, 0, 0, 0.498039) 2px 2px 4px; float: left; font-size: 13px; height: 32px; margin: 0px; padding: 0px; vertical-align: baseline; width: 32px;">
<a href="http://serverfault.com/users/49297/kenny-rasschaert" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"></a><br />
<div class="" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://serverfault.com/users/49297/kenny-rasschaert" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"><img alt="" src="http://www.gravatar.com/avatar/31cfce49b58ab051432b64572357369e?s=32&d=identicon&r=PG" height="32" style="background-color: transparent; border-width: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" width="32" /></a></div>
</div>
<div class="user-details" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #888888; float: left; font-size: 13px; line-height: 17px; margin: 0px 0px 0px 5px; overflow: hidden; padding: 0px; vertical-align: baseline; white-space: nowrap; width: 145px;">
<a href="http://serverfault.com/users/49297/kenny-rasschaert" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Kenny Rasschaert</a><br />
<span class="reputation-score" dir="ltr" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #444444; font-size: 13px; font-weight: bold; margin: 0px 2px 0px 0px; padding: 0px; vertical-align: baseline;" title="reputation score">4,877</span><span style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" title="3 gold badges"><span class="badge1" style="background-color: transparent; background-image: url(https://cdn.sstatic.net/serverfault/img/sprites.png?v=6); background-position: -40px -1034px; background-repeat: no-repeat no-repeat; border: 0px; display: inline-block; font-size: 13px; height: 11px; line-height: inherit; margin: 0px 1px 1px 3px; overflow: hidden; padding: 0px; vertical-align: text-bottom; width: 6px;"></span><span class="badgecount" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #808185; font-size: 13px; margin: 0px; padding: 0px 0px 0px 1px; vertical-align: baseline;">3</span></span><span style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" title="17 silver badges"><span class="badge2" style="background-color: transparent; background-image: url(https://cdn.sstatic.net/serverfault/img/sprites.png?v=6); background-position: -20px -1034px; background-repeat: no-repeat no-repeat; border: 0px; display: inline-block; font-size: 13px; height: 11px; line-height: inherit; margin: 0px 1px 1px 3px; overflow: hidden; padding: 0px; vertical-align: text-bottom; width: 6px;"></span><span class="badgecount" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #808185; font-size: 13px; margin: 0px; padding: 0px 0px 0px 1px; vertical-align: baseline;">17</span></span><span style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" title="39 bronze badges"><span class="badge3" style="background-color: transparent; background-image: url(https://cdn.sstatic.net/serverfault/img/sprites.png?v=6); background-position: 0px -1034px; background-repeat: no-repeat no-repeat; border: 0px; display: inline-block; font-size: 13px; height: 11px; line-height: inherit; margin: 0px 1px 1px 3px; overflow: hidden; padding: 0px; vertical-align: text-bottom; width: 6px;"></span><span class="badgecount" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #808185; font-size: 13px; margin: 0px; padding: 0px 0px 0px 1px; vertical-align: baseline;">39</span></span></div>
</div>
</td><td align="right" class="post-signature" style="background-color: transparent; border: 0px; font-size: 13px; height: 58px; margin: 0px; padding: 2px 0px 0px 5px; vertical-align: top; width: 175px;"><div class="user-info user-hover" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; height: 35px; margin: 0px; padding: 0px; vertical-align: baseline; width: 185px;">
<div class="user-action-time" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 2px 0px 4px; padding: 0px; vertical-align: baseline;">
answered <span class="relativetime" style="background-color: transparent; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" title="2012-03-23 02:40:06Z">Mar 23 '12 at 2:40</span></div>
<div class="user-gravatar32" style="-webkit-box-shadow: rgba(0, 0, 0, 0.498039) 2px 2px 4px; background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-collapse: separate; border: 0px; box-shadow: rgba(0, 0, 0, 0.498039) 2px 2px 4px; float: left; font-size: 13px; height: 32px; margin: 0px; padding: 0px; vertical-align: baseline; width: 32px;">
<a href="http://serverfault.com/users/23818/mike" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"></a><br />
<div class="" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://serverfault.com/users/23818/mike" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"><img alt="" src="http://www.gravatar.com/avatar/9d1484425e9a125402f85497c7f86a00?s=32&d=identicon&r=PG" height="32" style="background-color: transparent; border-width: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" width="32" /></a></div>
</div>
<div class="user-details" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #888888; float: left; font-size: 13px; line-height: 17px; margin: 0px 0px 0px 5px; overflow: hidden; padding: 0px; vertical-align: baseline; white-space: nowrap; width: 145px;">
<a href="http://serverfault.com/users/23818/mike" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Mike</a><br />
<span class="reputation-score" dir="ltr" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #444444; font-size: 13px; font-weight: bold; margin: 0px 2px 0px 0px; padding: 0px; vertical-align: baseline;" title="reputation score">9,856</span><span style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" title="3 gold badges"><span class="badge1" style="background-color: transparent; background-image: url(https://cdn.sstatic.net/serverfault/img/sprites.png?v=6); background-position: -40px -1034px; background-repeat: no-repeat no-repeat; border: 0px; display: inline-block; font-size: 13px; height: 11px; line-height: inherit; margin: 0px 1px 1px 3px; overflow: hidden; padding: 0px; vertical-align: text-bottom; width: 6px;"></span><span class="badgecount" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #808185; font-size: 13px; margin: 0px; padding: 0px 0px 0px 1px; vertical-align: baseline;">3</span></span><span style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" title="9 silver badges"><span class="badge2" style="background-color: transparent; background-image: url(https://cdn.sstatic.net/serverfault/img/sprites.png?v=6); background-position: -20px -1034px; background-repeat: no-repeat no-repeat; border: 0px; display: inline-block; font-size: 13px; height: 11px; line-height: inherit; margin: 0px 1px 1px 3px; overflow: hidden; padding: 0px; vertical-align: text-bottom; width: 6px;"></span><span class="badgecount" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #808185; font-size: 13px; margin: 0px; padding: 0px 0px 0px 1px; vertical-align: baseline;">9</span></span><span style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" title="22 bronze badges"><span class="badge3" style="background-color: transparent; background-image: url(https://cdn.sstatic.net/serverfault/img/sprites.png?v=6); background-position: 0px -1034px; background-repeat: no-repeat no-repeat; border: 0px; display: inline-block; font-size: 13px; height: 11px; line-height: inherit; margin: 0px 1px 1px 3px; overflow: hidden; padding: 0px; vertical-align: text-bottom; width: 6px;"></span><span class="badgecount" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #808185; font-size: 13px; margin: 0px; padding: 0px 0px 0px 1px; vertical-align: baseline;">22</span></span></div>
</div>
</td></tr>
</tbody></table>
</td></tr>
<tr style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><td class="votecell" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: top; width: 60px;"></td><td style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><div class="comments" id="comments-372617" style="-webkit-tap-highlight-color: rgba(255, 255, 255, 0); background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-top-color: rgb(170, 170, 170); border-top-style: dotted; border-width: 1px 0px 0px; color: #444444; font-size: 13px; line-height: 17.5px; margin: 10px 0px 0px; padding: 0px 0px 10px; vertical-align: baseline; width: 660px;">
<table style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-collapse: collapse; border-spacing: 0px; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline; width: 660px;"><tbody style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<tr class="comment" id="comment-380084" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><td style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-color: rgb(221, 221, 221); border-bottom-style: dotted; border-width: 0px 0px 1px; font-size: 13px; margin: 0px; padding: 5px 0px; vertical-align: baseline;"></td><td class="comment-text" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-color: rgb(221, 221, 221); border-bottom-style: dotted; border-width: 0px 0px 1px; font-size: 13px; margin: 0px; padding: 5px 6px 5px 7px; vertical-align: text-top;"><div style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span class="comment-copy" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">I +1'd this answer because it gave two solutions.</span> – <a class="comment-user" href="http://serverfault.com/users/155993/francois-beausoleil" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline; white-space: nowrap;" title="236 reputation">François Beausoleil</a> <span class="comment-date" dir="ltr" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-style: none; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; color: #999999; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><a class="comment-link" href="http://serverfault.com/questions/372575/application-deployment-with-puppet#comment380084_372617" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-style: none; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; color: #999999; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"><span class="relativetime-clean" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" title="2012-03-23 03:45:09Z">Mar 23 '12 at 3:45</span></a></span></div>
</td></tr>
<tr class="comment" id="comment-380877" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><td style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-color: rgb(221, 221, 221); border-bottom-style: dotted; border-width: 0px 0px 1px; font-size: 13px; margin: 0px; padding: 5px 0px; vertical-align: baseline;"></td><td class="comment-text" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-color: rgb(221, 221, 221); border-bottom-style: dotted; border-width: 0px 0px 1px; font-size: 13px; margin: 0px; padding: 5px 6px 5px 7px; vertical-align: text-top;"><div style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span class="comment-copy" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">I appreciate the pointers. We have Ruby elsewhere in our stack, so I'll look at Capistrano.</span> – <a class="comment-user owner" href="http://serverfault.com/users/87611/michaeltwofish" style="background-color: #f3f1d9; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13px; margin: 0px; padding: 2px 5px; text-decoration: none; vertical-align: baseline; white-space: nowrap;" title="119 reputation">michaeltwofish</a> <span class="comment-date" dir="ltr" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-style: none; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; color: #999999; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><a class="comment-link" href="http://serverfault.com/questions/372575/application-deployment-with-puppet#comment380877_372617" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-style: none; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; color: #999999; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"><span class="relativetime-clean" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" title="2012-03-25 22:28:54Z">Mar 25 '12 at 22:28</span></a></span></div>
</td></tr>
</tbody></table>
</div>
</td></tr>
</tbody></table>
</div>
<div class="everyonelovesstackoverflow" id="adzerk3" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px 0px 8px; padding: 0px; vertical-align: baseline;">
</div>
<a href="http://www.blogger.com/blogger.g?blogID=6287813703760221728" name="372603" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #10456a; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"></a><br />
<div class="answer" data-answerid="372603" id="answer-372603" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-color: rgb(102, 102, 102); border-bottom-style: dotted; border-width: 0px 0px 1px; font-size: 13px; margin: 0px; padding: 20px 0px; vertical-align: baseline; width: 728px;">
<table style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-collapse: collapse; border-spacing: 0px; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><tbody style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<tr style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><td class="votecell" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: top; width: 60px;"><div class="vote" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; text-align: center; vertical-align: baseline;">
<a class="vote-up-off" href="http://www.blogger.com/blogger.g?blogID=6287813703760221728" style="background-color: transparent; background-image: url(https://cdn.sstatic.net/serverfault/img/sprites.png?v=6); background-position: 0px -265px; background-repeat: no-repeat no-repeat; border: 0px; color: #10456a; cursor: pointer; display: block; font-size: 1px; height: 25px; margin: 0px auto; overflow: hidden; padding: 0px; text-decoration: none; text-indent: -9999em; vertical-align: baseline; width: 41px;" title="This answer is useful (click again to undo)">up vote</a><span class="vote-count-post " style="background-color: transparent; border: 0px; color: #555555; display: block; font-size: 24px; margin: 0px; padding: 2px 0px 0px; vertical-align: baseline;">4</span><a class="vote-down-off" href="http://www.blogger.com/blogger.g?blogID=6287813703760221728" style="background-color: transparent; background-image: url(https://cdn.sstatic.net/serverfault/img/sprites.png?v=6); background-position: 0px -300px; background-repeat: no-repeat no-repeat; border: 0px; color: #10456a; cursor: pointer; display: block; font-size: 1px; height: 25px; margin: 0px auto; overflow: hidden; padding: 0px; text-decoration: none; text-indent: -9999em; vertical-align: baseline; width: 41px;" title="This answer is not useful (click again to undo)">down vote</a></div>
</td><td class="answercell" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><div class="post-text" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13.5px; line-height: 17.5px; margin: 0px 5px 5px 0px; padding: 0px; vertical-align: baseline; width: 660px;">
<div style="background-color: transparent; border: 0px; clear: both; font-size: 13.5px; margin-bottom: 1em; padding: 0px; vertical-align: baseline; word-wrap: break-word;">
Puppet is used for deployments in many large organizations, but it's not always perfect. Much of it depends on your deployment methodology. Are you deploying to lots of machines at once? Do you do rolling deployments?</div>
<div style="background-color: transparent; border: 0px; clear: both; font-size: 13.5px; margin-bottom: 1em; padding: 0px; vertical-align: baseline; word-wrap: break-word;">
Some organizations use Puppet by building packages of their deployments and then having puppet enforce policy to be at the right version of that package. Because puppet has the concept of environments included, you can use environments to do deployments in stages (dev, test, prod for example).</div>
<div style="background-color: transparent; border: 0px; clear: both; font-size: 13.5px; margin-bottom: 1em; padding: 0px; vertical-align: baseline; word-wrap: break-word;">
Other organizations use puppet to orchestrate deployment by either firing off an rsync, git checkout or some recursively file copies using puppet (though that is rather slow).</div>
<div style="background-color: transparent; border: 0px; clear: both; font-size: 13.5px; margin-bottom: 1em; padding: 0px; vertical-align: baseline; word-wrap: break-word;">
There are other pretty good tools available for deployment too. I have used Whiskey Disk in the past (a simple ruby tool) and liked it a lot.</div>
<div style="background-color: transparent; border: 0px; clear: both; font-size: 13.5px; margin-bottom: 1em; padding: 0px; vertical-align: baseline; word-wrap: break-word;">
(Disclaimer, I work at Puppet Labs)</div>
</div>
<table class="fw" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-collapse: collapse; border-spacing: 0px; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline; width: 665px;"><tbody style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<tr style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><td class="vt" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: top;"><div class="post-menu" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 2px 0px 0px; vertical-align: baseline;">
<a class="short-link" href="http://serverfault.com/a/372603" id="link-post-372603" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #888888; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px 3px 2px; text-decoration: none; vertical-align: baseline;" title="short permalink to this answer">share</a><span class="lsep" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #cccccc; font-size: 14px; margin: 0px; padding: 0px; vertical-align: baseline;">|</span><a class="suggest-edit-post" href="http://serverfault.com/posts/372603/edit" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #888888; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px 3px 2px; text-decoration: none; vertical-align: baseline;" title="">improve this answer</a></div>
</td><td align="right" class="post-signature" style="background-color: transparent; border: 0px; font-size: 13px; height: 58px; margin: 0px; padding: 2px 0px 0px 5px; vertical-align: top; width: 175px;"><div class="user-info " style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; height: 35px; margin: 0px; padding: 0px; vertical-align: baseline; width: 185px;">
<div class="user-action-time" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 2px 0px 4px; padding: 0px; vertical-align: baseline;">
answered <span class="relativetime" style="background-color: transparent; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" title="2012-03-23 00:25:47Z">Mar 23 '12 at 0:25</span></div>
<div class="user-gravatar32" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; float: left; font-size: 13px; height: 32px; margin: 0px; padding: 0px; vertical-align: baseline; width: 32px;">
<a href="http://serverfault.com/users/115057/stahnma" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"></a><br />
<div class="" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://serverfault.com/users/115057/stahnma" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"><img alt="" src="http://www.gravatar.com/avatar/fa147ffde386565a674bc13453dc6f8d?s=32&d=identicon&r=PG" height="32" style="background-color: transparent; border-width: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" width="32" /></a></div>
</div>
<div class="user-details" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #888888; float: left; font-size: 13px; line-height: 17px; margin: 0px 0px 0px 5px; overflow: hidden; padding: 0px; vertical-align: baseline; white-space: nowrap; width: 145px;">
<a href="http://serverfault.com/users/115057/stahnma" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">stahnma</a><br />
<span class="reputation-score" dir="ltr" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #444444; font-size: 13px; font-weight: bold; margin: 0px 2px 0px 0px; padding: 0px; vertical-align: baseline;" title="reputation score">41</span><span style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" title="2 bronze badges"><span class="badge3" style="background-color: transparent; background-image: url(https://cdn.sstatic.net/serverfault/img/sprites.png?v=6); background-position: 0px -1034px; background-repeat: no-repeat no-repeat; border: 0px; display: inline-block; font-size: 13px; height: 11px; line-height: inherit; margin: 0px 1px 1px 3px; overflow: hidden; padding: 0px; vertical-align: text-bottom; width: 6px;"></span><span class="badgecount" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #808185; font-size: 13px; margin: 0px; padding: 0px 0px 0px 1px; vertical-align: baseline;">2</span></span></div>
</div>
</td></tr>
</tbody></table>
</td></tr>
<tr style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><td class="votecell" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: top; width: 60px;"></td><td style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><div class="comments" id="comments-372603" style="-webkit-tap-highlight-color: rgba(255, 255, 255, 0); background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-top-color: rgb(170, 170, 170); border-top-style: dotted; border-width: 1px 0px 0px; color: #444444; font-size: 13px; line-height: 17.5px; margin: 10px 0px 0px; padding: 0px 0px 10px; vertical-align: baseline; width: 660px;">
<table style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-collapse: collapse; border-spacing: 0px; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline; width: 660px;"><tbody style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<tr class="comment" id="comment-380050" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><td style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-color: rgb(221, 221, 221); border-bottom-style: dotted; border-width: 0px 0px 1px; font-size: 13px; margin: 0px; padding: 5px 0px; vertical-align: baseline;"></td><td class="comment-text" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-color: rgb(221, 221, 221); border-bottom-style: dotted; border-width: 0px 0px 1px; font-size: 13px; margin: 0px; padding: 5px 6px 5px 7px; vertical-align: text-top;"><div style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span class="comment-copy" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">Thanks. Deployments are to two balanced servers, currently using manual rsync.</span> – <a class="comment-user owner" href="http://serverfault.com/users/87611/michaeltwofish" style="background-color: #f3f1d9; background-position: initial initial; background-repeat: initial initial; border: 0px; color: #456583; cursor: pointer; font-size: 13px; margin: 0px; padding: 2px 5px; text-decoration: none; vertical-align: baseline; white-space: nowrap;" title="119 reputation">michaeltwofish</a> <span class="comment-date" dir="ltr" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-style: none; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; color: #999999; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;"><a class="comment-link" href="http://serverfault.com/questions/372575/application-deployment-with-puppet#comment380050_372603" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border-bottom-style: none; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; color: #999999; cursor: pointer; font-size: 13px; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"><span class="relativetime-clean" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: 0px; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" title="2012-03-23 00:52:21Z">Mar 23 '12 at 0:52</span></a></span></div>
</td></tr>
</tbody></table>
</div>
</td></tr>
</tbody></table>
</div>
</div>
</div>
</div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<br /></div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<b><br /></b></div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<b><br /></b></div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<b><br /></b></div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<br /></div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<b><br /></b></div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<b><br /></b></div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<b><br /></b></div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<br /></div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<b>__________________________________________________________</b></div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<b><br /></b></div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<b><br /></b></div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<b><br /></b></div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<b>Issues with MCollective as Deployment Orchestrator</b></div>
<span style="font-family: verdana, helvetica, sans-serif; font-size: x-small;"><a href="https://groups.google.com/forum/#!msg/mcollective-users/624ZMIOcNf0/0nHwcZ2FVqgJ">https://groups.google.com/forum/#!msg/mcollective-users/624ZMIOcNf0/0nHwcZ2FVqgJ</a></span><br />
<span style="font-family: verdana, helvetica, sans-serif; font-size: x-small;"><br /></span>
<br />
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<br /></div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<br /></div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<br /></div>
<div style="color: black; font-family: verdana, helvetica, sans-serif; font-size: 10pt;">
<br /></div>
</div>
Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-21629958805772213962013-03-25T04:56:00.000-07:002013-03-25T04:56:23.750-07:00Commit-Deploy-Test-Rollback Flowchart from IMVU<a href="http://www.slideshare.net/bgdurrett/learning-fast-with-ab-testing-and-continuous-deployment">Learning Fast With A/B Testing and Continuous Deployment</a><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-xH8MJ5UyXb0/UVA7PnHTKxI/AAAAAAAAAGw/vc8HmMVsDFk/s1600/2013-03-25_21-53-16_image.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://4.bp.blogspot.com/-xH8MJ5UyXb0/UVA7PnHTKxI/AAAAAAAAAGw/vc8HmMVsDFk/s320/2013-03-25_21-53-16_image.png" width="320" /></a></div>
<br />Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-36995232262708669892013-03-25T04:06:00.003-07:002013-03-25T04:09:09.374-07:00Larry Wall: 5 Programming Languages Everyone Should Know <br />
Video: <a href="https://www.youtube.com/watch?v=LR8fQiskYII">https://www.youtube.com/watch?v=LR8fQiskYII</a><br />
<br />
Back then it would have been: Fortran, COBOL, Basic, LISP, APL.<br />
<br />
Now...<br />
<br />
<b>Javascript</b><br />
<br />
<ul>
<li>If only to know if you should click the button that says to Enable Javascript</li>
</ul>
<br />
<br />
<b>Java</b><br />
<br />
<ul>
<li>The elephant in the room</li>
<li>The COBOL of the 21st century</li>
<li>heavyweight, verbose, everyone loves to hate it</li>
<li>Managers love it because it looks like you're getting a lot done... </li>
<ul>
<li>100 lines of Java vs 5 lines in another language</li>
<li>You can eat a 1 pound steak or 100 pounds of shoe leather - and feel a greater sense of accomplishment after the shoe leather but... maybe there's some downsides</li>
</ul>
<li>Considered industrial, programmers are considered interchangeable... parts.</li>
<ul>
<li>Managers like it for that reason and for that reason a lot of Java jobs have been outsourced</li>
</ul>
</ul>
<br />
<b>Haskell</b><br />
<br />
<ul>
<li>Functional... not as in other languages are dysfunctional</li>
<li>A language for geniuses by geniuses</li>
<li>Should know about it if only to be able to say "is this kind of like Haskell?" if so, you know you need to hire some really smart people to program in it</li>
<ul>
<li>A modern LISP in that sense</li>
</ul>
</ul>
<div>
<b>C</b></div>
<div>
<ul>
<li>Continues to be a fundamental language - if only because everyone is trying to reinvent it but not succeeding</li>
</ul>
<div>
<b>Scripting language - Python, Ruby, PERL</b></div>
</div>
<div>
<ul>
<li>Liveliest community</li>
<li>Redesigning to leapfrog other languages, removing warts</li>
<li>One chance to breaking backward compatibility, break things that need breaking, keep it a joy to use, useful/enjoyable for decades</li>
<li>I'd recommend PERL - but I'm known to be prejudiced in the matter</li>
</ul>
<div>
<br /></div>
</div>
Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.comtag:blogger.com,1999:blog-6287813703760221728.post-76579607725169287542013-02-11T14:16:00.002-08:002013-02-11T14:16:33.069-08:00Installing Python 2.7, Pip, Fabric on RedHat 5.6RedHat 5.6 comes with Python 2.4, Fabric requires Python 2.5+<br />
<br />
<b>Test Box Setup: </b><br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">vagrant box add centos5.6 http://dl.dropbox.com/u/15307300/vagrant-0.7-centos-64-base.box
</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">vagrant init centos5.6</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">vagrant up</span><br />
<br />
<b>Install ActiveState Python 2.7</b><br />
<span style="font-family: Courier New, Courier, monospace;">wget http://downloads.activestate.com/ActivePython/releases/2.7.2.5/ActivePython-2.7.2.5-linux-x86_64.tar.gz</span><br />
<span style="font-family: Courier New, Courier, monospace;">tar zxvf ActivePython-2.7.2.5-linux-x86_64.tar.gz </span><br />
<br />
<div>
<span style="font-family: Courier New, Courier, monospace;">rm ActivePython-2.7.2.5-linux-x86_64.tar.gz </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">cd ActivePython-2.7.2.5-linux-x86_64/</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">./install.sh </span></div>
<div>
<br /></div>
<div>
<b>Install Fabric in a virtualenv</b></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">cd ~</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">mkdir venvs</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">cd venvs</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">curl -kO https://raw.github.com/pypa/virtualenv/master/virtualenv.py</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">python virtualenv.py fab-env</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">vi ~/.bashrc</span></div>
<div>
<span style="font-family: inherit;">After "export PATH=/home/jenkins/ActivePython-2.7/bin:$PATH" insert:</span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">if [ -f ~/venvs/fab-env/bin/activate ]; then</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> source ~/venvs/fab-env/bin/activate</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">fi</span></div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace;">fab</span></div>
<div>
<br /></div>
<div>
<br /></div>
<br />
<br />
<b>Other Failed Attempt:</b><br />
<br />
<b><u>EPEL</u></b><br />
<b><br /></b>
<b>Install Python 2.6</b><br />
<br style="font-family: monospace; font-size: 12.666666984558105px;" />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;"><b>Install EPEL:</b></span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;">wget http://download.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;">sudo rpm -Uvp epel-release-5-4.noarch.rpm</span></span><br />
<br />
<div style="font-size: 12.666666984558105px;">
<span style="background-color: white; color: #333333; font-family: メイリオ, 'MS Pゴシック', osaka, arial, helvetica, sans-serif; font-size: 15.454545021057129px;"><b>Fix yum repo:</b></span><br />
<span style="background-color: white; color: #333333; font-family: メイリオ, 'MS Pゴシック', osaka, arial, helvetica, sans-serif; font-size: 15.454545021057129px;">sudo vi /etc/yum.repos.d/elff.repo</span></div>
<span style="color: #333333; font-family: メイリオ, MS Pゴシック, osaka, arial, helvetica, sans-serif;"><span style="font-size: 15.333333015441895px;">search for:</span></span><br />
baseurl=http://download.elff.bravenet.com/5/$basearch<br />
replace with:<br />
baseurl=http://rbel.frameos.org/stable/el5/$basearch<br />
<br />
<span style="font-family: monospace; font-size: 12.666666984558105px;">sudo yum </span><span style="background-color: #edeff1; font-size: 11.199999809265137px;">--nogpgcheck </span><span style="font-family: monospace; font-size: 12.666666984558105px;">install -y python26</span><br />
<span style="font-family: monospace; font-size: 12.666666984558105px;">sudo yum install -y gcc make python26-devel</span><br />
<span style="font-family: monospace; font-size: 12.666666984558105px;">mkdir ~/venvs</span><br />
<span style="font-family: monospace; font-size: 12.666666984558105px;">cd venvs</span><br />
<span style="font-family: monospace; font-size: 12.666666984558105px;">curl -kO </span><a href="https://webmail.wotifgroup.com/owa/redir.aspx?C=67ee40f3fea0442998fdfb6cf475cac3&URL=https%3a%2f%2fraw.github.com%2fpypa%2fvirtualenv%2fmaster%2fvirtualenv.py" style="font-family: monospace; font-size: 12.666666984558105px;" target="_blank">https://raw.github.com/pypa/virtualenv/master/virtualenv.py</a><br />
<span style="font-family: monospace; font-size: 12.666666984558105px;">python2.6 virtualenv.py fab</span><br />
<span style="font-family: monospace; font-size: 12.666666984558105px;">source fab/bin/activate</span><br />
<span style="font-family: monospace; font-size: 12.666666984558105px;">pip install fabric</span><br />
<span style="font-family: monospace; font-size: 12.666666984558105px;">echo "source venvs/fab/bin/activate" >> ~/.bashrc</span><br />
<span style="font-family: monospace; font-size: 12.666666984558105px;"><br /></span>
<span style="font-family: monospace; font-size: 12.666666984558105px;">This approach worked fine in Centos but not RedHat...</span><br />
<span style="font-family: monospace; font-size: 12.666666984558105px;"><b>Problem with this approach:</b></span><br />
<span style="font-family: monospace; font-size: 12.666666984558105px;">"pip install fabric" requires a newer version of Python (Python 2.4 currently installed in RedHat 5.6): ".. if .. else .." syntax does not work</span><br />
<span style="font-family: monospace; font-size: 12.666666984558105px;"><br /></span>
<span style="font-family: monospace;"></span><br />
<span style="font-family: monospace;">~ $ pip install fabric</span><br />
<span style="font-family: monospace;">Downloading/unpacking fabric</span><br />
<span style="font-family: monospace;"> Downloading Fabric-1.5.3.tar.gz (241Kb): 241Kb downloaded</span><br />
<span style="font-family: monospace;"> Running setup.py egg_info for package fabric</span><br />
<span style="font-family: monospace;"> Traceback (most recent call last):</span><br />
<span style="font-family: monospace;"> File "<string>", line 14, in ?</span><br />
<span style="font-family: monospace;"> File "/home/jenkins/build/fabric/setup.py", line 7, in ?</span><br />
<span style="font-family: monospace;"> from fabric.version import get_version</span><br />
<span style="font-family: monospace;"> File "fabric/version.py", line 57</span><br />
<span style="font-family: monospace;"> sha1 = (" (%s)" % sha) if sha else ""</span><br />
<span style="font-family: monospace;"> ^</span><br />
<span style="font-family: monospace;"> SyntaxError: invalid syntax</span><br />
<span style="font-family: monospace;"> Complete output from command python setup.py egg_info:</span><br />
<span style="font-family: monospace;"> Traceback (most recent call last):</span><br />
<span style="font-family: monospace;"><br /></span>
<span style="font-family: monospace;"> File "<string>", line 14, in ?</span><br />
<span style="font-family: monospace;"><br /></span>
<span style="font-family: monospace;"> File "/home/jenkins/build/fabric/setup.py", line 7, in ?</span><br />
<span style="font-family: monospace;"><br /></span>
<span style="font-family: monospace;"> from fabric.version import get_version</span><br />
<span style="font-family: monospace;"><br /></span>
<span style="font-family: monospace;"> File "fabric/version.py", line 57</span><br />
<span style="font-family: monospace;"><br /></span>
<span style="font-family: monospace;"> sha1 = (" (%s)" % sha) if sha else ""</span><br />
<span style="font-family: monospace;"> ^</span><br />
<span style="font-family: monospace;">SyntaxError: invalid syntax</span><br />
<br />
<span style="font-family: monospace; font-size: 12.666666984558105px;"><br /></span>
<span style="font-family: monospace; font-size: 12.666666984558105px;"><br /></span>
<span style="font-family: monospace; font-size: 12.666666984558105px;"><br /></span>
<span style="font-family: monospace; font-size: 12.666666984558105px;">Refs:</span><br />
<br />
<ul>
<li><span style="font-family: monospace; font-size: 12.666666984558105px;"><a href="http://stackoverflow.com/questions/1465036/install-python-2-6-in-centos">http://stackoverflow.com/questions/1465036/install-python-2-6-in-centos</a></span></li>
<li><span style="font-family: monospace; font-size: 12.666666984558105px;"><a href="http://fedoraproject.org/wiki/EPEL/FAQ#howtouse">http://fedoraproject.org/wiki/EPEL/FAQ#howtouse</a></span></li>
</ul>
<br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;"><br /></span></span>
<br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;"><br /></span></span>
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;"><br /></span></span>
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;"><br /></span></span>
<span style="font-family: monospace; font-size: 12.666666984558105px;"><br /></span>
<span style="font-family: monospace;"></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;">[vagrant@vagrant-centos-64 ~]$ python</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;">Python 2.4.3 (#1, Nov 11 2010, 13:30:19)</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;">[GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;">Type "help", "copyright", "credits" or "license" for more information.</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;">>>></span></span><br />
<div style="font-size: 12.666666984558105px;">
<span style="font-family: monospace;"><br /></span></div>
<div style="font-size: 12.666666984558105px;">
<span style="font-family: monospace;"><br /></span></div>
<span style="font-family: monospace;"></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;"><br /></span></span>
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;">[vagrant@vagrant-centos-64 ~]$ sudo yum install python26</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;">Loaded plugins: fastestmirror</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;">Loading mirror speeds from cached hostfile</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;"> * addons: mirror.aarnet.edu.au</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;"> * base: mirror.aarnet.edu.au</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;"> * epel: mirror.overthewire.com.au</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;"> * extras: mirror.aarnet.edu.au</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;"> * updates: mirror.aarnet.edu.au</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;">http://download.elff.bravenet.com/5/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404: Not Found</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;">Trying other mirror.</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;">Error: Cannot retrieve repository metadata (repomd.xml) for repository: elff. Please verify its path and try again</span></span><br />
<div>
<div style="font-size: 12.666666984558105px;">
<span style="font-family: monospace;"><br /></span></div>
<div style="font-size: 12.666666984558105px;">
<span style="font-family: monospace;"><br /></span></div>
<span style="font-family: monospace;"></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;">[vagrant@vagrant-centos-64 venvs]$ python2.6 virtualenv.py fab</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;">New python executable in fab/bin/python2.6</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;">Also creating executable in fab/bin/python</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;">Installing setuptools............................done.</span></span><br />
<span style="font-family: monospace;"><span style="font-size: 12.666666984558105px;">Installing pip.....................done.</span></span><br />
<div style="font-size: 12.666666984558105px;">
<span style="font-family: monospace;"><br /></span></div>
<div style="font-size: 12.666666984558105px;">
<span style="font-family: monospace;"><br /></span></div>
</div>
Matt Callananhttp://www.blogger.com/profile/11078559917985248989noreply@blogger.com