Saturday, March 31, 2012

AWS Cloudformation with mcollective

  • Create security group ("mcollective") with following ports open
    • 22, 6163, 80
  • Launch new EC2 instance in eu-west-1a
    • Search community images for "mcollective" choose only result (mcollective-120)
    • Add user values of 
      • mcollective=server
      • role=server
  • Login as root
  • start-mcollective-demo.rb 
    • Copy the following block of text output:
      • mcollective=
  • mc-ping
  • Launch 5 more ec2 instances in eu-west-1a by right-clicking existing image and "Launch more like this"
    • Number of instances = 5
    • Paste the block of text copied above into the "User Data" field
  •  Do the same again (another 5 servers) in eu-west-1b
  • Back to main server console
    • mc-ping --timeout 1   # should see all 11 servers
    • facter | grep ec2
    • mc-facts --timeout 1 ec2_placement_availability_zone
      • Report for fact: ec2_placement_availability_zone

                eu-west-1a                              found 6 times
                eu-west-1b                              found 5 times

        Finished processing 11 / 11 hosts in 580.78 ms
    • mc-package --with-fact ec2_placement_availability_zone=eu-west-1a status httpd
    • mc-package --with-fact ec2_placement_availability_zone=eu-west-1a install httpd
    • mc-service httpd status
    • mc-service --with-fact ec2_placement_availability_zone=eu-west-1a httpd start
    • (You'd use Puppet to deploy the app, but upgrade packages using mcollective)
    •  mc-nrpe check_load
    • mc-package --with-fact ec2_placement_availability_zone=eu-west-1b install httpd
    • mc-service --with-fact ec2_placement_availability_zone=eu-west-1b httpd restart
    • mc-service httpd status
    • facter
    • (Can provide your own data/plugins)

Old instructions:

  • facter
  • mc-ping
  • mc-inventory ip-10-239-131-241
  • mc-facts ec2_ami_manifest_path
  • mc-inventory ip-10-239-131-241 | grep ec2
  • mc-facts ec2_placement_availability_zone
  • mc-facts cluster
  • mc-facts cluster -v
  • mc-find-hosts -W cluster=alfa
  • mco service httpd status
  • mco service -W cluster=alfa httpd status
  • mco package install httpd
  • mco service httpd restart
  • mc-urltest

Patrick Debois Interview with Tom Parish

Tom Parish: What is DevOps?

Patrick Debois:

  • The multi-million dollar question everybody keeps on asking me.  
  • There is not one definition, and I doubt if there will ever be one.  
  • It can mean different things to different people.  
  • I can tell you about how I came to coin the term, what it means to me and how other people have begun to see it.
  • Before I came up with the term, I was very involved with projects that used Agile Development methodology. 
  • I was supporting them from a system administration point of view, and I felt a bit left out of their group. 
  • I wanted to start doing the same things and help them from my perspective.  
  • I first named it Agile System Administration, and then I thought it was more like a system administrator doing Agile, while development was doing Agile – we needed development and operations to work together.
  • I saw the term DevOps as collaboration, but in reality, this is a narrow definition.  
  • It's not just about development and operations collaborating, it's getting every silo, every part of the business/enterprise/organization collaborating to meet business goals.  
  • The term is short and is catchy; making it longer, like Dev-Test-Security-Ops-Business just didn’t make sense, so DevOps just started getting around.

Tom Parish:  You mentioned automation and tooling – can we refine this a little? Is DevOps, primarily an automation or a tooling issue?

Patrick Debois: It's not primarily automation; it's culture, as I said before.  To support that culture you use automation. If you take the automation step, you then can get workflow feedback from IT to code-check to development, to test, to production.  Automation makes that a very fast loop.   From the same perspective it could be also about other feedback and monitoring for testing stuff.

You need the philosophy, both at the people and the process level so that people collaborate. But if your technology doesn't support you to be flexible, you can have every good intention, but your feedback loop will be very long, or you don't see anything in that feedback loop. 

Tom Parish:  What's the role of change management in DevOps?

Patrick Debois: Traditionally, change management was a term within ITIL or CRMI, for approving things that would go into production.  What happens a lot of the time is that people were avoiding change, because they knew change was one of the primary causes of outages.

            Change management has always been meant that you have to make sure that nothing breaks, that you understand the risk and that you know that you can recover. With automation, a lot of the changes that we used to think were very scary become simple and safe.  It allows us to focus our time on the larger and the scary changes. Then the small items don’t have to pass through the change management cycle anymore.

Tom Parish:  I have a question I've heard from many managers in IT.  If all developers were to sysadmins and all sysadmins were developers, then would DevOps need to even exist?

Patrick Debois: I think the problems we are facing are just becoming more complex. To solve these problems, we need knowledge and skills that can't be found in one person.

            If you make everyone a generalist,  you just get bits of mediocre knowledge of development and system management. Instead, find people that are specialists because they can make a difference, but make them collaborate with other parts of the company.

The more knowledge you have from your collaborator, the  easier it is to translate your concepts, your ideas and what you need to do. But it's not about making everybody having the same skills.  Agile will be the same within their group.  There are people still doing tests, DBA, front-end, backend, and they were all specialists, but they're worried about the global code set.

You have all these people with different specialties collaborating, but they care about the whole delivery and production.  So I don't think everybody has to have every knowledge or skill.  That's not what I think DevOps is meant to be.

Tom Parish:  How important is monitoring for DevOps?

Patrick Debois: A lot of the initial focus of technical discussions in DevOps was about automation, and clearly, it's about how fast can we get things into production.  This makes sense and it’s a crucial part. But once things are in production, you will want to know how they're doing.  Do they make sense?  Are they valuable to the customer?  What's happening there? Are things breaking?  You need automation that provides reporting feedback from ops to dev, so that everyone knows what is happening.    How can we learn from what is happening there? 

From a business perspective and from a development perspective, how can we learn from all the feedback we are getting from customers?  It's a major part of this – you need to receive feedback once it's there.

Tom Parish:  Yes. It just has to be there as part of  the feedback mechanism.

Patrick Debois: Traditionally, but they are different tools; monitoring used to be kept within the operational site.

Tom Parish:  Oh, yes. I see what you're saying.

Patrick Debois: More and more people are beginning to relate that to their development and business groups, so they can learn from that as well.  It is a strange thing - why are we using different tools for debugging and testing things in our test labs environment?  Why are we not using the same monitoring tools to understand what's going on there?

These tools will start aligning too, but in the whole cycle we are using and viewing what's out there.  Why are developers not writing monitoring scripts, so they can see issues while they're writing the application as well as once it's in production you can see what's happening there.  And from what I've seen, developers actually love to see how their baby runs.  It just makes sense.

Tom Parish:  You spend hours in development, not just by yourself but with your whole team, and part of the satisfaction is to see how well it's performing. I think it's a really good way of working.  It's kind of odd that it hasn't happened sooner.

Patrick Debois: There's a kind of metaphor I use  – relating development to raising kids.  Just before the teenage years, the kids are sweet and nothing is happening.  But when you're putting things into production, it starts behaving differently, strangely, like a teenager.

            You don't know what's happening.

Then after a few years, the legacy code starts growing old, so in fact, they need to care about it  from inception tools through legacy,  not just when everything is sweet and performing well.

Tom Parish: In other industries, they have seen essential transformation of product supply chain activities, by focusing on build-to-manufacturing practices.  So is this transformation similar to how DevOps is transforming IT?

Patrick Debois: We have a discussion about how this - if manufacturing is lean, it is sort of similar to this way of thinking, but within manufacturing, you try to minimize change as well.  Within Ops, we do try to minimize the change in a way and we automate, trying to we make everything repeatable, but it's more about freeing up time to work out the most complex problems.

We are not doing this to make people redundant, but to free up those valuable resources so we can do more and more fabulous, awesome things. It's a strange thing that a lot of people drive automation just to reduce the number of people who manage servers, but in fact, all the time that gets freed up enables them to do things that actually differentiate them from their competitors.

It's not just about minimizing the number of people or automating things like a factory. That's why I don't like the metaphor of the factory, because it implies that we want to remove the human factor.  And I believe in the opposite.

Tom Parish:  We were talking about DevOps and how it needs to be a part of everything from the software development cycle all the way to release, but how far back into the product life cycle should DevOps go, or be involved?

Patrick Debois: If somebody has an idea,  you start discussing it determining whether it's a good idea or even feasible.   I guess that's not a good time to actually get engineering involved, because they are focused on getting it done, so you lose the creativity.  Once you get going on a project, it makes sense to have them involved, in part to have a reality check. 

From that moment on, it makes sense to get all these people aboard, and see what the impacts would be across their different specialty.  Don’t wait until the product is partly developed and then come in. You can start immediately at the project phase when that kicks off...

You can do it sooner, but like I said, the risk is they will just stop imagining.   You have to pick the right moment.

Tom Parish:  Hmm, interesting.  Let's say a group is definitely into the DevOps concept and moving in that direction. Can you provide some insights into DevOps success metrics for developers, operations and business people?

Patrick Debois: Ultimately, it comes to having people work together in way that helps to  make that business a success.  It’s hard to define a metric for that. Traditionally, the metrics have been winning for each silo. There would be things like the number of pointer codes, the number of services you manage, but what they don't take into account is how you measure the collaboration aspect between the different groups, and whether you're actually doing something useful.

If you look at more flow-based metrics, how long does it take to get something from development to production?  You can't take that alone as a metric, because you can do a lot of shitty things and push them out very fast, but you also have to look at how your flow rate is reduced doing that stuff.

I don't think metrics make sense if they are measured by group or by individual.  You have to look at the  flow from the beginning to the end.  Is it optimized? I If you reduce that time, you will notice that it means you are resolving collaboration conflicts you had in the past. 

Did we actually tackle that problem?  The metrics that exist within typical Scrum or Agile management may have some use but you have to extend them so that it makes sense when you go through the whole cycle.

Tom Parish:  By the way, do you need to do this internally in each group, as well?  You figure out what your metrics are, but then, you keep your eye on more than their metric.  You need to look at whether what you're doing is adding or subtracting to the end goal?

Patrick Debois: Yes.  I made the analogy with managing servers.  How do you know that your business is up?  If you ask your system administrator, he will tell you things like how the CPU, memory and disk areperforming.  But that doesn’t matter.  Your developer will say, "Oh, my code works.”   But the only monitoring that actually makes sense is from a business perspective.

Can a user log in? Can you accept this page?  But if you measure that and yet, something fails, what you need is the lower level metrics and monitoring to be able to understand where it's failing and why it's failing.  And you won't get that from a business perspective, obviously.  You still need to monitor from the business perspective, but you also need  the lower level metrics  to see where things are working or failing.

  • Within Ops, we do try to minimize the change in a way and we automate, trying to we make everything repeatable, but it's more about freeing up time to work out the most complex problems.
  • We are not doing this to make people redundant, but to free up those valuable resources so we can do more and more fabulous, awesome things... all the time that gets freed up enables them to do things that actually differentiate them from their competitors.

Tom Parish:  Patrick, last question.  DevOps seems to be at risk from suffering from an overload of media and analyst noise, so do you think this is a danger to the organic growth and the adoption of DevOps?

Patrick Debois: Obviously you have watched it growing at an increasing rate, But some companies are just taking that term, re-branding their processes or repositioning existing products.

Will it be a danger?  

The difference is that this doesn't just start from an institute or a company success - let's coin the term and drive it through.  It has to grow from a bottom-up approach, which takes longer.  If a vendor uses that term and he doesn't deliver, each person has to evaluate whether that was a good thing or not.