Continuous Delivery in the Cloud: Dynamic Configuration (Part 4 of 6)

In part 1 of this series, I introduced the Continuous Delivery (CD) pipeline for the Manatee Tracking application. In part 2 I went over how we use this CD pipeline to deliver software from checkin to production. In part 3, we focused on how CloudFormation is used to script the virtual AWS components that create the Manatee infrastructure. A list of topics for each of the articles is summarized below:

Part 1: Introduction – Introduction to continuous delivery in the cloud and the rest of the articles;
Part 2: CD Pipeline – In-depth look at the CD Pipeline;
Part 3: CloudFormation – Scripted virtual resource provisioning;
Part 4: Dynamic Configuration –  What you’re reading now;
Part 5: Deployment Automation – Scripted deployment orchestration;
Part 6: Infrastructure Automation – Scripted environment provisioning (Infrastructure Automation)

In this part of the series, I am going to explain how we dynamically generate our configuration and avoid property files whenever possible. Instead of using property files, we store and retrieve configuration on the fly – as part of the CD pipeline – without predefining these values in a static file (i.e. a properties file) ahead of time. We do this using two methods: AWS SimpleDB and CloudFormation.

SimpleDB is a highly available non-relational data storage service that only stores strings in key value pairs. CloudFormation, as discussed in Part 3 of the series, is a scripting language for allocating and configuring AWS virtual resources.

Using SimpleDB

Throughout the CD pipeline, we often need to manage state across multiple Jenkins jobs. To do this, we use SimpleDB. As the pipeline executes, values that will be needed by subsequent jobs get stored in SimpleDB as properties. When the properties are needed we use a simple Ruby script script to return the key/value pair from SimpleDB and then use it as part of the job. The values being stored and retrieved range from IP addresses and domain names to AMI (Machine Images) IDs.

So what makes this dynamic? As Jenkins jobs or CloudFormation templates are run, we often end up with properties that need to be used elsewhere. Instead of hard coding all of the values to be used in a property file, we create, store and retrieve them as the pipeline executes.

Below is the CreateTargetEnvironment Jenkins job script that creates a new target environment from a CloudFormation script production.template

if [ $deployToProduction ] == true

# Create Cloudformaton Stack
ruby /usr/share/tomcat6/scripts/aws/create_stack.rb ${STACK_NAME} ${WORKSPACE}/production.template ${HOST} ${JENKINSIP} ${SSH_KEY} ${SGID} ${SNS_TOPIC}

# Load SimpleDB Domain with Key/Value Pairs
ruby /usr/share/tomcat6/scripts/aws/load_domain.rb ${STACK_NAME}

# Pull and store variables from SimpleDB
host=`ruby /usr/share/tomcat6/scripts/aws/showback_domain.rb ${STACK_NAME} InstanceIPAddress`

# Run Acceptance Tests
cucumber features/production.feature host=${host} user=ec2-user key=/usr/share/tomcat6/.ssh/id_rsa

Referenced above in the CreateTargetEnvironment code snippet. This is the load_domain.rb script that iterates over a file and sends key/value pairs to SimpleDB.

require 'rubygems'
require 'aws-sdk'
load File.expand_path('../../config/aws.config', __FILE__)


file = File.open("/tmp/properties", "r")

sdb = AWS::SimpleDB.new

AWS::SimpleDB.consistent_reads do
  domain = sdb.domains["stacks"]
  item = domain.items["#{stackname}"]

  file.each_line do|line|
    key,value = line.split '='
      "#{key}" => "#{value}")

Referenced above in the CreateTargetEnvironment code snippet. This is the showback_domain.rb script which connects to SimpleDB and returns a key/value pair.

load File.expand_path('../../config/aws.config', __FILE__)


sdb = AWS::SimpleDB.new

AWS::SimpleDB.consistent_reads do
  domain = sdb.domains["stacks"]
  item = domain.items["#{item_name}"]

  item.attributes.each_value do |name, value|
    if name == "#{key}"
      puts "#{value}".chomp

In the above in the CreateTargetEnvironment code snippet, we store the outputs of the CloudFormation stack in a temporary file. We then iterate over the file with the load_domain.rb script and store the key/value pairs in SimpleDB.

Following this, we make a call to SimpleDB with the showback_domain.rb script and return the instance IP address (created in the CloudFormation template) and store it in the host variable. host is then used by cucumber to ssh into the target instance and run the acceptance tests.

Using CloudFormation

In our CloudFormation templates we allocate multiple AWS resources. Every time we run the template, a different resource is being used. For example, in our jenkins.template we create a new IAM user. Every time we run the template a different IAM user with different credentials is created. We need a way to reference these resources. This is where CloudFormation comes in. You can reference resources within other resources throughout the script. You can define a reference to another resource using the Ref function in CloudFormation. Using Ref, you can dynamically refer to values of other resources such as an IP Address, domain name, etc.

In the script we are creating an IAM user, referencing the IAM user to create AWS Access keys and then storing them in an environment variable.

"CfnUser" : {
  "Type" : "AWS::IAM::User",
  "Properties" : {
    "Path": "/",
    "Policies": [{
      "PolicyName": "root",
      "PolicyDocument": {

"HostKeys" : {
  "Type" : "AWS::IAM::AccessKey",
  "Properties" : {
    "UserName" : { "Ref": "CfnUser" }

"# Add AWS Credentials to Tomcat\n",
"echo \"AWS_ACCESS_KEY=", { "Ref" : "HostKeys" }, "\" >> /etc/sysconfig/tomcat6\n",
"echo \"AWS_SECRET_ACCESS_KEY=", {"Fn::GetAtt": ["HostKeys", "SecretAccessKey"]}, "\" >> /etc/sysconfig/tomcat6\n",

We can then use these access keys in other scripts by referencing the $AWS_ACCESS_KEY and $AWS_SECRET_ACCESS_KEY environment variables.

How is this different from typical configuration management?

Typically in many organizations, there’s a big property with hard coded key/value pairs that gets passed into the pipeline. The pipeline executes using the given parameters and cannot scale or change without a user modifying the property file. They are unable to scale or adapt because all of the properties are hard coded, if the property file hard codes the IP to an EC2 instance and it goes down for whatever reason, their pipeline doesn’t work until someone fixes the property file. There are more effective ways of doing this when using the cloud. The cloud is provides on-demand resources that will constantly be changing. These resources will have different IP addresses, domain names, etc associated with them every time.

With dynamic configuration, there are no property files, every property is generated as part of the pipeline.

With this dynamic approach, the pipeline values change with every run. As new cloud resources are allocated, the pipeline is able to adjust itself and automatically without the need for users to constantly modify property files. This leads to less time spent debugging those cumbersome property file management issues that plague most companies.

In the next part of our series – which is all about Deployment Automation – we’ll go through scripting and testing your deployment using industry-standard tools. In this next article, you’ll see how to orchestrate deployment sequences and configuration using Capistrano.


Continuous Delivery in the Cloud: CD Pipeline (Part 2 of 6)

In part 1 of this series, I introduced the Continuous Delivery (CD) pipeline for the Manatee Tracking application and how we use this pipeline to deliver software from checkin to production. In this article I will take an in-depth look at the CD pipeline. A list of topics for each of the articles is summarized below.

Part 1: Introduction – Introduction to continuous delivery in the cloud and the rest of the articles;
Part 2: CD Pipeline – What you’re reading now;
Part 3: CloudFormation – Scripted virtual resource provisioning;
Part 4: Dynamic Configuration – “Property file less” infrastructure;
Part 5: Deployment Automation – Scripted deployment orchestration;
Part 6: Infrastructure Automation – Scripted environment provisioning (Infrastructure Automation)

The CD pipeline consists of five Jenkins jobs. These jobs are configured to run one after the other. If any one of the jobs fail, the pipeline fails and that release candidate cannot be released to production. The five Jenkins jobs are listed below (further details of these jobs are provided later in the article).

  1. 1) A job that set the variables used throughout the pipeline (SetupVariables)
  2. 2) Build job (Build)
  3. 3) Production database update job (StoreLatestProductionData)
  4. 4) Target environment creation job (CreateTargetEnvironment)
  5. 5) A deployment job (DeployManateeApplication) which enables a one-click deployment into production.

We used Jenkins plugins to add additional features to the core Jenkins configuration. You can extend the standard Jenkins setup by using Jenkins plugins. A list of the plugins we use for the Sea to Shore Alliance Continuous Delivery configuration are listed below.

Grails: http://updates.jenkins-ci.org/download/plugins/grails/1.5/grails.hpi
Groovy: http://updates.jenkins-ci.org/download/plugins/groovy/1.12/groovy.hpi
Subversion: http://updates.jenkins-ci.org/download/plugins/subversion/1.40/subversion.hpi
Paramterized Trigger: http://updates.jenkins-ci.org/download/plugins/parameterized-trigger/2.15/parameterized-trigger.hpi
Copy Artifact: http://updates.jenkins-ci.org/download/plugins/copyartifact/1.21/copyartifact.hpi
Build Pipeline: http://updates.jenkins-ci.org/download/plugins/build-pipeline-plugin/1.2.3/build-pipeline-plugin.hpi
Ant: http://updates.jenkins-ci.org/download/plugins/ant/1.1/ant.hpi
S3: http://updates.jenkins-ci.org/download/plugins/s3/0.2.0/s3.hpi

The parameterized trigger, build pipeline and S3 plugins are used for moving the application through the pipeline jobs. The Ant, Groovy, and Grails plugins are used for running the build for the application code. Subversion for polling and checking out from version control.

Below, I describe each of the jobs that make up the CD pipeline in greater detail.

SetupVariables: Jenkins job used for entering in necessary property values which are propagated along the rest of the pipeline.

Parameter: STACK_NAME
Type: String
Where: Used in both CreateTargetEnvironment and DeployManateeApplication jobs
Purpose: Defines the CloudFormation Stack name and SimpleDB property domain associated with the CloudFormation stack.

Parameter: HOST
Type: String
Where: Used in both CreateTargetEnvironment and DeployManateeApplication jobs
Purpose: Defines the CNAME of the domain created in the CreateTargetEnvironment job. The DeployManateeApplication job uses it when it dynamically creates configuration files. For instance, in test.oneclickdeployment.com, test would be the HOST

Type: String
Where: Used in the StoreProductionData job
Purpose: Sets the production IP for the job so that it can SSH into the existing production environment and run a database script that exports the data and uploads it to S3.

Parameter: deployToProduction
Type: Boolean
Where: Used in both CreateTargetEnvironment and DeployManateeApplication jobs
Purpose: Determines whether to use the development or production SSH keypair.

In order for the parameters to propagate through the pipeline, we pass the current build parameters using the parametrized build trigger plugin

Build: Compiles the Manatee application’s Grails source code and creates a WAR file.

To do this, we utilize a Jenkins grails plugin and run grails targets such as compile and prod war. Next, we archive the grails migrations for use in the DeployManateeApplication job and then the job pushes the Manatee WAR up to S3 which is used as an artifact repository.

Lastly, using the trigger parametrized build plugin, we trigger the StoreProductionData job with the current build parameters.

StoreProductionData: This job performs a pg dump (PostgreSQL dump) of the production database and then stores it up in S3 for the environment creation job to use when building up the environment. Below is a snippet from this job.

ssh -i /usr/share/tomcat6/development.pem -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no ec2-user@${PRODUCTION_IP} ruby /home/ec2-user/database_update.rb

On the target environments created using the CD pipeline, a database script is stored. The script goes into the PostgreSQL database and runs a pg_dump. It then pushes the pg_dump SQL file to S3 to be used when creating the target environment.

After the SQL file is stored successfully, the CreateTargetEnvironment job is triggered.

CreateTargetEnvironment: Creates a new target environment using a CloudFormation template to create all the AWS resources and calls puppet to provision the environment itself from a base operating system to a fully working target environment ready for deployment. Below is a snippet from this job.

if [ $deployToProduction ]

# Create Cloudformaton Stack
ruby ${WORKSPACE}/config/aws/create_stack.rb ${STACK_NAME} ${WORKSPACE}/infrastructure/manatees/production.template ${HOST} ${JENKINSIP} ${SSH_KEY} ${SGID} ${SNS_TOPIC}

# Load SimpleDB Domain with Key/Value Pairs
ruby ${WORKSPACE}/config/aws/load_domain.rb ${STACK_NAME}

# Pull and store variables from SimpleDB
host=`ruby ${WORKSPACE}/config/aws/showback_domain.rb ${STACK_NAME} InstanceIPAddress`

# Run Acceptance Tests
cucumber ${WORKSPACE}/infrastructure/manatees/features/production.feature host=${host} user=ec2-user key=/usr/share/tomcat6/.ssh/id_rsa

# Publish notifications to SNS
sns-publish --topic-arn $SNS_TOPIC --subject "New Environment Ready" --message "Your new environment is ready. IP Address: $host. An example command to ssh into the box would be: ssh -i development.pem ec2-user@$host This instance was created by $JENKINS_DOMAIN" --aws-credential-file /usr/share/tomcat6/aws_access

Once the environment is created, a set of Cucumber tests is run to ensure it’s in the correct working state. If any test fails, the entire pipeline fails and the developer is notified something went wrong. Otherwise if it passes, the DeployManateeApplication job is kicked off and an AWS SNS email notification with information to access the new instance is sent to the developer.

DeployManateeApplication: Runs a Capistrano script which uses steps in order to coordinate the deployment. A snippet from this job is displayed below.

if [ !$deployToProduction ]


cap deploy:setup stack=${STACK_NAME} key=${SSH_KEY}

sed -i "s@manatee0@${HOST}@" ${WORKSPACE}/deployment/features/deployment.feature

host=`ruby ${WORKSPACE}/config/aws/showback_domain.rb ${STACK_NAME} InstanceIPAddress`
cucumber deployment/features/deployment.feature host=${host} user=ec2-user key=${SSH_KEY} artifact=

sns-publish --topic-arn $SNS_TOPIC --subject "Manatee Application Deployed" --message "Your Manatee Application has been deployed successfully. You can view it by going to http://$host/wildtracks This instance was deployed to by $JENKINS_DOMAIN" --aws-credential-file /usr/share/tomcat6/aws_access

This deployment job is the final piece of the delivery pipeline, it pulls together all of the pieces created in the previous jobs to successfully deliver working software.

During the deployment, the Capistrano script SSH’s into the target server, deploys the new war and updated configuration changes and restarts all services. Then the Cucumber tests are run to ensure the application is available and running successfully. Assuming the tests pass, an AWS SNS email gets dispatched to the developer with information on how to access their new development application

We use Jenkins as the orchestrator of the pipeline. Jenkins executes a set of scripts and passes around parameters as it runs each job. Because of the role Jenkins plays, we want to make sure it’s treated the same way as application – meaning versioning and testing all of our changes to the system. For example, if a developer modifies the create environment job configuration, we want to have the ability to revert back if necessary. Due to this requirement we version the Jenkins configuration. The jobs, plugins and main configuration. To do this, a script is executed each hour using cron.hourly that checks for new jobs or updated configuration and commits them up to version control.

The CD pipeline that we have built for the Manatee application enables any change in the application, infrastructure, database or configuration to move through to production seamlessly using automation. This allows any new features, security fixes, etc. to be fully tested as it gets delivered to production at the click of a button.

In the next part of our series – which is all about using CloudFormation – we’ll go through a CloudFormation template used to automate the creation of a Jenkins environment. In this next article, you’ll see how CloudFormation procures AWS resources and provisions our Jenkins CD Pipeline environment.


Continuous Delivery in the Cloud Case Study

A Case Study on using 100% Cloud-based Resources with Automated Software Delivery

We help – typically large – organizations create one-click software delivery systems so that they can deliver software in a more rapid, reliable and repeatable manner (AKA Continuous Delivery). The only way this works is when Development works with Operations. As has been written elsewhere in this series, this means changing the hearts and minds of people because most organizations are used to working in ‘siloed’ environments. In this entry, I focus on implementation, by describing a real-world case study in which we have brought Continuous Delivery Operations to the Cloud consisting of a team of Systems and Software Engineers.  

For years, we’ve helped customers in Continuous Integration and Testing so more of our work was with Developers and Testers. Several years ago, we hired a Sys Admin/Engineer/DBA who was passionate about automation. As a result of this, we began assembling multiple two-person “DevOps” teams consisting of a Software Engineer and a Systems Engineer both of whom being big-picture thinkers and not just “Developers” or “Sys Admins”. These days, we put together these targeted teams of Continuous Delivery and Cloud experts with hands-on experience as Software Engineers and Systems Engineers so that organizations can deliver software as quickly and as often as the business requires.

A couple of years ago we already had a few people in the company who were experimenting with using Cloud infrastructures so we thought this would be a great opportunity in providing cloud-based delivery solutions. In this case study, I cover a project we are currently working on for a large organization. It is a new Java-based web services project so we’ve been able to implement solutions using our recommended software delivery patterns rather than being constrained by legacy tools or decisions. However, as I note, we aren’t without constraints on this project. If I were you, I’d call “BS!” on any “case study” in which everything went flawlessly and assume it was an extremely small or a theoretical project in the author’s mind. This is the real deal. Enough said, on to the case study.      

AWS Tools

Fast Facts

Industry: Healthcare, Public Sector
Profile: The customer is making available to all, free of charge, a series of software specifications and open source software modules that together make up an oncology-extended Electronic Health Record capability.
Key Business Issues: The customer was seeking that all team members are provided “unencumbered” access to infrastructure resources without the usual “request and wait” queued-based procedures present in most organizations
Stakeholders: Over 100 people consisting of Developers, Testers, Analysts, Architects, and Project Management.
Solution: Continuous Delivery Operations in the Cloud
Key Tools/Technologies: Amazon Web Services  - AWS (Elastic Computer Cloud (EC2), (Simple Storage Service (S3), Elastic Block Storage (EBS), etc.), Jenkins, JIRA Studio, Ant, Ivy, Tomcat and PostgreSQL

The Business Problem
The customer was used to dealing with long drawn-out processes with Operations teams that lacked agility. They were accustomed to submitting Word documents via email to an Operations teams, attending multiple meetings and getting their environments setup weeks or months later. We were compelled to develop a solution that reduced or eliminated these problems that are all too common in many large organizations (Note: each problem is identified as a letter and number, for example: P1, and referred to later):

  1. Unable to deliver software to users on demand (P1)
  2. Queued requests for provisioned instances (P2)
  3. Unable to reprovision precise target environment configuration on demand (P3)
  4. Unable to provision instances on demand (P4)
  5. Configuration errors in target environments presenting deployment bottlenecks while Operations and Development teams troubleshoot errors (P5)
  6. Underutilized instances (P6)
  7. No visibility into purpose of instance (P7)
  8. No visibility into the costs of instance (P8)
  9. Users cannot terminate instances (P9)
  10. Increased Systems Operations personnel costs (P10)

Our Team
We put together a four-person team to create a solution for delivering software and managing the internal Systems Operations for this 100+ person project. We also hired a part-time Security expert. The team consists of two Systems Engineers and two Software Engineers focused on Continuous Delivery and the Cloud. One of the Software Engineers is the Solutions Architect/PM for our team.

Our Solution
We began with the end in mind based on the customer’s desire for unencumbered access to resources. To us, “unencumbered” did not mean without controls; it meant providing automated services over queued “request and wait for the Ops guy to fulfill the request” processes. Our approach is that every resource is in the cloud: Software as a Service (SaaS), Platform as a Service (PaaS) or Infrastructure as a Service (IaaS) to reduce operations costs (P10) and increase efficiency. In doing this, effectively all project resources are available on demand in the cloud. We have also automated the software delivery process to Development and Test environments and working on the process of one-click delivery to production. I’ve identified the problem we’re solving – from above – in parentheses (P1, P8, etc.). The solution includes:

  • On-Demand Provisioning – All hardware is provided via EC2’s virtual instances in the cloud, on demand (P2). We’ve developed a “Provisioner” (PaaS) that provides any authorized team member the capability to click a button and get their project-specific target environment (P3) in the AWS’ cloud – thus, providing unencumbered access to hardware resources. (P4) The Provisioner provides all authorized team members the capability to monitor instance usage (P6) and adjust accordingly. Users can terminate their own virtual instances (P9).
  • Continuous Delivery Solution so that the team can deliver software to users on demand (P1):
    • Automated build script using Ant – used to drive most of the other automation tools
    • Dependency Management using Ivy. We will be adding Sonatype Nexus
    • Database Integration/Change using Ant and Liquibase
    • Automated Static Analysis using Sonar (with CheckStyle, FindBugs, JDepend, and Cobertura)
    • Test framework hooks for running JUnit, etc.
    • Reusing remote Deployment custom Ant scripts that use Java Secure Channel and Web container configuration. However, we will be starting a process of using a more robust tool such as ControlTier to perform deployment
    • Automated document generation using Grand, SchemaSpy (ERDs) and UMLGraph
    • Continuous Integration server using Hudson
    • Continuous Delivery pipeline system – we are customizing Hudson to emulate a Deployment Pipeline
  • Issue Tracking – We’re using the JIRA Studio SaaS product from Atlassian (P10), which provides issue tracking, version-control repository, online code review and a Wiki. We also manage the relationship with the vendor and perform the user administration including workflow management and reporting.
  • Development Infrastructure - There were numerous tools selected by the customer for Requirements Management and Test Management and Execution including HP QC, LoadRunner, SoapUI, Jama Contour. Many of these tools were installed and managed by our team onto the EC2 instances
  • Instance Management - Any authorized team member is able to monitor virtual instance usage by viewing a web-based dashboard (P6, P7, P8) we developed. This helps to determine instances that should no longer be in use or may be eating up too much money. There is a policy that test instances (e.g. Sprint Testing) are terminated no less than every two weeks. This promotes ephemeral environments and test automation.
  • Deployment to Production – Much of the pre-production infrastructure is in place, but we will be adding some additional automation features to make it available to users in production (P1). The deployment sites are unique in that we aren’t hosting a single instance used by all users and it’s likely the software will be installed at each site. One plan is to deploy separate instances to the cloud or to virtual instances that are shipped to the user centers

    System Monitoring and Disaster Recovery – Using CloudKick to notify us of instance errors or anomalies. EC2 provides us with some monitoring as well. We will be implementing a more robust monitoring solution using Nagios or something similar in the coming months. Through automation and supporting process, we’ve implemented a disaster recovery solution.

The benefits are primarily around removing the common bottlenecks from processes so that software can be delivered to users and team members more often. Also, we think our approach to providing on-demand services over queued-based requests increases agility and significantly reduces costs. Here are some of the benefits:

  • Deliver software more often – to users and internally (testers, managers, demos)
  • Deliver software more quickly – since the software delivery process is automated, we identify the SVN tag and click a button to deliver the software to any environment
  • Software delivery is rapid, reliable and repeatable. All resources can be reproduced with a single click – source code, configuration, environment configuration, database and network configuration is all checked in and versioned and part of a single delivery system.
  • Increased visibility to environments and other resources – All preconfigured virtual hardware instances are available for any project member to provision without needing to submit forms or attend countless meetings

Here are some of the tools we are using to deliver this solution. Some of the tools were chosen by our team exclusively and some by other stakeholders on the project.

  • AWS EC2 - Cloud-based virtual hardware instances
  • AWS S3 – Cloud-based storage. We use S3 to store temporary software binaries and backups
  • AWS EBS – Elastic Block Storage. We use EBS to attach PostgreSQL data volumes
  • Ant – Build Automation
  • CloudKick – Real-time Cloud instance monitoring
  • ControlTier – Deployment Automation. Not implemented yet.
  • HP LoadRunner – Load Testing
  • HP Quality Center (QC) – Test Management and Orchestration
  • Ivy – Dependency Management
  • Jama Contor - Requirements Management
  • Jenkins – Continuous Integration Server
  • JIRA Studio - Issue Tracking, Code Review, Version-Control, Wiki
  • JUnit – Unit and Component Testing
  • Liquibase – Automated database change management
  • Nagios – or Zenoss. Not implemented yet
  • Nexus – Dependency Management Repository Manager (not implemented yet)
  • PostgreSQL – Database used by Development team. We’ve written script that automate database change management
  • Provisioner (Custom Web-based) – Target Environment Provisioning and Virtual Instance Monitoring
  • Puppet – Systems Configuration Management
  • QTP – Test Automation
  • SoapUI – Web Services Test Automation
  • Sonar – code quality analysis (Includes CheckStyle, PMD, Cobertura, etc.)
  • Tomcat/JBoss – Web container used by Development. We’ve written script to automate the deployment and container configuration

Solutions we’re in the process of Implementing
We’re less than a year into the project and have much more work to do. Here are a few projects we’re in the process or will be starting to implement soon:

  • System Configuration Management – We’ve started using Puppet, but we are expanding how it’s being used in the future
  • Deployment Automation – The move to a more robust Deployment automation tool such as ControlTier
  • Development Infrastructure Automation – Automating the provisioning and configuration of tools such as HP QC in a cloud environment. etc.

What we would do Differently
Typically, if we were start a Java-based project and recommend tools around testing, we might choose the following tools for testing, requirements and test management based on the particular need:

  • Selenium with SauceLabs
  • JIRA Studio for Test Management
  • JIRA Studio for Requirements Management
  • JMeter – or other open source tool – for Load Testing

However, like most projects there are many stakeholders who have their preferred approach and tools they are familiar in using, the same way our team does. Overall, we are pleased with how things are going so far and the customer is happy with the infrastructure and approach that is in place at this time. I could probably do another case study on dealing with multiple SaaS vendors, but I will leave that for another post.

There’s much more I could have written about what we’re doing, but I hope this gives you a decent perspective of how we’ve implemented a DevOps philosophy with Continuous Delivery and the Cloud and how this has led our customer to more a service-based, unencumbered and agile environment. 


DevOps in the Cloud LiveLessons (Video)

DevOps in the Cloud LiveLessons walks viewers through the process of putting together a complete continuous delivery platform for a working software application written in Ruby on Rails along with examples in other development platforms such as Grails and Java on the companion website. These applications are deployed to Amazon Web Services (AWS), which is an infrastructure as a service, commonly referred to as “the cloud”. Paul M. Duvall goes through the pieces that make up this platform including infrastructure and environments, continuous integration, build and deployment scripting, testing and database. Also, viewers will learn configuration management and collaboration practices and techniques along with what those nascent terms known as DevOps, continuous delivery and continuous deployment are all about. Finally, since this LiveLesson focuses on deploying to the cloud, viewers will learn the ins and outs of many of the services that make up the AWS cloud infrastructure. DevOps in the Cloud LiveLessons includes contributions by Brian Jakovich, who is a Continuous Delivery Engineer at Stelligent.

DevOps in the Cloud LiveLessons

Visit www.devopscloud.com to download the complete continuous delivery platform examples that are used in these LiveLessons.

Lesson 1:
Deploying a Working Software Application to the Cloud provides a high-level introduction to all parts of the Continuous Delivery system in the Cloud. In this lesson, you’ll be introduced to deploying a working software application into the cloud.

Lesson 2:
DevOps, Continuous Delivery, Continuous Deployment and the Cloud covers how to define motivations and differentiators around Continuous Delivery, DevOps, Continuous Deployment and the Cloud. The lesson also covers diagramming software delivery using spaghetti diagrams and value-stream maps.

Lesson 3:
Amazon Web Services covers the basics of the leading Infrastructure as a Service provider. You’ll learn how to use the AWS Management Console, launch and interact with Elastic Compute Cloud (EC2) instances, define security groups to control access to EC2 instances, set up an elastic load balancer to distribute load across EC2 instances, set up Auto Scaling, and monitor resource usage with CloudWatch.

Lesson 4:
Continuous Integration shows how to set up a Continuous Integration (CI) environment, which is the first step to Continuous Integration, Continuous Delivery and Continuous Deployment.

Lesson 5:
Infrastructure Automation covers how to fully script an infrastructure so that you can recreate any environment at any time utilizing AWS CloudFormation and the infrastructure automation tool, Puppet. You’ll also learn about the “Chaos Monkey” tool made popular by NetFlix – a tool that randomly and automatically terminates instances.

Lesson 6:
Building and Deploying Software teaches the basics of building and deploying a software application. You’ll learn how to create and run a scripted build, create and run a scripted deployment in Capistrano, manage dependent libraries using the Bundler and an Amazon S3-backed repository, and deploy the software to various target environments including production using the Jenkins CI server. You will also learn how anyone on the team can use Jenkins to perform self-service deployments on demand.

Lesson 7:
Configuration Management covers the best approaches to versioning everything in a way where you have a single source of truth and can look at the software system and everything it takes to create the software as a holistic unit. You’ll learn how to work from the canonical version, version configurations, set up a dynamic configuration management database that reduces the repetition, and develop collective ownership of all artifacts.

Lesson 8:
Database, covers how to entirely script a database, upgrade and downgrade a database, use a database sandbox to isolate changes from other developers and finally, to version all database changes so that they can run as part of a Continuous Delivery system.

Lesson 9:
Testing, covers the basics of writing and running various tests as part of a Continuous Integration process. You’ll learn how to write simple unit tests that will run fast tests at the code level, infrastructure tests, and deployment tests – sometimes called smoke tests. You will also learn how to get feedback on the test results from the CI system.

Lesson 10:
Delivery Pipeline demonstrates how to use the Build Pipeline plug-in in Jenkins to create a delivery pipeline for the commit, acceptance, load & performance and Production stages so that software can potentially be delivered to users with every change.


Continuous Delivery with Jenkins, CloudFormation and Puppet Online Course

This 3-hour online course is ideal for Developers or Sys Admins who want to know how to script a complete Continuous Delivery platform in the Cloud using Jenkins, AWS and Puppet. By the end of the course, you will have a working Continuous Integration system on AWS with all of the source code. The course is led by award-winning author of Continuous Integration and Stelligent CTO, Paul Duvall. The next course is Tuesday August 14, 2012 at 1PM EDT.

Attendees should be at least a junior-level Developer or Linux Sys Admin

In the course, you will:

  1. Use AWS CloudFormation to script the provisioning of AWS resources such as EC2, Route 53 and S3
  2. Script the entire Jenkins and target environment infrastructure using Puppet
  3. Create a fully-functioning Continuous Delivery platform using Jenkins and AWS
  4. Participate in discussions on Continuous Delivery patterns and best practices

You will also be able to email the trainer after the course with questions on setting up your CI platform in AWS.

Register now

There will be 15+ separate exercises that you will go through in creating a working Jenkins system in AWS. If you do not have an AWS account, you will need sign up by registering with Amazon Web Services as a part of the course pre-work (a credit card to pay AWS is required). The course material will be provided to students prior to the session. You will spend approximately $10 on AWS fees during the 3-hour training session.

You will be provided access to all of the source code and instructional material prior to, during and after the course If you have any questions, contact us at training@elasticoperations.com. A website with some of the example content for the course is located at http://onlinecd.s3-website-us-east-1.amazonaws.com/


Test-Driven Everything

In a lot of ways, developer-time testing is a solved problem. Not to say that test-driven development is always easy. There are still plenty of people and technical issues to sort through when you decide to start testing your application aggressively. That said, I think that bringing testing into every aspect of your software development lifecycle presents a much greater challenge, and therefore fun to tackle.

I get asked frequently how to automate testing for every team member. The reasoning usually goes something like this, “If test-driven development is good for programmers, why wouldn’t it be good for business analysts?” Feel free to replace “business analysts” with anyone on the project team – customer, project manager, quality assurance engineer, etc. It may seem a little radical. Fortunately, I agree with them in principle.

In fact, I like to say that tests are the primary artifacts that drive the communication from one team member to another. More precisely, I mean an executable test that gets run with every build and fails when the application no longer does what we want. In this post, I describe how my team and I got this working on a very successful project.

I’ll set the stage by describing the application, team makeup and development process. The application was a “back-end” system, meaning that it had no graphical user interface. Messages flowed into the system, which in turn updated the state of the database and orchestrated various interactions with other external systems. After much processing, the application spit out a message to indicate success, failure or some variation of the two. The entire project ran fulltime for a little more than one year. The team consisted of about 15 people consistently, sometimes going as high as 25. We had a fantastic war room, large enough for everyone on the team. And, we had a very well defined workflow for our Stories. At each step in this process, there was a heavy focus on testing.


Yep, that’s right, two passes through Quality Assurance. I’ll come back to why. And, just in case you think this sounds a little heavyweight, a Story would flow through this process in less than 2 weeks with lots of opportunity for the all-important feedback loop. We usually had about 6 Stories flowing in parallel to keep everyone on the team fully utilized.

Step one was prioritization with the customer. As the business customers reviewed and prioritized Stories, they would conduct meetings to determine what each Story really meant. Inevitably during this meeting interesting edge cases would arise. If the edge case was in scope for the Story, it would be documented in the body of the Story card. Those edge cases were the first stab at testing. Each case was essentially a comment from the customers saying “…and don’t forget to test that the application will handle this scenario”. Over time, the customers learned that by documenting these interesting variations right alongside the Story, they were helping to make the application better.

Next, the Story would get a thorough analysis. That meant looking into the guts of the functionality to make sure that every aspect was written down for the developer. As a part of that, the analysts would create tables of test scenarios. In fact some Stories consisted mostly of test cases. It’s surprising how well one can communicate complex functionality by giving examples. The amount of verbiage in the Story decreased as the analysts started to think more about how to test the functionality, instead of how to describe the functionality.

The analysts would also incorporate those edge cases identified by the business customers into the test cases. Frequently, the analysts would realize even more unique variants, and those too…would be documented in the Story as an example test case.

Ok, so far, this doesn’t sound much different than what everyone does. The difference was, those tables of test cases identified by the analysts were executable! Ok…you caught me…they were nearly executable. We taught the analysts how to create test cases consumable by the testing framework Fit, and FitNesse. The output from the analysis process was a textual description of the desired functionality with embedded executable (nearly) example test cases.

The third step in the process was a stop by the quality assurance folks. They would review the Story and think about how to test the functionality. They frequently discovered even more variations and potential impact to the rest of the system functionality. They possessed an intimate understanding of the custom FitNesse fixtures defined by the team. Therefore, they were able to streamline the testing, by using meaningful variables, random number generators to increase the test’s completeness, etc.

At this point, the Story was truly executable. Meaning, we could run it in FitNesse. It would of course fail, because the functionality hadn’t been implemented by the development team, but we would get a red bar. Red bars make a good starting point for writing code.

And here we are! Developers get to write some code to make that test pass.

We referred to our development approach as “Fit-First Development”. Just like test-driven development, where you start by writing a broken unit test, we would start with a broken Fit test and write just enough code to make it pass. Actually, we would drill down from the Fit test into the code, writing tests all the way along. We would sometimes expand the Fit test. We would always write unit tests, in this case using JUnit. As we got the unit tests passing we would pop our way back up to the Fit test. Sometimes, we would get the Fit test working one table row at a time until the correct functional implementation emerged.

As the tech lead on the team that pair programmed to write code all day long, I found the approach to be exhilarating. Never before had I felt so assured that the code my pair and I wrote actually did what the business wanted.

After the developers completed the code, i.e. made the Fit test green bar, they would pass it back to the QA team. One point here…The QA team was sitting in the same room. As mentioned, everyone including the customer was sitting in the same war room. Although my description might make it sound like chunks of work were being lobbed over the wall, that wasn’t the case. We talked back and forth continuously. There were times when the developers would encounter a contradiction between the plain English description and the test scenarios. In those cases, it was a matter of minutes before the analysts and QA people were hunched over the developer work station determining the correct answer.

So, the Story, along with the code flowed back into the hands of the QA team. They would do a thorough review of the functionality. Sometimes they would add more test scenarios that had become obvious, which might in turn result in the Story heading back into development (the aforementioned feedback loop). Once QA was satisfied, they would call the customer representative over to do a final functionality walk-through. Since it was a back-end system, the Fit test plan provided a visual mechanism that even a technically challenged person could relate to. When the customer representative was satisfied, the final sign-off was provided. In this manner, each and every Story went from start to finish.

A side-effect of this approach was that a huge regression test suite was created step-by-step throughout the course of the project. The executable Stories were stored in Subversion, right alongside the code. The Stories, and the tests contained therein, became first-class artifacts that warranted ongoing maintenance to ensure that functionality developed months ago didn’t inadvertently get broken.

The other aspect of this was that we included the entire FitNesse wiki in Subversion, too. We used lightweight, in-memory containers to simulate the production environment. That meant that anyone on the team could checkout the project from Subversion and run the entire regression test suite, locally. Nobody had to connect to a central server. Nobody needed their own database schema. You could checkout from Subversion, double-click a batch/shell script, and off you go. Everything just worked!

Was it easy to make everything just work? Yes. It was easy because we started the project knowing that we wanted to make everything just work. It was the culmination of a couple years of trying to get better and better about automating the entire process. But now, I would seldom knowingly do it any other way. Automate everything. Check in everything. Make your Stories an executable description. And treat those Test Scenarios just like you would your application’s code.

Originally authored by Paul Julius at testearly.com


Continuous Delivery Tools List

I usually shy away from giving a list of tools that we use because people have their particular tool preferences and are sometimes indignant in considering others. However, I realize it’s helpful for people to understand the tool landscape when it comes to Continuous Delivery in the Cloud just so they know where to start looking. After reading my Continuous Integration book, this is often the most common question I get from readers.

I want to say up front that I’m not advocating the use of any of these tools, just that we’ve used some of the tools or investigated when creating Continuous Delivery systems. I’m sure some of the tools that we use on a daily basis won’t make it to this list.

The precise toolset a team may choose to use depends upon numerous factors including project, cost and customer constraints – to nTool Chainame a few. Therefore, I suggest that you focus more on the type of tool and determine which one meets their particular needs for their Continuous Delivery ecosystem. Just because I’m not mentioning a particular tool doesn’t mean I’m not using it or that I don’t think it’s a good tool; these are meant to be illustrative. We tend to focus more on freely-available tools because people can download and use them quickly. There are good reasons to choose commercial tools. As implied before, you don’t need to be using all of these tools to get significant benefit from Continuous Delivery. Start small and build it up. I’ve listed some of the tools in each category for the Java, .NET and Ruby platforms. Since, we lean heavily toward Cloud tools, you’ll see that we opt for the SaaS-based tools, when applicable. Let me know if your preferred tool didn’t make the list. Ok, there’s my disclaimer. On with the list:

Application Containers – JBoss, Tomcat, IIS, Mongrel. NOTE: there are so many app containers, I’m not going to try to list all of them.
Build ToolsAnt, AntContrib, NAnt, MSBuild, Buildr, Gant, Gradle, make, Maven, Rake
Code Review - Crucible
Code InsightFisheye  
Continuous Integration – Bamboo, CodeShipJenkins, AntHill Pro, Go, TeamCity, TFS 2010
Cloud IaaS -  AWS EC2, AWS S3 ,  Windows Azure
Cloud PaaSGoogle App Engine, AWS Elastic Beanstalk, Heroku
Database – Hibernate, MySQL, Liquibase, Oracle, PostgreSQL, SQL Server, SimpleDB, SQL Azure, Ant, MongoDB
Database Change Managementdbdeploy, Liquibase
Infrastructure AutomationCapistrano, Cobbler, BMC Bladelogic, CFEngine, IBM Tivoli Provisioning Manager, Puppet, Chef, Bcfg2, AWS Cloud FormationWindows Azure AppFabric NOTE: There are many names and overlap for this tool “category”.
Dependency ManagementIvy, Archiva, Nexus, Artifactory, Bundler
Deployment AutomationJava Secure Channel, ControlTier, Altiris, Capistrano, Fabric, Func
Information SharingConfluence, Google Apps
InstallerInstallShield, IzPack
Integrated Development Environment (IDE)Eclipse, IDEA, Visual Studio
Issue Tracking - Greenhopper, JIRA
PasswordsPassPack, PasswordSafe
Protected ConfigurationESCAPE, ConfigGen
Project ManagementJIRA, Pivotal Tracker, SmartSheet
Provisioning - JEOS, BoxGrinder, CLIP, Eucalyptus, AppLogic
Reporting/DocumentationDoxygen, Grand, GraphViz, JavaDoc, NDoc, SchemaSpy, UmlGraph
Static Analysis - CheckStyle, Clover, Cobertura, FindBugs, FxCop, JavaNCSS, JDepend, PMD, Sonar, Simian
Systems MonitoringCloudKick, Nagios, Zabbix, Zenoss
TestingAntUnit, Cucumber, DbUnit, webrat, easyb, Fitnesse, JMeter, JUnit, NBehave, SoapUI, Selenium, RSpec, SauceLabs
Version-Control SystemSVN/Subversion, git, Perforce


ROI Calculator for 100% Automated Cloud Operations

We've worked in numerous types of organizations, from multi-billion dollar corporations to non-profits and small startups. We've worked with Operations teams, been part of the Operations team and comprised the entire Operations team. We've learned what works and what doesn't work in various types of organizations. What we've found is that while advances in Cloud Computing and automation continue to surge, many Operations teams still operate like they were 20 years ago. There is a better way…a MUCH better way: Continuous Delivery in the Cloud. This means 100 % automation of the delivery process (build, test Roicalc , deployment and release) and the utilization of Cloud resources. Cloud by itself is just a buzzword. Cloud coupled with 100% automation is where you get huge productivity gains enabling you to delivery software as quickly and as often as the business desires. 

While the Cloud ROI Calculator we developed is geared toward how much costs are reduced (for example, over $3 million USD per year for a 20+ application organization), the most considerable value, in my opinion, is when implementing Continuous Delivery with 100% automation in the Cloud, you can release more quickly and more often. 

How to Enter Information

There are four items of data entry: Average Engineer Hourly Rate, Number Of Applications In Portfolio, Average Size Of Projects and Average Technical Architecture Complexity. Since all data entry uses sliders, you can use the default values if you want to see how it works and then go back and modify values to see different results based on your organization. A medium sized project is approximately a 25K-100K SLOC code base, small is less than 25K and large is greater than 100K. Keep in mind that it's an average of all of the projects in your organization. The technical architecture complexity considers the number of application servers, number of database tables, configuration and other technical complexity. Again, it's the average technical complexity of all your applications.

Explanation of ROI Results

Once you click the Calculate ROI button, you'll see six rows of information, which are explained below. For each row, you'll see the number/cost when using 100% Automated Operations (what Stelligent provides for our clients) vs. the Traditional Operations team that manually performs and queues human tasks.   Roicalcresults

Number of hardware instances - For medium size projects and complexity, we assume 10 ephemeral instances per application and 20 fixed instances.
Hardware costs per year - Because of commodization and economies of scales, cloud instances are approximately 1/4 the cost of managing your own data center. Source: A combination of The Economics of the AWS Cloud vs. Owned IT Infrastructure and work performed by Stelligent.
Number of engineers – This is where the real cost savings is: number of engineers required to create and support a 100% automated cloud environment. using 100% automated cloud, you pay 1/3 the costs in terms of the human capital required to create and maintain the infrastructure. Source: Work performed by Stelligent.
Engineering costs per year - The total cost based on the average engineer rate in your organization, the number of applications and whether it's automated cloud or traditional Operations.
Organizational cost per year - The hardware plus the engineering cost per year.
Total savings per year – The amount in savings between automated cloud and traditional Operations per year.

About ROI Calculator Development

We used the Platform as a Service (PaaS) offering, Google App Engine (GAE) to develop the calculator. My friend Andy Glover developed the first release of the calculator using GAE, Groovy, Gaelyk, etc. We've since provide new features. The application is essentially stateless, but we're using BigTable to manage certain configurable values. Because it's a PaaS offering, we don't worry about hosting, uptime, etc. It just works. GAE also provides a comprehensive dashboard that has extensive logging, etc.


Screencast on AWS Elastic Beanstalk

Amazon Web Services released their Platform as a Service offering on Wednesday, January 19th. I've gotten an opportunity to play with it and I'm quite impressed. I created a seven-minute screencast that takes you through the steps to deploy and configure an application/environment using Elastic Beanstalk. In this screencast, you'll see how easy it was to get a Hudson CI server up and running in an EC2 environment. Furthermore, Elastic Beanstalk provides automatic scaling, monitoring, configuration right 'out of the box'. It's worth checking out.



Continuous Delivery: How to deliver software to users at the click of a button

Continuous Delivery – part of the Martin Fowler Signature Series – was published last month and if delivering software to your users quickly and often is important to you, you must get this book. Continuous Delivery is what we provide our clients at Stelligent – delivering complete software systems to users at the push of a button. Jez Humble and David Farley go into great detail in describing everything you need to do to create a continuous delivery system. 

This is one of theContinuous_delivery_book most important software books published in years. From the beginning and throughout the book, the authors emphasize the importance in establishing one delivery team consisting of various experts throughout the software lifecycle – developers, DBAs, Systems/Operations, network specialists, testers and so on. The overarching pattern the authors describe is the Deployment Pipeline, which is basically a staged process consisting of all of the steps to go from bare/virtual metal to a working system whenever there is a change to source files. Of course, the only way this can be done is through copious amounts of automation. The other key point the authors make is that this automated delivery system – itself – is versioned with every change. Not just the custom source code, but also the operating system(s), tools, configuration and everything necessary to create a working software system. It's the first book I know to describe this concept in such great detail and not (really) provide any exceptions to the rule. In Continuous Integration (also in Fowler's series), I made a point of putting 'everything' in version control (referring to Berczuk and Appleton's Repository pattern), but Humble and Farley made it a core theme of their Deployment Pipeline and the overall book.

To sum up the book in a few bullets:

  • The purpose of Continuous Delivery is to reduce the cycle time between an idea and usable software such that it's approaching zero
  • Automate (almost) everything necessary to create usable software
  • Version complete software systems (not just source code) for every change committed to version control system
  • Employ a Deployment Pipeline in which the entire system is recreated whenever a change is committed to the version-control system and provide continuous feedback
  • Identify one delivery team consisting of various delivery experts – build, deploy, provisioning, database, testing, etc. – a concept emphasized in the DevOps movement

The authors go into great detail in describing each of these themes. So, if you want the process of delivering software to any target environment – including production – to be a click of a button and something that can be accomplished as often as the business requires, get this book. When you employ the practices in this book, no longer will you need to artificially throttle changes delivered to users for months or even years because of the expense and risk required to deliver software.

As a note, I plan to extract and describe a combination of patterns from this book, the Continuous Integration book, and other patterns we use at Stelligent in upcoming blog posts.