I want to share with you the improvements we’ve made to our deployment automation to support Continuous Delivery (CD).

In this blog post, I’ll describe how we deploy our applications using CD. Today, our development teams are looking for ways to get features and bug fixes out into production as quickly as possible. A small number of our Development teams have adopted CD with many more looking to adopt it in the near future.

CD has allowed our teams to develop features and bug fixes rapidly. To support this rapid develop and deploy model, it was time to find ways to improve the tools and scripts we use in our deployment automation.

Our environment

Today, we have 100+ JBoss/JRuby applications being worked on by 13 different development teams. In production, all of our JBoss/JRuby applications are deployed on 500+ servers running JBoss Application Server (AS). We are in the process of implementing Torquebox for our JRuby applications. Torquebox is a Ruby application server that uses JBoss AS 7 as an application platform. Torquebox includes built-in support for messaging, scheduling, running of background jobs, and clustering.

We have 6 environments that we support and deploy to:

• Development – Deploys from the development code branch. Used by development teams to test changes before check in to integration code branch.
• QE1 – Deploys from the integration code branch. Used to verify the build before promoting it in the functional QE3 environment.
• QE2 – Deploys from the release code branch. Used to verify the build before promoting it in our stage environment.
• QE3 – Deploys from the integration code branch. Builds are promoted from the QE1 environment when acceptance testing passes.
• Stage – Deploys from the release code branch. Used to verify the build in a pre-prod environment.
• Prod – Deploys from the release code branch.

Our current CD deploy automation story

Our CD deploy automation is written in Python and uses Func to execute commands remotely, much like our non-CD process. The Jenkins jobs use Python scripts for deployment execution, which retrieve information about application servers and the apps installed on those servers from a MySQL database. The deploy automation also checks the health of applications and kicks off the quality engineering (QE) verification tests.

Technologies used:

 Puppet

We use Puppet to manage server configuration. Puppet handles server specific configurations including network interfaces, NFS mount points, and user account & permissions. For JBoss and JRuby applications, Puppet deploys the packaged artifact (either a war or knob file) along with all application and JBoss specific configuration files. Puppet manages JBoss and JRuby configurations such as JNDI data sources, database log ins, and JAVA_OPTS.

Nexus

Nexus is a repository manager. The war and knob file artifacts are stored in nexus under various branches, with each branch mapped to one of our environments.

• Development (Development environment) – Build artifacts from the development code branch are pushed to this branch.
• Integration (QE1 environment) – Build artifacts from the integration code branch are pushed to this branch.
• Integration-tested (QE3 environment) – The build is promoted from the integration branch when the acceptance test passes on QE1.
• Release (QE2 environment) – Build artifacts from the integration code branch are pushed to this branch.
• Release-tested (Stage environment) – The build is promoted from the release branch when the acceptance test passes on QE2.
• Master (Production environment) – The build is promoted from the release-tested branch when the acceptance test passes on Stage.

Func

Func is an essential tool that we can’t live without. I think Func is an absolute must have in anyone’s toolbox. Func is open source software used to run remote commands to other servers from a central location. We no longer have to set up SSH keys on each individual server. Func takes care of this for you and it also provides a detailed audit log of all server commands run. Func also has a python client that makes it easy to integrate into our deployment automation. To use Func, you need to:

• Set up a master server where all commands are executed from.
• List the minions it can talk to.

Here are some examples using Func from the command line:

• Get a list of java processes
  func appServer0[1-4]* call command run ‘ps –ef | grep java’
• Restart apache
  func appServer0[1-4]* call service restart httpd

Python-scripting language

Our deploy automation is written in python. We choose python because tools such as Jenkins and Func integrate well with it. Here’s an example of some python code integrating with Func. Remember, the script must run from the Func master, as shown here:

#!/usr/bin/python

 import func.overlord.client as fc

 svr=”your client server”

client=fc.Client(svr,nforks=10,timeout=600)

cmd = “/sbin/server httpd restart”

info = client.command.run(cmd)

for (host,details) in info.iteritems():

   print “Host: %s ” % host

   print “Details: %s ” % details

Jenkins

Jenkins is an open source continuous integration server most people use for building and packaging software. While we use Jenkins as our build/packaging server, we also set up Jenkins on a dedicated server for deployment and QE verification jobs. With some plugins and clever coding of our scripts, we successfully use it to launch our deployments and QE tests. Here are some of the key Jenkins plugins we use:

• Dashboard View
  Dashboard view gives us a good way to organize our deploy jobs by team. With this plugin you can see things such as the status of the last 25 jobs, build statistics, and a list of unstable jobs.
• HipChat
  We use the HipChat plugin to allow the deploy jobs to write entries into specific HipChat rooms, indicating the status of the job. Our deploy jobs use a development team-specific room and a general deployment room.

• Groovy Postbuild
  This plugin runs a groovy script as a post step in the deploy job. We use this to gather job statistics from Jenkins.

• Log Parser
  When deploying to large environments, anything we can use to help find an issue is a big plus. Log Parser allows us to color code INFO, WARN, and ERROR log levels in our logs.

• Python
  Our deploy automation is written in python. This plugin gives us the ability to use python code in a build step of our deploy job.

Conclusion

In this blog post, I described how we deploy our applications using CD and I listed the technologies we use for our deploy automation of CD apps. I hope that you’ll find this info useful as you start thinking about deploy automation.

In part 2 of this blog post, I’ll share some of the python methods we created for our deploy orchestration. Plus, I’ll walk you through the process of setting up a deploy job in Jenkins using this deploy automation.

Do you have questions about how we deploy our applications using CD? Ask me in the comments below. 

The post Deployment Automation for Continuous Delivery Part 1 appeared first on Constant Contact Tech Blog.