Is Continuous Delivery still a hostage to (manual) testing?

In traditional software development, you spend weeks, even months, developing code. Then comes the dreaded “integration cycle” and finally the “testing” or “QA” phase. We all know the story. At first, nothing works. But what’s even more interesting is that nobody even expects it to work! It takes days and sometimes weeks before a stable, working build is available and QA can start testing.

Agile, with all its promises of delivering us from slow and error prone SDLC didn’t get us very far. An enterprise development lead described his team’s process as “Scrum Waterfall”. I have also heard “Water Scrumfall” and “Scrumfall”. The point he was making is that irrespective of how fast Development moves with Agile and/or CI , the mandatory testing phase drags the whole process back to the slow lane.

From what I have observed, there are 3 primary reasons why despite all the automation tools, we have been unable to accelerate testing:

  1. Lack of environments – several self service environment management tools have been available for a while but unfortunately they have done little to alleviate the real pain of environments – ability to closely mimic Production with all its configurations AND services (and this is an important but missing piece of the puzzle)
  2. Delivery of “untestable” code – Developers may be running 2 week Sprints to deliver components that make no sense unless other “driver” components are delivered. For example, delivering db schema (not data) that will be used later
  3. Reliance on “Manual Testing” during development – since the code changes every day (several times a day), current automated tools are ineffective unless presentation layer is locked down right in the beginning (and this almost never happens)

While we can improve on all three,  “Reliance on Manual Testing during development” is what any forward looking IT organization should be addressing right away.

Imagine a system where requirements are turned to business process, which in turn generates test cases automatically and then loads test data that can be generated and managed automatically too. Such a process can bring down the end to end feature/app testing cycle from several days/weeks to minutes!

This is not as far fetched as it sounds.Using current TDM tools (and there are several in the market) and integrating them with already available testing tools like Selenium and QTP, some customers have already realized the true potential of Continuous Delivery. They have (more or less) eliminated a separate test phase making every change deployable.  Even if you want to keep the UAT a separate phase, it is still super fast – lasting no more than a few days rather than months it used to take earlier!

So what do testers do? They spend their time on more productive tasks like exploratory testing.

The key takeaway here is that unless we automate testing end to end – including test case creation and update, test data generation, test execution and defect filing, CD will remain hostage to manual test cycles.  Changing the mindset from record-replay method of test development to more of a model-based system will go far in addressing this challenge.

“Deployment Pipeline” – Is this it?

The term “Deployment Pipeline” is often used to imply different stages/environments your release candidate has to be deployed to before it can be released (go in “Production”).

When you hear it, you probably think of this


A heavy viscous liquid running through a long pipeline such that every single drop has to pass through every single inch of the pipeline sequentially.

However, Humble and Farley had a very different idea when they coined the term and it looked something like this:


Yes. It is a processor.

They borrowed the idea from the world of hardware where a sequential set of instructions is broken down in parallel threads of execution such that in the worst case there won’t be any time loss over sequential execution. In the best case though, instructions could be parallelized and speed up the execution.

How do they do it? Processors open parallel threads all the time but only when they are confident that they have enough data to predict the outcome of another thread. For example thread 1 is executing. It is the only thread. At some point the processor can predict that the outcome of thread 1 is going to be “X”. Processor takes “X” and hands it to thread 2. thread 2 starts execution.

When Thread 1 completes execution it verifies the actual result. If the result indeed was “X”, processor continues with thread 2. It just managed to parallelize the execution of otherwise serial instruction set.

If however outcome of thread 1 was “Y”, processor simply dumps thread 2 and continues with sequential execution.

Nothing was gained by opening thread 2. But even more importantly, nothing was lost. So there’s no downside and the upside is huge.

Brilliant! isn’t it?

“Deployment Pipeline” works exactly the same way. The commit stage in deployment pipeline is automated to run very quickly. We can now “guess” that all subsequent testing and verification will pass. With that assumption we can open a new thread by picking new items from the backlog to work upon. If the tests succeed, we have a new set of features coming down the pipe – parallel execution. If however, tests fail, no harm done. we simply send the errant code back to developers.

At the end we have achieved parallel execution upstream by being optimistic about the outcome of the activities downstream. 

When you talk about Deployment Pipeline, think of it as an enabler to parallel execution to achieve maximum efficiency of all resources across dev, test and release. Then, and only then, you have a true  Deployment Pipeline as envisioned in the Continuous Delivery paradigm.

Continuous Delivery Vs Continuous Deployment

A number of customers I talk to about Continuous Delivery say they have no intention of deploying to the production without proper approvals.

The nay sayers often use the “won’t work for us” excuse.

“Broken Facebook feed causes you inconvenience. Broken checking account could destroy your life.” say IT workers of any bank.

They have a point. However, I wasn’t talking to them about deploying an untested code in production. I wasn’t even talking to them about deploying to production and herein lies the challenge of distinguishing Continuous Delivery from Continuous Deployment.

Continuous Delivery doesn’t propose deploying every change to production. It only proposes that you ensure that every change is “deployable” . 


This means that the developers can’t churn out bug ridden code that is unstable and will most like crash any environment it gets deployed to. It also means that the testers can’t rely on old archaic testing methodologies that involve a lot of testing at the end of the dev cycle but don’t necessarily lead to better quality release! Similarly, the deployment can no longer be a set of manual steps that can’t be repeated unless handed over to “Dan on 6th floor who is out sick today”.

Continuous Delivery relies upon automation (and not the traditional automation).

Continuous Deployment, on the other hand, asks us to boldly go where many have gone before – to the world of automated deployment all the way to the production.


Continuous Deployment is the next step to Continuous Delivery. Once you have established good automated processes to develop, test and deploy every code change in production like environment  within acceptable quality threshold, you can take the plunge and auto approve for deployment to production.

Neither Continuous Delivery nor Continuous Deployment is about swashbuckling developers deploying half-baked  ideas to production. On the contrary, it is about highly disciplined teams that are mature enough to communicate with each other, committed to standing behind their work to deliver high quality (not just well tested) software for end users.




Continuous Delivery Vs Release Automation

Many people confuse Application Release Automation (ARA) with Continuous Delivery (CD). Many people confuse Continuous Delivery with Continuous Deployment but that will be discussed later.

ARA automates deployment on a target environment. ARA tools primary purpose is to push the build with a single click. As long as it can do that, in a repeatable, predictable fashion, the ARA tool has done its job.

Continuous Delivery however has much more to it than just pushing the bits out. CD can only be achieved if you can build, deploy and test every change while providing feedback at every stage. The fundamental principles of CD demand constant feedback and an almost sacred oath to not pass the defects forward.

Here’s my view on RA vs CD (inspired by Nhan Ngo’s sketch)

Screen Shot 2015-03-05 at 5.17.26 PM