Category Archives: Uncategorized

Goodbye GPS. Hello Self-Driving Vehicles!

Screen Shot 2016-04-11 at 12.18.56 PM

Caution: This blog will teach you nothing about GPS (or Self-Driving Vehicles)

===================================================

Remember the days when a road trip meant heading to AAA, getting triptiks, planning your stops and exits carefully and then hoping for the best?

That’s what Release planning looked like with long and tedious spreadsheets that listed dev modules, tests, defects, environments and action items. Dedicated program managers would pore over these spreadsheets, identify owners and remind them of their tasks in weekly status meetings.

Long, 24-48 hour phone meetings during release cycles were the norm and a  release could take anywhere from a few months to several quarters. I remember engineers who bragged about how long they stayed on the call. Inefficiency was a badge of honor.

(Semi-automated)  Release Management and Deployment Automation solutions discarded that approach. Integrated with CI tools such as Jenkins they acted like an integrated GPS system, orchestrating every single action and cutting the release time and the number of errors by up to 60%.  Certainly impressive improvement from the good ole’ spreadsheets.

But it wasn’t until PaaS burst onto the screen that people realized that a fully automated continuous delivery with little to no effort was not only a possibility but a great way to test out new ideas in the market.

PaaS however, came with its own shortcomings – lack of supported dev frameworks and platform lock-in being the 2 most obvious ones. PaaS was great for new, web native apps but didn’t do anything for existing applications.

That said, if Triptik was the manual navigation with a paper map, PaaS was the self-driving vehicle of Continuous Delivery.

However, the need for quick deployments without sacrificing developer flexibility is real and so crucial that enterprise IT would rather do a few tasks manually rather than use all encapsulating PaaS.

Enter, containers. Ability to package up your apps and ship them to any target environments freed the developers up to compose new apps from a repository of containers. It not only accelerated app delivery and deployment but also allowed architects to rethink their service architecture.

As we go forward, more and more apps will move to container delivery. Docker Cloud, Shippable, Nirmata etc solve the problem of container delivery but leave customers holding the bag when it comes to providing operator services. More on this in the next post.

 

Not your grandfather’s Continuous Delivery

Guinness

“What’s a seven-course Irish meal?” the old joke goes.

“Six pints of beer and a potato.”

Continuous Delivery is beginning to sound more and more like that. An upstream-heavy continuum of activities from developers who eventually toss the deployable package to ops, the idea is that the package is self-contained, requires no additional configurations, and can be applied to the production environment with little to no help from dev.

The reality, as always, is quite different. Prod never looks like dev/test and seldom even like pre-prod. Even if all the services, configurations, and apps are the same, the data, usage, traffic, and data center are not.

Continuous Delivery needs to evolve into a mix of developer and operation activities, where developers are not only responsible for the code but also for the deployable image. The image in question is testable, portable, and deployable across a range of infrastructures. It is self-contained, so starting up a new instance is simply a question of instantiating the image on the target server — an ops activity.

For this “cattle” workflow, ops doesn’t need to define specific servers and lay OS, config, and app on top. Instead, they request available compute and memory resources from IaaS and lay the image on the provisioned machine so that you have a running app within seconds, not the minutes or hours it could take for each component of the app.

When a PaaS claims they can bring up new instances of your app in seconds, this is what they are attempting to accomplish.

The problem with PaaS is their inherent limitation of supported developer frameworks and services, their unavailability as on-premise services (though both Cloud Foundry and Open Shift provide that option) and, above all, the steep cost of hosting apps (or having to manage the infrastructure).

While we won’t go into the hotly debated topic of PaaS adoption, it is clear that enterprises are attempting to realize this vision through a variety of tools.

Atlas, Otto/Vagrant, Packer, Terraform, Consul, Serf from Hashicorp, vRealize/vCloud from VMware, Ansible/Salt/Chef/Puppet, ElasticBox, CliQr, Jenkins etc. all solve part but not all of the problem.

However, who wants to adopt a tech stack where you have to define the workflow for each new technology? As the number of scripts and complexity grows, who wants to manage the “management platform” like Chef and Puppet?  Who hires people with tool-specific expertise and manages them into what is now being called the “DevOps engineering” team?

Nobody, that’s who.

The time to democratize the Continuous Delivery process is here. The time of opinionated workflows is not far behind.

The new implementation of Continuous Delivery should (and will):

  1. integrate (not just orchestrate) developer and operation activities into a single platform,
  2. automate the entire process through opinionated workflows,
  3. accelerate essential tasks like deployment and infrastructure automation through immutable infrastructure approach, and finally
  4. control the app delivery (start, stop, reset) and make it transparent to minimize risk.

Not only does this make Continuous Delivery faster, but also lighter, with fewer tasks and technologies to manage.

Do it right and that Irish meal will become  a lighter, five-course meal — two pints of beer, colcannon, onion soup, and a coffeecake.

Complexity of Pricing PaaS

In the early days of PaaS, the focus was on simplicity.  Simplicity of using the platform, developing apps, getting services and pricing. “No bill shock” was the mantra that almost all PaaS players embodied. However, those days of idealism and transparency are long gone.

In a desperate effort to differentiate for each other, PaaS players have added a lot more services and reengineered both, their platforms and their pricing.

Figuring out how much it will cost you to move your apps to PaaS is more difficult now than actually moving the darned app!

About 4 years while working on CloudFoundry price, I did some research on what the competition was charging. It was difficult and I had to make a lot of assumptions but I had a pretty good idea of what it takes to host an application on a competitor’s platform. Here it is

Screen Shot 2015-05-14 at 3.00.27 PM

Now, it is more difficult. I am yet to calculate the price for Heroku (it is confusing to me) and I am not considering EngineYard for this round. Here’s a quick take on 3 major players – GAE, Azure and AWS (yes, I know EBS is not real PaaS but lets consider it anyway).

Screen Shot 2015-05-14 at 2.45.44 PM

 

Like cars, pricing PaaS with options has become really difficult, if not downright impossible. The vendors are trying to keep things complicated to avoid a price war but enterprises are still working to do their own research before they pick a platform. After talking to a few startups and enterprise IT workers, I found that AWS becomes the default choice because it is a “safer bet” in every way, especially pricing.

Is Continuous Delivery still a hostage to (manual) testing?

In traditional software development, you spend weeks, even months, developing code. Then comes the dreaded “integration cycle” and finally the “testing” or “QA” phase. We all know the story. At first, nothing works. But what’s even more interesting is that nobody even expects it to work! It takes days and sometimes weeks before a stable, working build is available and QA can start testing.

Agile, with all its promises of delivering us from slow and error prone SDLC didn’t get us very far. An enterprise development lead described his team’s process as “Scrum Waterfall”. I have also heard “Water Scrumfall” and “Scrumfall”. The point he was making is that irrespective of how fast Development moves with Agile and/or CI , the mandatory testing phase drags the whole process back to the slow lane.

From what I have observed, there are 3 primary reasons why despite all the automation tools, we have been unable to accelerate testing:

  1. Lack of environments – several self service environment management tools have been available for a while but unfortunately they have done little to alleviate the real pain of environments – ability to closely mimic Production with all its configurations AND services (and this is an important but missing piece of the puzzle)
  2. Delivery of “untestable” code – Developers may be running 2 week Sprints to deliver components that make no sense unless other “driver” components are delivered. For example, delivering db schema (not data) that will be used later
  3. Reliance on “Manual Testing” during development – since the code changes every day (several times a day), current automated tools are ineffective unless presentation layer is locked down right in the beginning (and this almost never happens)

While we can improve on all three,  “Reliance on Manual Testing during development” is what any forward looking IT organization should be addressing right away.

Imagine a system where requirements are turned to business process, which in turn generates test cases automatically and then loads test data that can be generated and managed automatically too. Such a process can bring down the end to end feature/app testing cycle from several days/weeks to minutes!

This is not as far fetched as it sounds.Using current TDM tools (and there are several in the market) and integrating them with already available testing tools like Selenium and QTP, some customers have already realized the true potential of Continuous Delivery. They have (more or less) eliminated a separate test phase making every change deployable.  Even if you want to keep the UAT a separate phase, it is still super fast – lasting no more than a few days rather than months it used to take earlier!

So what do testers do? They spend their time on more productive tasks like exploratory testing.

The key takeaway here is that unless we automate testing end to end – including test case creation and update, test data generation, test execution and defect filing, CD will remain hostage to manual test cycles.  Changing the mindset from record-replay method of test development to more of a model-based system will go far in addressing this challenge.

“Deployment Pipeline” – Is this it?

The term “Deployment Pipeline” is often used to imply different stages/environments your release candidate has to be deployed to before it can be released (go in “Production”).

When you hear it, you probably think of this

landscape_sketch___pipeline_by_jbn0s0rus-d5hk5zi

A heavy viscous liquid running through a long pipeline such that every single drop has to pass through every single inch of the pipeline sequentially.

However, Humble and Farley had a very different idea when they coined the term and it looked something like this:

stock-vector-circuit-board-pattern-vector-illustration-with-brain-sketch-80789035

Yes. It is a processor.

They borrowed the idea from the world of hardware where a sequential set of instructions is broken down in parallel threads of execution such that in the worst case there won’t be any time loss over sequential execution. In the best case though, instructions could be parallelized and speed up the execution.

How do they do it? Processors open parallel threads all the time but only when they are confident that they have enough data to predict the outcome of another thread. For example thread 1 is executing. It is the only thread. At some point the processor can predict that the outcome of thread 1 is going to be “X”. Processor takes “X” and hands it to thread 2. thread 2 starts execution.

When Thread 1 completes execution it verifies the actual result. If the result indeed was “X”, processor continues with thread 2. It just managed to parallelize the execution of otherwise serial instruction set.

If however outcome of thread 1 was “Y”, processor simply dumps thread 2 and continues with sequential execution.

Nothing was gained by opening thread 2. But even more importantly, nothing was lost. So there’s no downside and the upside is huge.

Brilliant! isn’t it?

“Deployment Pipeline” works exactly the same way. The commit stage in deployment pipeline is automated to run very quickly. We can now “guess” that all subsequent testing and verification will pass. With that assumption we can open a new thread by picking new items from the backlog to work upon. If the tests succeed, we have a new set of features coming down the pipe – parallel execution. If however, tests fail, no harm done. we simply send the errant code back to developers.

At the end we have achieved parallel execution upstream by being optimistic about the outcome of the activities downstream. 

When you talk about Deployment Pipeline, think of it as an enabler to parallel execution to achieve maximum efficiency of all resources across dev, test and release. Then, and only then, you have a true  Deployment Pipeline as envisioned in the Continuous Delivery paradigm.

Continuous Delivery Vs Continuous Deployment

A number of customers I talk to about Continuous Delivery say they have no intention of deploying to the production without proper approvals.

The nay sayers often use the “won’t work for us” excuse.

“Broken Facebook feed causes you inconvenience. Broken checking account could destroy your life.” say IT workers of any bank.

They have a point. However, I wasn’t talking to them about deploying an untested code in production. I wasn’t even talking to them about deploying to production and herein lies the challenge of distinguishing Continuous Delivery from Continuous Deployment.

Continuous Delivery doesn’t propose deploying every change to production. It only proposes that you ensure that every change is “deployable” . 

CONTINUOUS DELIVERY

This means that the developers can’t churn out bug ridden code that is unstable and will most like crash any environment it gets deployed to. It also means that the testers can’t rely on old archaic testing methodologies that involve a lot of testing at the end of the dev cycle but don’t necessarily lead to better quality release! Similarly, the deployment can no longer be a set of manual steps that can’t be repeated unless handed over to “Dan on 6th floor who is out sick today”.

Continuous Delivery relies upon automation (and not the traditional automation).

Continuous Deployment, on the other hand, asks us to boldly go where many have gone before – to the world of automated deployment all the way to the production.

CONTINUOUS DEPLOYMENT

Continuous Deployment is the next step to Continuous Delivery. Once you have established good automated processes to develop, test and deploy every code change in production like environment  within acceptable quality threshold, you can take the plunge and auto approve for deployment to production.

Neither Continuous Delivery nor Continuous Deployment is about swashbuckling developers deploying half-baked  ideas to production. On the contrary, it is about highly disciplined teams that are mature enough to communicate with each other, committed to standing behind their work to deliver high quality (not just well tested) software for end users.

 

 

 

Continuous Delivery Vs Release Automation

Many people confuse Application Release Automation (ARA) with Continuous Delivery (CD). Many people confuse Continuous Delivery with Continuous Deployment but that will be discussed later.

ARA automates deployment on a target environment. ARA tools primary purpose is to push the build with a single click. As long as it can do that, in a repeatable, predictable fashion, the ARA tool has done its job.

Continuous Delivery however has much more to it than just pushing the bits out. CD can only be achieved if you can build, deploy and test every change while providing feedback at every stage. The fundamental principles of CD demand constant feedback and an almost sacred oath to not pass the defects forward.

Here’s my view on RA vs CD (inspired by Nhan Ngo’s sketch)

Screen Shot 2015-03-05 at 5.17.26 PM

 

 

Is DevOps Just a “Rehash”?

Gartner refuses to define DevOps as a segment or a market. Forrester takes largely the same view though they have started to work on ARA (Application Release Automation) as a market.

DevOps, according to many enthusiasts is not about using new tools. It is about changing mindsets and the culture within an organization. Developers working closely with testers and operations staff and everybody owning the application collectively is not such a revolutionary idea provided you understand the history of enterprise software.

rehash

Earliest software was written on mainframe by the folks who both developed it and then provided support to their customer. As software grew and developers’ social skills were put to a test, it was fairly obvious that we needed a new breed of technology workers who will empathize with the end user and will dedicate their time in supporting their needs. That was the birth of operations.

DevOps has mashed developers (and testers) with operations again and tools that we considered to be old and archaic are suddenly at the centerstage again.

Lets take a look at APM. After rapid growth in late 90s and all through 2000s, the growth in this sector crawled to almost zero. With DevOps, resurgent operations teams started looking at the performance data again and for that they revived their APM tools. The result was the rise of the new vendors like New Relic and App Dynamics even as legacy APM providers continued to take a beating in the market.

I am observing the same trend in Test Automation. DevOps demands IT to move closer to Continuous Delivery and one of the biggest hinderance to that has been old automation paradigm (More on this in next post).

DevOps is not a new space or a market – it is a better model, a new philosophy on how to leverage tools (some old and some new) and how to change paradigms that have brought IT to a stage where more time is wasted on upfront planning and downstream fixing of critical issues rather than developing new and better quality code that could be called “production ready.”

BTW, going back to the title of this post, somebody decided to rehash coding. At Livecoding.tv you can watch developers code for hours! Try it. Let me know if you like it.

Velocity without risk

There’s an age old saying in the world of Project Management – every project has three aspects – Risk, Cost and Time. The paradox is that you can deliver on no more than 2 at any given time.

DevOps however, promises to deliver all three – provided the practitioners understand what what each means. In this post, I will talk about what Release Risk means in today’s world.

  1. Release Risk talks about Integrity of the “Release Package” – a blackbox of disparate assets/artifacts that may be deployed to hybrid environments but together make up the whole Release Package. Enterprises today can guarantee that a particular Release artifact (lets call it “Deployment Unit” or “Release Unit”) was tested and released but as far as the whole package is concerned, that remains a mystery.  Ability to create this immutable object, promote it from environment to environment and report on it is the key to determine Release Risk.
  2. Historical Data – how successful/problem ridden this particular Release has been determines how risky it will be this time around. Collection and display of historical data is a must to determine the risk of this release. 
  3. Number of changes – The (new) content in the release is a critical factor in determining the risk. The greater the number of changes, the more risky the release is.

The three factors above, combined with the preliminary test results give a fairly accurate and quantifiable Risk associated with your releases.

Who is buying Application Release Automation

In one word – The operations guys (Directors/Leads/VPs)

While ARA refers to applications being deployed on target environments and hence should be the focus of the developers, the fact is that most IT organizations see this as an Ops function. Their logic is simple – any deployment to the production environment is a “change” to that environment and hence has to be managed by Infrastructure and Operations (INO) team.

Developers still influence the tool but it is the operations team that doles out the budget for any such tool.

Surprisingly, too many vendors don’t understand this nuance and keep positioning their ARA tools to developers – who have an opinion but little purchasing power in this space.