ABD: Always Be Delivering
How Continuous Delivery Unlocks Elite Performance at the Edge
The well-known sales motto ABC or Always Be Closing, popularized in the movie Glengarry Glen Ross, promotes aggressively closing deals at all times, rather than letting them slip away.
I promote a more positive & altruistic mantra for software: ABD or Always Be Delivering.
The idea is simple: continuously deliver the latest & greatest software to customers. The moment we add new functionality, fix a bug or improve the product - we should deliver it. Minimize the time-to-value, the time-to-recovery, the time-to-impact.
Deliver early & often. Not only does it minimize time-to-market & cost-of-delay - it produces better software. The “The State of DevOps Report” indicates that high-performing teams who produce highly reliable software are 46% more likely to practice continuous delivery.
Other related practices are key as well, ie. version control, continuous integration, automated testing, trunk-based development, loosely-coupled architecture, and a culture of high-performance collaboration. But in many ways, continuous end-to-end delivery is the pinnacle of these practices.
Continuous delivery doesn't come for free. It requires investment into culture, practices, tooling & automation. As the saying goes: go slow, to go fast. But the inflection point can come quickly. These investments can be made gradually to match a project's growing pains and accelerate its growth in the process.
In his book "Continuous Delivery Pipelines", Dave Farley summarizes several other staggering performance benefits from The State of DevOps Report captured by teams practicing continuous delivery:
"44% more time spent on new features"
"50% higher market cap growth over 3 years"
"8000x faster deployment lead time"
"50% less time spent fixing security defects"
"50% lower change-failure rate"
"21% less time spent on unplanned work and rework"
These are great outcomes for every software-powered business. Why wait? Why do so many companies take the opposite approach of deploying software into production as little as possible?
Common Objections & Challenges
Let's contextualize the discussion for those of us delivering software to edge devices scattered out in the real world. Here are some of the reasons I hear for not employing continuous delivery to the edge:
It's too costly or risky to update our devices all the time.
We don’t need to update our products frequently or at all. We’ll deal with it if/when it becomes necessary.
We need weeks of testing to update our systems. Failure is not an option.
We’re in a heavily regulated industry. Documentation & sign-offs are needed every step of the way.
We’ve always developed & released our software this way. It's infeasible to change or automate it.
According to a survey by Puppet, the leading challenges to adoption are:
Skills shortage
Legacy architecture
Organizational resistance to change
Limited or a lack of automation
I’ll spare you an overly generalized rebuttal to each of these points. In reality, a nuanced approach is necessary. Each company possesses some unique needs & challenges. We’d love to discuss any specific challenges you’re facing. Let's chat! ;)
I'll instead address some of the key principles & misconceptions and perhaps you'll see how the above objections can be overcome to achieve the benefits of continuous delivery.
Delight your users by continuously improving your product
Key Concepts
What is continuous delivery? It may seem more nebulous than it really is, but the definition is as simple as it sounds: deliver software continuously. Effectively practicing continuous delivery, however, is not so simple, as highlighted by the Objections above.
Releasability
Is there really value in delivering new software several times a day? Maybe, maybe not - that's not really the point. The goal is to minimize time-to-value.
A key principle here is to keep our software releasable. That may sound obvious, but I've seen many projects that are rarely releasable - where teams must spend days, weeks, or maybe months preparing their code to be released. Responding quickly to needs in the field can result in a mad scramble that sometimes leads to circumventing the release process altogether.
We can maintain the code base in a releasable state by qualifying every small change to the code base as it gets committed. Maintaining the code base in a releasable state requires performing a series of verification activities each time changes are being committed. If a change causes the code base to fail these releasability checks, then the change gets reverted. By keeping changes small, enforcing good practices, and leveraging automation, the software will always be ready to release while facilitating a high pace of development.
Small Increments
A major enabler of keeping software releasable is ensuring changes to the project are small and incremental. Batching large, complex changes not only slows development but complicates validation and makes releasing risky. In the worst cases, it spirals into a doom loop of progressively harder & slower releases as a team batches increasingly more changes and further delays subsequent releases.
Practices like iterative trunk-based development, feature flags, and branching by abstraction help break down big batches into smaller safer increments. Small increments are easier & faster to verify. They're also easier to roll back if a regression is introduced. And even if those changes aren't immediately deployed into production, developers can still get rapid & valuable feedback from code analysis, review & testing provided by their CI/CD pipeline.
CI/CD Pipelines
Continuous delivery requires significant technical & process discipline. Automation makes it efficient & repeatable. CI/CD (Continuous Integration / Continuous Delivery) pipelines are used to manage, automate, and enforce software integration, quality control, and release practices. Pipeline activities may include:
Static code analysis and review
Building, configuring, and packaging software
Automated unit, integration, and end-to-end tests
Security scans & verification
Compliance checks, records management, and sign-off points
Deployment to systems/devices
The software on an edge device is often the output of a federation of multiple projects & pipelines. Commonly, a manifest is used to version control the integration of all the components, and a master pipeline integrates the system, performs end-to-end testing, and delivers a complete verified system. Controlling the complete system as code is referred to as IaC (Infrastructure as Code) - or as I prefer in the case of edge devices: DaC (Device as Code). DaC together with the right update technology and practices allows treating software in the field as immutable, such that the only way to deploy software to production devices is via the pipeline. This ensures that software is centrally controlled, doesn't become fragmented or drift, and provides much better traceability & auditability.
I often depict such a pipeline like this:
Always Be Delivering
In reality, pipelines can take many forms and should be constructed according to the needs of their project. This pipeline is segmented according to the rules it is enforcing (ie. Dev, Biz, Ops). The goal isn't to create organizational silos in the pipeline - a pipeline should be able to flow freely according to the rules of the pipeline - but it helps clarify which stakeholders are engaged in defining those rules at various stages.
Each stage is monitored, recorded, and provides rapid feedback to developers & stakeholders. This pipeline allows developers to execute the dev-centric portion of the pipeline more frequently than the business can/wants to release to production. A fast CI cycle for developers maximizes the pace of their learning & iteration.
A well designed & instrumented CI/CD pipeline achieves substantially better:
Speed & Efficiency
Consistency & Repeatability
Quality & Compliance
MLOps
The need for continuous delivery is especially important when delivering AI solutions to the edge. Good fresh data is vitally important for training effective machine learning models. The world around us is constantly changing - sometimes slow, sometimes fast - but often in ways that will adversely affect our models. Therefore, it's crucial that we can observe the performance of our ML models in the field, detect drift, collect new data from the edge, and rapidly deliver refreshed & improved models.
Continuous Compliance
Although security, quality control, and compliance are often named as reasons for slow & infrequent software releases, many verification, traceability, and compliance activities can be automated to unlock dramatically shorter cycle times. End-to-end instrumentation & traceability allows for tracking software, model & data provenance, lineage & verification while a system is continuously improved & evolved.
According to Puppet's 2023 survey, automating governance, compliance, and information security policies is a priority for 41% of platform teams over the next 12 months. Minimizing change is no longer a viable compliance strategy as emerging cybersecurity regulations such as the EU Cyber Resilience Act mandate immediate cyber incident response and timely software updates for connected devices.
Even manual sign-offs can be greatly accelerated by automating the activities & information gathering required for sign-off to occur, including record collection, report generation, and verification activities.
Conclusion
These concepts are key to continuous delivery, but this article only scratches the surface. Achieving continuous delivery in your business will require deep introspection and a willingness to challenge & transform everything.
While I was CTO at Toradex, I drove our adoption of CI/CD & software test automation and with Torizon helped our customers adopt modern DevSecOps practices to securely & reliably deliver software to their connected edge devices. The transformation took significant commitment & investment, but the results are indisputable. Toradex now builds hundreds of embedded Linux images and runs thousands of software tests every day to supercharge its delivery of high-quality software to its customers & their devices while greatly minimizing manual test & release efforts.
The most important factor for success is to start the adoption and remain committed to the results you seek. It doesn't happen overnight. Culture, skill, speed & quality will improve as adoption deepens - accelerating delivery & innovation, improving operational resilience, and delighting customers. Elite performance is worth the investment.
ABD: Always Be Delivering.
Example CI/CD Pipeline
