Like many companies with remote staff, it is not always easy for the team to share our learnings and experiences. To help build community and share those lessons, we have a weekly tradition of holding a Continuous Learning meeting where engineers, designers, sales, and leadership all come together to learn from each other.
During the meeting, we show-and-tell our work, demo new tools we dig, or host work-related book clubs. It is one of the few times where the entire team convenes and something I know many of us look forward to at the end of the week.
The Phoenix Project
Recently, we read The Phoenix Project, a novel published in 2013 — by Gene Kim, Kevin Behr, and George Spafford — about the growing but still esoteric phenomenon of (at the time) “DevOps.” The Phoenix Project takes a stylistic page from Eliyahu M. Goldratt’s The Goal — a novel written about management philosophy set in the backdrop of manufacturing. The authors of The Phoenix Project set up a clever (if not somewhat overwrought) approach to take a dry topic and, using the novel format, make Information Technology a more human, more compelling, and more accessible story and series of lessons.
Only five years ago did IBM’s CEO tell employees to adopt a DevOps and cloud-based mindset. Seven years ago, DevOps was the Wild West.
Ownership, not fancy project management processes or tools, is the key to successful teams.
But it is now 2020. The authors’ visions of a continuously delivered future have made themselves manifest through digital transformation of cutting startups and enterprises across the globe.
During that time, developers pushed for quicker release cycles and Information Technology sought out new workflows and tools in the midst of relinquishing hardware control in favor of harnessing cloud computing. It is the clash of those two practices, which we now refer to as DevOps, and which the book explores in a somewhat Not Invented Here fashion.
For Simple Thread, the book presented a great opportunity for us to discuss the historical processes of enterprise IT, while also affirming patterns we encourage with our clients around continuous integration and continuous delivery (CI/CD).
Continuous Delivery means you ensure every change can be deployed to production. Continuous Deployment means you deploy every change.
— Martin Fowler (@martinfowler) May 14, 2015
What follows are some of our team’s favorite learnings from the book.
The Theory of Constraints
Director of Engineering Alex Baldwin shared that one of the most important lessons he took from the book is Goldratt’s Theory of Constraints. In short, the axiom is that “any improvements made anywhere besides the bottleneck are an illusion.” Said elsewhere “a chain is only as strong as its weakest link.”
Through the main character’s trial and error, The Phoenix Project shares practical learnings of Goldratt’s principles and the ways to make improvements against the bottlenecks.
Keep Calm, Work-In-Progress On
One of the biggest practical takeaways from the book was that when we’re feeling overwhelmed, we should take a breath and evaluate our work in progress (WIP). Are there small things to prioritize, finish, and get off our plates? Are there large things we can reset expectations on and postpone? As Simple Thread founding partner Al Tenhundfeld said “often, though not always, when I’m feeling overwhelmed, it’s not really a matter of how much stuff I have to do. It’s a matter of how many things I’m trying to focus on at once, i.e., WIP.”
Think about taking a hiking trip; it is usually not the size of the mountain, but the steepness of the path you’re currently climbing. Worrying about what’s on the trail two miles up the mountain doesn’t yet matter, focus on the footing of your next five steps.
Review Twice, Ship Once
Continuous delivery can sometimes feel at odds with continuous improvement. Our engineer Caitlin Munley reminded us that holding those two concepts in balance isn’t always easy.
“It is important to identify bottlenecks and increase flow but not at the expense of passing defects downstream,” Munley explains. “This sounds obvious; but when there’s pressure from upcoming deadlines, it’s tempting to pass ‘defects’ along for Quality Assurance to deal with — who may or may not catch them.”
Our founder Justin Etheredge wholeheartedly agreed, saying “that is the biggest differentiator between a junior and senior engineer. A junior engineer will often get something to ‘feature complete’ and declare themselves done. A senior engineer understands that nothing is done until it is working in production, and therefore optimizes for the overall flow of the team.”
In the end, shipping defective code knowingly is not worth the time wasted by QA or Ops for it to only come back to development anyway. Even if there is not a formal process, having some quality control at every step is important for things to run smoothly and be set up to scale. Bottom line: a quality code review process is extremely important.
Culture Matters
Throughout most of the book, we see a clear anti-pattern of team trust and psychological safety at the fictional organization, Parts Unlimited. But what are the right patterns to which technology and business leadership should aspire?
Trust, empowerment, and ownership is what allows teams to succeed.
It is important to allow for space for a developer to not feel pressured into sending questionable commits down the line even in crunch time; there has to be a culture fostered and encouraged by leadership which allows developers to know they won’t be punished for doing the right thing. As engineer Mike Hartman put it, having a team whose goals are at odds with quality is problematic. Otherwise, “you just end up with people flushing them downstream to make their own numbers look better because that has more impact for them than the overall damage done to the operation.”
Prioritize Humanity
Code reviews can be a vulnerable process for the developers. Engineer Sam Ehlers suggests being intentional with language during the process. “I’ve always had some hesitation about being too brutal in a code review. Having been on the receiving end it doesn’t always feel great,” explains Ehlers. “So if I find I’m dropping a lot of comments or questions, I try to soften and humanize them with a one-on-one chat or even pairing session. Synchronous communication is extremely important if there are reply ping-pongs happening.”
Old Dogs Can Learn New Tricks
One of the interesting things that happens towards the end of the book is that a new project is started (Unicorn) to help provide some new value back to the project under the focus of the book (Phoenix). Even though the codebase and infrastructure is not that old (a few years) we start to see engineers on Phoenix are learning and adapting the more modern continuous deployment patterns used by the Unicorn team on their own Phoenix codebase.
When working on legacy codebases, our friend Scott Ford of Corgibytes often asks “would a doctor treat you using only medical knowledge available in the year you were born? When working on an old house, would you limit yourself to only tools available the year it was built?”
It might not be immediately obvious, but modern techniques can be applied to legacy codebases and doing so will only extend the life of the application.
The Permeability of IT and Operations
In the last chapter, we get a bit of a reveal of the authors’ long-term hopes; a future in which general business operations and IT are more integrally linked. As the CEO explains to the main character:
You’ve helped me see that IT is not merely a department. Instead, it’s pervasive, like electricity. It’s a skill, like being able to read or do math … Understanding what technology can and can’t do has become a core competency that every part of this business must have. If any of my business managers are leading a team or a project without that skill, they will fail.
The Phoenix Project
In this vision, IT is embedded into all aspects of business and there is less of a distinction between business and the technology that empowers it. Since our work often puts us directly in partnership and service to empowering users and customers on behalf of the business. We are always excited to see the connections between business, design, and engineering draw closer.
All Together Now
The Phoenix project really has become a classic book in our industry. Years ago when it was released, it provided a wonderful glimpse into how most companies at the time had engineering and operations departments that were working with diametrically opposed goals. Operations was measured by how stable a system was, and engineering was measured by how much they could ship. Engineering owned the application, and operations owned the environment.
With the help of hindsight this situation seems a bit ludicrous.
- If engineering doesn’t have ownership of the deploy, then how can they possibly have a clean handoff to operations?
- If operations doesn’t have any stake in the business value of the application, then how can they get on board with the constant change that the business needs?
- If engineering doesn’t get paged when the system breaks in the middle of the night, then why would they push back against tight timelines that cut into their testing?
As time goes on, more and more businesses are seeing that this dysfunctional relationship will never allow the delivery of great software.
Where The Phoenix Project really shined was that it introduced a number of practices that can be used to meld development and operations (DevOps, surprise!) into a single team all rowing in the same direction. These practices are important, but the real transformation that occurs is the eventual trust that is put into the team to set their own standards, choose their own tools, and have a seat at the table when discussing how to deliver business value.
Trust, Empowerment, Ownership
Many people will come away from this book focused on the DevOps practices that they want to implement, when we think it is the trust, empowerment, and ownership supplemented by these techniques that allows teams to succeed. Until businesses realize that ownership, not fancy project management processes or tools, is the key to successful teams we will continue to head down similar (but maybe more efficient) paths to failure.
Loved the article? Hated it? Didn’t even read it?
We’d love to hear from you.
