Automating a SAFer OnceHub

Recently on twitter there was a discussion about SAFe, the Scaled Agile Framework, and why it isn't really agile and why Agile methods and their certifications are all horrible. No link is provided because such a conversation seems to repeat itself every few months.

There are three common objections to SAFe.

1. It's adopted by companies that prefer a command and control structure, which tends to be the antithesis of an agile culture.
2. Because of all the layers and some graphics they provide, people believe the customer is lost.
3. Some practices which SAFe modifies are not as good as the original idea which inspired SAFes version of the practice.

The third point is an interesting discussion, and as a company embraces continuous learning and continuous improvement they can look deeper into those practices and decide for themselves which version best fits their culture, but points one and two are more critical and we believe are worth having an open conversation about it. For full disclosure, we have adopted continuous learning and so we don't follow SAFe exactly by the book, but have implemented modifications that fit our company.

A few years back, OnceHub adopted a good chunk of the SAFe methods, and we have grown and flourished because of it. We will explain some important insights we learned about SAFe and also explore one area where SAFe allowed for a technological change which reinforced a cultural change which further reinforced a technological change. I suspect this virtuous cycle will continue for the years to come as well in other areas.

The first, and likely strongest, objection to SAFe is that it is inherently "not agile". This is because SAFe allows a command and control company to make minimal changes to their processes and having called themselves "agile" they stop improving. I have no doubt that many consultants have been called in to help an organization when this happens. But it is the companies which do not call in consultants that I believe are much more interesting. While many will argue that this is SAFe's biggest failure, we will argue that this is actually SAFe's largest asset. The designers behind SAFe have found a way to collect the most useful practices which our industry experiments with, and package them into a process that allows strict command and control organizations to grow and adapt with minimal risk or disruption to their large organizations. Many other agile methods will never be adopted by command and control companies because they feel too threatening, risky, or foreign. Only after getting their feet wet, and experiencing the benefits of shorter iterations, regular releases, and more open communication and collaboration will these companies feel safe adopting agile practices that makes the company a better place to work.

Let's look at a particular example to illustrate the point. As we mentioned in our Kubernetes series, we went through a migration which allowed us to release every week, instead of an irregular pattern, with roughly three releases a year. In addition to a faster release cadence we also moved from a manual deployment process to aided Jenkins builds , and then finally in our latest iteration, a CD process making use of Jenkins X.

First, a brief and high view description of our transformation.

We knew as we migrated to microservices we would need an easier way to deploy than using FTP and SSH to copy our build artifacts onto VMs. Jenkins was the obvious choice at the time to help us do this as we created our build pipelines.

From the beginning we used Jenkinsfiles to describe our pipelines, and while we started with a unique jenkinsfile in every repo, we eventually built centralized Jenkinsfiles and Jenkins groovy functions to allow for cross the board improvements to our CI process such as adding sonarqube, or adding bare minimum test coverage requirements. After moving everything to Kubernetes, we then migrated everything from Jenkins to Jenkins X, and adopted a more gitops oriented process.

While many of these changes were led by our engineers, if it wasn't for the PI planning process from SAFe and strong support from our PMs, we never would have transitioned as quickly or smoothly as we did.

When we first adopted SAFe and did PI planning, it was very hard to get our non functional requirements prioritized as we would have wanted, and before SAFe, such requirements were even harder to prioritize over a wanted feature. SAFe recommends a 10 week, 5 sprint iteration where the entire company can work towards a shared goal. During the 5th sprint, or "Sprint 5" as we call it, no work is allowed to be done on new features and instead our time is spent dedicated to improving quality and processes. Yes, I can hear you asking why we didn't work on improving quality and processes during the other four sprints as well, and the reality is that we did, but we didn't do it with such focus.

The structure imposed by Sprint 5 and PI planning, really allowed everyone to become aware of how much our slow release process was harming our ability to release features. When we tried to shoehorn non functional requirements, into normal sprints and stories, the need became even more obvious.

These alignment calls eventually led to our PMs proposing improvements to our CI and deployment pipelines in words and stories that related to real customer value. We called it the 'Zero downtime epic'. Believing we would be regulated to working on zero downtime in sprint 5 only, our teams worked with PMs to find stories and improvements that would push us to prioritize features that aided in Zero downtime, such as push notifications when a new version of the site had been deployed, urging users to refresh the page. After reviewing the DORA report, we got further alignment from business to improve our release processes, pushing us towards faster releases, with higher quality, and less operating expenses.

Yes, it's true that we could have gotten the same results if we had "simply" opened up communication, and had more human relationships between departments. However, it's not always that simple. A space is needed to take a break, relax, reflect, and share ideas. PI planning provided that break and spaces without creating objections from higher management about utilization and efficiency. The SAFe processes allowed for that communication to grow, the ideas to flow between departments, and the transformation of a desire into real customer value.

If you are looking to improve your digital transformation, we highly recommend finding a method that already matches your current culture as much as possible, and creating high cohesion between your departments, while lowering the coupling between decisions and hierarchy.

If you haven't already had a chance, read our 6-part blog series on Kubernetes. Though not directly related, we feel it helped us be more agile, and allowed us the space to slow down, think, and breathe, and we hope these articles can help you along your journey as well.

Avi Kessner, Software Architect

Avi started as a Flash Animator in 1999, moving into programming as Action Script evolved. He’s worked as a Software Architect at OnceHub since September 2019, where he focuses on Cloud Native Automated workflows and improving team practices. In his free time, he enjoys playing games, Dungeons & Dragons, Brazilian jujitsu, and historical European sword fighting.