Create a clear, shared understanding…
of parallel development streams and overlapping releases from a shared codebase across multiple development teams
Situation
I was the lead architect on a multiple team agile project producing three versions of an application built using both build and design time feature flags. These applications were tightly integrated but deployed in physically separate environments (networkless laptops, public internet, and classified internet) with strictly defined formats for integration via import/export of flat files. There was ongoing development to support emergent fixes in production, short term functional enhancements deployed approximately on a quarterly basis, and longer-term changes that involved significant architectural upheaval which defied use of simple feature flags.
Objective
A branching plan was needed to allow development teams to add features and fixes in different work streams targeting different releases. To ensure CI/CD functionality was preserved the developers and build engineers needed to have a clear, shared understanding of the development goals and targets, where different versions of the application were to be deployed for testing, and which releases and code branches would be automatically deployed in which environments. Production deployment often required transfer of physical media into secure environments.
Action
I’ve found that communicating basic direction such as “we’re using trunk based development” or “we’re using git flow” is not useful for communicating CI/CD needs to build engineers, does not cover key issues such as timing, and leaves too much room for ambiguity across multiple teams and development streams. Instead, I devised a notation for describing branching and tagging, use of CI/CD environments, deployment to the physically separate production environments, and added an overlay of the sprint calendar to depict timing. This communicated when CI/CD environments would be needed, when branches would be cut for code freeze, when local and global test events would occur, and when releases would be deployed to production. All of this information was shown on a single page diagram used by feature teams, build engineers, acceptance test teams, deployment teams, and project leadership to plan and adjust schedules as needed when there were sudden changes in priorities or constraints.
Result
With a single reference point to keep all teams (literally) on the same page we were able to work multiple streams and releases, identify critical dependencies and constraints, and communicate the complexity driven by new or adhoc release requests while also managing our work so as to meet the goals of the client organization. Going forward I was able to leverage this same notation and diagramming technique when facing similar challenges at other clients.
About reducing the need for (and cost of) multiple CI/CD environments: This can be especially difficult to manage with complex cloud based infrastructure. It is important to note that managing multiple branches of development can drive cost and complexity in the integration environments, so the simpler approach of feature flagging is highly desirable wherever possible. This notation and approach can be adapted to reflect feature flags rather than code branches. That said, when the nature of the changes makes feature flagging impractical I’ve found this approach to provide much needed clarity to everyone involved in the development and deployment efforts. This notation is useful also in identifying and communicating when CI/CD environments can be deactivated or removed to reduce infrastructure cost.