1

I'm working on a project that has no CI/CD. We have only one person who builds the code and deploys it to staging and production. In the past there have been discussions on implementing CI/CD, but to no avail.

Our one person has become sick. I want to use this as an opportunity to bring CI/CD up again. What would be the key points to show how CI/CD could ease the pain of staff losses or extended absences when the staff member is unreachable?

New contributor
user223364 is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
1
  • CI/CD is about automation. Staffing levels and having T-shaped people or on-call rotations for business critical support are more business continuity issues rather than a CI/CD or project management issue, per se.
    – Todd A. Jacobs
    Jan 13 at 18:20
3

What you're referring to is not CI/CD* - it's automation of build & deployment procedure. Basically a Deployment Pipeline. The way it alleviates your problems is:

  1. You stop needing your Ops constantly. If Deployment Pipeline is done right in many situations it could work for years without the maintenance. This allows your Devs to build and your QA to deploy without the Ops team.
  2. Many configuration changes can be done by devs alone. Without the need to access the environments.
  3. Scripts automatically document the procedure. So if something happens to the Ops team, then either Devs or a newly hired Ops can figure out what's happening there and modify the procedure.
  4. Scripts are getting tested again and again. So eventually they do the job right all the time. As opposed to "our Ops is on vacation, so one of the Devs was deploying, but did it incorrectly and now the env is broken".
  5. Ideally environment installation/configuration is also automated. In which case creating new environments can be done by Devs/QA.

*CI/CD is also about:

  • The actual CI - meaning integrating changes made by a team in a single VCS branch.
  • The actual CD - meaning releasing changes frequently

These usually are much tougher to achieve.

2
  • I think your answer here is good at addressing how an automated pipeline could be helpful, but I really think the OP's issue is really more of an X/Y problem based on a fundamental lack of business continuity planning and reliance on a single siloed resource. While a solid CI/CD pipeline could help, I think an answer without a frame challenge to the OP's assumptions about root causes and solutions provides only a partial answer. What you've put here is certainly part of a good approach to implementing a pipeline-based solution, though.
    – Todd A. Jacobs
    Jan 16 at 4:02
  • I think on this SE we too often see verbose answers that don't directly address the question of OP. Sure, we could add more of our thoughts about the situation in addition to the direct answer, but personally I think it's optional. He asked for arguments in favour of implementing a deployment pipeline - I gave him the arguments. Also, I don't fully agree that "single siloed resource" is a bigger problem than the absence of automation, but let me comment under your answer about this. Jan 16 at 9:39
1

It's not the use of CI/CD that makes it easier to handle the absence of key staff. Instead, it's the practices that enable CI/CD. Practices such as automated tests, automated builds, and pair and mob programming.

Having automated tests means that you have tests captured as some kind of code or script. Code needs to be very specific in terms of what to execute, the order of execution, and what to expect. Compare this to test cases that are executed by humans - even if they are written down, a knowledgeable person can use their knowledge to fill in gaps. By automating tests, you have a clear and complete documentation of what the developers believe the system should do.

Automating your builds means that you have code or scripts to take your source code and create executables or deployment artifacts. Similar to tests, these scripts need to specify every step of the process so you don't rely on an individual's knowledge to fill in steps that may be unclear, ambiguous, or missing in written instructions. These builds can also include running those automated tests to be able to get more rapid feedback about the changes.

Pair and mob programming can be used to share knowledge across the team. Although it is possible to use pull requests to achieve the same thing, pair and mob programming can be seen as a real-time code review, reducing the asynchronous nature. Some people are also more likely to point out alternatives or problems if they are part of the discussions on how to do the work the first time. This knowledge sharing can reduce single-points-of-failure on the team. Plus, reducing the volume of asynchronous pull requests can get changes into the CI/CD pipeline faster, once you have one.

These aren't the only practices necessary to enable CI/CD. However, if you are looking to mitigate the risks of people being unavailable, these are the practices that would mitigate those risks and get you closer to CI/CD.

7
  • Pair and mob programming don't relate to CI/CD, I think they should be excluded from this discussion. Jan 13 at 17:57
  • @StanislavBashkyrtsev Except they do relate to CI/CD. They allow for multiple eyes on changes synchronously, which is faster than an asynchronous pull request. That makes integration of changes faster, or more continuous. If you'd like more elaboration, perhaps you should ask a question about it.
    – Thomas Owens
    Jan 13 at 18:31
  • Hm, you even say they enable CI/CD.. And I don't get where you're heading with the "faster" argument. Does everything that make development faster is part of CI/CD? I double checked the original book by Jez Humble. I surfed the internet. Myself I've been doing CI/CD for the last 10 years and spent couple of years as a CI/CD engineer. So far I have nothing to support your words. If you have literature explaining this thought in more details, please share. If these are your own ideas, then I'm fine leaving it there. Just let my disagreeing comments stay here so that others won't be confused. Jan 13 at 19:30
  • @StanislavBashkyrtsev While it could have been worded better, CI is not just a tool chain. It's also a philosophy & a set of practices around the notion of (ideally) non-stop integration. That makes practices like TDD, BDD, and pair programming at least relevant. Thomas' answer also addresses the OP's low bus factor by suggesting pair- & mob-programming to encourage T-shaped people & knowledge sharing. While these practices don't comprise a CI/CD pipeline per se, I think you're taking a single phrase out of context rather than suggesting constructive improvements to the answer.
    – Todd A. Jacobs
    Jan 16 at 3:49
  • @StanislavBashkyrtsev You may also have misunderstood Thomas. While you say "Pair and mob programming don't relate to CI/CD," Thomas said that "reducing the volume of asynchronous pull requests can get changes into the CI/CD pipeline faster." This is true if PRs that pass CI can't get merged/deployed without gating from a single (and perhaps absent) person without knowledgeable delegates. In full context, I agree with Thomas' points from a process perspective, although CI/CD pipelines don't intrinsically mandate pair/mob practices, code reviews, non-CI gates, or manual merging at all.
    – Todd A. Jacobs
    Jan 16 at 4:25
1

CI/CD Tools and Processes Don't Mitigate Bus Factor Risk

I'm working on a project that has no CI/CD. We have only one person who builds the code and deploys it to staging/production...What would be the key points to show how CI/CD would ease the pain of staff losses or extended absences where the staff member is unreachable?

This is not really a question of CI/CD per se. The issue is that you have a single point of failure: your organization has a singular resource responsible for manually building and deploying code. Even if there's some level of automation involved, it's not accessible to anyone else, so it doesn't benefit the rest of the organization.

Automation doesn't preclude the need for adequate staffing levels, business continuity planning, knowledge sharing, or succession planning. However, a robust and automated CI/CD pipeline allows the organization to "shift left" and empowers others beyond your singular resource to build, integrate, and deploy code without necessarily needing to understand the pipeline's internals for "business as usual" operations. That is generally a significant win.

However, CI/CD pipelines and tools need care and feeding just like anything else in IT. So while pipelines may reduce bottlenecks or single points of failure during routine business operations or for short gaps in coverage, they don't really reduce the need for risk management. Anytime you have only a single person capable of performing a job, your bus factor is exactly one. This represents organizational risk, and such risk must be mitigated, transferred, or accepted by the organization.

Any residual risk should be clearly identified to upper management, who may then refer it to your legal and risk management teams if you have them. In smaller organizations, senior management is usually the right place to start. Regardless of organizational size, senior management and the board of directors are ultimately responsible for managing the organization's risk appetite and right-sizing any residual risk.

From a project management perspective, your responsibility is to inform. If senior management ignores the issue and things break, then they get to keep both halves.

2
  • Anytime you have only a single person capable of performing a job, your bus factor is exactly one. - if you add more resources you raise the bus factor to N. If an Ops Lead has a conflict with the management, he can still leave and possibly take the whole team with him. But a well documented and automated process raises bus factor to infinity. Because you can hire/redirect other resources and they will be able to continue the work. Jan 16 at 9:46
  • Also, even if the dev process gets interrupted, most of the time it doesn't affect business continuity (unless there's some critical bug in PRD), so short interruptions (days or weeks) can often be tolerated by business. If the process is automated & documented, this should be enough to fill the shoes of the missing Ops. So from this perspective I think automation is more important than simply having more people who share the tribal knowledge. Jan 16 at 9:49

Your Answer

user223364 is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.