Which is better: large feature branches or merging frequently? Which is better small work items vs large work items?

Question

My team has problems pushing work items through all the stages in the Kanban board to done.

The work items hit code review and pile up and stay there for a long time. The following stage (the QA stage) is starved of work until suddenly everything is in QA and the testers have too much work instead of too little. The testers complain that they are expected to test all the sprint's work items the last day of the sprint. However it is worse than that. It is more like two or three sprints' worth of work items on the last day.

Our scrum master says that the work items are too big, and wants to break the items down further.

Our Manager says that team members need to review each other's code more quickly and more frequently.

Our Team Leader (me) says that there is too much work in progress and we need to enforce column limits at each stage.

The bigger problem is that solving the three above smaller problems is necessary but not sufficient in order to change the outcome. The way the team treats collections of work items as indivisible renders the other problems moot.

The team is divided between the old guard and the new guard. The old guard has been with the company more than ten years and the new guard less than one year. While the new guard breaks work into smaller pieces and pushes each work item though the pipeline one at a time, the old guard refuses to move work items from review to QA until all items in the same epic have been reviewed. Worse they refuse to merge their code into the main branch until they are ready to move their items to QA, this causes merge hell when everyone's code is integrated into the main branch and errors are frequently introduced during the merging process.

The old guard is resistant to changing to a process with more flow. They complain bitterly about the new guard, saying that they are not doing things the right way. Complaining that the way the new guard frequently merges their code makes it difficult for the old guard to integrate their code into the main branch without merge conflicts. They also complain that in depth review is impossible without reviewing the entire epic as a whole. They also complain that they can not release the software if it contains a partially completed epic that is merged into the main branch.

The old guard have reached a local efficiency maximum of sorts where any small change will make things worse and only radical change will help. To improve the outcome there needs to be a lot short term pain for a lot of long term gain and I don't see a lot of appetite for paying that price.

Personally I prefer timely piecemeal feedback from reviewers and QA over delayed comprehensive feedback. I am also skeptical about how in-depth their epic wide feedback will be as my experience so far is that the feedback is poorer and more delayed the larger the pull request.

The merge problems are worse the longer the branches are kept unintegrated. I know a lot techniques to break dependencies and to keep new features hidden from customers, however all of those techniques are based on the new guard way of doing things. I don't know anything that will help if they want to keep doing things the old guard way.

I have made a lot of small incremental changes to how the team does things. However I have run out of small things. Real improvement is going to require bigger changes.

I feel the biggest blocker is the insistence on getting all of an epic reviewed before merging to the main branch. And the biggest blocker to changing this is fear of delayed releases due to partial completed features.

So which is better: large feature branches or merging frequently? Are there any arguments pro or con that I missed? Is there any way out of my dilemma?

The pushback from the old guard against my changes is significant. There is support from on high for some of their arguments.

Answer 1

There is a lot to unpack in your question, but from what I read I see two main issues:

• the old guard (as you name it) doesn't want to change their way of working. What they did might have worked before, or not, but they found themselves a certain pace of doing things and the changes you want to bring forth are not to their liking. Since you mention a new guard, it means the team has grown, right? The old way of working might have worked with fewer people but now you need something else.
• the second problem is I think you are in a company that's trying to be more Agile but not willing to make the necessary changes to achieve it. For example, you say that there is support from on high for some of the old guard arguments. And you as the team lead seem to be ignored? I am all for collaboration and listening out for everyone's ideas and input into matters, but as the team lead you have the last word. Apparently you do not have the authority also. And as an aside, and if you don't mind me saying this, your main pain point seems to be the merges and the conflicts, so why the heck are people "from on high" meddling into this? This is a pure technical issue for the team to resolve.

Now, to get back to the questions in the title:

Which is better large feature branches or merging frequently? Which is better small work items vs large work items?

The answer is small work items and frequent merges. This is not a preference of mine, it is something that has many advantages and is a subject for a few books on continuous integration/deployment/delivery or on why Agile advocates for incremental and iterative development.

You started your question by mentioning Kanban, but you then mention a Scrum Master, sprints, and you also tagged the question Scrum. In Kanban the idea is to have flow, to keep things moving. You keep things small, you use WIP limits, you build things, you deploy them. Rinse and repeat. This increases your throughput, and minimizes delays and dependencies. The longer you keep things around, the more delays you will have and the more dependencies will be kept alive while someone reviews stuff, or waits for other stuff to happen first.

One thing that make no sense whatsoever is why delay all work inside an epic. It's an all or nothing approach. If your epic has functionality ready then that should be released independent of the rest of things in the epic. The fact that you work on epics in a sprint is even more worrying and a sign that work decomposition isn't properly done. Items are to big to work on and get delayed as they all finish around the end of the sprint.

Another issue is this:

The work items hit code review and pile up and stay there for a long time. The following stage (the QA stage) is starved of work until suddenly everything is in QA and the testers have too much work instead of too little.

You do not have collaboration here. None. Stuff is being thrown over the fence to code review (CR) or to QA. Why is the CR delayed? Are people busy with something else or they don't like doing CR? Why is QA waiting until the end of the sprint with little work during the beginning of it? These are obvious signs of phase development within your sprint. Not good. Worse still, people are not swarming to unblock the CR and help QA. They've done their part, now it's someone else's problem. Again, not good.

The longer things stay un-merged, the more difficult it will be to integrate the work, and the bigger the chances of introducing bugs when fixing merging conflicts. If you are doing this at the end of the sprint then things are no longer fresh in people's minds and again a source of potential merging errors. When you do merges often, everything is fresh in people's minds and you have less surface for potential problems. And if something breaks there is again less code to search through to see what's going on, since the last merge probably caused it. When you wait for merges like it's some exceptional event that needs to be done only once, then people get incentivized to work as fast as possible to be the first person to merge, to then let everyone else deal with conflicts later on. It's a "not my problem anymore" mindset, which you don't want to have in a team.

Some things to consider:

• identify as many issues as possible and discuss them at the retrospective;
• find better ways to split the work. See INVEST.
• find a process to handle the merges. I've used this one (slightly modified) without any issues whatsoever in a team with nine people.
• realize that merging is not an exceptional event, it's part of everyone working together. Don't be scared of it (unless your source control tooling is bad - with Git for example that's easy to do)
• add WIP limits to your columns and stick with them.
• If something gets blocked focus your efforts on unblocking it before working on something else (even if you have not reached WIP limits).
• try to gain the support of your Scrum Master and Manager on making the changes (you already have the new guard on board).
• finally, lay out the new plan and ask for everyone's support in making it happen. Some will like it, some will not. Prepare to loose some people and even be prepared to let some others go. Don't go full on though at first, make the changes gradually. Start with splitting the work in smaller pieces and having some merge procedures in place to happen sooner rather than later, then try to increase collaboration with QA to minimize handovers.

As I mentioned in my comment above, you might not capture everything you need to do with just one question (my answer itself is all over the place because there is a lot of things that need to be done) so maybe ask further questions for clarification or ideas.

Answer 2

Long-Lived Feature Branches and Infrequent Merging are Anti-Patterns

Which is better large feature branches or merging frequently? Which is better small work items vs large work items?

This question has both engineering answers and project management answers. In general, though, the answers will be similar for both disciplines, although perhaps for different reasons.

From a project management standpoint, and especially from an agile perspective, the INVEST mnemonic lends itself more towards small, testable work items that can be quickly integrated into the main line of development. Additionally, the agile principle of keeping a product in a potentially-releasable state at all times (or at least at iteration boundaries) argues against large or long-lived feature branches that may be difficult and time-consuming to merge. Work that diverges significantly from the potentially-releasable branch can't be quickly tested, or quickly converted into even partial earned value; that makes it a poor choice from an agile perspective.

Additionally, queuing and batch theory strongly suggest that small units of work are more efficient from a lean flow perspective. If you think of small, short-lived branches as work units, it becomes obvious that they fit better into a modern flow paradigm than large work items or long-lived branches.

While engineering questions per se are out of scope for PMSE, the modern received wisdom from a project management perspective is to embrace:

• Short-lived branches that represent small, atomic units of work.
• Routinely merging that work into a mob-development or per-iteration development branch to ensure that it can be easily integrated with other work being done within the iteration.
• Continual integration testing of the development branch.
• Continual testing of the development and release branches to guard against regressions.
• Using feature toggles or other engineering techniques to ensure that the product's main line of development is always in a potentially-releasable state.

This certainly doesn't represent the only possible approach or set of techniques. There are always edge cases and unusual circumstances, so a fully-canonical answer isn't possible. However, long-lived feature branches and infrequent integration are such powerful anti-patterns that I'd be hard-pressed to recommend them for anything other than political reasons. Such reasons are more likely than not to result in technical debt, which in turn creates long-term drag on project delivery and product support.

Answer 3

There is no single answer to this question. The purpose of branching is to facilitate parallel development. But exactly what kind of parallel development you need to support could differ from company to company, team to team, and even project to project. It's like asking "what kind of vehicle is best?" Well, that depends on your needs. Do you need to haul four kids to school? Tow a boat? Win a drag race?

There are costs and benefits to every branching strategy. In the modern world of fast-paced, continuous delivery, you'll find many proponents of trunk-based development with feature toggles. That strategy works great for countless software projects across the globe. There's a reason it's the go-to branching strategy these days. But if you lack sufficient mechanisms to catch defects before they make it to the mainline, or your team lacks the discipline to implement the strategy correctly, you're likely to cause at least as many issues as you're preventing by blindly introducing trunk-based development. Or if you're developing software for airliner or medical device where bugs could literally cost lives, you need to think long and hard about what kind of things you're introducing into the mainline. I know I wouldn't want to fly on an airplane running code that Steve the intern checked in yesterday, even if it's supposedly toggled off.

There's a paper I really like called "Streamed Lines: Branching Patterns for Parallel Software Development". It was written in 1998, prior the wide adoption of distributed version control systems like Git, and prior to the modern internet era, but it should give you an idea of just how deep the topic of branching strategies goes.

So, I suggest you take a step back and really try to understand what your team is trying to accomplish and what problems you're trying to solve. There's likely some kind of compromise to be made that can address their concerns and yours, but you're unlikely to find it in an answer from an outsider.

And lastly, unless you're in a position to impose your will on the team, you need to prepare yourself for the possibility that you may never change the way this team works. They could be 100% wrong and you could be 100% right, but that doesn't mean that anything will change. If you're not okay with that, you may just have to look for a job where the workflows are more in line with the way you want to work. And now you have a better idea what kinds of questions to ask in a job interview!

Answer 4

The long-lived feature branches and the merge conflicts are both symptoms: you have a lot of people working on the same codebase, but coordination is lacking.

Now there are two ways of dealing with this:

• take the most recent version from the main branch, make the minimal change that implements the interface you need and that doesn't break anything else, and merge back before anyone introduces a conflicting change
• split off, develop your feature, and deal with merge conflicts afterwards.

The first approach reduces merge conflicts, but also has an annoying tendency to leave unused interfaces around when requirements change during development of a feature, because they have been committed already and removing them would require someone to verify that they haven't in the meantime been used by anyone else, while leaving them in only requires leaving them alone.

The second means that code that was superseded during development will not make it into the main branch, but it causes an annoying merge that is visible to project management as "wasted" time.

But: both of these are mitigation strategies for failed planning, because the first point that any project coordination takes place is during the merge, when it should be way earlier. Changes conflict because two teams felt the need to change the same component, and these teams should have been talking to each other.

Answer 5

Is there any way out of my dilemma?

You write about the "old guard":

They also complain that in depth review is impossible without reviewing the entire epic as a whole. They also complain that they can not release the software if it contains a partially completed epic that is merged into the main branch.

It sounds like the old guard may actually have some reasonable objections--if the new people are pushing in changes that are not releasable and are dependent on other not-done-yet changes. Also, just a guess, you probably don't have automated testing to a level that inspires confidence--so any kind of incremental changes could potentially break existing features, requiring a lot of re-review and retesting (fixing that is discussed thoroughly in about five million other questions).

1. Possibly your stories are not split up well, so that they are not independently releasable. (If you have a "back end" and "front end" story, but only "end to end" testing then that is a big problem, for example.) This is probably the easiest to solve, in some ways, because you can solve it at the team level by breaking up your epics in better ways.

2. Possibly there are some business processes that are not well suited to agile. If the business side only considers something "potentially releasable" if it is fully featured, vs. having incremental releases that build on one another. (Honestly a lot of people believe that their stuff couldn't possibly ship incrementally when really it could and would be a huge improvement.) You need strong business people who can persuade upwards of the value of (much) smaller potentially releasable increments. Fortunately with SaaS/cloud/etc., a lot of people are getting used to seeing small changes instead of the "big bang" approach, and the perils of "big bang" are well documented.

3. Possibly you have something that really can't ship incrementally, for some reason. In my case it was because of external dependencies with unknown/subject-to-change shipping dates--if the external dependency isn't there, my side can't possibly work. Maybe you have some other situation like this. In that case you can either implement feature flags (as discussed in another answer) which does work great for many cases (and I used that some of the time), or use multiple, minimally overlapping, shared branches and multiple test environments.

Multiple minimally overlapping, shared branches with multiple test environments requires somewhat more work and more discipline, but it did work really well for me for some cases. You have your main branch which contains whatever is currently in production, then your separate branches X, Y, Z... for features that can't commit to being released together. Everyone working on X checks small, testable commits in to the shared X branch, with no fear of polluting Y, Z... With separate test environments the X changes can be tested in parallel with Y, Z... Whenever X does get released, you merge it back to the main branch and then merge the main branch to the other unreleased feature branches. Since these are minimally overlapping the merges are fairly easy. It does require some extra testing at this point to confirm that the merges were all good and all the changes ended up merged--but testing feature flags also has some cost. It also requires the discipline to make sure that changes get merged to every unreleased branch after a release. (And it requires minimally overlapping, shared, and separate test environments -- did I mention that?) Note that even though X branch may get large before it is ready to merge, individual commits to X can still be small.

Answer 6

Good question and many good answers, including those that highlight that there is no one single correct answer :)

Many replies do stress that iterating in smaller chunks allows for a better flow, though coupled with lack of reliable planning (or mid-way change of plans - which is a large factor of what agile is about) it can lead to mess in code and project.

Having tried several approaches, our team settled on a mixed approach (like, using it for over a year without complaints): many of the sprint tasks are identified as non-breaking and needed by shared product baseline. These small features are iterated by (usually one) developer, and merged to main branch with a pull request after code review and quick developer testing ("this does what I expected it to"), and go to QA ("this does what users might want"), and become part of the updated shared baseline.

There are other cases which are deemed to require a feature branch and those can be or not be pathologically long-lived. Probably one common clue is when some piece of development requires coordinated changes in several components, possibly such that make them not compatible to something provided in the shared codebase (e.g. change of low-level library to provide a new call, so builds of updated consumers against master branch won't see it). These may also be linked to changed layout of the resulting product bundle, such as adding third-party dependencies into the image.

Main purpose of those is to either isolate experiments that the product team might evaluate and decide whether to include in the product at some point or abandon and try something else, or "breaking changes" with large impact on the codebase. Some such feature branches are indeed due to epics, where several teams should make consistent changes (e.g. backend, UI and integration tests). These are still iterated in small chunky PRs, just that those land to dedicated branches in numerous git repos, and produce a separate final product bundle that QA can test and allow the feature to be merged (a spree crossing dozens of git repos in a short timeframe, sometimes).

In good cases, our main branches and resulting product bundles go from old green to new green state, with commits landing for product and CI code alike, and with front- and back-ends matching each other. Maybe even docs and help are updated in sync ;)

In bad cases, this allows us to abandon "experiments" without impact to shared codebase for the published product. It can be not only due to PoC failure, but also because customers or PMs changed their mind or agreed to something else while devs were busy.

For several times now it also allowed large refactoring to be made by part of the team (like, changing a library, or moving to another OS distro) without keeping the shared sandbox unusable for weeks in a row for the whole extended team - this is still a coordinated and communicated change, so merge conflicts are minimized... but just as well as an experiment, if a refactor goes too wrong or way over its time slot, it is not a problem for main product baseline to suspend or abandon that effort. Time might be wasted, money... lessons learned. But we can come to work tomorrow and continue bringing value added over the working product baseline, and not spend a week reverting breakage.

A large part of minimizing the merge conflict here is that the main branches are trustworthy to the team most of the time (as proven by regular CI), and they represent a baseline that indeed can be delivered to customers or QA. Ideally, at any moment, perhaps at start of sprint, the active feature branches which still exist can be rebased over main branches (or merge their changes in) so even if there are conflicts - they are solved semi-frequently, usually are few in numbers and with well-understood context differences. As a consequence, feature branches represent the current product plus one feature, and can be QA'd as such until they become part of the current product.

Being, in case of successful build, complete (although custom) product bundles, our feature images can be demonstrated, or delivered to users (as a hot-fix with its well identified place in git history and support cycle). Technologically speaking, regular numbered releases are also just another case of a feature image. They can also get some commits to fix issues, they can also "backport" some fixes to shared codebase if that still makes sense (code is a moving target, maybe the main branch does not have that part of code at all now).