How to move towards Scrum when we don't really seem to be able to fit into 'The Scrum Guide'?

Question

I have previously asked a question on PM, which was not well focused, but I still received some useful tips. I do plan on putting some of the notions in front of my manager, but I would like to re-approach our issues here to see if I can't get any more insight.

I work for a mid-sized American company (300ish) on a very small, not officially/organizationally recognized development team (3). Only one of us has a degree in CS (and it isn't me). Our manager is an manufacturing engineer and has negligible experience programming, designing software, or managing a development team (except for the past two years of our 'covert developing'). On top of this, our developer with a CS degree tends to work on projects solo, separate from the other two of us.

We have an insane amount of projects our manager is looking to achieve. Essentially he is hoping to digitize most of our Quality & Inspection processes & paperwork, all the way to semi-automating our PPAP/PFMEA documentation. Frankly it is all way more than is chew-able by such a small, inexperienced team (in any sort of respectable time frame), in my opinion, but I admittedly tend to have a 'glass half-empty' attitude. So far our extended time frame hasn't been an serious issue to our company's President. Half of the projects are very large... our current project has more or less taken up about 70% of overall work-time for two of our developers, and has taken 2 years to develop, and still going (we had to restart the architecture and include unit-tests because we didn't know what we were doing when we started).

Now, our manager (who, to his credit spends a lot of time reading on relevant subject matter to try and understand our position as developers better) wants to try and work under a Scrum framework. I have taken on the role of Scrum-Master, our manager as Product-Owner, and of course the other two developers are... developers. Other mgmt. and our President would be the 'shareholders'.

My main issues that I am struggling with, and unsure how to approach are:

• Our 1 'team' actually consists of two separate development teams working on unrelated projects (mentioned in second paragraph) and either team can, currently, get our priorities changed independent of the other team's priorities as well. This seems like it should be 2 separate scrum teams but our manager/PO does not have the time for that, and I don't think I have the mental energy for it, currently. Also, how are we supposed to be accountable as a team for separately developed projects?
• Projects are all essentially interwoven into the same 'suite' and our projects kind of have blurred borders/edges.
• Even if we break projects into hard-defined boundaries... the projects are still quite large and our product backlog could be too much to navigate if it is fully populated
• Fully and accurately populating the product backlog seems daunting, as often times 'case-scenarios' are often overlooked by shareholders in favor of 'the ideal interaction' or they state outcome but struggle to define how to get that outcome even on paper. Sometimes processes get such a vague description such as : "The quality tech would look up the relevant specification and make sure the document conforms"... which you can't turn into programmable logic on its own, we aren't making magical PDF parsers.
• More on the previous point: Getting vague, generalized process outcomes from shareholders has required us to go and talk to individual employees that actually do the paperwork, often times the nuances in their process will slightly deviate the generalized mind's-eye picture of the shareholders, which then often triggers long, grueling conversations where we as the developers are just trying to get the shareholders to understand what we need to know and why we need to know it... this seems incredibly anti-Scrum and I don't see how we are supposed to essentially design an enterprise-level software-suite if we have to re-explain why we need to understand case-scenarios and the nuanced details of the process every time we outline a new software behavior. Is there a way I can get non-programmers to understand our needs better? Especially when I have a hard time outlining our needs because we are trying to digitize processes that often rely on 'tribal knowledge'
• We lack personnel to be able to handle documentation, training, and maintenance (we often have to drop development work to handle these "fires" as they come up)
• EDIT: some of my other points kind of speak to this, but how granular/detailed should a PBI be? Do all PBI's have to be finishable within a sprint?
• Also, can Scrum Planning, Review, and Retrospective be held without any PO or Shareholders if none of them have time?

Answer 1

That's a lot of information to digest my friend, I'm going to pick some items I've also encountered and answer them, hopefully someone will chime in to fill in the remaining blanks :)

Our 1 'team' actually consists of two separate development teams working on unrelated projects

It's possible yet not at all ideal to do scrum over multiple projects. The reason I'd absolutely stay away from doing it like that is the main reason scrum works: your sprint works towards a potentially releasable product increment. How are you going to do that with only 2 engineers when you need to constantly context switch between projects in a single sprint? I'd say that is a big no-no and you need to do all you can to convince your manager that this is not the way to go.

Build a product backlog (more on that later) for each separate project, determine the requirements for the "version 1" release and then plan out your sprints to reach that first goal: "version 1". In reality all your manager wants is to know when things will be done, if you can tell him "We'll be able to deliver stories 1 through 12 in 3 sprints, that totals 6 weeks" he'll be happy. After that you can move on to the next project or start work on v2 of the first project, etc ...

Projects are all essentially interwoven into the same 'suite' and our projects kind of have blurred borders/edges

This smells like a lot of technical debt, a failed microservices architecture? You should determine how to isolate your projects beter so they can be maintained easier and won't affect the whole "ecosystem" when they are updated. It's a daunting task but it has to be done if you want to keep your sanity in the future. Take it step by step, sprint by sprint, the most important thing is that you have a plan on how to tackle this problem, break that plan down in manageable steps and then plan them in sprints together with your regular work.

I've worked with a type of story called "Technical story". The product owner has no say over the priority of these stories and the development team can plan them in the sprint as they please, but at all times a sprint should still deliver a "potentially shippable product increment", so an "all-tech-story Sprint" is a no-go.

Fully and accurately populating the product backlog seems daunting How granular/detailed should a PBI be?

And you shouldn't. A good backlog respects the DEEP model. The thing you want to look at here is "Detailed appropriately". It's not a problem that the items at the bottom of your backlog are vague, you won't start working on them any time soon. If your product owner deems them important enough he should put them higher up in the backlog and then your team should pick them up in the backlog grooming sessions to break these large and vague stories (or themes, epics) into smaller and well detailed stories.

Your product backlog should look like an iceberg where only the tip contains fully detailed, discussed and documented stories, the deeper you go, the more vague and abstract your stories will be.

Do all PBI's have to be finishable within a sprint

Yes. Gonna slap another model on it again: Your PBI's should respect the INVEST model. The "S" in "INVEST" stands for "Small" which means that a story should fit in one iteration.

talk to individual employees that actually do the paperwork, often times the nuances in their process will slightly deviate the generalized mind's-eye picture of the shareholders

This might be a rather controversial opinion but imho you shouldn't go talk to the individual employees, that's something your product owner should do. They both also have a different goal. Your product owner wants to maximise the value you output each sprint, the employee wants to get his specific part of the work done.

This doesn't mean you should NEVER do User Research, but you should probably not directly talk about story scope with individual users.

Another thing you could try is changing your approach, your product owner cares about value, make the discussion about that. If the change you propose actually increases the value of the product increment or future increments you will probably have an easier time convincing him.

can Scrum Planning, Review, and Retrospective be held without any PO or Shareholders if none of them have time

No. Your PO is essential in all of those meetings (maybe he could skip a retro now and then but rather not). In the planning meeting the PO needs to be there to answer question and together with the team determine what stories are pulled into the next iteration. In the review you'll want to demo the product increment, the PO should actually want to be there at all costs to see what you have achieved.

Adopting agile means getting all parties on board, and that is the hardest part in large companies. Something that could help is strictly timeboxing your meetings, get a timetimer clock or some other visual timetracking device that counts down the time of your meeting, this also prevents long stretched discussions about certain topics and forces attendees to stick to the point.