Why is the middle ground between functional and matrix organizations so common?

Question

I'm a software developer, and most companies I've worked for are organized in a system that is a middle ground between functional and matrix. Much more functional than matrix, indeed.

The structure is something like this: there are several teams in the company, each one being responsible for a specific function. But one of these teams has a special characteristic. It is also responsible for a product and works in a project-oriented manner. Let's call it the "product team". All projects that this team carries out are intended to maintain or improve their product. As such, the product team is the project "driver", so to speak, requesting services from the other teams as needed. In general, the product team is responsible for programming functions, and requests services from Data Analysis, DBA, User Interface, Test, Deployment and so on.

Honestly, from my personal experience, I have only seen problems with this approach. To list some:

• Individuals in the "service teams" lack long-term commitment to the product or the project. Their mindset is to finish the service as quickly as possible and, once delivered, it becomes somone else's problem. Obviously, this is not their fault. They are only reacting to incentives, because if they do otherwise, they will probably be punished (or, at least, not be rewarded).

• For the same reason as pointed out in the previous item, the service team members lack contextual information that would help them to see the big picture. They may be able to perform their services well, but they are not able to suggest alternatives or challenge wrong assumptions. Hence, the overall performance is sub-optimal.

• Since the activities are inherently coupled to each other (the separation into functional teams is somewhat arbitrary and artificial), some problems are faced over and over by different teams, as the project evolves. Each team will have to learn the business rules from scratch, for example, and will probably have the same basic questions, which will demand the same basic answers. All teams will go through the same learning curve.

• From a management perspective, coordinating the schedules of all those teams is a very hard problem. It is very common to have to wait some weeks to get a job done because the members of some service team are all busy fulfilling requests from other teams.

• Depending on how the teams are partitioned, sometimes different teams have conflicting goals. For example, a good project team will have a high throughput (let's say, number of software features per month), which necessarily leads to changes in the production environment. The deployment team normally has the opposite goal: to maintain the production environment as stable as possible. So, they will create all kinds of obstacles to the deployment of those features.

• A certain "tribal" behavior emerges from this setup. Probably this is just human nature, but when people are separated into different groups with conflicting goals, they will start to see each other as rivals and not as co-workers.

• [edit to include another problem I remembered] Due to the arbitrary nature of this kind of partition, some problems that are experienced by one team can only be solved by another team, either for technical or political reasons. But such problems may be evaluated differently by each team. For the one that has the problem, it may be top priority. From the "solver" point of view, however, it may be less important. As a consequence, it will never be solved at all, or it will generate an enormous level of political conflict between the involved functional areas.

Despite all the drawbacks listed above, this kind of team organization seems to be overwhelmingly common. Why is this? Is there some big, obvious advantage that I'm missing, that surpasses all the problems I pointed out?

Answer 1

In my experience as an operations consultant, organization design is sloppily performed, if performed at all. Most organizations, I think, are very immature in its design of its structure and is more a result of reactions to adverse business results than a thoughtful design. When things are not going well, what is the first thing many orgs do? Re-organize! They will do this even if the organization had little to nothing to do with the result because it shows an active intervention and is seemingly easy to do (mostly because they do not do it well).

I would bet this middle ground you are observing was not created purposely. You have a lot of things going on here: reaction to marginal business performance, a great idea from a business manager who read a book and is trying to experiment, a other managers trying to build their fiefdom, to name a few.

Org design is a complex task, rooted in deep analysis, theory, culture, environment, and politics. It is no less complex than designing your business processes and system solutions. But I can guarantee that, in most cases, the effort behind it pails in comparison to the other two efforts.

Answer 2

From what I've seen, there is one common reason:

• It is easier to budget

The finance people have a lot of influence over organizational structure due to the way that they sort, categorize, and track expenses and they are normally non-technical. Since a specific function is easier for a non-technical outsider to identify, your financial people are more likely to group cost by function ($XX for e-mail, $YY for network connectivity, $ZZ for telephony, etc.) and then count and track all expenses along those lines - including staff.

This tends to match the management thinking, too. If there is an internal CRM tool, they want to know how much it costs to run it, program it, and manage it as a whole so it makes sense to place the ops, dev, and managers for that tool all together in a single pot. You can then use that in discussion/negotiation with other business groups, too - "Your CRM is costing us $XXX and is supporting your team of YY people, so we would like to take $ZZZ of your annual budget"

The cross-team, cross-function opportunities, costs, and associated friction get lost in this mode of thinking, as your question points out.

Answer 3

It is a lot easier for people to "get" a functional structure compared to projectized. "I am a developer and my boss is a senior developer and we are in the development group" is more intuitive to most people than "I am a developer and my boss from an HR perspective is different from my boss from a technical perspective and by the way the latter can change when I start new things".

Deviating from your original question, some suggestions for mitigating the issues you raised:

Individuals in the "service teams" lack long-term commitment to the product or the project. Their mindset is to finish the service as quickly as possible and, once delivered, it becomes somone else's problem. Obviously, this is not their fault. They are only reacting to incentives, because if they do otherwise, they will probably be punished (or, at least, not be rewarded).

The essence of a project is that it is temporary - it isn't necessary for a project team to have a long-term commitment to the product once the project is over. Realization of benefits is at a program/corporate level.

Commitment to the project is another issue altogether. My experience is that it isn't the team member so much as his manager that you have to worry about losing commitment to the project. Engage those key stakeholders early and often.

For the same reason as pointed out in the previous item, the service team members lack contextual information that would help them to see the big picture. They may be able to perform their services well, but they are not able to suggest alternatives or challenge wrong assumptions. Hence, the overall performance is sub-optimal.

I've found it helpful to have end-of-stage meetings shortly before a key milestone is met so that you can go over upcoming tasks, acceptance criteria, assumptions, lessons learned, risks, etc with those individuals that are going to be working on the next stage's products. More work for me in the short run but less rework in the long run

Since the activities are inherently coupled to each other (the separation into functional teams is somewhat arbitrary and artificial), some problems are faced over and over by different teams, as the project evolves. Each team will have to learn the business rules from scratch, for example, and will probably have the same basic questions, which will demand the same basic answers. All teams will go through the same learning curve.

There should be some level of consistency during a project (e.g. same project management team), if this doesn't exist the new team needs to make a point of talking to the old team during the handover process to avoid rework. Develop institutional knowledge and share lessons learned as much as possible. Make a point of asking questions when you get moved onto a new project.

From a management perspective, coordinating the schedules of all those teams is a very hard problem. It is very common to have to wait some weeks to get a job done because the members of some service team are all busy fulfilling requests from other teams.

Again, end-of-stage meetings and engaging stakeholders can be helpful. Provide teams scheduled to do work with the plan (that they would have agreed to up front) and find out what has changed and what might prevent them from keeping to the plan. If you organization has many projects that are competing for resources you need to advocate for your project with your executive management. If a business case for why your project needs top priority for resources can't be made then management needs to accept that it will run late.

Depending on how the teams are partitioned, sometimes different teams have conflicting goals. For example, a good project team will have a high throughput (let's say, number of software features per month), which necessarily leads to changes in the production environment. The deployment team normally has the opposite goal: to maintain the production environment as stable as possible. So, they will create all kinds of obstacles to the deployment of those features.

The best that we can do is to have a clear and shared vision of what the end product has to look like, what the acceptance criteria are, etc. If everyone is working towards the same, clear objective you should be able to minimize conflict.

A certain "tribal" behavior emerges from this setup. Probably this is just human nature, but when people are separated into different groups with conflicting goals, they will start to see each other as rivals and not as co-workers.

No good solution for this, other than trying to set the example of someone who sets the interests of the project and team first and departmental goals/ambitions second.

Answer 4

I'm not a historian by any stretch of the imagination, but here's my completely unscientific theory:

I would say most business started off as functional. Think about your most primitive teams. Men were hunters. Women were gatherers. Other specialties evolved, and over the centuries, teams and companies grew to business as we know it. Somewhere along the way, some really smart MBAs probably got together and decided matrix organizations were cool, and companies gravitated to the new fad. Some succeeded, some failed, most fell somewhere in between.

And here we are.

Answer 5

From a career development perspective most people's development follows a functional model. People develop within a functional career stream; accounting, marketing, project management, software development... This works well for developing resources, but it doesn't always support a business's needs, especially for projects.

The matrix model allows resources with the appropriate skills to be assigned to the appropriate tasks. This works for a number of cases. If you have a bunch of plants to manage, you need a plant manager for each plant. Functionally, these belong to the same group but are assigned to different units. It would be possible for someone to develop a career withing a plant, but in many roles there would be no functional peers.

Projects work the same way, but have a much shorter lifetime. A project should never be a career. However, a project usually needs people from a number of functional groups. This is why you see the matrix model used. Some of the problems you pointed out result from the short term nature of projects. Scheduling resources for projects is always difficult, and scare resources often cause problems.

Managing resources within a matrix model is difficult and as you have noted not without its risks. Many organizations have trouble developing and retaining the appropriate corporate knowledge. Matrix management can make this more difficult.

The requirements of production and development are very different. Production needs reliable processes, when tends to lead to a preference for stable processes. Development needs change and can disrupt production processes. Techniques that work well in development can be totally destructive in production. (Developers can recreate a new database at will, a process which is not possible in production.)

Despite its drawbacks, the matrix model makes projects work.