What is the correct process for assessing project management tools/solutions?

Question

We do cross-platform development with .NET, Java, and iOS. I have been tasked with selecting an Application Lifecycle Management (ALM) solution suitable for managing multiple small distributed teams.

I'm struggling to see how any one of the vendors have any major advantage over the others. It's not at all clear to me how to choose between any of the major vendors solutions as they all seem pretty good, all support cross-platform, etc.

Just for context, I'm trying to decide mostly between the following suites of ALM tools:

• Atlassian (JIRA, Greenhopper, Confluence, etc.)
• Microsoft Visual Studio 2012 ALM: (Team Foundation Server, etc.)
• IBM Rational Collaborative Lifecycle Management (Rational Team Concert, Requirements Composer, Quality Manager, etc.)

What process should be used to assess and pick the best solution? I'm struggling with how to proceed.

Answer 1

First: Document Your Process

We don't do shopping questions here on PMSE, but we're all about process. The correct way to comparison shop for any tool is to document your actual work-flows and project processes, and then find a tool that supports what you are actually doing.

Write down how your team does things right now. If your project is just forming, write down whatever process you'd follow if you had to do it with paper and pencil. The tools don't matter; the controls and processes are what count.

Second: Shop for Tools Conforming to Your Process

Now that you know what your process is (or will be), you can shop around for whatever tools will automate or simplify the things you plan to do. Never, ever build your process around a tool-chain; always build your tool-chain around your process!

Answer 2

Proceed as if this is a project - which it is. The end product that you want to produce is a series of business cases, one for each tool. These will document the costs, benefits, limitations, constraints, support costs, etc etc for each tool so that a business decision can be made on which to proceed with. In general you will need to:

• Identify stakeholders. Not only end users and suppliers, but also who should be involved in decision making.
• Identify the scope. Engage users and get their key requirements and constraints. Engage suppliers and find out what can be provided. Work with them both to figure out where assumptions are misaligned.
• Identify the timeframe. Find out not only when can a tool be delivered but also how long training will take, what timeframe post-implementation work will occur in, etc.
• Identify the costs. Not just costs for maintenance but also ongoing costs. And also document if you are going to do anything to defray the costs (e.g. pass them on to customer)
• Identify the benefits. Describe this in dollar terms if you can (e.g. things like productivity improvements) though for some things this may not be easy (e.g. ensuring regulatory compliance).

Answer 3

This is not a process, this is a choice analysis. Choose an easy but controlled method.

It is driven off of your requirements--functional, technical, business, and financial. If you do not already have these articulated, then do so. Bring in a team of SMEs that cuts across your organization. In other words, you need SME representation from your technical team, functional team, financial team, and user communities. Requirement definition is a difficult process of and in itself. If you have someone that has expertise in facilitating this, that would help. As a general rule, your requirements need to be in the same level of decomposition and you need to ensure you do not have requirements that contradict each other. And, minimize mandatory requirements to the degree possible. I personally use a 90/10 rule, where no more than 10% would be mandatory.

When that is finished, you need to put a weight on each non mandatory requirement. You can use a simple low, medium, high or numbers ranging from zero to 10 or something like that.

Your SME team will then evaluate each product against the requirements and provide a score. You can either facilitate a consensus for a score or use some type of blind recording method. There are pros and cons to each approach.

The sum of the products established by the score and weight will provide a product score. It will not take long to see the ranking.

Answer 4

If you are unfamiliar with the tools, you can follow the systematic and rigorous process as recommended by David Espina. I followed that exact process for a critical software selection a few years ago. The big challenge was in figuring out which requirements the tools only partially met. The tool vendors will 'never' say they don't meet one of your requirements. They will say 'yes' and provide explanations. Figuring out which tool truly meets those requirements effectively is hard. If you can manage that, it should get you the results.

However, you say "they all seem pretty good". I would like to recommend a slightly different process. Do bring in the team of SMEs that cuts across your organization. And do create the list of mandatory requirements. Eliminate those tools that don't meet your mandatory requirements. In addition, create another list of requirements that will help make your team shine. This should be a small list, say not more than 10 requirements. Put all of your effort in evaluating which tools meet these and to what extent. Further confirm it with an in-house trial of the shortlisted two tools.