1

It is known that software development tasks should be controlled by some project management metrics, that show the progress of work at each particular moment of time.

What metrics are better to be used if the project is a research and usual work consists of not only making a SW product, but (before that) reviewing and large corpus of scientific papers, making various PoC's? They key point here that at the beginning such project has no fixed architecture and just need to determine what's the best one.

In our project the "done" criteria are:

  1. Our decided architecture decision is best
  2. We have implemented a piece of this architecture and it works
  3. This approach is new thus our project team is the best who continue the work on it.
  4. We have published a paper in the top conference/journal about our research.

In order to achieve this criteria we plan to:

  1. search all possible directions in literature,
  2. investigate how to apply these directions to our particular problem (because methods from papers can be described for adjacent but different applications),
  3. estimate how much effort will it take to implement each of approaches,
  4. estimate how much quality of SW product we can obtain by each approach (we already have this metric, it's deterministic in mathematical sense),
  5. compare different approaches and choose the best one,
  6. implement the best approach as a proof of concept,
  7. write and publish a paper about this approach.

What I need to know is that metrics are better to start with in this case? We need metrics that could be tracked on a weekly basis, i.e. they must show how much part of work is done.

  • Hi, welcome to PM.SE! I believe your question is a dup of pm.stackexchange.com/questions/23251/… ... could you please confirm or make it more specific? Thanks! – Tiago Cardoso Feb 20 at 14:32
  • 1
    What is the expected outcome of your project. You mention it's a research project, but it sounds like your end goal is to create an application, so I'm a bit unsure. – Daniel Feb 20 at 15:00
  • @TiagoCardoso It seems to me that my question is different. I'm asking about metrics of quality that could be tracked throughout the project cycle. The question you've pointed asks about management and performance evaluation. These two are more high-level concepts: management is a high-level approach to manage the project, and performance evaluation is the event happening at the end of large time cycle (yearly or half-yearly). What we need is everyday metric that would track how much work is done from what is to be done. We need a metric that can be checked on a weekly basis. – Ilya Palachev Feb 20 at 19:05
  • @Daniel The outcome is two-part. First (1), we need to develop software architecture for a SW product or the collection of SW products that will be developed for some years later. In current year we need to show that (a) our architecture is feasible for future implementation, (b) our project team (and nobody else) is able to develop components of future product, (c) the future product will benefit to the company's process. (b) implies the second goal/outcome: (2) produce the proof of concept for the SW architecture that is new in scientific sense, and publish a paper about that. – Ilya Palachev Feb 20 at 19:13
0

I'm not entirely sure I understand the question, so please forgive me if my answer is tangential. That said, I would ask the questions in the opposite order.

First question is (for me) always, "What does done look like?" Sounds to me like the first phase of your project is to complete a literature survey/market survey/research effort. Based on your comment, it seems like "Done" is "Determine the best architecture"

Personally I'd construct the decision matrix and preliminary weighting first, and then use it to direct my research. When the next unit of research won't change the outcome of the decision matrix, research is done. When I last did literature review, I could estimate the time fairly accurately based on the amount of published research; doing the decision matrix first would allow me to report complete in less time than that estimate.

At that point the relevant metric/KPI is the amount of uncertainty reduced by each time period of research. Technically you're tracking risk/opportunity - the reduction in uncertainty. Research reduces epistemological uncertainty; assuming that you start with the higher weight factors, eventually the epistemological uncertainty is small enough that it won't affect your decision.

I like crunchy data, so I'd probably model the decision matrix with a component that measures the probability of finding new valuable papers, but I'm a frustrated quant. Although I'd find it fun, I think it is actually overegging the pudding.


Some notes while I organize my thoughts: what follows will be under revision There are a couple of tools that fit within the "decision matrix" grouping (there are others, and I'll add them as they occur to me, but the point is that there are multiple tools.

  • Analytical Hierarchy Process (AHP), which is fairly heavyweight, but may help if the factors affecting the decision are unclear. On the other hand, it is a front ended process; I'm not sure I'd want to revise the weighting if research were to change my understanding of the problem.

  • Decision Tree is the tool that was in every PMI course I took. Seems to be PMI's preferred tool. Probably not my preferred tool for this problem, since half the effort seems to be setting up the tool, and doing so implies to me that you have a deeper understanding of the problem than is appropriate for a research scenario.

  • Decision matrix - this is closer to what I've typically used. Unfortunately this is why I need to take time to organize my thoughts. Reading this page makes a relatively straightforward process seem like rocket science/PhD material. I need to find a way to simplify this to the actual process.

Fundamentally, all of these tools reduce the problem into a choice, and a set of factors that affect that choice. Creating a trivial example, if I have a set of software requirements (functional, and nonfunctional) {R}, and a choice of architecture A or B.

  • For each r in {R}, can it be expressed more simply in A or B? Are there examples in the literature of successful implementations? Are there known limitations?

  • Are there relevant resource constraints for A or B? (Do we have the relevant skills on staff? Does our intended hardware support? Will we need to provide any infrastructure/logistics support in order to market the software?) - NFR's such as performance, security and logistics may show up here.

I'm going to make the example really trivial here, but if the choice is between Architecture A and B, and the factors influencing the choice are R1, R2, R3 then we need to assign weights to the factors (R1 is 5 times as valuable as R3; R2 is half as valuable as R1). At the beginning of the search, A & 5 are each 50% likely. As we identify relevant papers, they will affect the weighting. If we find a set of papers that indicate that Architecture A has been used repeatedly to fulfill requirements like R1 & R3, then we apply the relevant weights to shift towards architecture A. On the other hand if there is zero evidence that Architecture A has ever been used to support R2, that will have a lesser, countervailing impact.

Returning to your fundamental question - how do I use this to bound my search? how do I measure progress towards "done"?

I still think the coarse measure, and the one I'd use for day to day, is a linear measure of exhaustion of my search space. Inelegant, but I'm willing to bet that it will be close enough. (It is early in the morning, and I can't think of a name for the class of metrics that are like yesterday's weather; embarrassingly simple, but good enough for practical purposes.)

I would also work from the assumption that I'm going to find the most valuable results early. I'd calculate the interval between finding useful results and plot the curve of days between interesting results - I'm willing to bet that curve will be a familiar long tail curve, and that I can use that to predict approximately how long it will take me to find the next interesting result. (it is 80% likely that it will take between 5 and 10 days to find the next interesting result).

I can look at the decision weight (Architecture B is leading 80% to 20%) and the predicted time to next interesting paper (80% likely to be up to 10 days), and estimate the probability that I will find one or more papers that will affect the weighting by over 16% is less than some threshold. And I can use that to terminate my literature search early.

  • In our project the "done" is the last version of ones you've proposed, the decision of best architecture. But for the decision to be best, we need to (1) search all possible directions in literature, (2) apply these directions to our particular problem (because methods from papers can be described for adjacent but different applications), (3) estimate how much effort will it take to implement each of approaches, (4) estimate how much quality of SW product we can obtain by each approach (we already have this metric, it's deterministic in mathematical sense), (5) compare different approaches. – Ilya Palachev Feb 20 at 19:23
  • Could you clarify what you mean by decision matrix? – Ilya Palachev Feb 20 at 19:36
  • I will; may take me a day to put my thoughts in order. – Mark C. Wallace Feb 20 at 19:37
  • It also would be great if you point to some paper/book about that, if that is some general concept. – Ilya Palachev Feb 20 at 21:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.