I am working as a Business Analyst in a project which target is to replace an old In-house developed application with a new COTS application. We have defined quite many requirements (~200) within several High Level Requirement (HLR) areas. After the RFQ (Request For Quotation) Response has been finalized with answers on additional questions and final evaluation of each requirement for each Supplier solution, we will consolidate in an Supplier Evaluation Matrix.
Requirement fulfillment Grades
We are using the same gradation in both Requirement Evaluation (by Supplier) and the Supplier Evaluation Matrix (Suppliers side by side on HLR).
- 0 = fail
- 1 = large deficiencies
- 2 = partly fulfills requirement(s)
- 3 = barely fulfills requirement(s)
- 4 = fulfills requirement(s) well
- 5 = exceeds expectations
Grading and consolidation
Within a HLR, e.g. how we work in projects, we have ~30 requirements. Most of them have grade 4, i.e fulfulling requirements, but some have other grades on above grade range.
Consolidation in Supplier Evaluation Matrix level will be on HLR and in above example need to consolidate to one figure for this HLR.
My thoughts regarding this is that there are different approaches, but I don't find any of them particulary good.
- Average - just sum all together and split by number of requirements graded.
- Lowest - So if one requirement fails (Grade 0) then the HLR level will be fail.
- Business Value weight - "Legal" highest, "Critical" second, "Important" third and "Wanted" lowest.
"Average" is very easy, but then not supported (failed) critical requirement may get unnoticable impact on the rating.
"Lowest" might have too high impact on the rating if everything is fine except that one not supported requirement.
"Business Value weight" is perhaps the way to go, but then we need to spend time to further develop templates etc.
What are the common approaches - that provides "fair grading", not taking so much time and easily can achieve common view between reviewers?
Note - For the Agile supporters - Yes, this is Waterfall, but once we have chosen solution we will have an agile implementation.