There are various methods that can be applied to undertake MCA. These Guidelines present a predominantly qualitative approach to MCA, as quantitative scoring of project options may lead to an overly positive view about the accuracy and rigour of the MCA and/or an incorrect interpretation of the results. Numeric weightings and scoring are also subject to a higher level of subjectivity than rankings and ratings.
The Project Team should follow the steps described below when undertaking MCA:
- Determine a set of appropriate evaluation criteria
- Assign a relative importance to each criterion
- Provide a rating for each option against each criterion
- Rank each option, review the results and provide a written justification.
For Tier 3 projects and ‘Business-as-Usual’ Tier 2 projects, the Project Team may not need to follow each of these steps to the level of rigour outlined in the following sections. However, they should ensure they undertake an objective assessment, considering the risks outlined in Risk section.
More information on each of these steps is provided below.
Step 1: Determine a set of appropriate evaluation criteria
The Project Team should use consistent and well-defined criteria to compare and evaluate project options or tender responses in an accurate and unbiased way. The Project Team should develop criteria based on the objectives of the project and the overall need, problems and opportunities, and benefits of the project outlined in the Strategic and Policy Alignment and Needs Analysis sections of the Business Case. The Project Team is encouraged to incorporate the relevant domains of the Wellbeing Framework as evaluation criteria. This will ensure that the set of benefits the project seeks to achieve include social progress and welfare benefits, as well as traditional economic measures.
The Project Team should establish clear measures that will underpin each criterion, which should be informed by quantitative data where available. These measures will provide the evidence base for scoring the criteria.
The Project Team should discuss, develop and agree the evaluation criteria with a range of key stakeholders to ensure the criteria cover the key objectives of the project and can be understood and interpreted consistently.
The Project Team should not over-complicate the analysis and should focus on the most important criteria. The number of evaluation criteria selected by the Project Team will vary depending on the complexity of the project, but should be at least five and not exceed ten in number.
As Risk section on the previous page notes, in the Options Analysis and Economic Appraisal, the Project Team should also be careful to not include evaluation criteria that may be assessed through the CBA, so as not to double-count benefits.
Examples of potential evaluation criteria are provided in the Options Analysis Guidelines.
Step 2: Assign a relative importance to each criterion
The Project Team should assign a relative importance to each evaluation criterion so that more important criteria have higher weights in the analysis. Weighting the evaluation criteria numerically is not recommended.
Instead, The table below provides an example scoring system to rank each criterion by their relative importance in a non-numerical way.
Relative importance | Description |
|---|---|
*** or High | These evaluation criteria are of the highest importance and therefore are comparatively more important than other criteria |
** or Medium | These evaluation criteria are comparatively more important than one-star evaluation criteria but less important than three-star criteria |
* or Low | These evaluation criteria are comparatively less important than other evaluation criteria |
The Project Team may assign the same relative importance ranking to several different evaluation criteria, but should aim to have at least one criterion in each of the relative importance ranks.
Step 3: Provide a rating for each option against each criterion
Each option should be rated against the evaluation criteria, independently of their relative importance. Where possible, the Project Team should involve other key stakeholders in the rating process to mitigate the risk of subjectivity. The rating process should be as objective, and as evidence based, as possible, using the measures that were agreed to underpin the evaluation criteria.
The table below provides an example rating system to rate each criterion according to the extent to which each project option achieves the criterion.
Rating | Summary | Description |
|---|---|---|
A | Excellent | For qualitative criteria, full achievement of the criterion in a manner demonstrating excellence, innovation and high-quality outcomes For quantitative criteria, the upper quartile range of the possible results; the top 25% of results, where 100% is the best possible, realistic result |
B | Good | For qualitative criteria, good or sound achievement of the criterion For quantitative criteria, the median to upper quartile range of the possible results; the 50% -75% range of results, where 100% is the best possible, realistic result |
C | Moderate | For qualitative criteria, minimal but potentially acceptable achievement of the criterion For quantitative criteria, the lower to median quartile range of the possible results; the 25%-50% range of results, where 0% is the worst possible, realistic result |
D | Poor | For qualitative criteria, no or unacceptable achievement of the criterion For quantitative criteria, the lower quartile range of the possible results; the bottom 25% of results, where 0% is the worst possible, realistic result |
Using the example provided in the above table, each option should be assigned a relative rating of A – D based on the extent to which it achieves each criterion. The Project Team should not use the ranking of options for each criterion to determine their rating, but instead use the descriptions in the rating categorisation system table above as a guide to the rating of options.
For criteria that can be quantified, the Project Team should map objectively the quantifiable results to the qualitative rating categorisation system as described in above table. For qualitative criteria, the Project Team should focus objectively on points of difference between options with reference to the Base Case, rather than trying to assess them against an arbitrary absolute standard.
When appropriate, the Project Team should use a ‘+’ or ‘‒’ to show differences between options that otherwise would have the same rating. For example, three alternative options each may achieve the required user functionality in a sound way, and hence have a B rating, but the options have a clear ranking, so they would receive ratings (in descending order) of B+, B and B-.
When rating each option, the Project Team should consider as a form of benefit any avoided costs or the avoidance of other negative impacts that would result from proceeding with each of the identified project options.
Step 4: Rank each option, review the results and provide a written justification
The Project Team should rank each project option based on the relative importance of each criterion and the rating of each option against the criteria, as described further in this section.
The Project Team, along with other key stakeholders, should review the results of the MCA to assess whether they are reasonable and have not been unduly affected by anomalies such as incorrect weighting of criteria.
Based on the review, the Project Team should document concisely the results of the MCA and articulate clearly the justification for why a project option, or a shortlist of project options, has been recommended; and why other project options have not been recommended or shortlisted. When documenting the results of the MCA, the Project Team should include:
- The project options assessed
- The evaluation criteria and underlying measures that were selected and the process through which these were agreed
- The relative importance of each evaluation criterion, and how and why it was established
- The rating categorisation system
- The rating of project options against the evaluation criteria and a clear justification for each rating, linking back to the selected measures and any supporting evidence
- The results of any sensitivity tests undertaken
- A description of the final ranking and a written justification.
The Project Team should be able to explain how the options have been compared in relation to their respective ratings against the evaluation criteria. For example, as shown in the Simplified example MCA ranking table below, a project option with an A rating against a three-star criterion and B or C ratings against all other two-star and one-star criteria (such as Option 1 in the below example) may be ranked lower than an option with a B rating against the same three-star criterion but with higher ratings against all other two-star and one-star criteria (such as Option 2 in the below example).
Criterion | Relative Importance | Option 1 | Option 2 |
|---|---|---|---|
Increase safety for Canberrans | ** | C – Moderate | B – Good |
Design functionality | *** | A – Excellent | B – Good |
Meet sustainability goals | *** | B – Good | B – Good |
Impact on facility users during construction | ** | C – Moderate | B – Good |
Impact on surrounding properties during construction | * | C – Moderate | B – Good |
Support local industry participation | * | C – Moderate | B – Good |
Rank |
| 2nd | 1st |
Option 2 has been ranked as the preferred option as, even though Option 1 ranked higher on the 3‑star criterion ‘Design functionality', Option 2 ranked B – Good on this criterion, equally on ‘Meet sustainability goals’ (the other 3-star criterion) and higher on all other criteria.
The Project Team should be able to demonstrate that all options have been given full consideration, using the best available evidence.
Footnotes: