Evaluation methodology (Tier 1 and Tier 2)

Supporting Templates

Summary

Although the PIR Advisor can use the evaluation methodology throughout the PIR process, they should primarily use it in Step 4.

Evaluation methodology in the PIR process

Step 4 of the PIR process, evaluation matrix assessment

The PIR Advisor is required to cover a range of Evaluation Areas in the PIR. These Areas are grouped into four Evaluation Categories:

Need: considers how well a project has met the needs described in the Business Case, its alignment with policy objectives and the extent to which its expected benefits have come to fruition
Outcomes: provides an analysis of the core delivered outcomes of cost, scope, time and sustainability
Implementation: considers whether the project was delivered as stipulated in the Business Case across the dimensions of risk management, delivery model selection, governance model and stakeholder and advisor engagement
Internal Processes: provides an analysis of the lessons learnt from the project, which should include how these can be applied to future processes. This should include consideration of the internal processes undertaken throughout the project’s development.

The diagram below identifies the Evaluation Areas that fall within each Evaluation Category.

Evaluation Areas by Category

To provide consistency in the review across each Evaluation Area, the PIR Advisor should answer the following questions for each of the Evaluation Categories, except for Internal Processes:

Completeness: to what extent was the Evaluation Area covered in the Business Case?
Outcomes Alignment: have the desired outcomes, as stated in the Business Case, been achieved?
Unplanned Outcomes: was the Business Case appropriate in light of outcomes or results that it did not anticipate?

For the Internal Processes Evaluation Category, the PIR Advisor should answer the following questions:

Existing Lessons Learnt: Were any lessons learnt from prior experiences applied? What were these?
New Lessons Learnt: What lessons have been learnt from this project?

Evaluation Questions and items for the PIR Advisor to consider have been developed for each Evaluation Area (refer to Need assessment section, Outcomes assessment section, Implementation assessment section and Internal Processes assessment section below). These items and questions cover the information that the PIR Advisor needs to assess in the review. The PIR Advisor should provide a response to each of these items to the extent possible.

A template Evaluation Matrix is provided within the PIR Report template. The PIR Advisor should use the Evaluation Matrix to record the detailed responses to the Evaluation Questions. The PIR Advisor should note any document referenced or interview conducted in the Evaluation Matrix.

Need assessment

The Need assessment table below provides the Evaluation Questions that the PIR Advisor should answer and items that the PIR Advisor should consider when assessing how well the project met its needs, achieved its policy alignment and realised its benefits.

Need assessment table
Evaluation Area	Evaluation Question(s)	Items to consider in evaluation response	Potential documents to review
Need	Completeness What information did the Business Case include regarding the project needs? Is the needs statement in the Business Case supported by adequate evidence, data and analysis?	Summarise the needs statement provided in the Business Case (i.e. what is the service need or gap being filled, the problem being solved by this project or the opportunity being realised) Summarise the basis for the need statements (e.g. underlying data, assumptions and/or analysis or modelling carried out) and assess the completeness of this basis	Business Case Investment Logic Map
	Outcomes Alignment To what extent were the project needs realised?	Determine if needs stated per the Business Case have been met based on the early indicators of the performance of the delivered asset (i.e. review the assumptions or modelling used to validate the need coming to fruition)	Business Case Data reviewed here will vary depending on the project. It will likely require an update of underlying data produced in the Business Case to compare outcomes against projected needs
	Unplanned Outcomes Were there any unanticipated outcomes or project needs that were not in the Business Case?	Indicate if there were unplanned outcomes related to the project needs identified in the Business Case Include items noted by stakeholders as needs currently not being met by the asset	Business Case PIR interviews Project specific data (see note above)
Policy Alignment	Completeness What information did the Business Case include regarding the project’s policy alignment?	Summarise the policy alignment provided in the Business Case	Business Case Wellbeing Impact Assessment Other relevant policy documents
	Outcomes Alignment To what extent were the policy alignment expectations realised?	Determine if the policy alignment expectations stated in the Business Case have been met based on early performance of the delivered asset	Business Case Data reviewed here will vary depending on the project. It may require an update of underlying data produced in the Business Case or consulting existing measurement frameworks adopted by the asset owner (i.e. the Sponsoring Agency)
	Unplanned Outcomes Were there any unanticipated project policy alignment outcomes not captured in the Business Case?	Indicate if there were any unanticipated outcomes related to the policy alignment of the project identified in the Business Case	Business Case
Benefits	Completeness What information did the Business Case include regarding the expected project benefits?	Summarise the project benefits anticipated in the Business Case Outline which of the project benefits were identified as being measurable and which were qualitative in nature	Business Case Investment Logic Map Benefit Realisation Plan
	Outcomes Alignment To what extent were the expected benefits (including social and environmental, stakeholder satisfaction) achieved?	Determine if the project benefits stated as per the Business Case have been achieved based on the early performance of the delivered asset Are the measurable or quantitative project benefit statements being measured and monitored and, if so, are the expected benefits being achieved? Summarise how the project benefits that have been realised are aligned with the Wellbeing Framework	Business Case Benefit Realisation Plan PIR interviews Wellbeing Impact Assessment Data reviewed will vary depending on the project. It may require an update of underlying data produced in the Business Case or consulting existing measurement frameworks adopted by the asset owner (i.e. the Sponsoring Agency)
	Unplanned Outcomes Were any benefits realised post-completion that were not in the Business Case?	Indicate if there were unplanned outcomes related to the project objectives identified in the Business Case	Business Case Benefit Realisation Plan PIR interviews

Outcomes assessment

The Outcomes assessment table below provides the Evaluation Questions that the PIR Advisor should answer and items that the PIR Advisor should consider when assessing a project’s overall cost, scope, timing and sustainability.

Outcomes assessment table
Evaluation Area	Evaluation Question	Items to consider in evaluation response	Potential documents to review
Cost	Completeness What information did the Business Case include related to cost and funding?	Summarise the following items in the Business Case: Preliminary Cost Estimate, including split of capital and recurrent amounts Contingency Whole-of-life analysis Budget/funding strategy including amounts to be spent per year Other categories that were included (if applicable)	Business Case Budgeted contingency amounts Documents of contract amounts (actual contract or accounting record) Final project financial summary
	Outcomes Alignment How did the final expended amounts compare to the cost estimates in the Business Case?	Compare the project costs to similar recently completed projects within the ACT and/or other jurisdictions. Projects should be compared across a common factor (e.g. buildings are often compared by cost per square metre) Compare estimates of capital expenditure as stated in the Business Case against two key phases of the project: contract amounts and final capital expenditure (use the Summary of Project Outcomes table in the PIR Report template) Examples of capital expenditure cost categories to consider include: design costs, construction cost, contingency amount, and total of other costs (e.g. internal government costs, project management, ICT, commissioning, etc.) Consider if budget and cost reporting used consistent reporting categories throughout the different stages in the project lifecycle to allow for a clear line of sight to initial approved costs in the Business Case Compare the amount of contingency spending against the planned contingency amounts Compare estimates of operating and/or maintenance expenditure as stated in the Business Case against two key phases of the project: contracted amounts, and any operations and maintenance expenditure to date	Business Case Budgeted contingency amounts Documents of contract amounts (actual contract or accounting record) Final project financial summary
	Unplanned Outcomes Was any additional funding required beyond the amount sought by the Business Case and approved or, conversely, were any budget appropriations returned?	If the final costs were significantly greater than those approved in the Business Case, at what point in the project were the additional funds requested and granted (if applicable)? If the final costs were significantly less than the those approved in the Business Case, at what point in the project was this expected to occur and what was the amount of returned funds (if applicable)? Determine if the funding appropriation was sufficient to deliver the asset. If not, identify where the discrepancy occurred	Business Case Record of additional appropriation or returned funds PIR interviews
Scope	Completeness What scope requirements were included in the Business Case, Functional Design Brief and related project documents?	The extent of scope requirements outlined in the Business Case Functional Design Brief	Business Case Functional Design Brief
	Outcomes Alignment How did the final delivered asset meet the scope requirements stated in the Business Case, Functional Design Brief and related project documents?	To determine if scope requirements were achieved, compare scope requirements in Business Case against two key phases of the project: contract stage and final delivered asset Examples of potential key scope items to consider are asset area (sqm), capacity requirements and key building spaces (if applicable) Provide commentary on reasons for scope changes and/or at what stage in the project the scope changed (if applicable)	Business Case Functional Design Brief Site tour and PIR interviews Contract document Initial and final design Value management documentation
	Unplanned Outcomes Were any additional scope items delivered that weren’t in the Business Case, Functional Design Brief and related project documents?	Indicate any additional scope items delivered that were not indicated in the Business Case	Business Case Functional Design Brief Site tour and PIR interviews Contract document Initial and final design
Timing	Completeness What key project stages and associated timelines were included in the Business Case?	To what extent an indicative timeframe and/or key project milestones are outlined in the Business Case The project timeline included with the Business Case (possibly as an appendix)	Business Case and project timeline (if included)
	Outcomes Alignment How did the actual timing of key project stages align with the dates set out in the Business Case?	Key documents produced at each key stage of the project to determine if the target timeline was met Examples of key dates/milestones include: Project budget approval Planning and environmental approvals Tender evaluation process, completion of design Construction award Start of construction Substantial completion Commencement of operations	Business Case Functional Design Brief Final sketch plans or other design deliverables Development application Tender documents Construction contract Construction start date Project Control Group or Project Steering Committee meeting minutes Updated project timeline(s)
	Unplanned Outcomes Were there any unplanned outcomes related to project timing and/or any key project stages that were not mentioned in the Business Case?	Indicate any major timeline changes Indicate any key project stages or milestones not listed in the Business Case that were later determined to be required during project implementation	Business Case Updated project timeline(s) Stakeholder interviews
Sustainability	Completeness What information did the Business Case and other related project documents include related to sustainability?	The extent of sustainability measures or indicators included in the Business Case Further climate change actions and asset management details included in the Business Case The environmental or sustainability ratings, assessments or reviews of the project	Business Case Sustainability rating tool/report
	Outcomes Alignment How did the final delivered asset meet the sustainability requirements stated in the Business Case and other related project documents?	Environmental or sustainability ratings, assessment and reviews Determine whether the environmental and sustainability requirements, measures and outcomes as stated in the Business Case (or other related project documents) were achieved (or have been achieved to date)	Business Case Sustainability rating tool/report
	Unplanned Outcomes Were there any unplanned outcomes related to sustainability that were not mentioned in the Business Case?	Indicate any additional sustainability requirements that were not anticipated at the Business Case Indicate any sustainability outcomes, climate change actions or environmental implications that were not accounted for at the Business Case	Business Case Sustainability rating tool/report PIR interviews

Implementation assessment

The Implementation assessment table below provides the Evaluation Questions that the PIR Advisor should answer and items that the PIR Advisor should consider when assessing a project’s overall risk outcome, appropriateness of the delivery model, governance structure and stakeholder and advisor plans.

Implementation assessment table
Evaluation Area	Evaluation Question	Items to consider in evaluation response	Potential documents to review
Risk	Completeness What risk assessments and analyses were used to inform the Business Case?	Risk management workshops – note if the risks workshops required were conducted and if they were held within a reasonable timeframe in relation to the date of the Business Case Did the risks workshops result in a Risk Register capturing all key information about risks (risk description, likelihood, impact, quantification, risk management plan)? The extent to which risks were considered in determining the delivery model The extent to which risks were considered in determining project contingency	Business Case Risk Registers (if included) Documents showing contingency calculation
	Outcomes Alignment What risk management activities occurred during project delivery and how did this relate to original risk management plans created?	Consider if there was a consistent project risk management approach applied throughout the project lifecycle If the Risk Register included mitigation strategies, note if there is indication that the mitigation strategies were implemented If there were any realised project risks, consider if these were anticipated by the Risk Register and/or contingency How risks and contingency have been tracked, reviewed and updated throughout the project lifecycle	Risk Registers Risk management reports Documents showing contingency spending PIR interviews
	Unplanned Outcomes Did any unplanned project risks arise, including any additional contingency requests above the amount sought in the Business Case and approved?	Unanticipated outcomes that were not captured in the initial risk workshops Realised risks that were not captured in the initial risk workshops or in advance of the risks during project implementation	Business Case Risk Registers Risk management reports Contingency documents PIR interviews
Delivery Model	Completeness What factors were taken into consideration in determining the delivery model stated in the Business Case?	Summarise the evaluation process that occurred to select the delivery model, including items such as: Items considered as justification for selecting a particular delivery model (such as project risks) Specific areas of risk transfer expected to be achieved through the selected delivery model The following items apply to PPP or DCMO projects only: Summarise the detailed Financial Analysis performed to support the selection of the delivery model Indicate if Public Sector Comparator analysis was completed Indicate if a market sounding occurred or if there was a plan for market sounding included in the Business Case	Business Case Delivery strategy report Risk Register
	Outcomes Alignment What delivery model was used and how did it perform against the desired outcomes stated in the Business Case?	The delivery model that was implemented and what contracting agreements were used to establish the delivery model If the delivery model objectives (stated in the Business Case) were met by the chosen delivery model If the risk transfer objectives stated in the Business Case were achieved via the established contracts Commentary from PIR interviews related to the effectiveness of the chosen delivery model for the project (and validate as needed)	Business Case Contract document (relevant to the chosen delivery model) Pre-execution legal analysis (which highlights the key contract components) Risk Register PIR interviews
	Unplanned Outcomes Were there any unplanned outcomes related to the delivery model used?	Outcomes that were not addressed by the chosen delivery model that could have been addressed by a different model	Business Case PIR interviews
Governance	Completeness What information did the Business Case include relating to a governance plan?	Indicate to what extent the governance arrangements were outlined in the Business Case Indicate if the governance section of the Business Case included mention of key roles and responsibilities	Business Case Governance Plan
	Outcomes Alignment How did the governance in place during the project match the governance structure and process outlined in the Business Case?	Compare how the governance committees during project implementation aligned with the governance arrangements presented in the Business Case	Business Case Copy of meeting minutes for the governance groups PIR interviews
	Unplanned Outcomes Were any unplanned governance structures implemented?	Governance groups that took on any roles that were different from those originally decided in the Business Case Governance groups formed that were not part of the original governance plan in the Business Case Governance groups that were planned for in the Business Case but did not proceed or were not required Governance committee participants that differed from those planned in the Business Case	Business Case Copy of meeting minutes for the governance groups PIR interviews
Stakeholders	Completeness What information did the Business Case include relating to stakeholder engagement?	The Business Case, including the: Identification of stakeholders Stakeholder Engagement Plan and Communication Strategy	Business Case Stakeholder Engagement Plan Communication Strategy
	Outcomes Alignment To what extent did stakeholder management and communications operate according to the plan included in the Business Case?	How stakeholder involvement matched the plan set out in the Business Case	Business Case Meeting minutes for the governance groups Meeting minutes from any stakeholder engagement sessions Revised Stakeholder Engagement Plan and/or Communications Strategy PIR interviews
	Unplanned Outcomes Did any stakeholder involvement occur that was not included in the Business Case Stakeholder Engagement Plan?	Indicate if the need arose for stakeholder engagement that was not anticipated in the Business Case	Business Case Copy of meeting minutes for the governance groups Revised Stakeholder Engagement Plan and/or communications plan PIR interviews Meeting minutes from any stakeholder engagement sessions
Advisors	Completeness What information did the Business Case include relating to the required advisors?	Advisors the Business Case required and, if any, further detail as to the role or scope of work of each advisor	Business Case
	Outcomes Alignment Were the advisors listed in the Business Case consulted as planned?	Determine if advisors were consulted as planned in the Business Case	Business Case List of contracts for the project Meeting minutes for the governance groups Deliverables from the advisor(s)
	Unplanned Outcomes Were any advisors needed that were not listed in the Business Case?	Indicate any advisors consulted that were not mentioned in the Business Case Advisors identified in hindsight as people that would have been useful to the project (regardless of if the advisor was actually hired during the course of the project)	Business Case List of contracts for the project Meeting minutes for the governance groups Deliverables from the advisor(s) PIR interviews

Internal Processes assessment

The Internal process assessment table below provides the Evaluation Questions that the PIR Advisor should answer and items that the PIR Advisor should consider when assessing the lessons that can be learnt from the project and the internal processes that were undertaken.

Internal process assessment table
Evaluation Area	Evaluation Question	Items to consider in evaluation response	Potential documents to review
Internal Processes	Were any lessons learnt from prior experiences applied? What were these?	Summarise the lessons learnt from previous projects that were used in the planning, development, procurement and implementation of this project	Data reviewed here will vary depending on the project
	What lessons have been learnt?	Summarise the internal processes that occurred throughout the project development: What worked well and what could have been improved? The lessons that can be learnt from all stages of this project, including its planning, development, procurement and implementation How these lessons can be applied to future projects	Data reviewed here will vary depending on the project

Footnotes: