Evaluation criteria

The Project Team should use MCA to select the delivery model that is likely to be most appropriate for the project, as described in Delivery model assessment. This section describes the criteria that are likely to be relevant for most infrastructure projects. These include:

• Project value
• Project scope and duration
• Ability to specify project outputs clearly
• Time to deliver
• Flexibility
• Price certainty
• Innovation and incentive
• Risk allocation to private sector
• Market interest
• Whole-of-life issues
• Procurement and supervision costs
• Stakeholder management.

Should a Project Team seek to use additional criteria, the Project Team must consult first with ICA.

Some of these criteria will not be relevant for all Tiers. For example, the Project Team for Tier 3 projects may only need to use the evaluation criteria project value, project scope and duration and time to deliver. The Project Team should carefully select the criteria to use in the MCA that are the most important and relevant for the project.

In addition, the delivery model selection methodology depends in part on the relative importance of these criteria, with more important criteria having a greater weight in the analysis than less important criteria.

Project value

Relevant for all Tiers.

There is no definitive link between the capital or whole-of-life value of a project and the most appropriate delivery model. However, some delivery models (Project Management Agreement, Design then Construct) generally are only appropriate for low value, Tier 3 projects, whereas others (particularly the Integrated models) generally are only appropriate for high value, Tier 1 projects, because of their complexity and relatively high procurement costs. However, User Charge PPPs can suit well some types of projects with low capital values where it is possible to define the required outputs very tightly (e.g. hospital carparks, waste treatment plants).

The diagram below provides an indicative guide to the delivery models that may be appropriate based on the capital value of the project. This guide is not definitive: for example, there may be specific features that would make the PMA model appropriate for projects with a higher value than $5 million, and there have been successful PPP projects with capital values below $100 million (in current prices).

Alignment of delivery models to project value

Project scope and duration

Relevant for all Tiers.

When selecting the recommended delivery model, the Project Team should consider if the project has a clear scope, opportunities to bundle services and is expected to have a long duration.

Integrated delivery models can be effective if there is scope for the project to combine non-core ongoing ancillary and/or asset-related services with design and delivery of the project assets within a single, long-term contract structure.⁠ (Footnote: Typically, at least ten years in duration and can be much longer. ) This bundling allows synergies between the different project components that can lead to better value for money.

Traditional and Flexible delivery models typically have separate contracts for each individual service, and the Project Team typically will only procure contracts for specific services when such needs arise. These models are more effective where:

• There is limited opportunity to bundle services together to create a long-term operational/ maintenance opportunity
• The Project Team is unable to specify long-term service requirements due to, for example, uncertainty about the long-term requirements of the project assets
• The project scope is likely to change significantly prior to project completion and the Project Team cannot provide satisfactorily for the potential change in the specification.

If the scope of the project involves the provision of Core Services (i.e. services to the public that require the use of the project assets), the Project Team must carefully consider the appropriate delivery models as Government’s policy is to not contract these services to the private sector.

Ability to specify project outputs clearly

Typically more relevant for Tier 1 and Tier 2 projects.

Where the Project Team clearly understands its output requirements from the project and can readily measure project outputs, the project is likely to be suitable for the more complex, Integrated delivery models. The Project Team can translate these measurable outputs into a performance specification and related payment mechanism, and hence align financial incentives for the contractor to key performance requirements.

An Output Specification is a distinctive feature of Integrated delivery models. It involves Government specifying the project’s requirements and scope in terms of functions and service quality standards in order to meet the project’s objectives, rather than in terms of inputs such as a prescriptive design, or required construction and/or maintenance methodologies, etc.

Traditional and Flexible delivery models are more effective for projects where the Project Team needs some control over the design and construction process; for example, because:

• The project output requirements are not clear or are not capable of ready measurement
• There are extensive statutory, regulatory or stakeholder requirements
• The Project Team has better experience than the private sector in managing the design and delivery of the project assets.

Time to deliver project

Relevant for all Tiers.

The Project Team generally should plan to ensure that project timelines do not constrain the selection of the delivery model. This may require taking into consideration the wider Government infrastructure program or interrelated projects. However, there may be circumstances where it is especially important to complete design and construction and commencement of services within a specified time or by a specified date; in these instances, this criterion will have a relatively high weight in the analysis of different delivery models.

There may be other projects where early commencement of construction will matter.

Although the procurement of Traditional and some Flexible delivery models generally is significantly quicker than that of Integrated models, there are other factors that, at least in part, offset this speed before the start of the tender period and/or during the design and construction stage.

The table below describes the key features of each delivery model that influence the time to deliver and gives a typical overall relative rating for each model.

Flexibility

Typically more relevant for Tier 1 and Tier 2 projects.

The ‘Flexibility’ criterion has two aspects:

• Operational flexibility: the extent to which the delivery model enables Government to retain flexibility in terms of the project’s operational profile
• Future scope changes: the extent to which the delivery model assists Government in managing and implementing changes to the functional requirements of the project over time (both during the design and construction stages and subsequently).

The separation of operational flexibility from scope flexibility will be appropriate for some projects, but a combined criterion is likely to be sufficient for most.

The Flexibility vs price certainty diagram below gives an indicative guide to where the different delivery models sit on the flexibility continuum. Flexible or Relationship delivery models are inherently more flexible and tolerant to changes in scope and uncertainties than are other types of delivery model but, as the Price certainty section below notes and the diagram illustrates, typically there is a trade-off between this flexibility and price certainty.

Integrated models are often more flexible than Traditional models, as described further in the Guidelines for Public Private Partnerships.

Flexibility vs price certainty

Price certainty

Typically more relevant for Tier 1 and Tier 2 projects.

As a general principle, the Project Team should seek price certainty where possible. However, there will be some projects that have a risk profile such that they will not be able to achieve price certainty, and hence Flexible or Relationship delivery models are likely to be more appropriate for these projects. The structures of these models facilitate delivery where uncertainty prevails. Uncertainty can arise for several reasons, such as:

• The project requires innovation in design, making the design outcome uncertain
• There is geotechnical instability or unknown contamination
• The Project Team is still developing the project scope
• There are overriding time pressures
• There are major interdependencies with other projects
• There are unknown latent conditions in existing assets that will form part of the final project
• There are complex and unresolved stakeholder issues.

However, this ability to respond to uncertainty comes at a cost in terms of price certainty. The Flexibility vs price certainty diagram above gives an indicative guide to where the different delivery models sit on the price certainty continuum.

Innovation and incentive

Typically more relevant for Tier 1, Tier 2 and some Tier 3 projects that are not Business-as-Usual.

This criterion considers the existence of opportunities and incentives within the delivery model to drive innovation. The relative importance of this criterion depends on the perceived opportunities for innovation that would help deliver best value for money.

The PPP models and, to a lesser extent, the DCMO model are most likely to have opportunities for innovation and to provide incentives to exploit them. Under these PPP models, as the contractor has long-term responsibility for the project infrastructure and facilities within an Output Specification, it has a strong financial incentive to minimise whole-of-life costs, including through innovation where possible at reasonable risk. The DCMO model is similar, although the financial incentives are less than under a PPP and the design, construction and services specifications are likely to be more prescriptive, limiting the scope for innovation.

The aim of the Alliance and, to a lesser extent, Managing Contractor models is finding ‘best for project’ solutions, but there is less direct financial incentive to do so. Under the other models, Government’s design and construction specifications are likely to be more prescriptive than under the DCMO or PPP options, further limiting the scope for innovation.

Traditional models (apart from the short-term DCM to an extent) have limited interfaces between the contractors responsible for construction, operations and maintenance, and hence reduced scope for innovation in relation to whole-of-life issues. There is opportunity for design innovation in the D&C and DCM models in as far as it affects constructability and value for money of construction.

The diagram below gives an indicative guide to where the different delivery models sit on the Innovation and Incentive continuum.

Innovation, incentive and risk allocation

Risk allocation to private sector

Typically more relevant for Tier 1 and Tier 2 projects.

One of the main criterion for determining the most appropriate delivery model is the extent to which the delivery model allocates project risks between the Government and the private sector contractor in the most optimal way. The Innovation, incentive and risk allocation diagram above gives an indicative guide to where the different delivery models sit on the continuum of risk borne by the private sector.

The relative importance of this criterion in the evaluation generally will depend on:

• The quantum of the project’s risks
• The complexity of these risks
• Whether the project provides material opportunities to allocate risks (particularly whole-of-life risks) to the private sector within a long-term contract.

In evaluating different delivery models, the Project Team should consider:

• Identifying the model that optimises project (or Program or Precinct) outcomes (not necessarily the model that has lowest risk)
• Conducting a detailed risk assessment incorporating the proposed delivery model
• Identifying risk treatments where appropriate
• The risk treatment strategy, risk ownership and risk allocation under the project delivery model. See the detailed Risk Analysis Guidelines for further information
• Considering another delivery model if, in the planning process, the Project Team identifies unusually high risks or possible consequences that lie outside Government’s tolerance for risk, and:
  • Either a management strategy is not available, or
  • The costs outweigh the benefits.

Traditional and Integrated delivery models allocate design and construction risks to the private sector to a significant degree (other than the Design then Construct model, which only does so for construction risks). However, Integrated models are more effective in doing so than are Traditional models, as described further in the Guidelines for Public Private Partnerships.

Market interest

Typically more relevant for Tier 1 and Tier 2 projects.

The availability of suitable contractors with the capacity and capability to undertake the project under the various delivery models is another relevant factor in determining the most appropriate delivery model. Strong market interest in specific delivery models for the project will improve competitive tension and help maximise value for money.

Government will generally prioritise the facilitation and involvement of local contractors where possible, which should be considered within this criterion.

The suitability of each delivery model to maximise this criterion will depend on the specific details of the project and of market circumstances at the time. The Project Team may need to undertake a market sounding exercise to determine whether the shortlisted delivery models (particularly the more complex, Integrated models) are likely to attract an appropriate number of prospective bidders to ensure competitive tension (see Step 1: Data gathering section on the previous page for further guidance).

Whole-of-life issues

Typically more relevant for Tier 1 and Tier 2 projects.

Under Integrated delivery models, Government can derive significant cost savings over the life of the project from the private sector party planning upfront the project’s lifecycle and other maintenance works. Well-designed PPP and DCMO contracts can allow for a highly effective allocation of the maintenance responsibility for a facility to the private sector, especially for major replacement works, which tend to be more costly than ongoing planned and reactive maintenance.

The allocation of long-term operation and maintenance responsibilities to the private sector creates an incentive to ensure that construction is of an appropriate quality, due to:

• The private sector bearing the risk of these cost elements over the full Contract period
• The competitive pressure in the bid stage to minimise total project costs over the whole life of the asset.

As a result, the private sector undertakes maintenance works, particularly major maintenance or replacement works, at the optimum time, rather than when there is availability of Government Budget funding for them (as is often the case under Traditional and Flexible or Relationship delivery models).

In contrast, Traditional and Flexible or Relationship delivery models usually separate the consideration of expenditure for the D&C components of a project from that for the components relating to operations and maintenance, particularly lifecycle and other major maintenance. Additionally, due to the mechanisms for public sector Budget appropriation, funding constraints for major replacement works and necessary infrastructure improvements can increase the chances of disruption of service delivery due to the eventual maintenance works being more extensive than had Government undertaken them earlier.

Procurement and supervision costs

Typically more relevant for Tier 1 projects.

Integrated delivery models (PPP and DCMO) and the Alliance model involve significantly higher costs during the procurement process than do Traditional models, both for the public sector and for bidders.

Private sector bid costs that are ‘at risk’ (i.e. incurred before selection of a Preferred Bidder) in an Availability PPP project typically have been a minimum of $2 million to $3 million per bidder for a project with a capital value of around $250 million.⁠ (Footnote: See PPP Procurement: Review of Barriers to Competition and Efficiency in the Procurement of PPP Projects, KPMG, May 2010, https://www.infrastructureaustralia.gov.au/sites/default/files/2019-06/Barriers_to_Competition_and_Efficiency_in_the_Procurement_of_PPPs_KPMG_May_2010.pdf) Bidders’ costs increase with the size of the project, although not proportionately: for a $1 billion project, costs ‘at risk’ have been around $5 million to $6 million per bidder. The winning bidder then has needed to incur further costs of perhaps $1 million to $3 million as part of addressing all outstanding issues before executing the PPP Contract and achieving contractual and financial close. However, the value for money resulting from PPP models historically has more than offset these costs, often by a considerable margin. Bidders’ costs for User Charge PPP projects have been substantially higher still, due to the need to analyse and assess usage risk: $20 million or more for a large toll road project.

Public sector costs during the procurement phase also are high, due to the need to employ specialist legal, commercial, financial and technical advisors, and to the relatively long duration of procurement. These costs typically are of a similar order of magnitude to those of the successful bidder. However, public sector management and supervision costs during project delivery and operations generally are significantly lower than under Traditional delivery models, offsetting the high procurement costs, as the PPP contractor monitors its own performance on behalf of the Project Team in normal circumstances.⁠ (Footnote: However, experience of the Territory’s two PPP projects shows that the Project Team still needs to undertake its own monitoring. )

Similarly, the costs to establish and maintain relationships between the parties to Alliance projects can be high, particularly as they require on-going involvement of appropriately senior staff with authority to resolve issues. However, there also are project management efficiencies through integrated management and the elimination of all claims within the ‘no fault, no blame’ culture.

Stakeholder management

Typically only relevant for Tier 1 and Tier 2 projects, but also Tier 3 projects that the Project Team has identified as requiring significant stakeholder engagement.

This criterion measures the extent to which each delivery model assists Government in managing stakeholders through the delivery of the project.

Flexible or Relationship and the Design then Construct delivery models typically are strong in this respect, enabling the Project Team to respond positively to stakeholder issues, whereas Integrated models in particular are less flexible, and lead to a need to educate stakeholders who are likely to be unfamiliar with these models to ensure that there is no compromise to other project success factors.