April 23, 2015

Not Another Metric

The release of Research Engagement for Australia by the Australian Academy of Technological Sciences and Engineering (ATSE) does not provide the needed balance to the Excellence in Research for Australia (ERA) assessment to recognise the excellence of university research in support of industry and public needs.

The IRU recognises the work ATSE has done to extract from current data metrics about engagement with research users in industry and the public sector. However, ATSE is wrong in its guiding assumption that a thorough qualitative assessment of the value of research for users outside the academic communities is not feasible.

With the third ERA to be completed in 2015, the Government should redirect the resources that support ERA to assessing the extent of industry driven research, with the next ERA to follow after the impact assessment.

ERA’s reinforcement of traditional assessments of standing has been major. Industry driven research has less standing in university culture than research that achieves great recognition amongst other researchers. Balancing those priorities, without diminishing the importance of investigator led research, will not be easy.

A parallel exercise to ERA to assess the real impact of a university’s research is a central element to changing the importance given to supporting industry driven research. Such an exercise must have a qualitative element. Like ERA, it must be benchmarked against an objective standard rather than being rankings-based. ATSE’s proposal does not do this.

IRU also has concerns about the detail of the ATSE approach:

  • it relies exclusively on financial input measures, when ‘impact’ is primarily about outcomes;
  • it ranks universities and then arbitrarily forces them into four sets with no clear basis to distinguish the groups. Comparisons with ERA ratings which it does are hence invalid. A true comparison requires a similar objective assessment of value
  • by design, the method will not capture all inputs, even for engagement with business, such as fee-for-service consultancy income;
  • there is much highly valuable research engagement, particularly with sectors other than business, that is transacted without substantial reliance on financial inputs.