October 15, 2020
ERA and EI review – IRU response
The IRU has provided a response to the Australian Research Council’s (ARC) review of Excellence in Research for Australia (ERA) and the Engagement and Impact Assessment (EI).
Overview
Australia’s university research system is performing strongly. It has directly improved the quality of life of all Australians.
Excellence in Research for Australia (ERA) provides a credible evaluation of the quality of our university research. Overall, ERA has met its objectives as an evaluation framework and national stocktake of research.
The Engagement and Impact (EI) assessment provided the first comprehensive guide to how well the good research demonstrated in ERA is translated into action by business, Government agencies and community bodies.
EI now requires substantive change to seriously build off the learning from the first assessment. This would refine EI towards the activities of greatest benefit with the least administrative cost.
Excellence in Research for Australia
Excellence in Research for Australia (ERA) provides a credible evaluation of the quality of our university research. Overall, ERA has met its objectives as an evaluation framework and national stocktake of research.
ERA has matured into a robust, evidence-based means for identifying and developing areas of strength against international benchmarks. Over its decade of four assessments, the ERA has encouraged universities to ensure research investment is well directed to produce quality outcomes.
ERA has identified research excellence across the full spectrum of fields, specialisations and university sizes. It has also provided assurance to stakeholders that Australia’s research has improved over time. This has narrowed the spread of results across the sector as all universities have shown their true capabilities.
The IRU’s preference is for limited change to ERA targeted towards enhancing the integrity, acceptance and use of ERA data, without compromising comparability to earlier and future iterations. As a mature evaluation system, focus should now be on how ERA can be refined for operational purposes and better used to meet its objectives.
The broad soundness of the ERA methodology means ERA is already used extensively for internal decision making, albeit with adequate caution when drawing comparisons across fields of different sizes and evaluation methods.
Use of ERA data could be developed further with greater transparency in reporting, including publication of detailed volume metric ERA data and feedback.
Engagement and Impact
The Engagement and Impact (EI) assessment provided the first comprehensive guide to how well the good research demonstrated in ERA is translated into action by business, Government agencies and community bodies.
EI requires substantive change to seriously build off the learning from the first assessment. This would refine EI towards the activities of greatest benefit with the least administrative cost.
As a first exercise, it was successful in identifying exemplar cases. With over 200 high impact studies and 170 high engagement narratives, EI provides a partial evidence base for how institutions have translated research into impact across the spectrum of disciplinary and interdisciplinary areas. This has the potential to assist universities to identify and refine processes to incentivise and enable greater impact, alongside processes already underway such as new career paths for academic staff beyond the traditional disciplinary-based teaching and research functions.
EI’s contextual engagement indicators, including the (increasing) share of research funding from business, Government and not for profit research end-users, are indicative of how universities have embraced the engagement agenda.
Following the first assessment of its type and with a targeted focus, priority should be to optimise EI based on what was learnt, with less concern about comparability to EI 2018. The complexity of identifying impact and attributing its causes, combined with a current lack of suitable quantitative indicators, means EI is primarily a qualitative and targeted exercise. This is unlike ERA, which is comprehensive and mostly quantitative-driven. This is appropriate, but it does make EI resource-intensive and difficult to generalise.
Main areas for action
The IRU proposes eight areas for action.
- Stronger collective advocacy of ERA and EI results
- Retain the ERA rating scale, but publish volume metric data
- Improve transparency through public availability of more ERA data
- Consider extending citation analysis to peer-reviewed ERA fields
- Combine engagement and approach to impact in EI
- Refocus EI at showing first the sections of Australia’s economy and society most using research to improve outcomes
- Continue to develop ERA and EI evaluations of Interdisciplinary and Indigenous research
- Alternate ERA and EI cycles every six years, with EI to be the next assessment.
