Council of Australian Law Deans

Name (Individual/Organisation)

Council of Australian Law Deans

Responses

Q8. With respect to ERA and EI:

(a) Do you believe there is a need for a highly rigorous, retrospective excellence and impact assessment exercise, particularly in the absence of a link to funding?

(b) What other evaluation measures or approaches (e.g. data driven approaches) could be deployed to inform research standards and future academic capability that are relevant to all disciplines, without increasing the administrative burden?

(c) Should the ARC Act be amended to reference a research quality, engagement and impact assessment function, however conducted?

(d) If so, should that reference include the function of developing new methods in research assessment and keeping up with best practice and global insights?

The Council of Australian Law Deans (CALD) comprises the 38 deans of Australia’s 39 university law schools. CALD’s objectives include the encouragement of legal research and, where appropriate, the adoption of common positions in relation to issues of mutual concern to Australia’s law deans and university law schools.

As the peak body for all Australian law schools, CALD wishes to provide feedback to the Panel that is undertaking the independent review of the Australian Research Council Act 2001 (Cth). Specifically, CALD wishes to respond to Section 8 of the Consultation Paper for ERA and the related EI. Each question in Section 8 is addressed separately.

8a. Do you believe that there is a need for a highly rigorous, retrospective excellence and impact exercise, particularly in the absence of a link to funding?

CALD submits that there is no need for ERA and EI. It agrees with the view expressed in the Consultation Paper that ERA achieved its initial purpose in refocusing Australian research, and specifically legal research, from an emphasis on quantity to quality of outputs. However, it should also be recognised that the Commonwealth Government’s termination of the component of block research funding that was calculated by reference to publication outputs has been just as significant, if not more so, in compelling the shift from quantity to quality. While no funding is directed to universities for their academics’ actual publication output, there is little risk that universities will return to an emphasis of quantity over quality. CALD also agrees that the EI ‘reference window’ is too narrow, and cannot necessarily capture the impact of legal research which, in many cases, may only have consequences in reform by legislation or adjudication decades after the research was published.

In addition, CALD notes:

- It is difficult to justify the public expenditure involved in ERA and EI when the quality assessments made in those exercises are decoupled from the public funding of universities.

- ERA in particular has given rise to unforeseen consequences that have had negative consequences for legal research. The ARC expressly recognises that law and legal studies (Field of Research 48), along with other humanities and social sciences, are ‘disciplines with more diverse outputs [for which] journal citation analysis may not be sufficiently robust’, (Australian Research Council, ERA 2023 Submission Guidelines (2022) 13) and so peer review is conducted without reference to any robust journal ranking metrics. CALD is aware that many, possibly most, law schools confront internal university quality evaluations (for the management of academics’ performance) that do not recognise the priority of peer review in these disciplines. There is now a plethora of bespoke or institutionally idiosyncratic university ‘Q1’ journal lists, which in the absence of robust metrics do not necessarily resemble any real assessments of publication quality. Although the ARC may consider that this is an internal matter for each university to manage, as the peak body for university law schools CALD must draw attention again to the disadvantage that many law schools, and legal research generally, suffer as a consequence of university managements that try to apply artificial metrics-based assessments to peer review disciplines. It is also known that, in assessing grant applications, some ARC assessors upgrade or downgrade the assessment of the track record of individual researchers by reference to the ERA score of the university in which they are currently employed. It is another example of the counter-productive ‘institutional comparisons and competition’ that are referred to in the Consultation Paper.

- There are plainly anomalies with the institutional attribution of the research assessed in EI when there may be lengthy periods, and changes in a researcher’s institutional affiliation, between the time of publication and the time of impact.

CALD considers that ERA and EI will not be justifiable as genuine assessments of the quality or effect of research in FoR48 and other FoRs relevant to legal publishing if there is a shift to data-driven approaches for evaluations in those FoRs.

8b. What other evaluation measures or approaches (eg data driven approaches) could be deployed to inform research standards and future academic capability that are relevant to all disciplines, without increasing the administrative burden?

CALD rejects the use of any metrics in assessing the quality of research outputs in FoR48 and other FoRs relevant to legal publishing. In response to a request from CALD, Professor Kathy Bowrey, UNSW, published a report in 2012 which recommended: 'That CALD issue a general advisory statement about the problems of using proxies for quality in research active and research quality assessment of individual researchers in law and warn of the detrimental effect on research culture in law that would be caused by using existing data collections to assess individual researcher performance.' (2012 – Assessing Research Performance in the Discipline of Law: The Australian Experience with Research Metrics, 2006-2011 (2012) 21-3)

In its response to the Bowrey Report’s recommendation, CALD stated (2015): 'In light of the serious concerns regarding metric-based research assessment raised by the Bowrey Report and the highly detrimental effect it may pose to Australia’s vibrant research culture in law, CALD condemns the blunt use of quantitative proxies for quality, such as (but not limited to) publishers/journals rankings or citation counts, and cautions against their indiscriminate application towards assessing research activity and/or the research quality of individual researchers in law. CALD notes that the ARC, itself, has expressed reservations about and resiled from certain ERA metric-based measures: notably, journal rankings, now in abeyance. Given this, CALD urges a holistic approach to the assessment of research activity and the research quality of individual researchers in law, utilising both quantitative and qualitative measures, without favouring one over the other, each being subject in the end to inclusive and respectful academic judgement.'

The profile of legal research internationally and nationally has not changed since then, and CALD continues to affirm the need for inclusive and respectful academic judgement in assessing the quality of legal research.

As noted above, as recently as this year the ARC expressly recognised that law and other humanities and social sciences are ‘disciplines with more diverse outputs [for which] journal citation analysis may not be sufficiently robust’. Metrics are not available to quantify the quality of non-journal outputs that are esteemed in legal publishing, such as research monographs, chapters in edited book collections and Non-Traditional Research Outputs (such as law reform submissions). There is negligible Scopus tagging of law journals and other publications. Even then, any partial citation analysis that is available does not accommodate the distinct issue of ‘jurisdictionality’ in legal publishing. Legal publishing in any country concentrates largely on the legal issues arising in that country, and so does not as easily attract citation in other countries. If ERA is genuinely Excellence in Research for Australia, significant proportions of quality legal publishing for Australia will be located in Australian journals or with Australian monograph publishers. While there is interest in some areas of Australian legal research in Commonwealth countries that share a more immediate legal tradition with Australia, there is very little interest in the United States. As a result, there is little prospect of citation in the world’s largest market of legal publishing, from which it would be expected that most citations would come if there were any comprehensive tagging of law journals (although there isn’t). This would not necessarily reflect the quality of the research - it would more likely reflect the ‘jurisdictionality’ of legal publishing.

8c. Should the ARC Act be amended to reference a research quality, engagement and impact assessment function, however conducted?

In light of its submission on Question 8a, CALD submits that the ARC Act should not be amended to reference any research quality, engagement and impact assessment function.

8d. If so, should that reference include the function of developing new methods in research assessment and keeping up with best practice and global insights

In light of its submission on Question 8a, CALD submits that the ARC Act should not include any reference to the function of developing new methods of research assessment.

However, if CALD’s submission on Question 8a were not accepted, and ERA in particular were to be conducted in future years, CALD will be making submissions to the ARC as to how the transparency of the peer review processes conducted in ERA can be improved.

Submission received

09 December 2022

Publishing statement

Yes, I would like my submission to be published and my name and/or the name of the organisation to be published alongside the submission. Your submission will need to meet government accessibility requirements.