Anonymous #51

Name (Individual/Organisation)

Anonymous #51

Responses

Q1. How could the purpose in the ARC Act be revised to reflect the current and future role of the ARC?

For example, should the ARC Act be amended to specify in legislation:
(a) the scope of research funding supported by the ARC
(b) the balance of Discovery and Linkage research programs
(c) the role of the ARC in actively shaping the research landscape in Australia
(d) any other functions?

If so, what scope, functions and role?

If not, please suggest alternative ways to clarify and define these functions.

Research, by its very nature, explores and creates knowledge. I hope that the Australia (through the ARC and other agencies) will continue to fund a broad scope of research across all disciplines, including fundamental and applied research.

The ARC is the primary source of Discovery program funding and Linkage infrastructure funding. Together this funding drives Australian university research and training including fundamental research. History has shown that we are not always good at recognizing how fundamental research will be applied in the future. For these reasons, I think it would be detrimental for the ARC to reduce funding in this already competitive area of research.

I am concerned it could create perverse outcomes to create a strict percentage of Discovery and Linkage programs in the ARC Act. For the more applied Linkage programs to serve the nation well, Australia may need to change the balance over time. For example, let’s say that a new commercial area emerges (e.g., CO2 drawdown, optical communication) that requires understanding fundamental theory (e.g., in physical sciences) to properly harness. If a strict percentage of Discovery vs. Linkage exists then the ARC might find itself unable to be agile enough to fund the research that is necessary to enable translation.

In terms of actively shaping the research landscape in Australia, the ARC does this to a limited extent through the Special Research Initiative (SRI) programs. Perhaps there are ways to increase SRI programs in partnership with other parts of government so that university researchers have clear access to funding from other government strategies and initiatives? Perhaps the ARC could look to broaden its research into areas like the US NSF’s RAPID, EAGER and Ideas Lab programs.

Q2. Do you consider the current ARC governance model is adequate for the ARC to perform its functions?

If not, how could governance of the ARC be improved? For example, should the ARC Act be amended to incorporate a new governance model that establishes a Board on the model outlined in the consultation paper, or another model.

Please expand on your reasoning and/or provide alternative suggestions to enhance the governance, if you consider this to be important.

Governance of the ARC would benefit from a Board to oversee a transparent process of decision-making. This would provide assurance to the Minister, Parliament and the research community that decisions have integrity. Such a Board would strongly benefit from including a significant proportion of academics.

Q3. How could the Act be improved to ensure academic and research expertise is obtained and maintained to support the ARC?

How could this be done without the Act becoming overly prescriptive?

The ARC evaluation process should include world-class, experience academics and mid(-early) career academics (to create experience throughout the academic ranks). Perhaps the Act should indicate specific areas where academic and research expertise is required: for example, the Board, the ARC Panel of Experts, on-loan from universities to the ARC Office (like the USA NSF) etc.

Q4. Should the ARC Act be amended to consolidate the pre-eminence or importance of peer review?

Please provide any specific suggestions you may have for amendment of the Act, and/or for non-legislative measures.

Yes, the ARC Act should indicate that peer reviewers (assessors and the ARC Panel of Experts) are essential for world-leading evaluation of research.

Q5. Please provide suggestions on how the ARC, researchers and universities can better preserve and strengthen the social licence for public funding of research?

As recently instituted by the ARC, the National Interest Test (like the ‘Benefit’ section) should be part of the expert evaluation. Expert peer reviewers should assess proposed research proposals for excellence as well as alignment with the National Science and Research Priorities.

Beyond the existing National Interest Test, the ARC could consider expanding the description of materials for the "Benefit” section to encompass broader impacts of research. This would create culture change in Australian research as well as strengthen social licence for the public funding of research. For example, see Section d(i) of the NSF guidelines for proposals:
https://beta.nsf.gov/policies/pappg/23-1/ch-2-proposal-preparation#2D2b

The ARC could also promote the training and mentoring aspects of its funded work to researchers, government and industry.

Q6. What elements of ARC processes or practices create administrative burdens and/or duplication of effort for researchers, research offices and research partners?

One type of CV/ROPE should be required for all grants (e.g., LIEF requires a different type of CV from a Discovery Program grant).

Research Offices spend an inordinate amount of time assessing “Eligibility”, especially in large team grants. This administrative burden is reduced if the penalty (losing the grant) was reduced and instead the ineligible team member was removed if the grant is funded.

Overseas PIs find the ROPE requirements challenging, which commonly results in lead CIs writing these documents. A simplified CV for overseas PIs would be better.

The section on any past leave/part-time work/moves in the application must be entered for each individual proposal and cannot be saved in the RMS system. Some people have more than a dozen items that must be entered manually for each grant. It would help to allow this section to be saved in the personal data section and updated as needed.

It is not clear why a letter of support is required from an organization for a Fellowship. A few simple questions to ensure any workload relief, space and infrastructure support, and confirmation that the proposal is supported by the unit would likely suffice.

Q7. What improvements could be made:

(a) to ARC processes to promote excellence, improve agility, and better facilitate globally collaborative research and partnerships while maintaining rigour, excellence and peer review at an international standard?

(b) to the ARC Act to give effect to these process improvements, or do you suggest other means?

Please include examples of success or best practice from other countries or communities if you have direct experience of these.

FUNDING
It is essential for Australia's research excellence that the funding for ARC programs and processes compare favourably with leading research countries and that funding is indexed and long-term. Funding is essential for research excellence, agility, collaboration, and rigour in the process.

TIME FRAMES
To improve excellence and collaborations/partnerships it would be helpful to continue the recent approach that the ARC provides predictable timelines that respect the academic cycle, and school holidays. These timeslines provide the certainty needed to develop collaborative research and partnerships. As indicated above, relaxing the rules around PI’s CVs would also help to promote collaboration.

COLLABORATIONS
To ensure that collaborators and partners can still be involved, it would help to announce grant outcomes in a short timeframe after submission (<6 months).

INTERNATIONAL APPROACHES TO PEER REVIEW
I have experience submitting and reviewing grants in the Canadian system (NSERC and CSA proposals) and the US system (NSF and NASA proposals and I have been on 1 NSF panel and 5 NASA panels). Due to larger volume of researchers these systems have panels that are more closely aligned to the discipline which allows for a more richly referenced and higher level proposal. Given the small volume of Australian researchers and fierce competition, I understand why the system is different here (although I prefer the USA system for many reasons).

The US system tends to grow and support early-mid career researchers well. I think that this is because it focuses more on the proposal quality than the researcher’s track record. For example, NASA proposal reviews ask for a summary and then strengths and weaknesses in intrinsic merit, relevance to the program and cost.

In Australia, early career researchers tend to need to be supported by a DECRA or senior mentor’s Discovery grant (since salary is not allowed for the lead CI). I feel that the ARC approach results in an unhealthy dependency of early career researchers on senior researchers and creates problems in the pipeline for underrepresented groups who get less sponsorship from senior researchers (e.g., junior women in STEM).

In the Discovery program, I think this could be alleviated by reducing the % in the Assessment Criteria on the Investigator and increasing the % on the Project Quality to ensure that innovative ideas are rewarded (future aspirations) rather than the past opportunities (track record and sponsorship). This will also allow partners with little academic experience to better contribute to the proposal (without a penalty by assessors in the "Investigator" section).

PILOT BLIND REVIEW PANELS
Last year, I experienced blind review on a NASA Panel. This truly focused us on the research proposed. One academic who had experience outside the discipline received a grant in this process; some panel members thought that this may not have been the outcome if their CV had been revealed. Two outstanding early career academics (one female and one male) did not initially receive grants in this process because they did not sufficiently demonstrate in the proposal that they had the experience to do the very challenging work proposed. Fortunately, there were several “reveals” for reassessment after the blind process and these two early career academics received grants once their identity was revealed and we realized they had underplayed their capabilities and past experience. All proposers were revealed at the end of the process. This process was time-consuming, but thorough and eye-opening. This demonstrated to me (again) that it is impossible to avoid bias, even in blind review, but blind review is better.

Q8. With respect to ERA and EI:

(a) Do you believe there is a need for a highly rigorous, retrospective excellence and impact assessment exercise, particularly in the absence of a link to funding?

(b) What other evaluation measures or approaches (e.g. data driven approaches) could be deployed to inform research standards and future academic capability that are relevant to all disciplines, without increasing the administrative burden?

(c) Should the ARC Act be amended to reference a research quality, engagement and impact assessment function, however conducted?

(d) If so, should that reference include the function of developing new methods in research assessment and keeping up with best practice and global insights?

PART A
I do not believe that there is a need for such an exercise. The question indicates some of the reasons that this exercise is not useful: it is highly rigorous (i.e., expensive and time-consuming), it is retrospective and so not indicative of how research is moving forward, and it does not link to funding outcomes.

PART B
Finding appropriate evaluation measures or approaches is a vexing challenge that is the topic of much research. Research standards tend to drive poor behavior (e.g., Hicks et al., 2015, Nature, 520, 429). For example, citation metrics drive self-citation, ‘clubs’, and tend to discriminate against minorities in a discipline (e.g., Chapman et al., 2019, Proc Royal Soc B). Quantity of publications tends to drive researchers towards more publications in lower ranked journals and does not account well for books and other large bodies of work. Non-traditional research outputs (e.g., media, art, collections) can have high impact, yet are not captured in data-driven approaches. The Leiden Manifesto (Hicks et al., 2015, Nature, 520, 429) is a possible approach, but it does increase administrative burden. For minimal administrative burden, research could be evaluated using existing ranking systems.

It is not clear to me that evaluating future academic capability is a) possible; and b) the role of the ARC. I strongly feel that if anyone has this role it is a hiring committee in a specific field – and sometimes a hiring committee gets it wrong, so it is an imperfect process.

PART C
No, the ARC should not be involved in assessing research quality, engagement and impact beyond grant programs.

PART D
I suggest that this question could be informed by Hicks et al. (2015) Nature, 520, 429 and the 1000+ citing articles.

Q9. With respect to the ARC’s capability to evaluate research excellence and impact:

(a) How can the ARC best use its expertise and capability in evaluating the outcomes and benefits of research to demonstrate the ongoing value and excellence of Australian research in different disciplines and/or in response to perceived problems?

(b) What elements would be important so that such a capability could inform potential collaborators and end-users, share best practice, and identify national gaps and opportunities?

(c) Would a data-driven methodology assist in fulfilling this purpose?

PART A
I think that this question is referring to generic evaluation of research excellence and impact, not specific grants. In this case, like Q8, I don’t think that it is valuable for the ARC to be *evaluating* Australian research (leave this to other government agencies). Furthermore, it creates a mixed message in how the ARC can transparently evaluate grants.

Instead, I think that it would be helpful for the ARC to concentrate on *demonstrating* the value of Australian research. Media releases / other communications related to ARC-funded research projects would highlight outcomes and benefits of research. Research meets Parliament type events would help show the value and excellence of Australian research to important stakeholders. A requirement that any university researcher media releases / communications acknowledge ARC funding where appropriate would help.

PART B
I don’t think that this is part of the ARC’s role. This work is more relevant within universities and through the Office of the Chief Scientist.

PART C
No, see previous question.

Q10. Having regard to the Review’s Terms of Reference, the ARC Act itself, the function, structure and operation of the ARC, and the current and potential role of the ARC in fostering excellent Australian research of global significance, do you have any other comments or suggestions?

I understand that this review is largely about the ARC Act and I hope that the following comments relate to the the ARC management functions and structures to ensure they are contemporary, fit for purpose, and meet the needs of stakeholders

Researchers are driven by funding and promotion and so the ARC’s requirements are quite important to Australian research culture. I feel that the ARC could do a lot more to encourage a healthy and collegial research community within Australia. The ARC can make simple changes to its structures that will more strongly support early career researchers and under-represented groups:

1. Let applicants know about awards in a timely manner (e.g., less than 6 months) so that early career researchers can plan their careers and families and so that collaborators partners can commit.

2. Follow best-practice of the US National Science Foundation by clearly indicating policies around discrimination and including a diversity statement in the eligibility section of grant proposals:
https://beta.nsf.gov/policies/pappg/23-1/ch-11-other-post-award-requirements#a-non-discrimination-statutes-and-regulations-751 Or, follow the approaches for diversity and inclusion implemented by the UK Wellcome Trust: https://wellcome.org/what-we-do/our-work/research-culture

3. Require a specific expectation that all grant applicants will indicate how their research project will promote diversity and inclusion and a supportive research culture. For example, specifically request example equity and diversity activities for the “Benefit” section (see above) and provide best-practice exemplars. This initiative will pave the way for increased understanding of the positive impact of a more diverse workforce on organisational and research outcomes. Consider supplemental funding for career life balance (like NSF: https://beta.nsf.gov/policies/pappg/23-1/ch-2-proposal-preparation#2F2)

4. Require that a mentoring plan be required for any postdoctoral fellows (https://beta.nsf.gov/policies/pappg/23-1/ch-2-proposal-preparation#2D2i-i)

5. The rule that an academic can only hold two grants at one time has some perverse outcomes with respect to sick/maternity/paternity leave and part-time work. If someone employed by a grant takes leave/part-time work, then the grant must be extended and the academic must wait to apply for a second grant. This policy could be revisited in the case where someone employed on a grant takes sick/maternity/paternity leave or part-time work.

Submission received

14 December 2022

Publishing statement

Yes, I would like my submission to be published but my and/or the organisation's details kept anonymous. Your submission will need to meet government accessibility requirements.