The research evaluation system, as well as the rewards and career aspects for researchers, are increasingly being discussed. Institutions, entities, and organizations at the international level have long been seeking new criteria capable of highlighting and rewarding research quality. Common points of discussion include the development of new indicators that go beyond quantitative measures and criteria that take into account practices of open science.
COARA – Coalition for Advancing Research Assessment
Agreement on reforming research assessment
In recent years, the European Commission has issued several recommendations and guidelines for research evaluation and the adoption of open science models. The Commission’s activities, along with the work of COARA, have led to the drafting of an agreement for the reform of the research evaluation system, published in July 2022. The Agreement aims at establishing a common action for the reform of evaluation, based on specific principles and commitments to be achieved within defined timelines.
general principles:
> Respect for ethics and scientific integrity.
> Safeguarding the freedom of scientific research.
> Respect for the autonomy of research institutions and organizations.
> Independence and transparency of data, criteria, and evaluation processes, ensuring access and ownership by the scientific community to critical infrastructure and tools used.
principles for evaluation criteria:
> Quality and impact.
> Diversity, inclusion, and collaboration.
In addition to these principles, there are 10 fundamental commitments to identify and promote good practices for quality research, as well as to introduce new criteria for evaluation, replacing the inappropriate use of quantitative metrics.
Among the aspects to be considered, the agreement emphasizes the importance of open science practices, such as result sharing, data openness, and sharing for increased transparency, collaboration, and impact of scientific research. Open science activities and skills are also considered for the enhancement of roles and careers.
As of now, over 300 members, including institutions and universities, have joined the agreement, including ANVUR and UNIMORE.
USEFUL LINKS
EC I Agreement on reforming research assessment
ANVUR I adesione all’Agreement on reforming research assessment
PREVIOUS European Commission ACTIONS:
EU Recommendation 2012/417 and 2018/790:
> In 2012, the European Commission recommended adjusting the recruitment and career evaluation system for researchers and allocating funding in a way that rewards those who embrace the culture of sharing research results. This includes developing alternative evaluation models, measurement criteria, and appropriate indicators.
> In 2018, an additional indication was made to use next-generation metrics that consider the openness of research, also understood as social impact.
Open Science Policy:
The European Commission emphasizes the need to develop new indicators for research and impact suitable for evaluating open science practices. The policy also suggests that open science activities should be recognized in the career progression system.
These needs and recommendations are reiterated in Horizon Europe, emphasizing the Commission’s ongoing commitment to fostering open science, alternative evaluation methods, and the recognition of open science activities in career advancement systems.
LINK UTILI
EC | RECOMMENDATION 2012/417 | Recommendation 2018/790
EC | OPEN SCIENCE POLICY
EC | REPORT: CONSULTATIONS 2021
UE | EVALUATION OF RESEARCH CAREERS FULLY ACKNOWLEDGING OPEN SCIENCE PRACTICES
DORA
Declaration on research assessement
The Declaration, produced by a group of editors and publishers of scientific journals and academics, was drafted in 2012 and is addressed to funding agencies, institutions, universities, journals, and individual researchers. It highlighted the shortcomings of traditional metrics, especially the Journal Impact Factor.
It recommends not using journal-based metrics as a surrogate measure of the quality of individual scientific articles or to assess the contributions of a single researcher, or in hiring, promotions, or funding allocations.
It suggests considering the value and impact of all research outputs (including datasets and software) and taking into account a wide range of impact measurement forms.
It emphasizes the importance of transparency: the criteria used in evaluating the scientific output of those seeking funding should be explicit, as well as those used for career progressions and hiring decisions.
USEFUL LINKS
Dora declaration | full text
LERU
League of European Research Universities
The theme of indicators and the need to identify new metrics is reiterated by Leru, which recommends:
> DEVELOPING A BIBLIOMETRIC POLICY BASED ON THE PRINCIPLES OF THE LEIDEN MANIFESTO;
> ADOPTING NEW EVALUATION MODELS IN INTERNAL PROCEDURES FOR CAREER ADVANCEMENT AND RESEARCH ASSESSMENT;
> INFORMING RESEARCHERS ABOUT BEST PRACTICES AND INAPPROPRIATE USE OF TRADITIONAL INDICATORS;
> TRAINING YOUNG RESEARCHERS, ALREADY IN THE EARLY STAGES OF THEIR DOCTORAL PROGRAMS, FOR RESPONSIBLE USE OF METRICS.
USEFUL LINKS
LEIDEN MANIFESTO
OPEN SCIENCE AND ITS ROLE IN UNIVERSITIES: A ROADMAP FOR CULTURAL CHANGE
SE – Science Europe
Regarding the evaluation processes, Science Europe has outlined some guidelines for institutions and universities. In particular:
> Make information about the evaluation process accessible and user-friendly.
> Publish the results of the assessments conducted.
> Focus assessments of research products on their content rather than on bibliometric indicators.
> Provide the possibility of introducing a “right to reply” system for candidates.
> Conduct evaluations without discrimination, conflicts of interest, or partiality.
> Monitor the progress, fairness, and validity of evaluation methods and reconsider them at regular intervals.
> Exchange information with other institutions about their evaluation systems and related best practices.
USEFUL LINKS
Science europe – Position Statement and Recommendations on Research Assessment Processes
UNESCO
Even UNESCO, in its Recommendation for Open Science, emphasizes evaluation systems, highlighting that current models are not compatible with those of open science.
The evaluation of research contributions and career progressions should take into account open science practices, thereby contributing to their dissemination.
Institutes should adopt the principles of the DORA Declaration, especially the principle that research should be assessed in terms of quality rather than quantity, and not based on traditional bibliometric values such as the journal impact factor.
There is a highlighted need to consider the external impact produced by research contributions and to take into account the specificities of disciplines and their respective open science approaches.
Encouragement is given to funders, research institutions, editorial committees, societies, and publishers to adopt policies that require and reward open access to scientific knowledge, including scientific publications, research data, software, source code, and hardware.
USEFUL LINKS
unesco | Recommendation on Open Science
Coalition S
The members of Coalition S have committed to promoting responsible and equitable approaches to research evaluation. Funders within Coalition S conduct assessments of contributions based on the intrinsic merit of research work, not on the publication channel, its impact factor (or other journal-level metrics), or the publisher.
The members have endorsed the DORA Declaration.
For funding, recognition is given to preprints, datasets, software, inventions, educational products, and services. A series of qualitative impact measures have been adopted, including the influence on society at large. Funders are working to eliminate the inappropriate use of journal-based metrics and are committed to using bibliometric indicators responsibly and judiciously.
Additionally, Coalition S has established a Research Assessment Task Force to help identify, exchange, and align best practices in research assessment.
USEFUL LINKS
Plan s principles
The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management
Monitoring the effects of Plan S on Research and Scholarly Communication
World Conferences on Research Integrity Foundation ↗
WCRI Fondationd aims at promoting research integrity.
Hong Kong Principles ↗
The research assessment principles produced in 2019.
OS I Open Science and Research Assessment