Impact and Evaluation Group activities

The Rugby Team

The mission of the Impact and Evaluation Group (previously the Rugby Team) is to to 'propose a meaningful and workable way of evaluating the effectiveness of skills development in early career researchers'.

The current terms of reference (2008 - 2012) are to:

• inform national and agency policies and practices relating to the evaluation of skills development of researchers
• provide sector input into shaping a programme to build an evidence base on the effectiveness of developing researchers' skills
• act as a sector ‘sounding board' to the new ‘researcher development' programme with respect to their engagement in helping to build the evidence base.

Each year it agrees a range of projects, based on the recommendations of the Roberts' Policy Forum.

Ongoing projects

• The Impact Framework
• STaRSS
• Understanding researchers' careers and the RT careers profiles project
• Competency framework for researchers
• International researcher evidence base
• Redefining the doctorate
• Other activities

The Impact Framework

The Impact Framework (IF) is an evaluation model for training and development activity specifically tailored to the context of training and development of researchers in higher education (HE).  It explores the potential benefits of investment in training and development activity for the many different stakeholder groups.  It proposes a model for evaluation and highlights example techniques for evaluation.  It is intended that the Impact Framework will foster, support and potentially guide existing and new ways of effectively evaluating researcher training and development, with the aim of encouraging further engagement in the evaluation agenda by higher education institutions (HEIs) and aiding the HE sector in building a more comprehensive evidence base. 

The RTIF was presented in draft form at the Roberts Policy Forum of 2008 and, after stakeholder consultation, in a finalised form at the Vitae Conference, September 2008.  Regular updates on evaluation activity in the sector are provided at the annual Roberts Policy Fora and Vitae Conferences.  A consolidated review of evaluation activity in the sector is provided at the 2010 Vitae Conference.  A JISCmail email network for sharing ideas, information and practice in evaluation in the sector has been set-up.  To join email a request to t.p.bromley@adm.leeds.ac.uk

Roberts Policy Forum 2009: Impact Framework Update Jan 2009

Activity Update March 2009: Evaluation and Impact Framework Update Mar 2009

Activity Update June 2009: Evaluation and Impact Framework Update June 2009

Evaulation update to the Vitae conference 2009 : Impact Framework one year on Sept 2009

For those of you who would like a quick guide to the RTIF please click here: Quick Guide to the Impact Framework

STaRSS

STaRSS (Skills Training and Research Supervision Survey) is designed to provide a national tool that can be used to measure the extent of the awareness and cultural acceptance of the skills agenda by academics. Three volunteer universities piloted it during the summer of 2007. The initial results from the successful STaRSS trial were presented at the UK GRAD conference in September 2007. Feedback from the trial and two workshops has been used to further refine the survey. The survey tool will be finalised and made available for institutions to use.

Understanding researchers' careers and the RT careers profiles project

The objective of this project is to devise a recommended skeleton template for the development of researcher career profiles which all agencies and institutions could use for both qualitative evaluation and in developing tools for supporting individual researchers' career. The framework is designed to provide an underpinning commonality in the collection of career profiles, such that a national picture can be better explored through analysis of larger datasets.

The next stage of the project is to look at the more strategic long term opportunities for the project and tracking researcher careers in general, particularly how this will contribute to gathering evidence for the Impact Framework. In the short term it will also link with other national agencies to agree broad areas of interest for profiling careers and a set of key reference questions. 

Competency framework  for researchers

The usefulness of developing a competency framework for researchers was recognised at the 2008 Policy Forum report. This project was not starting from the same point as with the development of the Joint Skills Statement, in that The Rugby Team believes that this project should be ‘driven' by the sector. There are many other organisations that are interested in this project and need to be engaged. Many institutions have started to develop internal frameworks and promotion criteria, often using the JNCHES role profiles and the HE Academy National Professional Standards Framework for Teaching and Supporting Learning in Higher Education. The Rugby Team will do a scoping study for a staged, realistic approach on how this project can be taken forward on a national level and by whom.

International researcher evidence base

The 2008 Policy Forum agreed that to stay internationally competitive the sector should build an evidence base on non-UK researchers in the UK. Specifically why researchers are attracted to the UK, how training programmes compare with other countries and undertake longitudinal surveys of the benefit of studying in the UK

The Rugby Team will explore this area with the UUK International Office, who are currently undertaking a review of ‘Supply and demand for international doctoral students in the UK'.

Redefining the doctorate

The Rugby Team acknowledged feedback from the sector that the ‘Roberts agenda' raises questions about the nature of the PhD qualification. An aspect of its work has been to encourage the sector to engage in an informed national debate on ‘What is a doctorate?' in the context of the skills and employability agendas. The HE Academy are organising a national conference on the doctorate, in partnership with UKCGE and Vitae, in November 2008, in London. The three core themes are likely to be the global standing, positioning and reputation of the UK doctorate; the appropriateness of doctoral graduates in contributing to the economy (national and local); and the contribution of doctoral graduates to the sustainability of research in their disciplines.

Other activities

The RT06, wherever possible, advise key stakeholders (particularly UUK, RCUK and the sector) on the implications for the skills and employability agendas of major initiatives, such as, the Bologna process, the European Charter and Code and the updated Concordat, and future funding mechanisms.