Evaluation of pre-16 outreach activities (2017-18)
This project was commissioned by the Office for Fair Access (now the Office for Students) in order to better understand what outreach activities are being provided by English universities for young people in order to increase the likelihood of their progression to higher education. The project focused particularly on the objectives of these activities and how the effectiveness of these activities is being evaluated. The project team, which I led, was drawn from UWE, Derby, Sheffield and Sheffield Hallam and the final report was published in December 2018.
The report highlights a range of logistical and epistemological issues with evaluating outreach activities that are intended to operate over very long time periods in a complex social fields. In particular, it questions whether self-report data collected from young people is sufficiently robust to draw conclusions about effectiveness and whether ‘aspiration raising’ is a suitable framework for understanding attitudinal change. Drawing on data collected from third sector organisations also offering outreach, it advocates for a ‘theory of change’ approach to devising and evaluating outreach activities.
The report also makes a series of recommendations, including increasing the focus on transformational pre-16 outreach work, developing a more robust epistemological foundation for evaluation and upskilling the evaluators employed by universities.
Moving On Up (2016-17)
This project was funded by the National Network for the Education of Care Leavers and was designed to shed light on the experiences of those care leavers and other care-experienced young people who participate in higher education. The project had two elements: (a) a statistical analysis of national data of over 650,000 young people in England turning 16 in 2007/08, and (b) an online survey of 212 current students who had been ‘in care’ as children. The final report was published in November 2017 and you can watch a one minute summary video below.
In particular, the data suggests that there are six elements that support success for care leavers in accessing and participating in higher education: (1) strong attainment at Key Stage 4, (2) a well-managed transition from care to university, (3) membership of the university community, (4) resilience and determination, (5) strong mental health and disability support at university, and (6) access to alternative pathways into higher education. You can watch my 20 minute presentation of the report below.
Evaluating Student Financial Support (2015-16)
This project was commissioned by the Office for Fair Access from an multi-university team led by Sheffield Hallam University. The purpose was to produce resources to allow institutions to evaluate the additional financial support that they provide to disadvantaged students – most commonly in the form of means-tested or other targetted bursaries. The final report was published in December 2016.
I led on the development of an analytical approach (based around a discontinuity logistic regression design) to allow institutions to explore whether bursaries had been successful in ‘levelling the playing field’ for disadvantaged students, drawing on their own student records data around retention, completion, degree results and graduate employment. This was converted into an easy-to-use toolkit that is now being used by most English universities.
Assessing Impact and Measuring Success (2014-15)
This project was funded by the Society for Research into Higher Education and was undertaken with my colleague Richard Waller. It aimed to explore changing perceptions of manager-practitioners engaged with widening participation in higher education, drawing on two sources of data: (a) extended telephone interviews with ten former senior managers within Aimhigher (the government’s flagship programme from 2004 to 2011), and (b) an online survey of then-current managers within individual institutions, with 57 institutions responding.
Among other things, we found that there had been an atomisation of effort since the end of Aimhigher, with the previous collaborative ethos giving way to more competitive practices. There had also been a decline in the activities aimed at younger age groups that the Aimhigher manager-practitioners had valued highly, in favour of a focus on shorter-term and more readily measurable programmes. We concluded that these might be successful at the university level, they were unlikely to have a transformative effect on the sector as a whole, representing a wasted opportunity.
Money Matters (2013-14)
This project was funded by the British Academy and was a collaboration with colleagues in New Zealand and the United States. We were interested to understand how younger students in the three countries conceptualised student debt and whether there were any differences. We collected data from 624 participants through an online survey and used factor analysis to explore the different meanings that debt can have, especially in a period when the amount of debt taken out by students is on the rise.
We found that the picture was more complex than had previously been appreciated, with students distinguishing between four different elements – rather than a simple debt aversion scale. We found that individual students saw debt as more or less useful to give them access to education or the lifestyle they wanted. Others took a greater or lesser interest in the details of their student debt (e.g. the repayment terms), while debt was a source of anxiety for some, but not others. Interestingly, we found that debt attitudes were not strongly related to social class or financial literacy, but they were related to gender and personality. There were also differences between the three countries.
Certificate of Personal Effectiveness and GCSE outcomes (2011-12)
This project was commissioned by the ASDAN qualification-awarding body and undertaken with my colleagues David James and Kathryn Last. We were asked to explore the impact of ASDAN’s Certificate of Personal Effectiveness (CoPE) on young people’s outcomes at GCSE. The final report was published in April 2012.
CoPE is a skills-led qualification that involves young people tackling a series of challenges that are negotiated with their teacher, often drawing on their own interests. It is usually undertaken during Key Stage 4 alongside GCSEs or vocational qualifications, with young people building a portfolio of evidence over one or two years.
We used logistic regression analysis alongside semi-structured interviews (with staff) and focus groups (with young people) in four case study schools. We were able to demonstrate that there was a relationship between taking CoPE and improved GCSE results, particularly in English. We identified three possible routes by which this occurred: (a) increased motivation for school due to the integration of young people’s interest, (b) the development of a reflective learning orientation through the plan-do-review cycle used by CoPE, or (c) increased written communication skills through the assembling of the CoPE portfolio.
You can watch a 30 minute presentation on the research at the Education and Employers Taskforce by me and David James below.