An update from Gordon Miles CEO & Romana Moss Director of Education

As CEO of the College, I wanted to set out for our membership a summary of where we are with our work to enhance the delivery of our examinations and the progress we are making.

For over a year now the College has been focused on a programme of internal review work to enhance the experience of those taking our examinations. This is a significant undertaking spanning three years that first involved the expansion of our Examinations Team and then involves a painstaking exploration of our processes and systems, including a review our IT suppliers who underpin our examinations delivery.

To help us with this work we have been informed by two reviews which I commissioned following the results error relating to the March 2022 FRCEM Single Best Answer examination. These were an Internal Investigation and an External Review.

Internal investigation in to the FRCEM results error

The Internal Investigation was conducted by the Chair of our Corporate Governance Committee supported by two members of this Committee: Chris Pickering Fellow and Jasmin Booth, an IT specialist.  This review was completed quickly in the wake of the results error. This identified a number of recommendations to help us reshape our culture to become more responsive, our IT to be more effective and our processes to be more reliable and efficient. All 13 recommendations were accepted in full and last year at our AGM I went through all we were doing to implement these. The recording of the AGM is available to watch here: Access AGM Recording. This activity is in itself a significant three-year change programme. Since then, our work on process and systems improvements has continued.

External review into RCEM examinations

As shared with you previously, the independent External Review commissioned last year into our examinations services was completed in the Spring of 2023 and is published here. Access External Review.

The external review was undertaken by Professor John C McLachlan who has significant medical education/assessment experience. After Professor McLachlan invited feedback from the College membership, more than 60 people were spoken with including candidates, a number of college staff and officers, as well as representatives from other groups such as the Emergency Medicine Training Association and the Emergency Medicine Speciality and Specialist Doctors.

The Review involved a close examination of the processes and procedures which had been in place at the time the error occurred, albeit many had already been improved or were being improved as the review was underway.

As a result of the External Review 17 recommendations have been made, which have also been accepted by the College in full. All 28 recommendations from both the internal investigation and the external review now form part of one work programme aimed at improving our examinations service delivery and are being monitored by the College Trustee Board as well as the General Medical Council.

Where are we now?

As we work though a complex change programme we are working on some 500 process improvements across four phases.

This change programme, called the ‘Examinations Transformation Plan’, is a programme of work encapsulating the recommendations of both the Internal Investigation and the External Review.

The outcome of the Transformation Plan will see candidates and examiners conducting exam-related activities via a portal which will have iMIS (IT system) as the central hub for data, processes and interactions with other platforms as appropriate. This will help to improve the operating efficiency of our systems, with integrations across platforms maximised to enhance our capacity and service delivery reliability, with the outcome being an improved experience for candidates overall.

Our Exams Transformation Plan spans the exam lifecycle:

• Registration
• Exam Preparation
• Exam Delivery
• Results
• Post-Results
• Other

Each process of the exam lifecycle was considered against the following areas of scope:

• Operations
• Quality
• Communications
• Systems
• Resources

The 500 work plan actions to deliver the Transformation Plan are split across four phases and we are working towards the timeline below:

Phase	Phase Completion	Phase Quarters	Phase Work Plan
	30/09/2023	2023 Q1 & Q2	Immediate All Priorities
	30/09/2024	2023 Q4, 2024 Q1, Q2 & Q3	Intermediate Priority 1, 2 & 3
	30/06/2025	2024 Q4, 2025 Q1 & Q2	Intermediate Priority 4, 5 & 6
	31/12/2025	2025 Q3 & Q4	Future All Priorities

So, there is a significant work programme of improvements which will take us to the end of 2025.  Part of this work also includes reviewing our IT support and online examinations services. So far we have completed 40 of the actions, with work ongoing on a further 29 for phase one which we aim to complete by 30 September this year.  We are currently 17% through the entire Examinations Transformation Plan.

We have now also upgraded our Ripley Systems software which is the platform used to manage our theory and OSCE exams. This upgrade was identified in the Internal Review as being necessary and the work to implement it was complex.  The operational demands on the fifteen strong team of delivering the examinations whilst also changing systems and processes is substantial.

Online examination delivery

When offering our theory examinations across a large number of sites around the world we have experienced occasional issues with the delivery.  Some of these are outside our control and that of our supplier, for example internet service deliver issues.  Our suppliers, Pearson VUE, do all they can to manage their own internet service providers to offer reliability, but on occasion issues can and do occur. When these happen, we do all we can to help the candidates with alternative options.

These issues are highly regrettable. We have sincerely apologised for the disruption and stress these issues have caused and are working with our suppliers to learn lessons to reduce the risk of these issue recurring. Pearson VUE has said that they take pride in the service they provide to their partners and candidates, and events such as these are dealt with very seriously and always with the care of our candidates and clients at heart.

The Spring saw issues affecting two diets. A technical error occurred at two test centres in Leicester and Nottingham on 15 March 2023 whereby both locations experienced a technical failure which meant the exam couldn’t be loaded. All candidates from Nottingham were moved to the Derby test centre and completed the exam. All candidates from Leicester were moved to online appointments and all were able to complete the exam.

In April, Pearson VUE and the Royal College of Emergency Medicine acknowledged that owing to unforeseen technical issues, some candidates sitting at overseas centres were examined on a retired version of the MRCEM SBA examination during the Thursday, 25 April 2023 sitting. An initial investigation into the cause of this incident found that there were technical issues that resulted in a retired version of the examination being available for a small number of candidates. Pearson VUE and the RCEM have assessed what corrective action is necessary and have agreed to several processes to ensure this issue is not repeated. The affected candidates have been contacted with information about what will be done to address this issue with the reassurance that the care for our candidates and the integrity of our examination remain our top priority.

We are sorry that these events have impacted our candidates. The disruption was very much regretted, and we apologised directly to all 19 involved.

We know that our supplier aims to deliver 99.99% effectiveness but for examinations sat using OnVue at home this reduces to around 92% largely because of internet delivery issues for home use.

We are working closely with our suppliers to identify and tackle service failures and candidates can help us by reporting during the examination when things are not working as they should, or noise or other environmental issues are impacting, so these can be logged and actioned.  Also, we need to know after the examination if there were issues so we can tackle these with our suppliers.  EMTA also would like to know of issues and on their website:  there is at the foot of the home page a ‘Tell EMTA’ button you can use to do this.  We liaise closely with EMTA on these matters to help improve the service delivery.

Candidate Numbers

In 2022 we delivered 9378 examinations across the UK and in over 45 countries around the world.

	2019	2020	2021	2022
MRCEM Primary	3193	2629	3272	3515
MRCEM SBA	2561	2083	2559	3193
MRCEM OSCE	376	104	667	979
FRCEM SBA	636	574	1025	1190
FRCEM OSCE	397	370	538	501
Intermediate SJP	762	894	497
Critical Appraisal	737	440	199
QIP	375	393	277
Total	9037	7487	9034	9378

A new examination structure was introduced in August 2021 following the start of the current RCEM Curriculum to reduce the assessment burden on trainees. The Intermediate SJP, Critical Appraisal and QIP examinations were withdrawn from the programme of assessment and the skills and knowledge assessed in those old exams are now tested elsewhere in the new examination structure. The new format MRCEM OSCE replaced the previous Intermediate SJP examination, and the knowledge tested in the old Critical Appraisal and QIP examinations are now assessed in the FRCEM SBA and FRCEM OSCE examinations.

As a result of the new simplified examination structure, the total number of examinations delivered only shows a steady increase between 2021 and 2022, whilst the number of individual candidates taking our examinations has grown significantly between in that same period.

Our Staffing

Our College team is a relatively small one with some 70 full time equivalent personnel delivering the full range of our activities. The Examinations Team is our largest single team. This was expanded over the past year from 11 to 15 full time equivalent, albeit we are not at full complement as we are carrying some vacancies which we are currently working to recruit to.

Candidate Experience

The table below outlines the overall experience ratings from candidates for the exams shown. The total is out of five. This rating will be produced for all exams.

Exam	Exam Date	Location	Overall Rating	Total Responses	Total Candidates
MRCEM Primary	Nov -22	UK & Int.	3.6	589 (33%)	1776
MRCEM OSCE	Dec-22	London	3.99	86 (34%)	255
MRCEM OSCE	Feb-23	London	4.06	49 (30%)	161
FRCEM OSCE	Nov-22	London	3.63	88 (35%)	251
MRCEM OSCE	Mar-23	Malaysia	3.85	52 (34%)	153
FRCEM SBA	Mar-23	UK & Int.	3.2	211 (38%)	561
MRCEM SBA	Apr-23	UK & Int.	3.65	759 (38%)	2025
MRCEM Primary	May-23	UK & Int	4.09	501 (29%)	1713

The overall candidate experience ratings for College exams are improving by exam type. Where there is a decrease in theory exam for the FREC SBA there were significant issues at test centres in Leicester and Nottingham.

Complaints

The table below highlights the number of complaints that the College receives and the number of complaints that are attributed to exams.

Year	Number of Exam Complaints
2019	7
2020	0
2021	2
2022	4
2023 (to date)	5

The complaints noted against exams are for the following reasons:

• Technical issue at a test centre
• Image quality
• Candidate not being allocated an OSCE place

Conclusions

The work to enhance our examinations performance is a complex and comprehensive programme of work spanning the next two and a half years. We are determined to tackle this as fast as we can, balancing the need to continue to deliver our examinations whilst changing our processes and systems. We have resourced up our team to help with this, we are working with our suppliers to underpin this change programme with fit for purpose systems and we are delivering on our Transformation Plan.

With any system involving high volumes there will be some bumps in the road.  IT systems sometimes experience faults and outages, people sometimes make mistakes.  To try to anticipate this we have a strong checking regime in place.  No system is guaranteed error free, but we are working to deliver reliable and effective service performance for our examinations. When dealing with systems that operate around 99.9% error free processing then there will be occasional service issues; these are not always within our direct control when they happen but we learn from each issue and work with our suppliers to manage the risks.

We hope that this update gives a sense of the amount of work, resource and management attention that is focused on our examinations service delivery and that you will see by our sharing in an open way as demonstrating our commitment to transparency in our operations.

Gordon Miles CEO & Romana Moss Director of Education.