TM RM Assessor Supervision Guide Marker Monitoring for units with Batch Checking of Live Marking OCR 2017
Table of Contents Introduction... 3 Marker monitoring and batch checking... 3 Batch checking on RM Assessor... 3 Routine batch checking of live marking... 4 Reviewing seeding scripts... 4 Using marker monitoring data... 5 Checking of live marking in response to an alert... 5 Reviewing live marking on RM Assessor... 7 OCR 2017 2
Introduction This guide replaces the section Marker monitoring in the OCR RM assessor Supervision Guide. It is written for principal examiners and team leaders of units where the quality of marking is monitored by marker monitoring data and batch checking of live marking. Typically, batch checked units are those with high mark tariff extended responses marked by levels-based mark schemes, and/or have a high degree of optionality. Marker monitoring and batch checking It is the responsibility of the principal examiner to ensure that examiners mark consistently by monitoring their work throughout the marking period. Where it is deemed necessary, corrective action must be taken and recorded fully to provide an audit trail. There are several sources of information available to principal examiners and team leaders, helping you to identify instances of poor marking and take appropriate action. These include seeding scripts, marker monitoring data and live marking. Batch checking on RM Assessor The introduction of on-screen marking has provided team leaders with more detailed information about the performance of markers. There is greater opportunity to give feedback where markers are deviating from the agreed standard. In some instances, however, increased feedback has led to less consistent marking, with markers reacting to comments from team leaders and over-compensating as a result. Therefore, where batch checking is done on RM Assessor, no specific feedback should be given to markers except on the incorrect use of annotations, or where there is a lack of annotations. Below is a flowchart illustrating how the batch checking process works. Marker monitoring flags issue No Issue with application of the mark scheme Complete batch check and record on batch check form Marking ok? No AE asked to stop marking Supervisor reviews seeding script Seeding script ok? No Issue with annotation Notify AE AE reviews scripts already marked Examiner continues marking OCR 2017 3
Note: Team leaders and principal examiners should complete batch checks on all markers, even if they do not see evidence in seeds or receive an alert suggesting that there is a problem with marking. Routine batch checking of live marking During the marking period, you will conduct batch checks for all markers in your team as part of the monitoring process. A minimum of 20 scripts must be reviewed across the batch checks conducted and these checks should be spread evenly thoughout the marking period. Where the marking period is 21 days or more, 4 batch checks of at least 5 scripts each should be conducted Where the marking period is shorter fewer batch checks may be conducted, but these should still total 20 scripts overall. A minimum of two batch checks of 10 scripts should be conducted. It is important to sample the range of optional questions, where applicable, and to look at scripts across the mark range. You must record these batch checks on the Batch Check Form. NOTE: The minimum of 20 scripts can be made up from scripts reviewed during batch checking and live scripts checked as a result of an alert. Once you have completed a batch check (50%,100%), take one of the following actions: If a batch check reveals the examiner to be marking in line with the agreed standard or is within tolerance, please email the completed batch check form to the Monitoring and Support team. If a batch check reveals the examiner to be marking either consistently harsh or consistently generous (in addition to evidence of the same consistency from the seed scripts), please identify this on return to the Monitoring and Support team. It may be possible in this circumstance for the examiner to be scaled. If a batch check reveals inconsistent marking outside of tolerance, please complete the Request to Stop Marker tab on the batch check form and return to the Monitoring and Support team. Please do not wait until the 50% or 100% batch deadline if you have concerns about an examiner s marking before this date. Reviewing seeding scripts Seeding scripts provide a snapshot of marker activity allowing team leaders to keep a check on each examiner s adherence to the mark scheme. However, because seeding script flags in RM Assessor are based on cumulative deviations of each item from the definitive marks, seeding scripts can flag when in fact there is no major problem with marking. Seeding scripts should be used on an on-going basis and as a prompt to investigate marking further if you have concerns. Where such checks are done, a Live Script Check Form should be completed to provide an audit trail. If a seeding script is flagged in RM Assessor, you should initially only look at the questions where the marker has deviated from the definitive mark, as set by the principal examiner. At this stage, you should not provide markers with specific feedback, except about the use of annotations. Having reviewed a seeding script, you may take one of the following actions: OCR 2017 4
If the seeding script does not raise concern, the marker may continue to mark. If the marker is not using annotations correctly, the team leader must provide feedback about the correct use of annotations the marker must review all scripts they have already marked and make corrections where appropriate the marker may continue to mark. If the seeding script seems to indicate a more significant problem with marking, the team leader should proceed to a check of live marking. Using marker monitoring data Marker monitoring data on seeding scripts and live marking is used to construct a profile of each marker s performance. This negates the need for Principal Examiners and Team Leaders to keep their own records of seeding script marks. This data is also used to provide alerts which will be generated at set points in the marking period if an examiner is potentially aberrant. Where an alert is received, this should act as the trigger for a more detailed review of live marking. There are two types of alert: Seeding script alert this is raised when the average number of deviations across a number of seeding scripts are beyond a set threshold Live marking alert the marking profile of live scripts is significantly different from that of the rest of the panel You must record these whole script checks on the Alert Response form, so that there is an audit trail of the evidence that informed your decision on next steps. Checking of live marking in response to an alert Where you have received an alert about seeding scripts or live marking, you should undertake a review of live marking. The following steps should be closely adhered to: A minimum of 3 live whole scripts should be checked in detail at any stage where an alert is raised These checks must be recorded on the Alert Response form, including your own marks for the scripts checked and relevant comments about any trends identified As a result of your script check, you may take one of the following actions: If the script check reveals the examiner to be marking in line with the agreed standard or is within tolerance, the team leader must complete an Alert Response Form and email it to the Monitoring and Support Team to tell them the maker may continue marking. If marking is consistently out of tolerance, but less than twice tolerance, the team leader must note this on the Alert Response Form and suspend the marker, who is identified as lingering doubt the team leader must email the Alert Response Form to the PE and the Monitoring and Support Team to consider the evidence and reach a decision if the examiner should continue marking, the team leader informs the examiner and removes the suspension OCR 2017 5
If the script check shows that the marker is aberrant; the Monitoring and Support Team confirms to the PE/TL that the examiner should be stopped the scripts marked so far will be put back into the pool for re-marking. Please be clear in your recommendation to the Monitoring and Support team and make the reasons for your decision explicit. NOTE: if you have concerns about an individual seeding script you can also look at live marking and if there is an issue with the use of annotation then the marker may be informed of this and allowed to continue. If this inspection reveals a wider problem then the above procedure should be followed or you can move to your next batch check. OCR 2017 6
Reviewing live marking on RM Assessor To review a subordinate's marking: 1. On your Team Overview page (see Viewing your team), select a marker in the list and then click View, or double-click a marker. 2. In the left-hand menu, select Marking and then select Closed responses. 3. From the worklist, open the response you wish to review. 4. If you are satisfied with the quality of the marking, close the response and select another response to review. 5. If you are not satisfied with the quality of the marking, choose one of these options: send a message (see Messaging within RM Assessor) or call the marker to explain that there are marking discrepancies, and also explain what you need them to do. If the status of the response is Pending the marker will be able to make corrections and re-submit the response. suspend the marker (see Suspending a marker). withdraw the marker altogether from this QIG (see Withdrawing a marker). 6. Select another response to review, and continue until you have finished reviewing. OCR 2017 7