The laborious task of marking examination papers has been made much easier and fairer with the introduction of a new on-screen marking process in Hong Kong. The Hong Kong Examinations and Assessment Authority (HKEAA) developed the system that will progressively be used for all public exams by 2012.
The most important examinations, for which the new system will be used, are the Hong Kong Certificate of Education Examination, taken by students after five years of secondary education, and the Hong Kong Advanced Level Examination after seven years. Both will be replaced by the Hong Kong Diploma of Secondary Education Examination in 2012.
For this project, the HKEAA has won an MIS Asia IT Excellence Award in the category of best change management (government). The manual marking of students’ scripts requires the services of thousands of teachers and other professionals. They collect the examination scripts from the HKEAA, and return the marked scripts to the HKEAA for checking and data capture before the final results are processed and released.
Human error
As with any large-scale manual process, human error is impossible to eliminate completely. Even if a small proportion of scripts were to go missing, it would create serious problems. Markers occasionally break their agreement and mark scripts in public places, or allow unauthorised persons to mark papers.
It is easy for errors to arise in totalling the marks for the parts of each question, and to prevent this, university graduates are hired to check the arithmetic. It is necessary to check the quality of marking, but the data arising from this operation is available only at the end of the marking period, so that the time for remedial action is rather limited.
Another problem is the time taken to mark the scripts and process the results. The schools want to hold the examinations as late as possible, to give students the maximum time for preparation, but the nine tertiary institutions want the results out as early as possible, to facilitate the selection of student entrants.
In late 2005, the HKEAA started trials of on-screen marking technology to improve the efficiency of marking, increase the security of the process, and enhance the quality and reliability of marking.
HKEAA commissioned Automated Systems HK (ASL), after a rigorous tendering process, as one of the key contractors for the on-screen marking system, responsible for the development, systems integration and implementation of both hardware and software. The core engine for on-screen marking was developed by ASL’s partner in China. In fact, China is the most advanced country for adoption of such a marking process, already using it in about 23 provinces.
“The on-screen marking engine was sourced from China,” says Tsang Kwong-nap, HKEAA’s IT general manager, “but substantial further development and customisation was required to provide the required performance for Hong Kong’s examinations.”
During this time, the HKEAA was also modernising its IT infrastructure and systems by phases. Part of the purpose was to provide the basis for the greater use of IT systems, including on-screen marking.
The application
The on-screen marking application runs on an “intraWeb”—HKEAA’s private IP network that is physically secure from hacking or other Internet security risks. At HKEAA’s Lai King data centre, nearly 50 Sun Microsystems servers running Solaris provide the marking system as a Web service, and also control the database of students’ scripts. A second data centre replicates the data for resiliency. The application is written in JavaScript and accessed through the Internet Explorer browser.
All students’ scripts are scanned and a barcode is attached to the original document for easy identification. The scanned scripts are stored in a database that enables individual questions to be retrieved separately, since they are often marked on-screen by different specialised markers. More than 4,000 teachers and professionals who serve as markers now carry out this task at one of three assessment centres, at Lai King, Wanchai and Tsuen Wan, which have a total of 1,000 seats. There are about 40 subjects studied at each level and each subject may have special marking requirements.
Obviously, the marking of a maths paper is different from language papers that have open-ended answers. An online booking system enables markers to reserve preferred timeslots for marking at the assessment centres, with readily available on-site support.
Testing for markers
Provision of facilities at the marking centres includes areas where markers could take a break, watch TV or eat snacks during intervals in the marking process. This is one of the essential measures to support good quality marking.
Before markers can do on-screen marking, they must pass a qualifying test by marking scripts with pre-assigned scores, which are checked. If the markers pass, they can proceed to mark live scripts at one of the marking centres. Actual use of the on-screen marking system has been made very simple, to avoid errors and reduce stress on the markers. The markers use ordinary PCs, with large screens that allow display of the marking guidelines alongside the students’ scripts. Allocating marks, or adding ticks or crosses, is carried out by mouse clicks. Totalling of marks is done by the computer. The marking screens display Chinese or English, according to the subject.
“If markers are too lenient or too harsh, this is detected by automatic statistical analysis of the marking, and in the case of double marking, by comparing the marks awarded by the two markers,” says Tsang. “The chief examiner on duty at the centre can assist and advise the markers concerned.” The HKEAA has also organised different training for markers, including specific instructions for individual examinations.
In February 2006, teachers and university students participated in trials of the on-screen marking system. Based on the results of the pilot, the HKEAA further refined the policies and procedures.
Last year, the HKEAA implemented the on-screen marking of papers for the Chinese and English language subjects. That year, about 560,000 scripts of the two languages were marked on-screen, amounting to 30 per cent of all the 1,880,000 digitally scanned scripts for the two public examinations. The target is to achieve full implementation for the introduction of the new Hong Kong Diploma of Secondary Education, to be launched in four years.
Security risks
The new system eliminates many security risks. Through script digitisation, the possibility of missing scripts has been eliminated, together with the problem of marking in inappropriate places. Mental calculation errors have been replaced by computerised calculation.
Apart from on-screen marking, the HKEAA plans to use CCTV cameras and access control at both the examination centres and the secure script stores. Other ideas being considered include radio frequency identification tags to monitor the movement of question papers. “The new technology addresses a range of problems, including the security of scripts, and supervision of marking quality by the provision of management information during the process,” says Tsang.
Management changes
The HKEAA has set up an IT committee (ITC) to advise on and monitor IT matters, which will include the proposed measures. The ITC is responsible for scrutinising and monitoring the planning and delivery of the proposed measures. The HKEAA’s IT division has been re-structured to segregate development and operational functions.
A quality assurance unit has been set up, reporting to the secretary-general of the HKEAA, to bring together various quality assurance and audit and risk management functions.
The HKEAA has made renewed efforts on staff development, and strictly enforced the procedures for incident management, including authorisation, verification documentation and reporting. The authority has also cultivated a more collaborative environment with closer communications at both the management and operation levels, so as to efficiently deliver the proposed measures.
The time involved in the marking process has been reduced by one to two weeks. “Both the Education Bureau and the University Grants Committee-funded institutions are keen to see the processing time shortened to allow more time for teaching and learning and also for university admission,” says Tsang.
The response of markers towards on-screen marking is generally positive. “Although some markers miss the more flexible and familiar manual marking process, they recognise the many advantages of on-screen marking” says Tsang. “Once markers become familiar with the features and operations of on-screen marking, they prefer it to manual marking.”
Better quality
Real-time monitoring of markers performance is carried out and remedial action can be taken immediately in cases where markers are found to be too harsh or lenient, or inconsistent. On-screen marking also provides detailed information on the marking of each question for feedback to schools and teachers.
The quality of marking has been improved and human error minimised. The cost of the marking operation has also been reduced. In addition to being cost-effective, online marking has earned the HKEAA several IT awards.
“We are contributing to the competitiveness of the Hong Kong education system by reporting reliable, fair and accurate results for the students,” says Tsang.
