Typically, written examinations consists of combinations of multiple choice (MCQ) or extended matching responses (EMQ), and short answer response examinations (known as key features problems). Written responses, especially short answer papers, often have to be decollated, scanned, distributed to a marker, marked, and then re-collated before an overall mark is determined for standard setting and quality assurance. In the case of MCQ and EMQ papers, electronic scanning needs to be manually performed either at the examination centre or on return of the completed scripts to a central location.
To date many associations, Universities and Medical Colleges have been conducting the examination process using a paper-based system. This has involved a significant logistical effort and cost in forming, distributing, collating and marking the examination papers, and is prone to significant levels of error necessitating careful oversight and revision to ensure the validity of the Candidates’ examination results. The risk management of these high stakes examinations is vital.
Transform Strategists and Genix Ventures have designed and built an on-line examination system for associations and healthcare organisations, through the development of a software platform to deliver, conduct and assess the examinations on-line over the internet using industry standard PCs and web browsers.
Based on a systems approach, the build included management of the examination platform to enable delivery of online examinations to multiple venues in Australia to accommodate many 100s of candidates per sitting.
Specifically, the objective the Online Examination system covered the following high level areas:
- Online exam delivery to candidates;
- Computer generated automatic marking of questions, and product of extract files (candidates answers and score) for revalidation by the College;
- Facilities to allow online marking of text based questions/answers;
- Facilities to allow Quality Assurance marking, and remarking;
- Investigation with the client into appropriate examination centres;
- Generation of appropriate handouts and control information for both candidates and markers;
- Contingency options in case of site failures;
- Appropriate security for a high stake, high risk examination;
- Intuitive user interface for the candidate, administrators and Examination Markers;
- Compatibility with current industry standards;
- Suitable for thin client and/or browser based delivery operations;
- Appropriate performance and concurrency assuming a minimum of 1000 concurrent users; and
- Suitable to meet the client’s needs for the next 5 years in the event of new examination modalities.
In addition the online examination system, Transform Strategists have worked with Genix Ventures to develop an innovative, effect and secure examination database that complements the online examination modality. Through the use of this approach, significant risk management mitigation, administrative efficiencies and examination timeliness have been implemented.
The design included a data driven approach instead of hard-coding to maximise flexibility, and enabled an SQL Server preference of the Database Management System.
The risk management advantages of moving to an electronic written examination are significant:
- Examination security;
- Question bank integrity and review;
- Rapid and secure electronic script marking and analysis without the introduction of potential for human error; and,
- The ability to conduct a secure examination across multiple time zones and locations using a secure hyper-text transfer protocol.