Exam Design And Lifecycle
OpenEDG Python Institute certification exams are designed to measure proficiency in specific areas of knowledge, skills and abilities. Main goal: create fair, valid, reliable, credible and secure certification examinations.
The exam development process consists of eight inter-related stages, which all have an impact on one another. These are: Market Analysis, Syllabus Development, Exam Design, First Exam Review, Small Market Trial (SMT), Second Exam Review, Exam Publishing, and Exam Maintenance.
The foundation of each process is the market analysis stage and the analysis of certification demand and identification of target groups and stakeholders, analysis of sibling certification programs already in existence, as well as an assessment of the deployment risk/success rate. This stage also includes the preparation of a design breakdown, publishing, and maintenance costs.
This stage involves identifying and recruiting SMEs (subject matter experts) for the purposes of defining exam objectives, defining the target audience (job task analysis and evaluation of professional standards), preparing exam documentation, and carrying out the initial process of pre-constructing the exam structure and content. The SMEs cooperate with psychometricians who assist in the exam development and test validation processes during the Syllabus Development, Exam Design, Exam Review, and Exam Maintenance stages.
The psychometric solutions adopted in the exam development process are compliant to the largest extent possible with the recommendations set out by the Association of Test Publishers (Applied Testing Technology and Test Design Guidelines), the Standards for Educational and Psychological Testing (the American Educational Research Association), ICE 1100 : 2010(E) – the Standards for Assessment-Based Certificate Programs from the Institute for Credentialing Excellence (ICE), American National Standards Institute (ANSI), International Standards Organization (ISO), as well as the European Qualifications Framework and the IT Professionalism Europe skills frameworks and standards (the CEPIS e-Competence Benchmark).
This is the most important stage, in which the exam contents are created. It involves simultaneous cooperation of certification program managers, stakeholders, psychometricians, SMEs, test publishers, and developers who work together in order to create the framework for an exam, its structure, item format, delivery mode and format, scoring, as well as designing the actual exam items. The Exam Design stage consists of five independent phases:
- Structure development: defining test specifications (the blueprint) such as number of sections, number of items, section/item weight, exam length, type of questions, scoring model, passing score and scale, rounding mode, graphics or exhibits, presentation order, forms (if applicable), defining reporting groups, etc.
- Item Development: defining the stem (question), and the set of correct answers and distracters; defining requirements for exhibits and graphics; first internal technical proof;
- Exhibit Development: constructing media, simulations, and graphics that supplement the exam items; exhibit implementation;
- Proofreading: language editing, second technical proof and scanning for typos; copyediting;
- Item selection and implementation: selecting the items that will be included in the question pool or exam forms; creating source files and implementing them in the local Exam Testing System for internal review.
First Exam Review
The first internal review of the exam is carried out in the simulated testing environment; the exam is launched and tested by the SMEs; instant corrections and amendments are made where necessary. The main purpose of this review is to locate and eliminate errors before releasing the exam for pilot testing (SMT).
Small Market Trial (SMT/Beta Testing)
The Small Market Trial is the Pilot Testing stage when the exam is administered, if possible, among three groups of individuals:
- test candidates with no prior knowledge of the subject matter who have completed a training course aligned with the exam (focus group)
- test candidates with prior experience in the subject matter who have completed a training course aligned with the exam (control group # 1)
- test candidates with prior experience in the subject matter who have not completed a training course aligned with the exam (control group # 2).
Test candidates are also asked to mark exam items that may be misinterpreted, have poor wording, or have either no or more than the expected correct answers.
The small market trial is the final stage when data can be collected for evaluation in order to adjust the desired level of difficulty to the skills and knowledge manifested by the target audience expected to take the exam in the future.
Second Exam Review (data analysis and final formatting process)
The second review comes immediately after the pilot testing stage (SMT), when the data collected from the focus and control groups allow for final quantitative and qualitative analyses of the exam content and the implementation of final formatting.
An insight is made into the performance of individual exam items, and the exam in its entirety, which includes looking into such aspects as: diagnosing potential issues with item difficulty levels, diagnosing the correlation between exam items, identifying possible scoring and answer key issues, determining the assumed estimates of pass rates and time allocated for the exam.
The final (amended/updated) exam version (the assembly of operational test forms or pools of questions) is the resulting product of this stage ready for publishing.
Exam Publishing is the stage when the exam has been designed and tested, and is now ready for release. The exam is labelled with a series code, and becomes available worldwide through the network of reputable testing centers. This stage requires close cooperation between the product manager and the exam publisher in order for the process to be effective, timely, and successful.
Exam design and development is a continuous process. The Exam Maintenance stage takes place once the exam is live and test candidates take it. At this stage, the exam’s performance is monitored and analyzed to ensure that it meets the goals and assumptions set forth in the Market Analysis and Syllabus Development Stage.
This stage involves continuous statistical evaluations of response data, monitoring of candidate comments, and the carrying out of marketing research.
The data collected allows for the carrying out of post-publishing analyses and the updating of exams, which means the exam contents are revisited, and the whole exam design process starts over. During this process, the candidate experience and exam reputation are analyzed in order to make necessary changes to improve the testing experience and strengthen credibility and recognition of the certification program.
Exam update frequency: once every two years after first publishing or after a specified number of test candidates has taken the exam; whichever happens first.