Peer Review of Exam Questions: Better Quality with Less Faculty Workload

By Sean P. Kane, PharmD, BCPS

Published September 14, 2025

The High Stakes of Exam Quality

In health sciences education, summative assessments using multiple choice questions are widely used to demonstrate student achievement and enable progression in professional programs. The quality of these exam questions is fundamental to fair and effective assessment.

Unfortunately, the quality of these exam questions in higher education is often poor. Multiple studies show that 50% or more of multiple choice questions in healthcare curricula (including nursing, pharmacy, and medical education) contain item-writing flaws that undermine reliability, validity, and fairness. (Downing 2005, Tarrant 2006, Caldwell 2013, Dell 2017)

Poorly written exams can cause capable students to fail and unprepared students to pass by guessing or exploiting test-wise cues. This results in an assessment that fails at its primary job: to accurately measure knowledge. In health sciences education, unreliable assessments lead to progression issues, lower board exam pass rates, and concerns about the quality of patient care provided by graduates.

Why Are Low-Quality Exam Questions So Common?

Despite abundant resources outlining best practices, faculty frequently write flawed questions due to:

1. Inadequate training – Most faculty have not received formal training in writing high-quality questions.
2. Limited time – Faculty juggle multiple responsibilities and have little time for item writing.
3. Low emphasis – Faculty often spend much more time on lecture slides than on crafting exam questions.

Peer Review Improves Exam Question Quality

Research shows that a peer review process is one of the most effective ways to improve exam question quality. Colleagues bring fresh perspectives, catch ambiguous wording, factual errors, and item-writing flaws.

Studies across health professions education programs consistently show that peer review: (Malau-Aduli 2011, Abozaid 2017, Roni 2022)

• Improves item discrimination (discrimination index and point biserial values)
• Increases exam reliability (KR-20 or Cronbach's alpha)
• Raises the number of functioning distractor answers

External peer review by clinicians or content experts outside the faculty has also proven effective. In one study, external reviewers flagged 21% of multiple choice questions for revision or replacement even after an internal peer review process. (Smeby 2019)

Significant Barriers to a Peer Review Process

Despite its proven value, peer review of exam questions is relatively uncommon in health sciences programs. Major barriers to implementation exist that make implementation difficult.

There are significant time commitments and logistical hurdles associated with peer review. It takes time to understand the content being tested, review the exam questions, and deliver focused, constructive feedback. Faculty may feel that their limited time is better spent with other responsibilities. Logistically, this also means that lecturers need to prepare their exam questions well in advance to provide sufficient time for a peer review and for questions to be improved based on feedback. The logistics of having exam items prepared well in advance often derail the process.

Peer reviewers need to be trained to review exam questions. Because peer reviewers are the same faculty who commonly write flawed questions, these peer reviewers need additional faculty development regarding item writing flaws and assessment best practices. This can be time and resource-intensive, and without training, the peer review process is far less effective.

Lastly, there are complex human dynamics associated with peer review. Giving and receiving critical feedback can be uncomfortable. Egos get bruised, professional relationships can be strained, and the fear of offending a colleague (or being offended) can lead to superficial reviews that can make the peer review process less effective.

Even when peer review exists, it is often not systematic, scalable, or sustainable due to these barriers.

The Benefits of Peer Review Without the Burden or Barriers

ExamEval offers AI-powered exam analysis for health professions educators. This automated, on-demand expert peer review addresses the barriers of a tradition review process:

• Comprehensive feedback is delivered in minutes without the need for complex timelines or logistical hurdles
• The review process is based on well-established item-writing best practices without needing for faculty development sessions
• Objective, constructive feedback is provided directly to the faculty writer -- no ego bruising or personal relationship strains
• Feedback includes specific, actionable steps to improve exam questions

Faculty retain full control over the exam writing process. While ExamEval offers detailed suggestions, the final decision to accept, reject, or modify feedback remains with the item writer.

By automating the most challenging parts of peer review, ExamEval enables the creation of high-quality, fair, and effective exams for every assessment. It achieves the core goals of peer review while sidestepping barriers that have held educators back for years. The result: better exams, fewer flaws, and more reliable assessments.

References

1. Downing SM. The effects of violating standard item writing principles on tests and students: the consequences of using flawed test items on achievement examinations in medical education. Adv Health Sci Educ Theory Pract. 2005;10(2):133-143. doi:10.1007/s10459-004-4019-5
2. Tarrant M, Knierim A, Hayes SK, Ware J. The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments. Nurse Educ Today. 2006;26(8):662-671. doi:10.1016/j.nedt.2006.07.006
3. Caldwell DJ, Pate AN. Effects of question formats on student and item performance. Am J Pharm Educ. 2013;77(4):71. doi:10.5688/ajpe77471
4. Dell KA, Wantuch GA. How-to-guide for writing multiple choice questions for the pharmacy instructor. Curr Pharm Teach Learn. 2017;9(1):137-144. doi:10.1016/j.cptl.2016.08.036
5. Malau-Aduli BS, Zimitat C. Peer review improves the quality of MCQ examinations. Assess Eval High Educ. 2012;37(8):919-931. doi:10.1080/02602938.2011.586991
6. Abozaid H, Park YS, Tekian A. Peer review improves psychometric characteristics of multiple choice questions. Med Teach. 2017;39(sup1):S50-S54. doi:10.1080/0142159X.2016.1254743
7. Roni MA, Berrocal Y, Tapping R. Improving summative assessment through a resource-efficient faculty review process. Med Sci Educ. 2022;32(5):979-983. Published 2022 Sep 13. doi:10.1007/s40670-022-01631-9
8. Smeby SS, Lillebo B, Gynnild V, et al. Improving assessment quality in professional higher education: could external peer review of items be the answer? Cogent Med. 2019;6(1):1659746. doi:10.1080/2331205X.2019.1659746