ExamEval Logo
  1. ExamEval
  2. Item Writing Flaws
  3. All of The Above

Why "All of the Above" Should Never be Used in Multiple-Choice Questions

Sean P. Kane, PharmD, BCPS
By Sean P. Kane, PharmD, BCPS
Published June 16, 2025

Three Core Flaws with "All of the Above"

There are three reasons why "all of the above" as an answer choice should never be used in multiple-choice questions:

  1. Research consistently shows that when this option is used, it tends to be the correct answer.
  2. Once a test-taker identifies two correct answer options, "all of the above" will be selected without needing to evaluate the remaining answer choices.
  3. If a test-taker confidently identifies an incorrect answer, "all of the above" can also be eliminated.

These issues introduce biases that compromise the validity and reliability of an assessment. Test-wise students are more likely to answer an "all of the above" question correctly even if they do not have a strong understanding of the concept being tested.

Consider the following example:

Medicinal Chemistry of Tricyclic Antidepressants
Which of the following statements about the medicinal chemistry of tricyclic antidepressants (TCAs) is correct?

A. TCAs typically contain a three-ring central structure
B. Most TCAs act primarily by inhibiting the reuptake of norepinephrine and serotonin
C. The pKa of most tricyclic antidepressants is between 9 and 10, causing the molecule to primarily exist in its ionized form at physiological pH
D. Desipramine is metabolized mainly by CYP2D6 to 2-hydroxydesipramine
E. All of the above ✓
Answer choices A and B are both true statements and are relatively easy facts regarding TCAs. Answer choices C and D are much more difficult concepts that fewer students will have mastered. If a test-wise student recognizes that options A and B are correct, the student will select "E. All of the above" without needing to evaluate or demonstrate knowledge of choices C and D.
Critical Issue

Using "all of the above" reduces the validity of an assessment by rewarding partial knowledge. Students can identify the correct answer without evaluating all options, which means the question fails to accurately measure their full understanding of the topic.

Exam Performance with All of the Above

In a study comparing the same concept between an "all of the above" answer choice and a "select all that apply" format (without partial credit), students answered "all of the above" correctly about 72% of the time, compared to correctly answering "select all that apply" about 15-20% (Harasym 1998). The reliability, validity, and discriminatory power of "all of the above" were also significantly worse than a "select all that apply" format.

Replacing "All of the Above"

The simplest method of addressing this item-writing flaw is to replace an "all of the above" answer choice with a "select all that apply" (SATA) exam question format. Although SATA is less preferred over a "single best answer" question structure, SATA is commonly implemented on board examinations in the health sciences.

In a 2024 publication by Dell and colleagues, replacing "all of the above" questions with a "select all that apply" format on a pharmacy school exam resulted in an increase in question difficulty (92% vs. 62%, p=0.007) but an improvement in the discrimination index (0.10 vs. 0.21, p=0.03). Consistent with the NAPLEX (pharmacy board exam), no partial credit was given when scoring these questions.

The following example utilizes this approach to revise a flawed question with a "select all that apply" format:

Flawed Question
A patient presents with symptoms of a common cold, including a runny nose, sore throat, and a worsening, productive cough. Which of the following over-the-counter medications could provide symptomatic relief?

A. Cough suppressant (e.g., dextromethorphan)
B. Analgesic (e.g., acetaminophen)
C. Expectorant (e.g., guaifenesin)
D. All of the above ✓
Test-wise students are more likely to guess the correct answer because "all of the above" is commonly a correct answer choice. Additionally, if a student identifies at least two options that are correct (e.g. A and B), the student will select "all of the above" even if they are not sure whether choice C is correct or not.
Corrected Question
A patient presents with symptoms of a common cold, including a runny nose, sore throat, and a worsening, productive cough. Which of the following over-the-counter medications could provide symptomatic relief? (Select all that apply)

A. Analgesic (e.g., acetaminophen) ✓
B. Antibiotic (e.g., amoxicillin)
C. Cough suppressant (e.g., dextromethorphan) ✓
D. Decongestant (e.g., pseudoephedrine)
E. Expectorant (e.g., guaifenesin) ✓
By revising this question to a "select all that apply" format, the student must demonstrate knowledge of each answer choice as a true/false statement. This approach increases the difficulty of the question but improves the discrimination index.

Other Corrections of This Item-Writing Flaw

In addition to converting a question to a "select all that apply" format, the following methods can be used to modify a question with an "all of the above" choice to more accurately assess student knowledge:

  • Single Best Answer Format: Revise the answer choices so that only one option is clearly correct. This is the preferred question format in most item writing guidelines.
  • Specific Integration: If multiple concepts need to be tested together, write a specific answer choice that integrates these concepts rather than relying on "All of the above."
  • Separate Questions: Break complex topics into multiple questions, each focusing on a specific aspect of the content.

This flaw can be automatically detected and corrected using ExamEval, an AI-powered exam analysis platform for health professions educators. ExamEval eliminates item-writing flaws and improves assessment reliability by ensuring exams are free from issues like "all of the above" answer choices.

References

  1. Harasym PH, Leong EJ, Violato C, Brant R, Lorscheider FL. Cuing effect of "all of the above" on the reliability and validity of multiple-choice test items. Eval Health Prof. 1998;21(1):120-133. doi:10.1177/016327879802100106
  2. Dell KA, DeVries DM. Effect of changing multiple choice questions from "all of the above" to "select all that apply". Curr Pharm Teach Learn. 2024;16(3):174-177. doi:10.1016/j.cptl.2023.12.034
  3. National Board of Medical Examiners (NBME). Item-Writing Guide. Philadelphia, PA: National Board of Medical Examiners; February 2021.
  4. Haladyna TM, Downing SM, Rodriguez MC. A review of multiple-choice item-writing guidelines for classroom assessment. Appl Meas Educ. 2002;15(3):309-334. doi:10.1207/S15324818AME1503_5
  5. Balaha MH. Simplified guidelines for multiple-choice question writing to increase faculty compliance and ensure valid student results. Educ Med J. 2019;11(4):1-17. doi:10.21315/eimj2019.11.4.1

Take the First Step Toward More Reliable Exams

✓ No credit card required
1,000 free credits each month
1,000 bonus credits for joining
✓ Cancel anytime