- ExamEval
- Item Writing Flaws
- All of The Above
Why "All of the Above" Should Never be Used in Multiple-Choice Questions

Three Core Flaws with "All of the Above"
There are three reasons why "all of the above" as an answer choice should never be used in multiple-choice questions:
- Research consistently shows that when this option is used, it tends to be the correct answer.
- Once a test-taker identifies two correct answer options, "all of the above" will be selected without needing to evaluate the remaining answer choices.
- If a test-taker confidently identifies an incorrect answer, "all of the above" can also be eliminated.
These issues introduce biases that compromise the validity and reliability of an assessment. Test-wise students are more likely to answer an "all of the above" question correctly even if they do not have a strong understanding of the concept being tested.
Consider the following example:
Using "all of the above" reduces the validity of an assessment by rewarding partial knowledge. Students can identify the correct answer without evaluating all options, which means the question fails to accurately measure their full understanding of the topic.
Exam Performance with All of the Above
In a study comparing the same concept between an "all of the above" answer choice and a "select all that apply" format (without partial credit), students answered "all of the above" correctly about 72% of the time, compared to correctly answering "select all that apply" about 15-20% (Harasym 1998). The reliability, validity, and discriminatory power of "all of the above" were also significantly worse than a "select all that apply" format.
Replacing "All of the Above"
The simplest method of addressing this item-writing flaw is to replace an "all of the above" answer choice with a "select all that apply" (SATA) exam question format. Although SATA is less preferred over a "single best answer" question structure, SATA is commonly implemented on board examinations in the health sciences.
In a 2024 publication by Dell and colleagues, replacing "all of the above" questions with a "select all that apply" format on a pharmacy school exam resulted in an increase in question difficulty (92% vs. 62%, p=0.007) but an improvement in the discrimination index (0.10 vs. 0.21, p=0.03). Consistent with the NAPLEX (pharmacy board exam), no partial credit was given when scoring these questions.
The following example utilizes this approach to revise a flawed question with a "select all that apply" format:
Other Corrections of This Item-Writing Flaw
In addition to converting a question to a "select all that apply" format, the following methods can be used to modify a question with an "all of the above" choice to more accurately assess student knowledge:
- Single Best Answer Format: Revise the answer choices so that only one option is clearly correct. This is the preferred question format in most item writing guidelines.
- Specific Integration: If multiple concepts need to be tested together, write a specific answer choice that integrates these concepts rather than relying on "All of the above."
- Separate Questions: Break complex topics into multiple questions, each focusing on a specific aspect of the content.
This flaw can be automatically detected and corrected using ExamEval, an AI-powered exam analysis platform for health professions educators. ExamEval eliminates item-writing flaws and improves assessment reliability by ensuring exams are free from issues like "all of the above" answer choices.
References
- Harasym PH, Leong EJ, Violato C, Brant R, Lorscheider FL. Cuing effect of "all of the above" on the reliability and validity of multiple-choice test items. Eval Health Prof. 1998;21(1):120-133. doi:10.1177/016327879802100106
- Dell KA, DeVries DM. Effect of changing multiple choice questions from "all of the above" to "select all that apply". Curr Pharm Teach Learn. 2024;16(3):174-177. doi:10.1016/j.cptl.2023.12.034
- National Board of Medical Examiners (NBME). Item-Writing Guide. Philadelphia, PA: National Board of Medical Examiners; February 2021.
- Haladyna TM, Downing SM, Rodriguez MC. A review of multiple-choice item-writing guidelines for classroom assessment. Appl Meas Educ. 2002;15(3):309-334. doi:10.1207/S15324818AME1503_5
- Balaha MH. Simplified guidelines for multiple-choice question writing to increase faculty compliance and ensure valid student results. Educ Med J. 2019;11(4):1-17. doi:10.21315/eimj2019.11.4.1