- ExamEval
- Item Writing Flaws
- Non-Preferred Question Format
Non-Preferred Question Formats in Multiple Choice Assessments
Multiple-choice items can broadly be divided into two main families of questions:
- One best answer: The test-taker ranks and identifies a single, most accurate answer choice.
- True/false: The test-taker identifies whether each answer individually is correct or incorrect.
Within these families, there are a variety of different formats. Historically, these question formats were assigned arbitrary letters (e.g., "A-type questions") to distinguish each format. Assessment experts strongly recommend the use of the A-type question format whenever possible. As described below, A-type contains a single answer choice and requires the test-taker to rank or order choices from least to most appropriate or correct.
Multiple-choice questions that use other question formats are less preferred because they tend to increase test-taker confusion, are more prone to item-writing flaws, and are more difficult to write. For that reason, the National Board of Medical Examiners (NBME), the organization responsible for writing USMLE exams for physician licensure, only uses one-best-answer format items on exams.
Non-preferred question formats, such as true/false or multiple-response, can introduce construct-irrelevant variance by measuring skills other than the intended content knowledge. For example, true/false questions have a high probability of guessing, while multiple-response questions can be confusing and difficult to score accurately.
True/False Family Questions
In contrast to the "one best answer" format in which answer choices are ranked or ordered, true/false questions require the test-taker to decide the cutoff or extent to which a question must be "true" to be marked as "true." In many cases, particularly in health sciences education, answer choices are usually not completely true or completely false in all circumstances.
Questions within the true/false family are more likely to have subtle difficulties or ambiguities that lead to this format of question being rewritten or discarded by instructors compared to other multiple-choice formats. Unfortunately, when exam writers remove ambiguity from true/false questions, this tends to focus on recall of facts as opposed to application of knowledge (e.g., a lower Bloom's taxonomy target), which is typically not the desired learning level for health sciences education.
Examples of True/False Questions in Health Sciences Education
Best Practices for Multiple Choice Question Formats
- As the NBME's guide to item-writing emphasizes, the single-best-answer format is the most reliable and valid way to assess a student's knowledge.
- Use scenario-based or case-based multiple-choice questions that reflect realistic clinical scenarios.
- Avoid true/false and simple recall formats except for prerequisite knowledge checks.
- Align question formats with professional standards and real-world practice.
Non-preferred formats can undermine the validity of assessments, leading to unreliable results and poor measurement of student learning. ExamEval, an AI-powered exam analysis platform for health professions educators, automatically detects item-writing flaws, including the use of non-preferred question formats, and helps faculty draft more reliable assessment questions.
References
- National Board of Medical Examiners (NBME). Item-Writing Guide. Philadelphia, PA: National Board of Medical Examiners; February 2021.