ExamEval
Item Writing Flaws
Heterogeneous Answer Choices

Heterogeneous Answer Choices and Non-Parallel Question Construction

By Sean P. Kane, PharmD, BCPS

Published June 20, 2025

Understanding Heterogeneity

Answer choices in multiple-choice assessments should be homogeneous and use parallel construction, meaning the format, length, style, category, and verb tense of the answer choices should be similar. Heterogeneous answer choices lack this parallel construction. Options that are unique or stand out are more likely to be identified as the correct answer by test-wise students, even if they lack actual knowledge of the content.

Common forms of heterogeneity include variations in length (one option significantly longer or shorter), different levels of specificity (some options general, others highly detailed), mixed formats (some numerical, others descriptive), or inconsistent grammatical structures. As Haladyna, Downing, and Rodriguez (2002) emphasize, maintaining parallelism in answer choices is a fundamental principle of effective item writing. When this principle is violated, the question is more likely to be flawed.

Critical Issue

Heterogeneous answer choices introduce construct-irrelevant variance by providing clues that are unrelated to the content being assessed. This flaw allows test-wise students to identify the correct answer based on structural anomalies rather than genuine understanding, which undermines the validity of the assessment.

Consider the following example. Even for a student with no knowledge of computer science, a test-wise student can make an educated guess at the correct answer:

Operating System Concepts

Which of the following best describes how a modern operating system manages virtual memory during process execution?

A. It swaps processes.

B. It allocates RAM.

C. It uses paging and segmentation to isolate process memory, manages page tables for address translation, and handles page faults by loading pages from storage. ✓

D. It schedules tasks.

Option C is much longer and more detailed than the other choices, making it stand out as the likely correct answer based on length and specificity alone. The other options are brief and vague, creating a clear heterogeneity that provides test-wise cues.

Why Longer, More Detailed Answer Choices Are Usually the Correct Answer

Longer, more detailed answer choices are often correct in multiple-choice assessments because item writers, consciously or not, tend to include more qualifiers, explanations, or specific details in the correct answer to ensure its accuracy and defensibility. This results in the correct option standing out due to its length or complexity, while distractors are typically shorter and more general.

Exam writers often write the correct answer choice first, naturally including more details or qualifiers to ensure accuracy and defensibility. When drafting plausible but incorrect distractors, it is much harder to add specific or detailed qualifiers that remain believable. This challenge frequently results in distractors that are shorter, more generic, or less specific, further contributing to heterogeneous answer choices that can cue test-wise students to the correct response.

Example of Heterogeneous Answer Choices

Flawed Question

What is the mechanism of action of atorvastatin?

A. Inhibits cholesterol synthesis by blocking HMG-CoA reductase enzyme in the liver, thereby reducing hepatic cholesterol production and upregulating LDL receptor expression

B. Blocks calcium channels

C. ACE inhibition

D. Beta-blockade

Option A is significantly longer and more detailed than the others, making it stand out as likely correct based on length alone. Options B, C, and D vary in format (complete phrase vs. simple terms), creating additional heterogeneity that provides test-wise cues.

Corrected Question

What is the mechanism of action of atorvastatin?

A. Inhibits HMG-CoA reductase enzyme

B. Blocks calcium channels in cardiac muscle

C. Inhibits angiotensin-converting enzyme

D. Blocks beta-adrenergic receptors

The corrected version creates parallel answer choices that are similar in length, specificity, and format. All options describe enzyme or receptor mechanisms using consistent grammatical structure, requiring content knowledge rather than pattern recognition to identify the correct answer.

Homogeneous Options Improve One-Best-Answer Question Formats

Assessment experts recommend a one-best-answer (or single best answer) format in which the answer choices can be ranked or ordered from least to most true along a single dimension. If answer choices are heterogeneous across multiple dimensions, this ranking or ordering process is not possible.

Consider this example in which the flawed question has heterogeneous answer choices across multiple dimensions:

Flawed Question

Which of the following is true regarding semaglutide?

A. It is a GLP-1 receptor agonist. ✓

B. It is FDA approved for non-valvular atrial fibrillation.

C. It is an injectable medication for use in the hospital setting.

D. It is contraindicated in patients with a history of pancreatic cancer.

Although the answer choice text is homogeneous in length and format, all four answer choices are heterogeneous with respect to the dimension being tested (e.g., answer A is the mechanism of action, B is FDA approved indications, C is the administration and setting, D is contraindications). The heterogeneous concepts prevent ordering or ranking of answer choices from least to most correct.

Corrected Question

Which of the following is the most accurate description of the mechanism of action of semaglutide?

A. GLP-1 receptor agonist ✓

B. Dihydropyridine calcium channel blocker

C. Dipeptyl peptidase 4 (DPP-4) inhibitor

D. Sodium-glucose cotransporter 2 (SGLT2) inhibitor

This revised question stem focuses specifically on the mechanism of action, allowing the answer choices to follow a common conceptual thread and be homogeneous across a single dimension.

Creating Homogeneous, Parallel Answer Choices

Effective answer choices should be homogeneous across multiple dimensions:

Similar Length: Options should be approximately the same number of words or characters to avoid length bias.
Consistent Specificity: All options should operate at the same level of detail—either all general or all specific.
Uniform Format: Use consistent formatting, capitalization, and punctuation across all options.
Parallel Grammar: Ensure all options follow the same grammatical pattern relative to the question stem.
Equivalent Complexity: Options should require similar levels of sophistication to understand and evaluate.
Same Domain or Concept: Options should all share a concept or domain (e.g., all are medications, all are risk factors for a disease, all relate to pathophysiological processes, etc.)

Assessments with heterogeneous answer choices can compromise exam validity by enabling test-wise students to guess correctly without demonstrating true content mastery, eroding confidence in exam results. ExamEval's AI-powered exam analysis platform automatically identifies and corrects this type of item-writing flaw to boost assessment reliability and improve student learning outcomes. Discover how ExamEval can support health professions educators at ExamEval.

References

National Board of Medical Examiners (NBME). Item-Writing Guide. Philadelphia, PA: National Board of Medical Examiners; February 2021.
Haladyna TM, Downing SM, Rodriguez MC. A review of multiple-choice item-writing guidelines for classroom assessment. Appl Meas Educ. 2002;15(3):309-334. doi:10.1207/S15324818AME1503_5