[Each day in October, I analyze one of the 31 item writing rules from Haladyna, Downing and Rodriquez (2002), the super-dominant list of item authoring guidelines.]
Content: Avoid over specific and over general content when writing MC items.
Their fifth rule has two fundamental problems. First, it is supported by just 15% of their 2002 sources and not even that many of their 1989 sources. Furthermore, fully 25% of their 1989 sources argue against this rule. But that general lack of support is not the biggest problem.
The biggest problem is that this rule does not mean anything. Do not do something too much is all they are saying. Sure, some people say you can never be too rich or too thin, but other than that sort of formulation, saying too (i.e., as in too much) or over (i.e., as in too much) makes the rule a tautology. That is, it is a logical circle. Yes, it would be bad for the item to be too hard. Don’t do that. It would be bad for the item to be too specific. Don’t do that. It would be bad for the font to be too small, too big, too baroque. Again, that’s just what “too” or “over” mean!
So, the real question is what it means to be over specific or over general. They seem to be saying that such things exist, but provide no guidance whatsoever for what they mean. They are providing an objection, but no basis for when it applies—of even what it really means. Nothing. Just nothing.
I do not think that I hate any of Haladyna et al.’s rules more than this one. It epitomizes the problem with their whole approach. Perhaps there is value in their original articles as literature review. Perhaps. But they have brought these lists forwards into handbooks and their own books. Others cite them and quote them all the time. They called their list, “a complete and authoritative set of guidelines for writing multiple-choice items.” Is it really? Does this example from their “Guidelines/Rules/Suggestions/Advice” (as they called them in 1989) actually help anyone to write or evaluate items?
Obviously not. I do not think it could be any more obvious.
[Haladyna et al.’s exercise started with a pair of 1989 articles, and continued in a 2004 book and a 2013 book. But the 2002 list is the easiest and cheapest to read (see the linked article, which is freely downloadable) and it is the only version that includes a well formatted one-page version of the rules. Therefore, it is the central version that I am taking apart, rule by rule, pointing out how horrendously bad this list is and how little it helps actual item development. If we are going to have good standardized tests, the items need to be better, and this list’s place as the dominant item writing advice only makes that far less likely to happen.
Haladyna Lists and Explanations
Haladyna, T. M. (2004). Developing and validating multiple-choice test items. Routledge.
Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. Routledge.
Haladyna, T., Downing, S. and Rodriguez, M. (2002). A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment. Applied Measurement in Education. 15(3), 309-334
Haladyna, T.M. and Downing, S.M. (1989). Taxonomy of Multiple Choice Item-Writing Rules. Applied Measurement in Education, 2 (1), 37-50
Haladyna, T. M., & Downing, S. M. (1989). Validity of a taxonomy of multiple-choice item-writing rules. Applied measurement in education, 2(1), 51-78.
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied measurement in education, 15(3), 309-333.
]