The biggest mistake that people make when thinking about item alignment is focusing on the charged task, rather than thinking about the clarity of the observable evidence that an item generates or about the cognitive path that test takers might follow.
True item alignment is about quality of the evidence that the item generates. Does the item present strong evidence that the successful tester does indeed have proficiency with the Targeted Cognition (i.e., the KSAs revealed by a close reading of the standard)? Does the item present strong evidence that the unsuccessful test that lacks proficiency with the Targeted Cognition?
Mistaken thinking about item alignment tends to reduces those questions mere to questions of relevance – rather than focusing on the clarity of the evidence. That is, some look at what test takers are asked to do or think about (i.e., the task), and consider whether it is relevant to the standard, or how relevant. From this perspective, an item that suggest tasks which depend on cognition that is closer to what is described in the standard and/or that are more dependent on that cognition are deemed to be more strongly aligned. However, items that merely make use of cognition that is merely related the what is described in the standard cannot provide clear affirmative and/or negative evidence of test takers’ proficiency with the Targeted Cognition. Simply put, it is not enough.
This mistaken approach often accepts items that test takers get wrong for reasons other than the Targeted Cognition as aligned. This mistaken approach often accepts items to which that test takers can respond correctly without the Targeted Cognition as aligned. It allows for the mere possibility that the test taker used the Targeted Cognition or misapplied the Targeted Cognition to count as alignment, in spite of the ambiguity of the such evidence.
(With multiple choice items, quite often, this is exacerbated by focusing only on whether the Key is accurate, whether the distractions are incorrect and/or – at best – whether they capture the kinds mistakes that test takers might make. However, these mistakes do not have to be mistakes in or misunderstandings of the cognition described in standard, itself. Thus, these items frequently max out at level 3 (i.e., Task Alignment).)
The RTD Alignment Scale addresses this mistake, head on.