One might think that the main determinant of a multiple choice item’s difficulty is the set of KSAs that a an item is targeting. One might think that item difficulty can be spotted through an examination of the stem (i.e., the item’s question or prompt). But one would be wrong.
The most important determinant of item difficulty is the distractors (i.e, the incorrect answer options).
An item without plausible distractors is going to be an easy item. That is, an item whose distractors can all be quickly and easily dismissed—even by those without the command of the targeted cognition—will be easy. We call this low bar for plausibility shallow plausibility or surface plausibility. Distractors must at least be shallowly plausible, and yet they often are not.
An item whose distractors are all shallowly plausible and deeply plausible will be a more difficult item. Deeply plausible distractors are those that require working through the item to dismiss, because they follow from mistakes in applying that targeted cognition.
The most difficult items often have distractors that are quite similar to the key (i.e., the correct answer option). They might differ in some subtle way from their corresponding key. They might rely on a minor point in a text to differentiate. They might be a good answer option, just not the best answer option. For example, they might not be false, and yet they might not contain as much truth as the key. Therefore, they might look like a good answer to a test taker who does not check all the answer options.
None of these possibilities involve changing the targeted cognition, the stem or the key. And yet, these different sorts of distractors or distractor sets can radically alter the empirical difficulty of an item.
Heck, distractors are so powerful that they shift the meaning of the evidence that an item collects from the targeted cognition to some other KSA(s).