I was just looking at simple math problem with a group of people and we came up with four different ways to solve this multiple choice item. This is not the actual item, but it was ver similar to this:
Jeremy goes to the store to buy a jar of peanut butter and a jar of jelly. If he starts with $15.85, the peanut butter costs $2.95 and the jell costs $3.65, how much money does he have when he leaves the store with his purchases?
This really was a very simple item. Sure, the items (and the starting amount) were expressed in both dollars and cents. But they were all multiple of five.
Pretty simple, right? But when we each went though the item ourselves, we naturally and authentically did it in four different ways.
One of us was lazy, and just rounded all the numbers and did the math in their head. They then added the two purchases (~$7) and subtracted that from the original ~$16, leaving around ~$9. Only one answer option was around $9, so they picked that.
Another one of use did the straightforward two step math problems on a piece of paper. $15.85 - $2.95 = $12.90, and then $12.90 - $3.65 = $9.25.
A third person tried to do the two step math problem, but in their head. They doubted their mental math skills, so they pulled out a piece of paper to double check — and it was good that they did. They ended up with $9.25.
The last person stuck with addition. They added the two items to get $6.60, and added that to each answer option until they found an answer option that yielded a total of $15.85.
Four very capable and very education professionals addressed a single simple math problem, but had four different strategies. And that was for a VERY simple problem.
Imagine how many different cognitive paths more complex problems might prompt. Imagine how many different ways including more context for a problem could lead test takers to different obstacles, distractions and even mistakes. To help with that, we have put together a set of a few dozen personas to try to think about as hypothetical test takers who might approach an item differently than you.