Cognitive complexity can be a powerful lens through which to examine items. It can highlight gaps between the richness and depth of instruction, of standards, of classroom learning and what appears on assessment. Unfortunately, the idea of cognitive complexity is too often just waved at, perhaps resented, and only cursory effort made to consider it. More thoughtful use and consideration of cognitive complexity can contribute to the development of assessments that educators feel better reflect their efforts and the learning goals for students. In other words, tests that they feel have greater (facial) validity.
Perhaps the fundamental mistake in the assessment context in thinking about cognitive complexity – very much like thinking about other aspects of alignment – is assuming (or insisting) that items have some fixed or inherent complexity and should be evaluated as such. The RTD central tenet is that valid items elicit evidence of the targeted cognition for the range of typical test takers. RTD is deeply rooted in the idea that assessment is about cognition and that cognition varies across the range of typical test takers. RTD is clear that to understand cognitive complexity, one must start with the cognitive paths of a variety test takers.
So, why is the idea of some innate item complexity wrong? Well, it depends on what is meant by that claim.
If item complexity refers to the complexity of the result or product, there are many reasons.
The actual final work product on most items on most large scale standardized tests is merely the selection of one offered option among a small handful. A is not a complex product. C is not a complex product.
If one argues that it’s not the label on the answer option but rather than contents of the answer option, one has hardly made any progress. Those answers are almost always quite short and straightforward. They certainly lack the range of complexity and depth of answers that might be offered in a classroom – be it in writing or orally. This leaves very little range of complexity for standardized test items.
Furthermore, this approach would suggest that any math item whose answer is a number is not very cognitively complex and that math problems are generally less cognitively complex that even fairly simple ELA items. We do not believe any of that.
Some argue that the final product is evidence of the complexity of cognition of the test taker. Frankly, whether they realize it or not, this simply concedes that cognitive complexity is a trait of the cognitive path (see below) and capitulates on the claim that complexity is in the final product.
Clearly, this idea simply is not productive or useful in the context of large scale standardized assessment. This means that item complexity must somehow be about the process through which test takers arrive at their final answers. And, again, we ask whether this idea is compatible with the idea that items have some important fixed innate complexity.
Put another way, is there some singular decisive process or path through an item to a solution/response that should be focus of cognitive complexity classification decisions? Is that idea productive as a general approach?
One simply must acknowledge that all items can be responded to with multiple cognitive paths. All. Test takers can respond with nonsense and/or can just guess, rather than working through the problem. Classroom teachers know well that many students respond to stress by losing confidence that they can work through a problem and revert to guessing. One might posit that the singular decisive path is the one that yields the correct response, but guesses can be correct.
Clearly, there exist multiple potential cognitive paths, so the question is really Which cognitive path is the singular decision process or path through an item to a solution/response that should be focus of cognitive complexity classification decisions? If, of course, such a thing even exists.
It seems obvious to us that if you had to choose one cognitive path as the most important one, it would be the one that most test takers use – or at least that a plurality of test takers use. But that is an empirical question that can only be determined through massive amount of quite difficult data collection (i.e., ideally through the development of mind reading technology that be used at large scale to best assure that the sample whose minds are read are appropriately representative of the testing population). But even were that research possible, this would not simply be a feature of the item. Different testing populations may choose different cognitive paths. Moreover, if standards, curriculum or textbooks change in a state or district, students may be influenced to select different paths. There is no singular decisive path, here.
Perhaps the shortest and most direct path to the answer is the singular decisive path to examine for cognitive complexity. Well, one would clearly have to put aside the actually shortest paths. Guessing is renders everything low cognitive complexity. Having already seen the problem and simply remembering the answer is not guessing or cheating, and is also very short cognitive path. But that is clear not the singular decisive path. Very many math problems on standardized tests can be answered through backsolving, because of the modality of selected response items. Is that backsolving path the singular decisive path of these item for these purposes? If proponents of this approach would accept that and would call for serious efforts to find the shortest path to a correct response (i.e., still excluding guessing and already knowing the answer), we could almost respect that. The problem is that that still leaves questions of how much experience and prior knowledge to disregard. Some test takers might not have seen this exact question, but have seen the exact same type of question and therefore can more quickly cut to the answer. Which of these paths does one exclude from consideration and which are candidates for the singular decisive path? And once again, issues of variation across test takers and their classroom – and other – experiences must be considered. There is no decisive answer here.
Perhaps the singular decisive path is the one that is built around the KSAs of the desired aligned standard? That certainly would be convenient. Unfortunately, this idea quickly falls apart. In order for this approach to have any merit, one would have to assume that items are correctly aligned and that simply begs the question (new school) of what alignment means. It certainly forestalls the reality that items often have multiple paths to a response and begs the question (old school) of which path one should base such determinations upon. Otherwise, it is tantamount to saying, “Well, if they do the item the way we want them to – which might not be the easiest, most obvious or most appealing path for a test taker to taker – then this is how complex the item is.” In other words, cognitive complexity is a product of the path that content development professionals (CDPs) would like test takers to take. However, just as different test takers might see different paths as preferable, so might different CDPs. This simply becomes an arbitrary decision.
The previous option is just one way to get to a very common problematic view: the singular decisive path is the one that I would take, that I imagine that I would have taken and/or that I imagine that most (or typical) test takers would take. All of those are projections of the CDP’s own thinking, habits and/or preferences. Because different CDPs can come to different answers here, there certainly is no singular decisive path here to base determination of cognitive complexity upon.
Once you acknowledge that there are multiple cognitive paths through an item, there simply is no way to identify one of them as the singular and decisive path. If items are to have some fixed innate complexity that should be the starting point and focus of cognitive complexity recognition, it is not found in the cognitive paths of test takers
There is one more notable approach to identifying cognitive complexity – one that is actually quite commonly used. This approach says that skills and standards themselves can be classified by cognitive complexity. Unfortunately, this approach also collapses under the weight of reality and simple practical considerations.
The simplest application of this approach suggests that more advanced skills are of greater cognitive complexity. But this simply turns cognitive complexity into a recapitulation of grade level. That cannot be right, as it renders it useless for its duplicativeness.
A second application might consider the sophistication of the application of the skill or standard, but this generally becomes a standard-specific recapitulation of the grade level. Alternatively, it might be about the proficiency with which the has been (or must be) applied. But – again – proficiency is supposed to be a different construct. (In fact, IRT puts item difficulty on the same scale as test taker proficiency/latent ability. Cognitive complexity needs to be something else, to be useful.)
A third application of this approach might consider the difficulty of the skill or standard, but difficulty is a product of instruction, practice and preparation. Different teachers can emphasize different skills or standards and different curricula can set of better instructional paths towards some standards over others. Again, this is not simply a function of the Aristotelian ideal of the skill or standard. That is, difficulty is a population-specific result. Furthermore, we already collect empirical measures of item difficulty and this approach is largely duplicative of that – or at least is largely duplicative for well aligned items.
Where does this too long discussion leave us?
There is no innate item complexity to be found in the final product.
Test takers always have multiple paths to a response, and most every item has multiple paths to a correct response.
There is not particular singular decisive path that one can use determine item complexity
Classification of individual standards or skills by cognitive complexity either falls apart for redundancy with other measures or is population- (and their educational experience) dependent.
Every route out of this conundrum of how to determine a fixed or innate item complexity resorts to projection and preferences of the CDP (or other human judge of its complexity).
Which means that such a thing does not exist. Which means, cognitive complexity must be grounded in something else. RTD says that items often have a range of cognitive complexity because they prompt a range of typical test takers to take a range of cognitive paths to their responses.
So, what does RTD offer for cognitive complexity determination? Well, that’s rDOK.