Should we be paying this much attention to NAEP? I don’t think so.
Differing Standards
Are you an expert in anything? What do you think the important knowledge and skills in that topic are? Could you make a list of them—an organized and detailed enough list to guide years of instruction on that topic?
Let’s imagine cooking. Here are some questions you’d need to figure out
How important is baking? How much might you want to focus the skills and knowledge of baking breads? Cookies?
Roasting? (What is the difference between roasting and baking, anyway?) What are all the important skills of roasting meat? Roasting vegetables? Roasting pastas—or is that baking?
Grilling? Is that the same as barbecuing? What are the important skills and knowledge there? Still gotta cover gas, charcoal and wood?
What are the important skills and knowledge around salads? What is a salad, anyway?
Old school skills: aspic? Jello mold? What about them?
What about principles of healthy cooking? What are those? Are they worth including? In what year? What are the skills and knowledge?
Blooming spices? Is that on your list? Should it be?
Reusability of parchment paper? How to clean cast iron pans? How to season them? How to clean a blender properly? How to make clear ice cubes?
The thing is, my year by year list of critical knowledge, skills and abilities would be different than my co-author’s, and different from my wife’s. And different than yours. Two really good lists could still differ significantly—even radically.
We used to have more variation across the states when it comes to state learning standards for math and reading. We have less variation today, in large part because of the widespread adoption and adoption of the Common Core State Standards (CCSS). One would think that that eases the problem.
NAEP is Not Aligned to Common Core
The problem is that the widespread influence of CCSS has not really gotten to NAEP. For NAEP to align to Common Core would break comparability over time. Measure something else, even something only moderately different, and you should not compare the old scores to the new scores. All those longitudinal sequences would break—and longitudinal sequences is a big raison d'être of NAEP.
There is lot that I really like about NAEP. It is very well designed, produced and implemented assessment. It’s just testing the wrong thing.
But it does not measure what teachers are told to teach.
Does it generally measure the right things? Sure. Generally. But not exactly the right thing. It’s kinda measuring the wrong thing. Far from totally wrong, but not really the right thing.
Like my wife’s sister, or maybe her identical twin sister. If my wife and her (fictional, btw) identical twin sister were raised in the same family and took all the same classes, would it be ok to only test her sister and then say that the scores and grades applied to my wife? Would that be accurate? How far off do you think the scores might be?
NAEP is measuring over there, but teachers are told to teach over here. They are close, but they are not the same. So, how much can we trust NAEP to reflect that real state of learning and proficiency of our students?
There is No Good News Here
There is no way to spin the latest NAEP results (math, reading) as good. The downward trends are concerning. But frankly, I have no idea the extent to which they merely represent divergence between CCSS and NAEP’s own alignment references. None. And I’ve seen few serious efforts to figure that out. The fact that the downward trends predate COVID, but postdate the widespread adoption of CCSS really concerns me.
But down is down. It is not up. NAEPs measured constructs are similar enough to CCSS’s that I would hope to see increases in NAEP, even if they are attenuated from the actual learning and proficiencies of students. The problem is that it is quite easy to imagine and more and more refined efforts to address CCSS’s versions of the constructs result in more drift from NAEP’s foci.
Nonetheless, the trends are national in scope. Virtually across the board. That ain’t good. Perhaps it is just a reflection of testing the wrong thing, but there’s no good evidence here, no good story to be told.
I care most about making sure that the tests actually measure what they claim to measure—well, other than caring about the learning, development and health of children, of course. I think there a broad crisis in standardized tests misrepresenting what they actually mean with an audience that is hungry for particular meanings from those tests. If the tests are not measuring the right thing, the usefulness of that entire endeavor is questionable. The coverage we see of NAEP results does not account for this, which is perhaps evidence that we should not be paying any attention at all to them. If we cant’ get it right with NAEP, what hope is there for other assessments?
When state standards were so varied, NAEP offered a common yardstick to judge them all against. But the NAEP team’s conclusions on what to measure turned out different than the National Governors Association's and Council of Chief State School Officers’ team's conclusions about what to teach. So, my biggest concern is that NAEP’s supplementing other assessments with its own special strengths, NAEP is arrogantly sticking to its own construct definitions.
There is a path forward for NAEP. In a country without federal power over standards or curriculum, NAEP should acknowledge the hard work of states and their leadership—and the goals of schools and teacher. Then, we might actually get more value from it.