One of the earliest motivations of the Rigorous Test Development Project was the disparity of standing between psychometrician’s and content development professionals (CDPs). The people who edit and refine the items that appear on tests simply did not have the same seat at the table as the people who did the statistical analysis of test results.
Thus, an original goal of the RTD project has been to raise the standing of CDPs – as they are the ones who work on item validity, a necessary foundation for test validity. (note: CDPs are not just item writers — who write initial drafts of items. CDP work is much more complex and extended than mere item drafting.)
The most respected professions combine copious technical knowledge with well honed professional judgment. Aspirants spend years, studying their discipline before engaging in some form of apprenticeship, in which they gradually build their own professional judgment as they observe, and or supervised by more experienced practitioners.
Less respected professions are not recognized as having the same quantity and difficulty of technical knowledge, nor requiring the same degree of professional judgment, in order to be practiced at a high-level. Thus, respect is often a function of perception and disrespect a function of ignorance.
Professional certification exams exist in many many fields, and yet every one we have ever met who has taken such an exam complains that it fails to assess what it really takes to perform the work. We believe these exams tend to focus too much on the technical knowledge (often taken out of authentically complex and interconnected contexts). Thus, they fail to address the higher level skills and professional judgment that is the mark of true expertise – – or even successful practice.
There are are bad lawyers, bad doctors, bad architects, bad, beauticians, bad plumbers, and bad project managers. In many of these fields, one cannot even be a bad practitioner without being professionally certified. Licensed professional certification does not ensure quality, and it certainly does not ensure respect for a the profession.
In our view, there are two things that truly distinguish the work of psychometricians and of contact development professionals that contribute most strongly to this disparate standings.
First, psychometricians study their angle(s) on assessment in school and obtain advanced degrees in their field. They can earn masters degrees and PhD’s in measurement. Content development professionals may or may not have advanced degrees, but they are not degrees in assessment. (They may be in teaching or in their subject matter field.) Moreover, not every CDP holds an advanced degree.
Second, many people have more respect for STEM fields than other fields. They perhaps perceive those fields as harder than other fields – confusing the fact that many STEM problems have obviously and definitively correct and/or incorrect answers with the ideas of rigor and difficulty. They seem not to understand the difficulty of coming to a truly high-quality result when it is not always easily obvious whether the result is correct or incorrect. The need to balance multiple criteria to find a truly good – – or even great – – answer is not always as respected as the kind of technical work that is built on predetermined routines and literal formulas, if those routines and formulas involves lots of numbers.
So, what can we do about the lack of standing of CDPs?
Professional licensure is no guarantee of respect from those with PhD‘s or degrees from the most respected professional schools. Certification exams do little to ensure respect or high-quality work. (In fact, regulation is usually about providing a floor for quality, rather than raising the ceiling on performance.) Of course, that presumes that there is someone who could establish and perhaps require such licensure, which is questionable in the first place.
Our efforts in the RTD project are focused on codifying the kinds of knowledge, skills and decision making that CDPs engage with every day, both the assessment-specific examples and the ways that subject matter expertise and knowledge of the perspectives of test takers inform the assessment work. We can envision this body of knowledge becoming a formal course of study as an option in educational measurement option, someday. But that is form of credentialing is not the goal of our Project. We are still trying establish the body know knowledge and skills in a form that CDPs and non-CDPs both can see.
When if comes to the standing of CDPs in test development circles, we ask that non-CDP test developers, take note of the variety of technical knowledge that CDP tap into, and their use of professional judgment to balance issues that psychometrics often barely has language for.
Perhaps this way that can engage the kind of professional humility that is so important for learning and for collaborative work, and will better understand what their CDP colleagues can bring to the table.