Creating standards and assessing learning

Results of the New York State Math test, a standardized test given to children in grades 3-8 (8-14 year olds), was announced yesterday. What was amazing was the analysis of what these test scores meant and why students in the NY City area had done so well. Theories included that the test was too predictable to the fact that New York City schools were now under control of the Mayor. Surprisingly, the fact that a new formula to calculate state aid and greater investment into urban schools was never mentioned! This despite the fact that grades increased in all urban areas.

What this demonstrates is that it is difficult to assess learning solely on the basis of a standardized test results. Even more important, there was no analysis of what standards students were meeting and the impact that would have on higher ed and workplace readiness. Instead, Merryll Tisch, chanceler of the New York State Board of Regents announced that perhaps the passing score should be raised.

What do standardized assessments really measure?

A standardized test if designed correctly, measures the retrieval of information set out by the curriculum. In some cases, it may measure performance of a standard (i.e. the lab section of the NY state science test). However, often it measures the knowledge of processes rather than a level of understanding in performing a task.

For a standardized test to be successful, it should:

1) Be aligned with the stated standards
2) Have some sort of mechanism to ensure consistent evaluation and grading
3) Ignore other learning or knowledge that has not been defined in the standards

Standardized tests do not always have to be an objective test. For example, the GRE has a critical writing component in which evaluators are trained and each written answer is evaluated by 3 different people to ensure consistency in grading.

Standardized tests therefore do not grade a student's learning, but rather if they are familiar with the intended material, content, or curriculum the standards reflect. They also do not measure if the standards will prepare a student for college or the workplace if the standards do not reflect the skills needed. (See the related article on the White House push to improve curriculum to prepare students for the 21st Century).

Finally, I would think that the goal of assessment would be to have a high maintained level of passing as it demonstrates that more and more students are achieving the stated standards. However, from Chancellor Tisch's comments it would appear that either the standards are inadequate (which is why students need remedial classes) or it is being used to rate students, excluding a certain % (which is why the passing score would be raised). Standards should be changed if they no longer are aligned with the needs of the learning outcomes, not because "too many passed the test."

Choosing a better way to assess

It seems that there is a disconnect between

1) learning standards and learning outcome needs
2) what is methods of measuring learning and instructional design
3) knowledge and application of that knowledge in multiple contexts
4) having content and understanding the content

The first step in any instructional design should be to establish learning goals and develop measures to assess learning outcomes. In developing these learning goals, it is not enough to identify what someone should "know" or be "familiar with" unless these are aligned with how learning will be used.

For example, training for new regulations in the insurance industry can use standards such as "know" or "be familiar with" because it is knowledge of the regulations that is important to regulatory agencies.

However, training for FERMA personnel in procedures for first responders for a natural disaster will need to have an understanding of the context and environments in which they will be working. In addition, there is a level of tacit knowledge which will need to be used (they need to know the practice of disaster relief, not just the process).

A good example of melding these two types of assessment can be found in the driver's test. The first stage, getting the learner's permit, requires a written test about the laws required for driving. The road test, taken after practical training, looks at a driver's ability to assess the driving conditions, react to changes in the driving environment, and showing judgment in using certain driving skills based on environmental conditions. A student who is driving too closely so that they need to slam on their breaks when someone in front makes a quick stop is assessed differently than someone that slams on their breaks when ever they are startled. Both are slamming on the breaks, but in the second case, when there is no environmental reason, there would be a perception of less driving skill and knowledge.