Should feedback be anonymous?

Imagine this evaluation for a student:

John is one of the laziest students I have ever had. He sat in the back of the class and slept through most of my presentations. When I called on him, he simply answered, "I don't know." Not only is he lazy, but he's stupid.

Compare this to one of the comments I received on my teacher evaluation.

The instructor did not know anything about the topic. When I asked her a question in class, she wasn't able to answer it so I could understand it.

In both cases, the feedback is not really useful. In the first case, an instructor would be slammed if they wrote this type of comment on a student report card, if they were even asked for feedback on the student (it does not happen at the university level). In both cases, it would be important to have more information on both the writer and the person being evaluated. What is their relationship? How long have they known each other? How well was the student doing in the class? This would at least give some context.

More importantly, however, is that the person receiving the feedback would be able to critique the feedback and determine if it was justified or not.

Who is responsible for learning?

As you can probably tell, I received student feedback for my fall courses this semester. In addition to this feedback, which I like to evaluate and help direct me in preparing for my current courses, I also received direction at the one school on what our syllabus should include. These included such items as:

• school vision and learning goals,
• university vision and learning goals
• attendance policies
• policies for making up tests
• policies for late assignments
• accommodations for students with disabilities
• teaching methods
• classroom atmosphere
• work load expectations
• academic integrity
• grading policies
• Cell phone use

And this is only a partial list! I started to think, who's responsibility is it to learn? Shouldn't the student be accountable for knowing school wide policies and shouldn't the administration make sure students know things such as school mission, accommodations for students with disabilities, academic integrity, and workload expectations? My syllabus is currently 9 pages long. Last semester, when a student complained that something I asked them to do was not in the syllabus, I went straight to it and showed them the assignment. However, could I really fault her for not seeing it on pg. 5 of an 8 page syllabus?

There are two reasons for this level of detail in a syllabus:

1. As opposed to when I started teaching in college (and was one of the few to have a syllabus of more than 1 page, some teachers not even having a syllabus), the syllabus IS a contract (isn't even considered LIKE a contract) which can be used in a court of law.
2. New federal laws and guidelines passed in 2008 (NCLB for higher ed) requires that students be given this information. Rather than the administration spending time and money doing this in orientations (many times not required or students may be unable to attend), they incorporate it into the classroom. While this ensures a student will be informed of their rights, it also puts legal responsibility on the instructor, and gives the instructor even more administrative work to do (so why aren't instructors paid twice the salaries of top administrators?).

Student learning outcomes

This can have very disturbing outcomes for how we are preparing students for the workplace and student work ethics. As students are given detailed instructions on how to learn, and expect only to be assessed on those "standards", they will not stretch beyond what is given them. Why should they?

Likewise, assessing the "softer" skills will be more difficult and could result in a law suit. This means that all creativity will be knocked out of the students. Expect to see creativity in the workplace from those WITHOUT education. The entrepreneurs of tomorrow will probably be the drop-outs of today because the educational system has not supported the expansion of ideas.

Interestingly enough, I see our society becoming more like those of eastern Europe during communist rule. While working in Hungary, I found that the most successful workers were those that were used to doing what they were told to do, always deferring to the person above them (as they would get the blame). The entrepreneurs in Hungary were those that had been unsuccessful in school and the civil service.

Anonymity in evaluations

This brings me back to the question in my title. In the former system, universities implemented student questionnaires to get a feeling for how students perceived their learning. As most courses were not standardized, one measure for effectiveness was student satisfaction. However, if a student did not feel that evaluation was "objective" and their grade might be affected by giving frank feedback, most schools made these evaluations anonymous. Of course, back when these were first implemented, students were not used to giving teachers feedback. Instructors were still perceived as "gods" and students were reluctant to give critical feedback.

Additionally, most of the evaluations were hand written (as were assignments). A good teacher could still determine who wrote what, if they were engaged with their students. I often was able to contextualize the comments based on who the student was who wrote it. Because I received the comments long after grades were due, even if I was unprofessional enough, I couldn't change the student's grade. I received some valuable feedback through these evaluations, especially if I was able to figure out who the student was. In some cases, I was also able to disregard the comments, knowing that a certain student would never like my teaching style.

Now we come to the age of computerized evaluations. First, this means that students self-select if they will fill out the questionnaire. When it was given in the class (the teacher leaving the classroom, but college staff administering the evaluation), more students were apt to participate, even if they were indifferent. With the evaluations being online, the indifferent student will be less likely to take the time out to fill it out. Second, there is no context anymore with the evaluations. I am given a summary of the results and a summary of the comments. So did the student hate everything about my class or just the fact that they had to take the class (they would have hated it regardless of the teacher--typical for public speaking classes). These disembodied comments make it difficult for me to determine what I could have done better. Finally, these evaluations are now used to justify the hiring or retention of faculty member, the "effectiveness" of a college, and even strategic planning.

However, my students have never felt intimidated to express their displeasure in my class. I often hear complaints about how my assignments are "subjective" (I think the word they are really looking for is complex), they have too much work to do (even though I have had to outline my work expectations at the beginning of the course, and this does not change), and how other classmates aren't as smart, hardworking, etc... So why should they worry about putting their name to a course evaluation? Especially if the instructor will not see the evaluation until after final grades have been given?

So back to my opening comment. What would happen if students were not given individual grades for each course, but were asked to evaluate (anonymously) each student on skills such as critical thinking, class preparedness, content mastery, communication skills, ability to express themselves in text, technology, etc... Each semester, these evaluations would be aggregated for a grade, and comments would be summarized. At the end of four years, these evaluations would be sent to the school to review and decide if the student qualified for a degree. In addition, these aggregated scores would be sent to all potential employers. What kind of uproar do you believe would result? Wouldn't students want to know who graded them how and why? Why is this different for instructors?

Assessment and preparing for the 21st century

As I am trying to get finish up my grading, issues of assessment and evaluation keep coming up.

Some of the questions/problems/issues I am dealing with include:

Should we be using rubrics? Don't they limit creativity and motivation? My students will only do what is on the rubric. I am restricted to grading them only on the rubric. So some students who just do a mediocre non-creative job may get the same grade (or higher because I used the rubric) than those that used more creativity, not really doing what I outlined as expected, but addressing the problem in a whole new way.

If this is what we are training our future workers to do, we will have the "best" workers (those with the highest grade) doing only what their managers told them to do, thus stifling the possibilities and creativity of our workers.

Should we be telling our students our expectations for every assignment, giving them detailed instructions? How many times have you received detailed instructions on the job? Shouldn't we be creating the skill to negotiate "outcomes" with our superiors? Situations change, factors we can't control require us to change outcomes, and there might be a disconnect between strategists and front line workers. Companies would benefit if workers and managers began to communicate about outcomes be and open to the changing environment.

If graduates only expect to do the work that a manager has outlined for them, an organization loses incentive, ideas, and reality checks from both those at the top and those on the front line.

Should all grading be "fair"? How is "fair" defined and aren't there instances when a standardized means of evaluation is in fact "unfair"? I have students that excel in class. They are engaged, apply concepts or take ideas to new levels. But they don't "evaluate" well. Neither papers nor tests get at their level of understanding that I see in class. They do the work, but it does not translate into their grades. Is this fair? They will be great employees on day. In addition to their academic work, they are good team players, contribute to the class, have a good work ethic, act responsibly in class, and add to an overall good environment. Others that test well are over critical, stifle others ideas, create a negative environment, and really don't do much work, but are good writers and/or good test takers. I'd rather have the ones that don't grade well than those that do grade well. So how fair am I in my assessment?

So, a word of warning to all those in the training and HR departments: make sure you are using other measures to identify the ideal employees. GPA's don't really reflect a student's potential. A word of warning to educators: we need to re-evaluate how we assess in the current climate of "standardization." We may be doing our society a dis-service.

A Call to Reassess Assessment

Okay, so my top 10 list will be delayed. But the topic of assessment is screaming at me and just needs to be put down on this blog!

Two things happened over the last week that just gets my blood pressure up. First, my son got his PSAT scores back. For those that are not familiar with the PSAT's, these are the preparatory tests given a year or so before you are expected to take the SAT. Now, my son did well, not great, but well. But what really upset me was his impression that he wouldn't get into school or perhaps the school he wants to go to because of one test. He asked the question that I have asked the past 40 years when I took my PSAT's: how can doing the work FASTER mean that you are SMARTER? As he said, "They didn't test if I was a good reader, only faster."

Meanwhile, a project I have been following is finally ready for implementation. The developers were very proud of the way the pre-test and post test looked, with explanations given when a user gets the multiple choice question wrong. However, the project's goal is to change attitudes and professional culture. So how can a pre-test/post test multiple choice test with the "correct" answers assess these attitude or cultural belief changes? More importantly, how can we expect changes over the course of a week? Doesn't it take time?

I keep reading about the changes in Instructional Design, changes in the tools we use, even changes in the theoretical basis of instruction. Yet, we still fall back to the old measures. This is like using a magnifying glass to verify nano-technology research. We just can't see everything that is going on.

A New Way to Assess Complex Learning

I have written previously about the need to change assessment. I especially liked Ewa's expansion of my original idea:

test those who were not present in the training but worked on a project / assignment, with the ones who were. Assessment after 3-6 months of the project and the methodology used should be team-based: teams assess each other in front of the other teams that participated in the training. This way you'd see the advantages of networking and flow of information as well as assessment on how useful your training was: if people didn't use the methods you might want to reconsider its adaptation for the organization. I think this should be the model for business school or any other applied science education. Project- based Companies that want to practice continuous learning could also apply this model.

In terms of the SAT's, why must the tests be timed? If it is so important that students be ranked depending on the timing, then students need only finish the test, then punch in when they are finished and a time is generated. Is this just to generate a false sense of pressure so schools can see who does well under exam pressure? And shouldn't the SAT's include measures of the use of tools? I don't want to imply that those who do well on the SAT will not do well in college. But many of my "best students" come ill prepared for the college classroom as they are used to doing only what is given them to do. They also are paralyzed when they have to deviate from the written material and give their own insights. Education is becoming much more complex and the assessment tools should be also. Why isn't there a movement towards standardized portfolio assessment as a means of measuring high school student work (as NEAP does)? I wonder if this would be a better measurement of student preparedness?

The Future

I hope that students realize that there are alternatives in getting into good colleges. How many students are encouraged to enter national academic competitions (often this is reserved for only the "top students", those that take tests well)? Certainly demonstrating your ability in a National Writing competition, science fair, or even having an article published on a top the student has an interest in counts for more than a timed multiple choice test.

My hope is that the schools my son is interested in looks at the overall picture. But from my own personal experience, I know this doesn't happen. It is easier to cull the pool of applicants using a superficial mechanism when you have 10-20,000 of them. However, I am proof that there are many ways to achieve the same goal. Although it has taken many years and many rejection letters, I have straight A's and at the end of my Ph.d. program, with only the data analysis and my committee between my completing my degree.

I also would like organizations to stop looking at employees as nothing more than computers which just need the correct "programming" (training) to spit out the desired output (testing). There is a lot of untapped protential, and until companies learn how to measure this, they will wasting resources on busy work. They need only look at Coke's use of this old "numbers" model in developing New Coke, without really looking at the complexity of customer behavior.

And for all of you out there that say that this is too much work, I have never used a multiple choice test in my 17 years of teaching at the university level, and I only used multiple choice tests that were imposed on me by the organization before that. I have had up to 50 students in a class where I was the sole teacher, and still I used essay exams or projects. It can be done.

How to assess learning in different contexts

One problem that I see in the transition from school to the workplace is the question of assessment. My students complain when they don't agree with my grading, asking "how they lost points." How does this translate into the workplace? How often have we had performance reviews that we might not agree with? How much negotiation then goes into the final review? How does this translate into "action"? What is the accountability of the managers (rather than the individual). I think you won't be seeing a comparable move towards manager accountability that you now see in schools (it now has infiltrated universities so now professors are accountable for student learning rather than the "adult" students). So what will that mean for the transition to the workplace?

Already, we begin to see businesses complaining that new graduates do not take initiative, do not have critical thinking skills, and have a poor work ethic. However, teachers and schools are not entirely to blame for this. The curriculum is developed with many stakeholders in mind (including those business leaders demanding more accountability). Even within corporate training, the model of discreet testing (pre-test, post-test) is being used to measure the effectiveness of a program. Hasn't the corporate world learned in using surveys for marketing research that it is not what a person puts on a piece of paper but what their behavior is? Coca-cola learned that in the 1980's. They use the same complex models when analyzing training outcomes.