Better Than A Trained Monkey

Before the FSC meets, the Foundation wants to make sure we are all on the same page when it comes to reading and scoring files. To help with this, Tara has sent each of us a binder that includes instructions on how to read files, three sample files that have already been scored as examples, and then four files for us to read and score in advance of arriving in Annapolis later this week. The samples are all actual files from the 2010 competition which, in some way, proved challenging for last year’s FSC members to rate. When we arrive later this week, we’ll start with an hour-long orientation to help us understand the process further, which will include discussing these files.

As a FacRep, I found it interesting that the FSC is explicitly told that because “Faculty Representatives can appeal this committee’s decision and request reconsideration,” we should “take a conservative approach in deciding whom to advance to the interviews.” If candidates are borderline, then, it appears that the FSC usually sides with not selecting them as finalists, and depends on us as FacReps to signal the foundation through use of our appeal whether or not we really, really think they should advance. I’d be interested in hearing other FacReps’ thoughts on the appeals process; Grinnell uses our appeal every year for the non-selected candidate we think was the strongest, but I know (and deeply respect) other FacReps who rarely use the appeal and only do so when they believe a true oversight has occurred. How do others use the appeal?

Reading the three sample files that came pre-scored was humbling. I thought all three were excellent, but only one had been advanced as a finalist last year. Key to understanding the Truman selection process is the Truman Nominee Rating Form, which uses a 12-point scale. Note, though, that a score above 8 is considered “Outstanding,” and finalists usually end up in the 6-8 point range. In other words, FSC members are hugely stingy with points. The three sample files had scores of 4.0, 4.5, and 6.0, and only the last of those three was advanced as a finalist (and was ultimately not selected as a scholar). And these were great files! I think the early lesson here is that if you are a strong Truman candidate, you are average within the context of the pool. Only those whose applications are really stellar will advance as finalists.

The most helpful document in the binder, though, is a one-page document that describes what a 3, 2, 1, and 0 means in each of the three categories (public service, leadership, and appropriateness for graduate study program). As this document notes, a 3 is rare, and a 0 in any one category “probably should not advance as a finalist.” Still, I’ve already seen that this document can’t cover every scenario, that readers have to learn an instinctual feel between a 1.5 and a 2.0, and that scoring a Truman application is not about checking off boxes.

After reading the samples, I finally read and scored the four files, which was no easy task. One was included, I think, as an example of a really good kid, well-respected as a leader on campus, good letters of rec, but missing crucial elements that make up a Truman. I scored it as a 0.5. I’m taking the conservative advice to heart! I thought the next one wasn’t a lot better – the letters seemed tepid, the leadership results were all things that seemed to be promised in the future, and the goal statements seemed overly broad and a bit naïve. I scored it as a 2.0, but then (after I was done) looked up the finalist list from 2010, and the student was indeed named as a finalist. Hmm. So I emailed Tara to figure out what I was doing wrong, and said that I was struggling to figure out the difference between poor/good/excellent/wow. Her reply – in that distinctive Tara voice, was “That confusion is intentional - because there is no one answer to what is poor/good/ excellent/wow. Hence why there are so many of you readers instead of just me and a trained monkey tossing files at a wall somewhere.”

So lesson #1: it’s subjective. Lesson #2: that’s ok. Lesson #3: I will learn my own subjective reading of the files as I read more and more of them.

The third file was interesting, and I can see why it was included as a “challenge.” The student had a lot of accomplishments, but I never got a sense of his passion or what inspired him. His letters also seemed vague, and his vision of the future (Qs 12 and 13) was not particularly visionary. I scored it as a 4.0, and did not recommend it for finalist status.

The final file seemed to be head-and-shoulders above the rest, and was pretty clearly a finalist (and, I later learned, the student was indeed selected as a 2010 Scholar). The leadership was pretty amazing (I scored it as a 2.5, reserving that 3.0 for even better cases, which may be mythical), and the rest of the application was above-average – except for the writing, which I found oddly stilted, and so I took off a half-point on “quality of the application.” I also thought the future plans were a bit muddled, and his recitation of his public service record was unclear. I scored it as a 5.5, and recommended it for finalist status.

Scoring these files is really difficult. I recognize that these sample cases were chosen specifically because they are challenging, but a) there is a lot of information in the Truman application, b) it’s hard to compare very different records and types of accomplishments, and c) there’s a learning curve to understanding the Truman point scale, where a 6/12 is considered excellent, and an 8/12 should make you stand up and cheer. I can only guess at how much harder it will get once we add on a time pressure, too. I really took my time with these sample cases, and feel sure we won’t have that luxury once we’re faced with stacks of them later this week.

Lastly, I think I knew that the FacRep’s letter of nomination was the first way that the reader encounters the candidate, but seeing them at the front of these files (the files look just the way they do when you print them off the Truman website) brings home to me what a huge responsibility it is to write these. It really helps to frame the candidate, set-up what they have done, and contextualize their accomplishments from the start. I am going to try to think more about what I learn from reading a lot of them in the coming weeks, and hope to have some advice to share on this topic by the end.

This is very likely my last blog post until I am in Annapolis on Thursday. I leave Grinnell at 4:00am, we meet at 1:30pm for orientation, and start diving in to files by 2:30 for four-and-a-half hours. We’ll also be reading on Friday and Saturday from 7:00am to 6:30pm.