We've made it to Annapolis, and have already dived into the files.
The FSC convened at 1:30 this afternoon, sitting in a circle in one of the hotel's "ballrooms" (not as grand as you're imagining) for an hour-long discussion of the reading process. There are 18 of us -- about half are previous Truman Scholars, and the rest come from a variety of policy, educational, and foundation backgrounds. NAFAns might recognize Jennie LaMonte from Mitchell, for example. I knew a few of the Trumans from having met them at TSLW in the past when they were Senior Scholars, but knew very few of the others in the room when we introduced ourselves. Interestingly, three of the FSC members are from the admissions staffs of some of the top law schools in the country.
The group conversation was freewheeling, focused somewhat on the sample cases we read, but also talking us through the process of reading. My "conservative reading" was clearly too much so, as I scored the files 2-4 points lower, on average, than almost everyone else did, and my 0.5 score was pointed out to the amusement of all. I may have wrested the title of "dreamkiller" from last year's apparent title-holder. The good news, though, was that my recommendations of finalist vs. non-finalist were not out of line with all the others', and that's the really important part -- determining who moves on from here.
We have 597 files to read and score. They have been split into 16 regions, and we are instructed to attempt to select up to 11 finalists from each region. Each region has been pre-assigned to a 3-person team of readers. The regions are geographically arranged and are are called by the city in which finalists interview. Regions range from a high of 49 applicants (the region my team starts with, incidentally), to a low of 23 applicants (the region my team reads second, incidentally), but all will have about 11 finalists emerge from the pool -- that region with 23 may have significantly fewer, depending on the quality of the pool. Each three-person team has a "box captain," an experienced reader who organizes the team, keeps us on-schedule, and verifies that all of our review comments are complete and are helpful to Tara when she has to give feedback to FacReps. My team has been assigned three regions, with a total of 103 files.
The orientation was pretty informal. There is a feeling that it's hard to train someone to read these files -- that you just have to dive in and experience it, that you'll learn what you are looking for in your gut, and that this is a subjective process, and that's OK. Leadership, in particular, was pointed out as an area in which you have to go by intuition. "Read it as a novel," we were told (a piece of advice that I actually found quite helpful -- the candidate's story has to make sense, compel you forward, and engage you). And -- much to my surprise -- "We have no GPA guidelines." I had to raise my hand on that one -- really? NO GPA guidelines? "No, none....but you're not going to see a lot of people below a 3.5, and rarely below a 3.2." The key, instead, is that the candidate's grad school proposal needs to match up with their academic record.
A FacRep emailed me to ask what the one discretionary bonus point is supposed to be used for. It's discretionary. We're not given guidance. I have given a bonus point twice so far -- once to a non-traditional student who I wanted to reward (was not named a finalist anyway), and one to a student who "felt like a 7.5" to me but only scored out as a 6.5, and I wanted to make sure that he ended up as a finalist (he did).
After the orientation, we headed to our rooms to read. This is a very different experience than reading for Madison or Udall, where we were all in the same room at 6' conference tables, all facing the front of the room. Here, we each have a room -- imagine your typical Marriott hotel room, take out the King-sized bed, and insert a 6' round table with three chairs around it. That's what we are reading in. We're isolated from all of the other teams, but do see them at mealtimes and occasionally in the snack room to get some coffee.
The first region we are reading has three states in it, unevenly distributed -- 3 from State A, 16 from State B, and 30 from State C. We started with State A. We each read one of the files, then swapped so that each file got a second read. One of the files was significantly better than the other two, so we named it as a finalist from that state -- we are supposed to have more than one finalist per state as possible, but it just wasn't in this state. My teammates were initially concerned because there was an apparent split between my score and the other score on a file, but it was just a case of me not yet understanding the rating system -- we were in complete agreement that the candidate was not a finalist, despite my scoring it as a 6.0. One state done! One finalist selected!
We also read the second state -- same method. Every file gets two reads and scores, and a third read if the first two deviate from one another. There were two files that really stood out (one of which I read, the other of which I did not), and we have decided on them as finalists. Of the remaining 14, I would say that about four or five more were OK and are still in the running to be finalists -- and the rest have been determined not to be finalists. We're not making any decisions on that group of "OK files" until we read the third, much larger state. We're rooting for a ton of very strong files out of it to fill the remaining eight finalist spots, but know that some of these OK ones might advance.
What have I learned from reading so far?
1) A sense of passion matters. I remember Louis Blair once saying "I'm looking for students who are piiiissed off." That sense of urgency needs to really leap off the page at you -- why do they do this? What have they done to prove that they care? Have they given thought to what a career in this area looks like?
2) The story needs to make sense. Question 9 - the need of society they want to address - is a turning point in the application. When I get to that point, I should already know what you want to do, and when I read Q9, it shouldn't surprise me at all.
3) Don't set things up in the FacRep letter that aren't borne out in the application. There have been a few letters I have read by FacReps that point out that this student's most impressive leadership activity has been X, and then X doesn't get explicated anywhere in the file. That's frustrating for me as a reader -- I want to learn more!
4) Explain, explain, explain. A lot of these candidates seem to have done really terrific things, but I can't completely tell you what they are. "She transformed our leadership curriculum" doesn't tell me what the leadership curriculum was, what changes she made, why that is important, or what role the leadership curriculum has on your campus. There are a few candidates whose files I have read where I thought, "this campus LOVES this guy -- and I am really not sure why." You want to take the FacRep's word for it, but the proof has to be in the application.
5) GPA really doesn't matter much, as they said in the training. I am spending very little time on transcripts -- I get a sense of the student's smarts from their writing, and the letters of rec are much more illustrative of the student's abilities than are a GPA or transcript. The GPA is part of the overall story, and needs to make sense in the context of the story -- where they have come from, and where they are going.
That's day one. Really just a half-day -- from 2:30 until we knocked off around 6pm. We wade back into them early tomorrow morning. Questions are welcome!
Thursday, February 17, 2011
Tuesday, February 15, 2011
Better Than A Trained Monkey
Before the FSC meets, the Foundation wants to make sure we are all on the same page when it comes to reading and scoring files. To help with this, Tara has sent each of us a binder that includes instructions on how to read files, three sample files that have already been scored as examples, and then four files for us to read and score in advance of arriving in Annapolis later this week. The samples are all actual files from the 2010 competition which, in some way, proved challenging for last year’s FSC members to rate. When we arrive later this week, we’ll start with an hour-long orientation to help us understand the process further, which will include discussing these files.
As a FacRep, I found it interesting that the FSC is explicitly told that because “Faculty Representatives can appeal this committee’s decision and request reconsideration,” we should “take a conservative approach in deciding whom to advance to the interviews.” If candidates are borderline, then, it appears that the FSC usually sides with not selecting them as finalists, and depends on us as FacReps to signal the foundation through use of our appeal whether or not we really, really think they should advance. I’d be interested in hearing other FacReps’ thoughts on the appeals process; Grinnell uses our appeal every year for the non-selected candidate we think was the strongest, but I know (and deeply respect) other FacReps who rarely use the appeal and only do so when they believe a true oversight has occurred. How do others use the appeal?
Reading the three sample files that came pre-scored was humbling. I thought all three were excellent, but only one had been advanced as a finalist last year. Key to understanding the Truman selection process is the Truman Nominee Rating Form, which uses a 12-point scale. Note, though, that a score above 8 is considered “Outstanding,” and finalists usually end up in the 6-8 point range. In other words, FSC members are hugely stingy with points. The three sample files had scores of 4.0, 4.5, and 6.0, and only the last of those three was advanced as a finalist (and was ultimately not selected as a scholar). And these were great files! I think the early lesson here is that if you are a strong Truman candidate, you are average within the context of the pool. Only those whose applications are really stellar will advance as finalists.
The most helpful document in the binder, though, is a one-page document that describes what a 3, 2, 1, and 0 means in each of the three categories (public service, leadership, and appropriateness for graduate study program). As this document notes, a 3 is rare, and a 0 in any one category “probably should not advance as a finalist.” Still, I’ve already seen that this document can’t cover every scenario, that readers have to learn an instinctual feel between a 1.5 and a 2.0, and that scoring a Truman application is not about checking off boxes.
After reading the samples, I finally read and scored the four files, which was no easy task. One was included, I think, as an example of a really good kid, well-respected as a leader on campus, good letters of rec, but missing crucial elements that make up a Truman. I scored it as a 0.5. I’m taking the conservative advice to heart! I thought the next one wasn’t a lot better – the letters seemed tepid, the leadership results were all things that seemed to be promised in the future, and the goal statements seemed overly broad and a bit naïve. I scored it as a 2.0, but then (after I was done) looked up the finalist list from 2010, and the student was indeed named as a finalist. Hmm. So I emailed Tara to figure out what I was doing wrong, and said that I was struggling to figure out the difference between poor/good/excellent/wow. Her reply – in that distinctive Tara voice, was “That confusion is intentional - because there is no one answer to what is poor/good/ excellent/wow. Hence why there are so many of you readers instead of just me and a trained monkey tossing files at a wall somewhere.”
So lesson #1: it’s subjective. Lesson #2: that’s ok. Lesson #3: I will learn my own subjective reading of the files as I read more and more of them.
The third file was interesting, and I can see why it was included as a “challenge.” The student had a lot of accomplishments, but I never got a sense of his passion or what inspired him. His letters also seemed vague, and his vision of the future (Qs 12 and 13) was not particularly visionary. I scored it as a 4.0, and did not recommend it for finalist status.
The final file seemed to be head-and-shoulders above the rest, and was pretty clearly a finalist (and, I later learned, the student was indeed selected as a 2010 Scholar). The leadership was pretty amazing (I scored it as a 2.5, reserving that 3.0 for even better cases, which may be mythical), and the rest of the application was above-average – except for the writing, which I found oddly stilted, and so I took off a half-point on “quality of the application.” I also thought the future plans were a bit muddled, and his recitation of his public service record was unclear. I scored it as a 5.5, and recommended it for finalist status.
Scoring these files is really difficult. I recognize that these sample cases were chosen specifically because they are challenging, but a) there is a lot of information in the Truman application, b) it’s hard to compare very different records and types of accomplishments, and c) there’s a learning curve to understanding the Truman point scale, where a 6/12 is considered excellent, and an 8/12 should make you stand up and cheer. I can only guess at how much harder it will get once we add on a time pressure, too. I really took my time with these sample cases, and feel sure we won’t have that luxury once we’re faced with stacks of them later this week.
Lastly, I think I knew that the FacRep’s letter of nomination was the first way that the reader encounters the candidate, but seeing them at the front of these files (the files look just the way they do when you print them off the Truman website) brings home to me what a huge responsibility it is to write these. It really helps to frame the candidate, set-up what they have done, and contextualize their accomplishments from the start. I am going to try to think more about what I learn from reading a lot of them in the coming weeks, and hope to have some advice to share on this topic by the end.
This is very likely my last blog post until I am in Annapolis on Thursday. I leave Grinnell at 4:00am, we meet at 1:30pm for orientation, and start diving in to files by 2:30 for four-and-a-half hours. We’ll also be reading on Friday and Saturday from 7:00am to 6:30pm.
As a FacRep, I found it interesting that the FSC is explicitly told that because “Faculty Representatives can appeal this committee’s decision and request reconsideration,” we should “take a conservative approach in deciding whom to advance to the interviews.” If candidates are borderline, then, it appears that the FSC usually sides with not selecting them as finalists, and depends on us as FacReps to signal the foundation through use of our appeal whether or not we really, really think they should advance. I’d be interested in hearing other FacReps’ thoughts on the appeals process; Grinnell uses our appeal every year for the non-selected candidate we think was the strongest, but I know (and deeply respect) other FacReps who rarely use the appeal and only do so when they believe a true oversight has occurred. How do others use the appeal?
Reading the three sample files that came pre-scored was humbling. I thought all three were excellent, but only one had been advanced as a finalist last year. Key to understanding the Truman selection process is the Truman Nominee Rating Form, which uses a 12-point scale. Note, though, that a score above 8 is considered “Outstanding,” and finalists usually end up in the 6-8 point range. In other words, FSC members are hugely stingy with points. The three sample files had scores of 4.0, 4.5, and 6.0, and only the last of those three was advanced as a finalist (and was ultimately not selected as a scholar). And these were great files! I think the early lesson here is that if you are a strong Truman candidate, you are average within the context of the pool. Only those whose applications are really stellar will advance as finalists.
The most helpful document in the binder, though, is a one-page document that describes what a 3, 2, 1, and 0 means in each of the three categories (public service, leadership, and appropriateness for graduate study program). As this document notes, a 3 is rare, and a 0 in any one category “probably should not advance as a finalist.” Still, I’ve already seen that this document can’t cover every scenario, that readers have to learn an instinctual feel between a 1.5 and a 2.0, and that scoring a Truman application is not about checking off boxes.
After reading the samples, I finally read and scored the four files, which was no easy task. One was included, I think, as an example of a really good kid, well-respected as a leader on campus, good letters of rec, but missing crucial elements that make up a Truman. I scored it as a 0.5. I’m taking the conservative advice to heart! I thought the next one wasn’t a lot better – the letters seemed tepid, the leadership results were all things that seemed to be promised in the future, and the goal statements seemed overly broad and a bit naïve. I scored it as a 2.0, but then (after I was done) looked up the finalist list from 2010, and the student was indeed named as a finalist. Hmm. So I emailed Tara to figure out what I was doing wrong, and said that I was struggling to figure out the difference between poor/good/excellent/wow. Her reply – in that distinctive Tara voice, was “That confusion is intentional - because there is no one answer to what is poor/good/ excellent/wow. Hence why there are so many of you readers instead of just me and a trained monkey tossing files at a wall somewhere.”
So lesson #1: it’s subjective. Lesson #2: that’s ok. Lesson #3: I will learn my own subjective reading of the files as I read more and more of them.
The third file was interesting, and I can see why it was included as a “challenge.” The student had a lot of accomplishments, but I never got a sense of his passion or what inspired him. His letters also seemed vague, and his vision of the future (Qs 12 and 13) was not particularly visionary. I scored it as a 4.0, and did not recommend it for finalist status.
The final file seemed to be head-and-shoulders above the rest, and was pretty clearly a finalist (and, I later learned, the student was indeed selected as a 2010 Scholar). The leadership was pretty amazing (I scored it as a 2.5, reserving that 3.0 for even better cases, which may be mythical), and the rest of the application was above-average – except for the writing, which I found oddly stilted, and so I took off a half-point on “quality of the application.” I also thought the future plans were a bit muddled, and his recitation of his public service record was unclear. I scored it as a 5.5, and recommended it for finalist status.
Scoring these files is really difficult. I recognize that these sample cases were chosen specifically because they are challenging, but a) there is a lot of information in the Truman application, b) it’s hard to compare very different records and types of accomplishments, and c) there’s a learning curve to understanding the Truman point scale, where a 6/12 is considered excellent, and an 8/12 should make you stand up and cheer. I can only guess at how much harder it will get once we add on a time pressure, too. I really took my time with these sample cases, and feel sure we won’t have that luxury once we’re faced with stacks of them later this week.
Lastly, I think I knew that the FacRep’s letter of nomination was the first way that the reader encounters the candidate, but seeing them at the front of these files (the files look just the way they do when you print them off the Truman website) brings home to me what a huge responsibility it is to write these. It really helps to frame the candidate, set-up what they have done, and contextualize their accomplishments from the start. I am going to try to think more about what I learn from reading a lot of them in the coming weeks, and hope to have some advice to share on this topic by the end.
This is very likely my last blog post until I am in Annapolis on Thursday. I leave Grinnell at 4:00am, we meet at 1:30pm for orientation, and start diving in to files by 2:30 for four-and-a-half hours. We’ll also be reading on Friday and Saturday from 7:00am to 6:30pm.
Sunday, February 13, 2011
Credit where credit is due
I've learned that the charming Truman image accompanying this blog is the work of artist Dustin Harbin. Thanks to Dustin and past Truman Scholar Ann Gavaghan for letting me know the origin of this image, and for giving me the chance to credit Dustin for his terrific work.
Tuesday, February 8, 2011
Reading Harry
Welcome to the 2011 Truman Finalist Selection Committee (FSC) blog. The Truman Foundation’s Tara Yglesias has been using Twitter for several years to update all of us during the FSC’s work; this year, I will be live-blogging throughout the FSC process.
Some Truman FacReps may recall that I also blogged the Udall Scholarship committee reading process in 2009. This blog will follow much that same format (including this first post, which is eerily similar to the first post on that blog!).
The blog will begin in earnest on Thursday, February 17, when the FSC members convene at the Marriot Waterfront Hotel in Annapolis, MD for an afternoon orientation. I hope to post frequently – perhaps several times a day – for the three days of the FSC’s deliberations, concluding on Saturday evening, February 19. I would recommend reading this blog in concert with Tara’s Twitter feed to get the best, up-to-date take on what the FSC is doing.
My goal with this blog is twofold: 1) to “lift the curtain” on the reading process that goes into the Truman FSC, letting everyone see what it looks like when the FSC meets, what the committee’s procedures are, and how Truman goes about winnowing the approximately 600 applications to name about 200 finalists, and 2) to share what I learn about the Truman Scholarship from the perspective of a scholarship advisor that will help me advise my own candidates in the future.
What I won’t be doing, obviously, is discussing anything specific about candidates, or giving away details that could be tied to any one applicant. I’ll stick to generalities about how the process is going and what I am learning in the big picture.
I’d like to thank Tara Yglesias at the Truman Foundation for her cooperation in letting me do this – it’s a sign of the Truman Foundation’s continuing trust in FacReps and commitment to transparency that they are allowing me to blog. I will add that I did offer, and Tara has accepted the offer, for her to read and approve each post before it goes up (including this one). I don’t want to accidentally share confidential information, or accidentally post something untrue.
A little about me, just so you know who you are reading – I’ve been the director of social commitment at Grinnell College in Iowa for about 12 years, and as such serve as the college’s scholarship/fellowship advisor. I also advise students on post-graduation service opportunities, administer our own post-grad service program (the Grinnell Corps), advise our Student Government Association, and mentor our first-year Posse from Washington, DC. I’m the vice-president of the National Association of Fellowships Advisors (NAFA), and am looking forward to seeing other Truman FacReps at our conference in Chicago in July. I’ve also previously been a committee member for the Udall Scholarship, Madison Fellowship, and Dell Scholarship, and blogged the NAFA UK trip in 2006.
I hope you’ll check back frequently during the dates of the FSC (again, February 17-19) as the blog gets going in earnest. You should also feel free to email me questions at cutchins@grinnell.edu, and you can share this URL with your Truman nominees as well. They’re not going to learn anything about how we read their individual application, but they might get an interesting look into how the decision-making process goes.
Subscribe to:
Posts (Atom)