Still on the topic of reference letters (a.k.a. letters of recommendation), today I have a few comments about the evaluation forms that the person providing a letter (the referee) is expected to fill out and attach to the letter. Almost any graduate-school application includes such an evaluation form. Typically, the form has a few questions about what the referee thinks of the applicant’s abilities and potential. For the most part, these questions pertain to the same things that a good letter writer would put into his or her letter, anyway. So, the fact that there is an evaluation form like this is neither surprising nor worrying for most grad-school applicants; most assume that their referees have very good opinions about them.
There is another part to most evaluation forms, however, which most first-time applicants do not anticipate, and thus a lot of them suddenly get very nervous when they discover it. It is a list of general characteristics, abilities, or tendencies, on which the referee is asked to rank the applicant relative to an appropriate comparison group (such as all senior undergraduates they have known during their career, for example). For each characteristic, ability, or tendency listed, the referee places a checkmark to indicate a rough percentile ranking for the applicant within the comparison group. For example, the referee may indicate whether the applicant ranks in the top 2%, 10%, 25%, or 50% within the group; there is usually also an “unable to judge” category. The same ranking scale used for each attribute.
Some of the attributes or dimensions on which applicants are commonly ranked include: critical thinking, analytical thinking, communication skills (oral and written), integrity, judgment, organizational skills, work habits, interpersonal and leadership abilities, maturity, ability to work with others, ability to work independently, originality, ability to adapt to changing conditions or unexpected events, motivation toward a successful and productive career.
This is not an exhaustive list, by any means, and different programs have their own particular subset of such attributes on their evaluation form, but most will have variants of at least a few listed here. There is some variability across disciplines in terms of what tends to be on evaluation forms, but not much.
(You can see an example of a graduate-applicant evaluation form at the very end of this PDF file, which is used by many of the graduate programs at Concordia University).
Why does this part of the evaluation form cause students so much anxiety? For starters, many are quite surprised to discover the kinds of things on which they will be ranked, and they doubt that their referees know them well enough to make a ranking on most of the attributes. After all, does a professor who taught you a few classes and always gave you excellent marks, liked your essays, and with whom you have a friendly and chatty relationship, really have any basis for ranking you in terms of ability to work with others, critical thinking, leadership abilities, or integrity? In most cases, it’s not likely. At least, not if that’s the only capacity in which the professor has known you. (Typically, the form will also ask referees to indicate the capacity in which they have known you – such as, teacher, academic advisor, research supervisor, for example).
The point I’m hoping to make clear is that the fate of your graduate-school applications may depend a lot on how well you set up your letters of recommendation – that is, whether you do enough beyond your coursework to prepare for letters of recommendation from academic sources (ie., professors). By discussing these evaluation forms, I’m hoping to give you a better idea of what your referees will be expected to consider when writing a letter of reference for you. Hopefully, this will help you decide whether you have done enough to prepare for these recommendations, and whether the individuals you are asking to provide them are appropriate in terms of how well they know you. Remember, you’ll need at least two and maybe three of these letters. They must come from the right people, and those people will not only be describing things about you that they think are relevant, they will also be ranking you on personal dimensions that the graduate schools care most about. If you are getting worried and you want to know more about arranging to have the most effective letters of recommendation, I recommend this article that I wrote for myGraduateSchool
You may be wondering, … How important are these rankings? And how do they come into play when decisions are made about who gets in and who doesn’t. Let’s just say, they are extremely important – in fact, in most cases, even more influential than the reference letter!
I’m sure that last statement comes as surprise to many, but it is true. How the referee ranks the student within a relevant comparison group is usually more decisive than the actual content of the reference letter. It’s not difficult to understand why, once you understand the purpose of the rankings, and how admissions committees and prospective graduate supervisors use them.
Here is the logic behind why referees are requested to provide these rankings:
Other than requesting referees to comment on the applicant’s abilities or potential in specific areas, graduate schools have no control over what kinds of things a particular referee decides to include in his or her letter. Not surprisingly, there ends up being a lot of variability among the letters in terms of what things are commented on. All letters will generally say positive things about the applicant, but how are we to compare different positive comments, about different attributes, and from different people? If two letters discuss relevant albeit different things about two applicants, and both use similar superlatives, how is one to decide which letter makes a stronger endorsement? Not only that, but the two referees will also have their own style of letter writing; perhaps one of them makes liberal use of superlatives when describing their favorite students, whereas the other reserves the use of such words for only the most truly exceptional students. Not only that, but two different people reading the same letter may be left with somewhat different impressions, if not in kind, at least in magnitude. Think about how difficult all these foregoing factors make the task of discriminating between applicants on the strength of their reference letters. It is hard enough for one person to discriminate between applicants this way – how are the different members of an admissions committee supposed to compare their assessments of different applicants’ letters?
The main purpose of the evaluation form is to ensure that all referees provide assessment on a common set of attributes, using a somewhat objective scale, so as to get around the problems of ‘comparing apples and oranges’ inherent in reference letters. By requesting that referees rank the applicant on specific attributes, the graduate schools get information about the things that matter most to them. Everyone understands what it means to indicate that the student is judged to be among the top 2% versus among only the top 10%, or the top 25%.
Its easy to see how the rankings ‘level the playing field’ a bit, because they supposedly yield comparable information about each applicant. As I already mentioned, the ranking also provide the graduate program with some control over what attributes are actually assessed by the referees, because reference letters are not uniform in that respect. And there is still another way in which the rankings are of great utility for the members of an admissions committee or a professor: The rankings can be a major time-saver for some, by providing a concise and ready overview of how the referee views the student’s potential.
Someone faced with a large pile of applications to go through may only bother to carefully read the reference letters for students who have been rated by their referees as being among the best on all, or at least most, of the important dimensions. The rankings actually provide most of the information needed from the referee, and a letter of reference is generally used mainly to substantiate the basis for high rankings. (Yes, I am implying here that many of the reference letters written for grad-school applicants are never read. Instead of implying, let me just be explicit and tell you that many of the reference letters written for grad-school applicants are never read).
Keep in mind that different people have their own ways of completing a ranking form, and admissions committees and experienced professors all know this. Some professors consistently provide overly high rankings, placing virtually every student for whom they have written a letter of reference in the top 5%. Different professors will be more stringent, or honest, or discriminating, in their rankings. No doubt, the practice of evaluation-by-ranking is not without its pitfalls, but it’s probably true that no method of assessing a graduate-school applicant is ever going to be perfect.
Although you have no direct control over how a referee ranks you on a form, you do have control over whom you ask for a letter of reference, and if there is still enough time before you will need them, you might also have some control over how well your referees know you and how highly they are able to rank you on key dimensions. Now that you know about the evaluation forms and the importance of these rankings, are you confident that you have asked the right people to provide your reference letters?
And another thing…
In most cases, the evaluation form has a section that the student needs to fill out before giving it to the referee. It’s just basic information, like the name of the applicant and the specific program to which he or she is applying. It goes without saying that you need to properly fill out this section before giving the form to your referee. Still, it’s surprising how often students omit some of the pertinent information, such as the particular program. This can be annoying for the referee, who may be uncertain of exactly what the student is intending, and it can hamper the ability of the referee to customize the letter according to the specific program.
If there is more than one page to the evaluation form, look for a place at the top of the second page (and any subsequent page) where you are supposed to indicate your name and the anticipated program of study. In my experience, students are just as likely to miss this on the second page of a form, as they are likely to actually fill it in. That’s right – about half of people fail to fill it in! Is it because they think the referee is supposed to do it? It’s not the referee’s responsibility. It is yours. So be sure to fill in your information on every page that has a place for it. Failing to do so can make you seem careless.