Calibrated Peer Review is 'developmental' for instructors as well as students

I've set aside the original plan of having the students in my new genetics course write letters to the editor about genetics reporting errors, because finding suitable errors turned out to be too difficult for them (but thanks for the suggestions).  The new plan (they voted and chose it) is that they will instead write short reports titled 'My Favourite Human SNP'.  This looks like it will work well both for this small pilot class and for the ~500 students we expect in September, because the pool of SNPs with associated phenotype information is large and growing fast.  The assignment still has the benefit of letting each student choose their own topic, and of being very suitable for Calibrated Peer Review (CPR).

I've already posted on our course-management system a page of instructions to the students about what is expected in this assignment, but now I'm creating the assignment within the CPR system.  Although the CPR interface for the students is run locally (i.e. at UBC) under the new CPR4 system, assignments are generated centrally on the CPRCentral server at UCLA.  This allows the central server to maintain consistency (all authors have to use the same structure) and to provide a library of past assignments that any instructor may use or adapt.

In principle I could have adapted one of the many existing assignments from the CPR library, but none of them were suitable.  Instead I'm writing my own from scratch, using some of the library assignments as models.  Now I'm working through the surprisingly many steps of creating an assignment from scratch, using the detailed Authoring Guide that CPR provides.  This turns out to be much less daunting  than I had expected, and much more enjoyable and educational (for me).

The first step is choosing a title and descriptive information for the assignment.  Because all assignments are put it the open library for others to later adapt, it's more important that this title be informative for future users than that it be the best title for the students.  A suggested student title can be included in the explanatory notes.  The assignment is also given a topic area (mine is Biology - Genetics), keywords, and a user level (mine is Lower-division undergraduate).  This information is used by other instructors but I don't think it's seen by the students.

The next step is writing explicit Learning Goals for the assignment.  I hadn't done this for an assignment before, so thinking through what I wanted the students to learn (guidelines are provided) helped to educate me about the value of setting such goals as well as providing the students with clear expectations (students see these at the top of the assignment page).

Writing the Learning Goals was the first place where the lack of formatting power raised its ugly head.  The CPR interface accepts only plain text with or without html codes (e.g. you have to manually insert
wherever you want a line break).   I expected this to be very frustrating but the combination of a nice page of html tags and my kindergarden-level html skills let me format lists of points and boldface headings without a hitch.  However I anticipate that most of the students will have more difficulty with this - at a minimum they'll have to put in the line breaks.  (Yes, I know paragraph breaks are better, but if you don't have any idea how html tags work, line breaks are easier to understand.)  I'll need to create a short example page for them on Vista showing how to do this.  They can also just use a web page I've found that lets them paste their Word-generated text into a box that converts it to HTML.

The next steps create the resources the students should use to carry out their assignment (to generate the reports that will be assessed by their peers).

I. Guidance for Studying Source Materials:  This tells students how they should gather the information and understanding that will go into their report.  It has two parts, some text describing the source materials (handouts, textbook, articles, etc) and then some hyperlinks to web pages.  Here I just used modified text from the first part of the instructions I'd already posted for my students. I didn't initially notice the second part of this section, so I hand-coded the hyperlinks into my text section - that worked fine.

II. Guidance for Writing Your Text:  This tells students what their reports should say and how this should be presented.  Here I used modified text from the second part of my posted instructions.

III.  A 'writing prompt'.  This appears above the text entry box and gives the student specific instructions about format and text entry.

Now comes the big task of preparing the 'calibration essays'.  Three of these are required; each student will evaluate these using the series of rubric questions that you create in the next step.  One calibration essay is tagged 'high quality', one 'middle quality', and one 'low quality', but they can differ along several axes, with the caution that students will have a hard time evaluating the content of an essay with too many writing errors.  I had already decided on the SNPs I would write these about, but they're only about half done right now so I'll write more about these later.

The next step is creating the series of questions that students will use to evaluate the essays.  The interface makes this quite easy; it lets you specify what kind of answer is expected (yes/no, none/some/many, A/B/C (where you specify what A, B and C mean), and whether the student is expected to enter some text in a box.   I had some questions in mind from the original instructions I'd given the students, and I thought of more while I was working on the calibration essays.  Now I have 17 questions, which I suspect may be too many, even though most of them are very simple: "Does the report contain spelling errors? (none/some/many)"; "Does the report say how common the phenotype of interest is? (yes/no)".  I'll no doubt refine these once I'm into the next step.

The final big step is, for each evaluation question, designating the correct answer for each of the three calibration reports and writing a brief explanation of why this is the correct choice in each case.  This is going to take a while, and I can't begin until I finish writing the calibration essays.  But I'm looking forward to it, because I can see how valuable it will be.

So the above is a lot of details - what's the big picture?  UBC's learning technology people agreed to set up this CPR trial because they saw CPR as much more 'developmental' for the students than our existing peer-review options (iPeer and the peer review component of Turnitin).  I agree, but I'm finding that the experience of creating the assignment is also developmental for me - I'm being gently led through a series of steps that greatly improve the learning experience I'm providing for my students, with instruction at each step so I see both why the step is valuable and how best to implement it.