Saturday, April 23, 2011

Results of student survey: no need to have a focus group

I've analyzed the preliminary results of my student survey.  It provides some ideas of ways the course could be improved, but my focus-group experts agree that it doesn't raise any issues deserving focus-group investigation.

What they said:

Agree/disagree (~Likert scale):
  1. I had the necessary background for the course.  Most agreed
  2. The readings and reading quizzes prepared me for the lectures. Neutral
  3. The iClicker questions were not challenging enough. Most disagreed
  4. The Genetics in the News slides took too much time away from course material. Neutral
  5. The homework increased my comprehension of the lecture material. Most weakly agreed
  6. The tutorials helped me learn to solve genetics problems. Most agreed
  7. Having two mini-midterms and a midterm was too much testing. Most disagreed
  8. The course grade was based on too many different components. Most disagreed
  9. The workload was much higher than for other courses. Neutral
  10. I feel prepared to deal with genetics issues that may arise in my life. Most agreed

 Written Answer Questions:
  1. Should any topics be cut from the course material?  Most said no.
  2. Were any topics missing from the course that you wish had been covered? Most said no.
  3. A pizza-lunch focus group will be held later this month; all students are welcome to attend. Please mention below any specific issues that should be raised then.  Below is what they said:
  • Workload, discrepancy between difficulty of lecture material and what was tested (in tutorials, reading quizzes, midterms, etc.)
  • The tempo of the class. The first half seems like a review, and all the new stuff are in the second part. 
  • Methods of assessing learning in this course. 
  • No specific issues. 
  • How much we liked the format of the lectures -the methods in which we tried to prepare for exams
  • Mini-Midterm format. I think that the midterm was a fair examination however, the second mini midterm had a multiple choice question that had about 9 choices and was worth about 6 marks. I felt I did good on the rest of the exam but still didn't get a great mark because of 1 MC question.
  • How to study for the final. Every test has been a different format, what to expect. 
  • I would like to suggest ways to make homework more helpful in preparing us for the exams. Also, maybe investment
  • into custom booklets with some notes and problems sets like Bio 201. 
  • They are too little guidance in this course
  • Tested materials --> what to expect in midterms/exams weren't very clear 
  • Overall structure of how the course will be run next year. Textbook assignment and readings. Better formatting for the meiosis/mitosis content from the beginning of the year - personally I am still fuzzy, even though the concepts were stressed to be very important. 
  • I think going over online homework and reading quiz questions in class would help. Or perhaps explanations for the answers could be posted online because there are still questions that I don't understand. I also think the amount of work this course requires should be re-evaluated. The amount of reading is quite heavy and having two quizzes (homework and reading) PLUS peerwise PLUS tutorial each week is a lot.
  • How this course and its changes (234 vs. 334) related to other courses, such as Biol 335. 
  • How to study genetics
  • I think that the easiness of this course should be covered. I felt that this course reviewed a lot of material and didn't cover that much new material.



Ranking the course components:
Many components of this course contribute to the final grade. Please try to rank them according to how valuable you found them, taking into account your learning gains and the amount of time you invested in them. For example, an activity that took a lot of your time but resulted in little learning would score low.

  • Tutorials  High
  • Peerwise questions  Low
  • SNP report  Low
  • Calibrated Peer Review  Low
  • Online homework  High
  • Reading quizzes  No consensus
  • Studying for midterms  No consensus
  • Attending lectures  High

Only 21 of the 38 students have completed the survey so far.  That's certainly enough to go on, but I'll reanalyze the responses after the final exam marks have been posted (that's the last time the students will give any thought to the course).

Tuesday, April 12, 2011

The focus group plan

OK, I've consulted with the local experts.  They had excellent advice on how to proceed, and will be able to run the focus group for us if we decide it's what we need.

The first step is to analyze the responses from my student survey.  The survey questions are pasted below - for the purpose of the focus group the most important question was the one asking for topics for a focus group.  Once I've consolidated the responses I'll send them to the local experts and we can decide whether  issues were raised that should be considered by a focus group.  For a one-hour group we only want two or three such issues, and maybe one in reserve.

An ideal focus group would be about 6 students, and as few as three would be OK, so I think we can safely schedule it in May rather than before the final exam.  (And we do have money for pizza in the course budget.)


Survey Questions:
Agree/disagree (~Likert scale):

  1. I had the necessary background for the course.
  2. The readings and reading quizzes prepared me for the lectures.
  3. The iClicker questions were not challenging enough.
  4. The Genetics in the News slides took too much time away from course material.
  5. The homework increased my comprehension of the lecture material.
  6. The tutorials helped me learn to solve genetics problems.
  7. Having two mini-midterms and a midterm was too much testing.
  8. The course grade was based on too many different components.
  9. The workload was much higher than for other courses.
  10. I feel prepared to deal with genetics issues that may arise in my life.

 Written Answer Questions:

  1. Should any topics be cut from the course material?
  2. Were any topics missing from the course that you wish had been covered?
  3. A pizza-lunch focus group will be held later this month; all students are welcome to attend. Please mention below any specific issues that should be raised then.

 Ranking the course components:
Many components of this course contribute to the final grade. Please try to rank them according to how valuable you found them, taking into account your learning gains and the amount of time you invested in them. For example, an activity that took a lot of your time but resulted in little learning would score low.

  • Tutorials
  • Peerwise questions
  • SNP report
  • Calibrated Peer Review
  • Online homework
  • Reading quizzes
  • Studying for midterms
  • Attending lectures


Saturday, April 09, 2011

Focus group?

I've been advised that the best way to collect useful feedback from the students in my genetics pilot course is to have a focus group.  This initially seemed like a great idea (book a room, order pizza, tell the students), but I'm gradually realizing that implementing it will be difficult.  Several issues need to be dealt with.

First, I don't even know what running a focus group involves.  I expect that the discussion would need to be coordinated, and some record of the discussion kept.  This might just be notes, but an audio or video recording would be better.  But if a recording was made, then someone would have to later go through the recording, pulling out the important information.  And if there's no recording, would the person who is coordinating the discussion also be able to take the notes, or would a separate note-taker be needed?   How much expertise is needed to coordinate the discussion - can the needed skills be picked up in 5 minutes, or is formal training desirable?

Second, who is available to do this (call them the 'facilitator')?  To get uninhibited discussion the facilitator shouldn't have been involved in teaching the course or grading the students.  The person who recommended having a focus group initially suggested having the course TA run it.  This would be only slightly better than having me run it, and the TA quickly pointed out that she was not an appropriate facilitator.

There are other instructors who I could ask to act as facilitator, but I have no idea  (1)how much work I would be asking of them; (2) whether this would be considered a personal favour or part of their job; (3) whether any of them have whatever skills or experience a facilitator needs.  Might there be a Faculty of Science teaching/research person who could do this?  Should I contact our Centre for Teaching, Learning and Technology for help?

Third, I also don't know when we should hold the focus group.  I was initially thinking that we should do it in the week before the final exam.  The exam is scheduled for April 28, the very last day of the three-week exam period.  But the TA thought we should have it in May - she says many students will still be around.  And do we want to ask students to sign up for this, or just run it as a drop-in group?

Finally, who pays for the pizza?  Is there a special fund for course-development activities, or should it come out of the course's photocopying/petty cash budget?

I think I had better turn this post into an email to the person who suggested a focus group and to the head of the teaching-research group, so I can get their advice.

Thursday, April 07, 2011

Classes are over

Yesterday was the last lecture of the genetics pilot course.  I combined a review with specific everyday cases where knowledge of genetics would be useful, all framed as 'a friend or family member asks your advice, because you're now the genetics expert'.

Now I just need to prepare a final exam (and a sample final), and pull together as much feedback as possible to use in preparing for September.  I've given the class an on-line survey, and they'll do the usual post-course teaching evaluation, but we're also going to have a focus group in the week before the final, with pizza. 

Friday, April 01, 2011

Planning next year's tutorials

The other day I sat down with my favourite pedagogy expert to discuss how to improve the tutorials for the new genetics course.  I'm quite happy with the problem-solving component of the present tutorials, but they still need a component that develops students' reading-interpretation and connection-making skills.  We came up with a plan that I think will work well.

Any plan has to deal with the big practical problems.  First, students much prefer activities that they see as directly useful, and activities that will improve such fuzzy and poorly defined skills as reading-interpretation and connection-making fall far below activities that will directly improve their grade.  Second, most of the students are very anxious about speaking in class.  Third, even experienced and skilled teaching assistants are understandably reluctant to impose tutorial activities that the students don't like.  Fourth, many of our teaching assistants will be inexperienced and unskilled.

Under the new plan, students will spend the first part of each tutorial analyzing one or two short readings taken from textbooks.  They won't have to do any extra preparation for this, as the material they'll be analyzing will be part of the preparatory readings recommended for that week's classes (motivated by the weekly Reading Quiz).  They should see the analyses as directly useful, because the texts (and associated figures) will be about topics they need to master to pass the course.

In each tutorial, about 30-45 minutes will be spent on activities using these readings.  These are 2 hour tutorials, and the rest of the time will be spent working on a complex genetics problem (described in the last paragraph below).  They'll first work in pairs or small groups with a clear goal, such as
  • 'Identify a question you'd like to ask the author of this paragraph.' 
  • 'What are the important differences between the information presented by the paragraphs from two different textbooks?' 
  • 'How does the figure clarify the text?  What potential confusion does it clear up?'
Then the ideas from the groups will be discussed by the class.  I think that students are more comfortable reporting what their group came up with than describing their own ideas directly.  They could either form groups on their own, or the groups could be pre-assigned by the TA.

In the first few weeks of the class, the TA will then use one of the two texts to demonstrate one way to diagram the relationships of ideas in a text (one week hierarchical diagrams, another week flow charts, another week concept maps).  The students will then individually create this type of diagram for the other text they've been analyzing, and hand this in.  In later weeks the TA demonstration won't be needed, and the students can diagram the text in any way they like.  The TAs will mark the diagrams out of 2 points (1 for any attempt, 2 for something good), and return them to the students at the next class..

My pedagogy expert and I considered ways to let the students also look over what other students had done.  We didn't decide on anything, but later I came up with something that might be good.  The TA could hand the marked diagrams to random students (not to their authors).  Each student gets a minute to look over the diagram they've been handed before finding its author and giving it back to them.  This will be an ice-breaker, a minute of chaos that will get students talking and help them meet each other.  If we wanted to designate the pairs or groups that will discuss the new assigned text (rather than letting students pick their friends), returning the diagrams could also assemble the groups  - each report could be given to another member of the designated group.  One other possibility we discussed was having the TA choose one diagram from each tutorial and give photocopies to all the students.

Because the students should see this text-analysis process as valuable and non-threatening, the TAs should be comfortable leading it.  In the weekly TA meeting we'll prepare them by going over the readings with them, pointing out ways to help students develop their ideas.  We'll also show them how to teach the diagram-creating activity.  We'll make sure they know how to do the marking very quickly, without worrying about details.

A note about the problem solving part of the tutorials:  We've developed a complex genetics problem for each tutorial.  Students get the introductory information and one or two relatively simple questions ahead of time, and are expected to hand in the answer(s) at the start of the tutorial.  After the text-analysis, they spend the rest of the tutorial working through this problem in groups (mostly at the chalkboards) and discussing the answers to the questions it poses.  Finally the chalkboards are erased and the students are given a sheet with one or two of these questions, which they answer and hand in.  As with the text-analysis activity, the TAs are given lots of preparation for this problem-solving activity and for grading the answers (again 1 point for any attempt, 2 for a good answer).  The intent is that the TA meeting will fully prepare the TAs for their tutorials, and that the grading will not take more than one hour for each tutorial.

Once classes end next week, we're going to spend time developing the materials we have into a draft set of TA materials for each week.