Virtual Conversation Hour | Assessment & Reporting with Anne Shiell (2019)

On November 15, 2019, Anne Shiell guest hosted a Virtual Conversation Hour on assessment and reporting in OWC tutoring. As one of the managers for instructional services at the Walden University Writing Center, Anne discussed metrics collected at her center, as well as the benefits, barriers, and difficulties of collecting this data. The following is a summary of her presentation and her answers to attendee questions.

Assessment and Reporting at Walden University

The Walden University Writing Center is fully online and offers one-on-one paper review (asynchronous) appointments. The following data is collected in WCOnline and later used for assessment.

  • Total number of appointments completed, on a monthly, quarterly, and yearly basis.
  • Total number of appointments completed with students who self-identify as English Language Learners.
  • Appointment information by degree level and program (e.g., number of undergraduate appointments, percentage of total appointments that are made by undergraduates, trends in undergraduate writing)
  • Factors impacting the number of appointments offered, such as number of staff, residency staffing requirements, etc.

This data is reported to several individuals: a spreadsheet of center-wide metrics is sent to the Director of the Writing Center; all WCONLINE appointment data per quarter is sent to the
Associate Director for Budget and Operations at the Center for Academic Excellence; and program-specific metrics are sent to program directors and faculty.

In addition to the data above, the Walden Writing Center collects the following metrics, which are used internally and are not reported out unless by specific request:

  • Appointment availability (how far in advance the schedule is booked)
  • Any potential FERPA violations (e.g., wrong paper getting attached to an appointment)
  • Average time per review
  • Utilization rate (appointments offered vs. reserved)
  • Appointments that take place outside of the student learning agreement (i.e., papers that are not returned within 2 days)
  • Data for new initiatives, such as:
    • Number of students participating in pilot programs
    • Number of students opting in to new registration questions on pronouns and preferred names
    • Targeted marketing (specific programs, repeat students)

Collecting this data has allowed the writing center to:

  • Successfully argue for the university to fund staff increases
  • Explain/justify any decreases in appointment offerings
  • Better predict busy and slow times, which leads to  better staff capacity management
  • Assess marketing success
  • Make informed adjustments to appointment offerings by degree level
  • Confirm that appropriate time is spent on appointments (e.g., students are generally receiving the same amount of time from their tutors)

However, the following barriers and difficulties still exist in data collection:

  • Imperfect utilization rates. Because asynchronous appointments are not tied to a specific time, utilization rates in WCOnline are inaccurate.
  • No easy way to compare utilization rates with other data in WCOnline.
  • No way to track outside factors that impact appointment offerings.

Attendee Questions

How do you change your schedule in response to metrics that show when your center may be busy or less busy?

When appointment hours are underutilized, many centers use that time for additional training or meetings. If your online instructors don’t have the availability for face-to-face or synchronous meetings/training, you can keep a running to-do list of items that can be completed asynchronously when staff have free time. In addition, you can document what your instructors have accomplished when they don’t have appointments and track how these completed tasks have benefited students or faculty, to show that important work is still being accomplished. We also try to schedule team meetings, training, professional development, etc. during historically slower times.

If your schedule is too full, how do you keep track of the student complaints and use that data to leverage for more staff?

One way is to invite students to take a survey after each appointment and ask how long they had to wait to get an appointment. You can manually track comments and complaints via email as well, by using tags and sub-folders for organization. If you can track the percentage of emails that are related to limited appointment availability, as well as the time that you are spending responding to those complaints, you can use that data to argue for additional resources. Another option is to use a JotForm or similar technology to collect requested dates and times for appointments, which can then be automatically loaded into a Google Sheet for analysis.

How have you been able to get students to take surveys after appointments?

Students are often less inclined to complete surveys after an appointment because they have already gotten the feedback they wanted. Some options to increase participation include linking to the survey after each online appointment and communicating that survey responses will help inform the writing center. You can also clarify for students that each appointment is a new experience, so multiple survey responses are welcome. Emailing students individually to take a survey after their very first appointment could also increase engagement.

What kinds of questions would you put on a survey for student satisfaction?

If students say they are satisfied in a survey, it doesn’t mean they got what they actually needed. If you’re gathering data to prove your center’s worth, your data will look different than if you are collecting data on what works for students and what doesn’t. Rich responses can come from open-ended questions that ask what students find valuable or what their experience has been. You could also ask whether students were able to directly apply any of the feedback to their current writing assignment, which could show the direct impact of that appointment.

Leave a Reply