1. Home

Cleveland Clinic's First UX Study

Blog Date
Author(s)
Stuart O. Smith, Jr.
Tags
Usability
UX
Cleveland Clinic
NEOUPA
User Experience

I was very fortunate to be able to attend the  April 25, 2012, NEOUPA (Northeast Ohio Usability Professionals' Association – neoupa.org - @NEOUPA) meeting on Clinical Trial: Challenges Faced & Lessons Learned Conducting Cleveland Clinic's First UX Study with Kaitlan Chu (www.linkedin.com/in/KaitlanChu - @KaitlanChu). I was having a busy week and did not know if I could attend, but decided to make the time.  I am glad I did!

Here is the description of the talk from the NEOUPA website:

April 25, 2012 - Challenges Faced & Lessons Learned conducting Cleveland Clinic's First UX Study

Join NEOUPA at the Cleveland Clinic downtown for a presentation and tour.  Kaitlan Chu will be sharing her experiences of conducting the Clinic's first-even UX study, which included a large open card sort.  Kaitlan will discuss the study itself, as well as how she initiated the first in-house usability study; challenges faced and lessons learned throughout the process; the implication of this study and its impact on the UX efforts in the organization.

Before the presentation, you’ll have the opportunity to see some of Cleveland Clinic’s high-tech patient care facilities, visit the rooftop terrace, and see the remarkable underground robotic supply transport system.

About the speaker:

Kaitlan Chu is the sole IA for Cleveland Clinic's enterprise website whose 13,000 pages receive 120,000+ visits daily.  She provides thought leadership on usability on the Clinic's web and mobile platforms.  Kaitlan believes in letting data settle design debates from her 9 years of experience in information architecture, interaction design, and usability research in the health care, consulting management, academic and non-profit segments.  She conducts usability studies and evangelizes the research findings throughout the organization, one of the top 4 hospitals in the United States.

Meeting Time: 6:30-8:30pm

6:00 - 6:30 pm - Gather & network (Conf. Room Q1-300)
6:30 - 7:15 pm - Optional Tour
7:15 - 7:30 pm - NEOUPA Announcements & Speaker Intro 
7:30 - 8:30 pm - Presentation with Q&A

Cleveland Clinic Tour

Before Kaitlan Chu started her presentation, we were very fortunate to be given a quick tour of The Cleveland Clinic (ClevelandClinic.org - @ClevelandClinic).

We started the tour by going through the first floor and lobby, and going out to the Entry Plaza Fountain. There we learned about the original hospital building and its history.

Cleveland Clinic Entry Plaza Fountain
Enlarge Entry Plaza Fountain Photo

Next, we went from the top of the clinic with it rooftop view, to the basement with its army of robotic carts distributing hospital supplies through four miles of tunnels.

Cleveland Clinic rooftop view
Enlarge Rooftop Photo

 

Cleveland Clinic robots
Enlarge Robot Photo

 

 

Returning to the main floor, we noticed the many works of art displayed throughout the building. Our guide indicated that all the artwork is “high tech” to reflect that this is a high tech hospital.  

Cleveland Clinic “high tech” artwork
Enlarge High Tech Art Photo

This YouTube playlist of three videos are not from the NEOUPA meeting, but they do give you a flavor of what we saw. Here are the links to the original videos: 

I had never been in the Cleveland Clinic before this month, and leaned that you can take an organized tour of the clinic's artwork.  Here is information on the art tour: 

Thanks to NEOUPA president Cathleen Zapata (@catzap) from Metrics Marketing Group (www.MetricsMarketing.com - @MetricsMrktg) for suggesting and arranging the tour. The tour was a great start – it really set the mood to learn a little of the history and philosophy of the Cleveland Clinic before looking at the hospital's website user experience (UX) plan.

Cleveland Clinic’s First Usability Study

 

Cleveland Clinic’s First Usability Study NEOUPA Meeting with Kaitlan Chu

Cleveland Clinic’s First Usability Study NEOUPA Meeting with Kaitlan Chu
 

Cathleen Zapata (@catzap), in introducing Kaitlan Chu (www.linkedin.com/in/KaitlanChu - @KaitlanChu), let the attendees of the NEOUPA meeting know that this is the same presentation that Kaitlan will be presenting at the Usability Professionals Association International 21st Annual Conference  (@UPA_Intl).  If you want to attend Kaitlan's Large Card Sort Analysis Made Easy talk, select this link for details.

Here are Kaitlan's slides, which can also be found at:  http://www.slideshare.net/kaitlanchu

Challenges Faced & Lessons Learned Conducting Cleveland Clinic's First UX Study: a quantitative open card sort analysis

View more presentations from Kaitlan Chu

Kaitlan Chu started her presentation with the background from which the study was developed. It is very important to understand the organizational culture and history before starting a usability study of this type.

Historically, each department head would establish the structure their individual sections of the website. This did have an advantage in that the doctors who headed these departments knew the needs of their patients and insured that the website addressed those needs. However, this decentralized the management of the website, which led to difficulties for users in navigating between department areas. The website development team needed to develop a quantitative method to document the visitor's (normally patients and their family members) needs and then develop a hospital-wide navigation structure.

From my notes - Planning the study:

  • Quantitative
  • Open card sort
  • Online, remote study
  • Put together a team
  • Budget of $300 
  • Had 200 people in study. 
  • Analysis guidelines
  • Was hard to find guidelines

Since this was the Clinic's first-ever usability study, it was important to use methods that would be accepted by the key decision makers. Thus, the research needed to be based on quantitative studies using authoritative sources to develop a hospital-wide website site map. The Cleveland Clinic website team decided to use an open card sort, since it offered the flexibility they desired to accommodate more study participants' input.

Open card sort vs closed card sort. See: http://en.wikipedia.org/wiki/Card_sorting

  • Open card sort - In an open card sort, participants create their own names for the categories. This helps reveal not only how they mentally classify the cards, but also what terms they use for the categories. Open sorting is generative; it is typically used to discover patterns in how participants classify, which in turn helps generate ideas for organizing information.
    http://en.wikipedia.org/wiki/Card_sorting#Open_card_sorting 

  • Closed card sort - In a closed card sort, participants are provided with a predetermined set of category names. They then assign the index cards to these fixed categories. This helps reveal the degree to which the participants agree on which cards belong under each category. Closed sorting is evaluative; it is typically used to judge whether a given set of category names provides an effective way to organize a given collection of content.
    http://en.wikipedia.org/wiki/Card_sorting#Closed_card_sorting

Kaitlan went on to talk about the challenges of the project and how the team was developed. In addition to budget constraints, the website project team discovered that while the pilot test using internal staff went quickly (approximately 20 minutes), testing with people that better represented visitors to the hospital took much longer (approximately 45 minutes). Kaitlan also reported that there were no guidelines on how to do this type of testing in a setting like the Cleveland Clinic, so she used usability books as a starting point to develop standards appropriate for the Clinic. (I highly recommend meeting with Katilan to learn about this.) 

Book recommended by Kaitlan: Card Sorting by Donna Spencer (@maadonna)

Card Sorting by Donna Spencer

Kaitlan presented the statistical models used in the study. Take a look at her slides for the graphs. Here are my notes from the meeting to which I added links for more information: 

  • Statistical Analysis Methods: 179 completes, 71 cards => 12,710 lines in Microsoft Excel
  • Cluster Analysis 
  • Multidimensional Scaling
  • Each “bubble” represents one card. 
  • Only use this method as a secondary method.
  • First look at 2D then look at 3D chart. 
  • Cell Plot (Jieying Jane Chen) 

The impact of the study included these three success outcomes:

  1. Established a standard site map
  2. Stakeholders see the value
  3. Increase perceived value of Kaitlan's team & UX

At this point in Kaitlan's presentation, I took a look at the Cleveland Clinic (@ClevelandClinic) website, and thus I recommend this to you: my.clevelandclinic.org

In my notes, I listed the following as lessons learned from the project:

  • Tailor to corporate culture
  • Important to have decent participant incentives
  • Conduct pilot test with participants who represent real users
  • Analyze data in as many ways as possible
  • What number of cards is best?  (No real answer, but try to have a number of cards that allows the study to be completed in 20 minutes)

Thank you to Kaitlan Chu for sharing her vast knowledge at this NEOUPA meeting. Be sure to look for her presentation at the Usability Professionals Association International Conference at 11:00 am on Thursday, June 07, 2012.

 

@NEOUPA Meeting at @ClevelandClinic Tweets

Thanks to those who shared during the @NEOUPA meeting. Twitter is a great way to share ideas.