| Guidelines in implementation |
What is "Think-Aloud Protocol "?
"Protocol analysis or, more literally, the think-aloud protocol approach,
aims to elicit the inner thoughts or cognitive processes that illuminates
what’s going on in the person’s head during the performance
of a task, for example, painting or solving a problem. The point is to
undertake interviewing as close to the action as possible. While someone
engages in an activity, the interviewer probes to get the person talk
about what the person is thinking as he or she does the task.
The basic strategy of think-aloud interviewing involves getting people who are doing something to verbalize their thoughts and feelings as they do whatever they are doing." (Patton, 2002)
According to Ericsson & Simon (1993), three assumptions are made while adopting the methodology of protocol analysis.
i) Only the output of the cognitive processes are available
to the consciousness and are verbalizable,
ii) Performance on any given task can be conceptualized as a set of sequential
processes with intermediate products that are available to working memory.
The intermediate products are conceptualized as being directly associated
with observable indicators, in this case, verbal protocols.
iii) The total time to generate a response is equal to the sum of the
individual cognitive processes.
Hence, concurrent verbal reports do not change cognitive
processes although concurrent verbalization does increase the total time
to perform tasks.
Advantages
and Limitations
Advantages
"Think-Aloud" protocol has an advantage over simple observation as evaluator may gain valuable insights into what the participant is thinking on the spot.
There would be less discrepancy in the verbal response of the participant and what he or she actually thinks, as the participant does not need to recall from long-term memory events that have taken place earlier.
Limitations
Verbal protocol methods including "Think-Aloud Protocol" are designed to tap into certain types of thinking but not all.
Pure "Think-Aloud Protocol" may not help evaluator gather sufficient information to diagnose a problem without the use of probing.
"Think-Aloud" may modify the way participants perform their task as participants may feel uneasy hearing their own voices throughout the whole process.
In view of the limitations, "Think-Aloud Protocol"
has evolved over the years and probing is now commonly used to gather
more information form participants although probing may influence the
reliability of the verbal protocol. Ericsson and Simon (1984) recommend
that additional information should be collected in the form of retrospective
reports after the task to avoid any interruptions of task flow. "Think-Aloud
Protocol" is often used with other methodologies to gather more in
depth response from participants.
There are specific guidelines to follow for the evaluator to be successful.
1) First, a "Think-Aloud Protocol" can only be useful if you begin by determining a purpose. Specifies the task (this can be more than one) the participant need to accomplish during the session. Make sure participant understand what is to be done before proceeding.
2) Make it clear to the participant that it is not the participant but the learning system, which is being evaluated.
3) Ask the participant to "think aloud" while attempting the task so that you can understand what he or she is thinking about. Often it is useful to give an example of what you mean by this.
4) Then proceed with the task.
5) While the task is being attempted, it is important
to let the participant talk and to listen very attentively to what is
being said.
Intervene only in cases of extreme duress (e.g. if the participant is
completely stuck and has given up) or if you need to remind the participant
to think aloud.
6) After the participant has accomplished the specified task (or has given up), you should take a few moments to ask the participant to summarize his or her difficulties with the task and to give you any additional comments.
Note:
i) It is useful to audiotape or videotape the interview to aid analysis. By recording the interview, you have the opportunity to listen to the interview again so that you do not rely solely on notes or memory.
ii) The above guidelines should be modified according
to the context in which "Think-Aloud Protocol" is used.
Since "Think-Aloud Interviewing" is commonly used in evaluating usability of interactive learning systems or even websites, we will try something similar in class.
The purpose of this class exercise is to get us be acquainted with “Think-Aloud Interviewing” as a way to understand learning process, mental models and aspects of system's usability. Due to time constraint, we will evaluate just one or two aspects of the usability of a website. In order to do this, you will need to work in pairs. One will assume the role of the evaluator and the other will take the role of a participant in each pair (Don't worry: we will switch role if time permits). The URL of the website will be given in class. Prepare to have some fun while learning!
(Here's the PPT slides used in class.)
References
Ericsson, K. A., & Herbert A. S. (1984). Protocol Analysis: Verbal
Reports as Data. Massachusetts: MIT press.
Ericsson, K. A., & Herbert A. S. (1993). Protocol Analysis (Revised
Edition). Overview of Methodology of Protocol Analysis. Massachusetts:
MIT press.
Patton, M. Q. (2002). Qualitative Research & Evaluation Methods (3rd
ed.), Variety in Quality Inquiry (pp.385). California: Sage Publications.
Reeves, T. C., & Hedberg, J. G.(2003). Interactive Learning Systems Evaluation. Chapter 7: Formative Evaluation (pp.139-171). New Jersey: Educational Technology Publications.
Resources
I have found the following resources to be helpful and have used them
to create this web-based resource.
Papers
Probing and its Effects on the Validity and Reliability of Verbal Reports
Abolrous S.
Available
http://www.abolrous.com/sally/projects/probing_paper.htm
Verbal Reports Are Data! A Theoretical Approach to Cognitive Interviews.
Conrad. F., Blair. J. & Tracy. E.
Available http://www.fcsm.gov/99papers/conrad1.pdf
Expert Performance and Deliberate Practice
An updated excerpt from Ericsson (2000)
Ericsson, K. A.
http://www.psy.fsu.edu/faculty/ericsson/ericsson.exp.perf.html
The Trouble With Think Alouds: Generating Data Using Concurrent Verbal
Protocols
Jennifer L. B.
Available http://www.slis.ualberta.ca/cais2000/branch.htm
Guidelines and templates for conducting interviews
Usability Techniques
Methods for Successful "Thinking Out Loud" Procedure
Ramey. J.
Available http://www.stcsig.org/usability/topics/articles/tt-think_outloud_proc.html
Beta Testing and Report
White. J.
Available http://edweb.sdsu.edu/courses/edtec541/assignments/assign_beta.htm#example
Related Articles
These are articles which provide examples where "Think-Aloud protocol"
is used or articles with suggested alternatives which are used to overcome
the limitations of using "Think-Aloud protocol" for evaluation.
Constructive interaction and Collaborative work: Introducing a method
for testing collaborative system
Kahler. H.
Available: http://www.tu-chemnitz.de/informatik/Medieninformatik/mitarbeiter/download/kahler_constructive_interaction.pdf
Understanding Common Interviewing Pitfalls: An Evaluation of Legal Interviewing
Skills
Lynch. K.
Available: http://www.tlc.murdoch.edu.au/archive/cutsd99/finalpdfreports/Ch10Lynch.pdf
Usability Assessment Report: Remote Sensing Using Satellites
Hughes. M.
Available: http://it.coe.uga.edu/~treeves/RSUSeval/UTreport.html