In this video, we're gonna talk about running studies in-person. And to give you

a preview, in the next video, we're going to talk about running studies online. The

web has made it much easier to run controlled experiments and I think that

there's a lot of people who run studies that they wouldn't have been able to

previously do. But there's also a lot that you can get from watching people. Don't

miss out on that. In particular, at watching people live, you can talk to them

about what their doing, get their feedback, see points of confusion, and in

general, have a much higher bandwidth engagement with the participants in your

studies, then you would get just out of a click stream online. So, today we're going

to talk about planning, executing, and analyzing in-person studies. The first

step is to make clear goals, for example, say, you were building a room reservation

system for a Computer Science department. One strategy would be to put all the

information on one page. Another strategy would put different pieces on different

pages. And you might be particularly interested in if we split things up. Does

that change whether people will book a right size room or not? So, before running

your study, lay out its scope. You don't need to cover an entire system, especially

if you're working on a huge piece of software. It's okay to have a narrow

scope. And that scope will be guided by the purpose of your study, what you hope

to learn. I encourage you to come up with a hypothesis ahead of time and figure out

a way to know whether your hypothesis is true or not. Set up your study in way so

that you can see whether or not people's behavior supported your hypothesis. Book

an appropriate place for your study. If the interface is something that will be

used in a quiet room, book a quiet room. If, on the other hand, it's gonna be used

at a train station, test it at a train station. Figure out who you're gonna

recruit and how, and how many people. And come up with scenarios that you give

people when they come in. You want to m ake sure that these scenarios are

realistic and ideally something that these participants can be motivated to care

about. At least put themselves in the shoes of the actual user of the system.

So, here we have some questions, like will people book the right sized room and for

the right length of time. And this implies to measures. The size of room that they

booked, how long it was booked for. You can also get measures like task completion

time, how long it took and especially if you have realistic users, you can see how

this task interweaves with other things they do. So, if people are actually

booking a room for an actual meeting that they need to have, you might see whether

people makes notes to themselves to also order food or whether they need to

coordinate with people over e-mail first to, for example, get an approximate

attendance count. And if you think things like screen size might affect how users

behave, have a screen that's appropriate for what they'll have so. The simplest

thing to do obviously is to have a stock setup for everybody that is pretty normal.

One danger that you can see is that developers often have fancier computers

than normal users. So, don't test people on the latest greatest fanciest thing if

what people are going to be looking at your site with is a 5-year-old laptop. If

you can, try to get at least two people to be present for the study. That way you can

have one person who is the facilitator and another one who can take notes. It's

important to have concrete tasks among other things that lets you compare the

performance between people. And I encourage you to write them down as

opposed to speaking them so that everybody gets the same experience, making it even

easier to compare. Here's an example of a creativity task asking people to draw

creatures from a novel planet. And you can see that the instructions are pretty

clear. And you can see that the instructions are pretty specific, even

though the output of the task is really open ended. And here's an example of the

meeting room t ags. You can see how it gives some concrete details and context.

I've been amazed and heartened by how seriously most of the participants in our

studies take what they do. And the flip side of that is that sometimes people have

left in tears when they've been unable to perform well. In studies, people can feel

like they're being put under a microscope. And you have a responsibility to alleviate

that stress. For starters, make consent voluntary and make sure that consent is

informed. Avoid pressuring people to participate and let them know that they

can stop at anytime. And maybe the most important rule of all for minimizing

stress and increasing the quality of the interaction that you have with

participants is remind people multiple times that you're testing the site or the

user interface, the system, not them and that's an important frame change because

when you're testing the system, if the user is unable to complete the task, the

fault is the system's fault. If participants believe that they are the

ones being tested, then for example, if I am being unable to perform a task that

reflects poorly on me and my abilities and I may be stressed out. If, on the other

hand, I, as a participant believe that I am helping the designer test the system,

then when I encounter a roadblock, it's not a poor reflection of my abilities.

It's I've done something good. I've found a bad part of the user interface that I

can then help the designer make better. There's several important considerations

in a study like this. For example, what's going to be your order of tasks? One good

approach to this is to start out with a really simple task to get people warmed

up. And then you can move to more complex tasks as you go through the study. Other

times you may shuffle the order so there's no effect of one of the tasks on another

one. How much training, if any, are you going to provide? It depends on how the

real system's going to be used. If you have a ticket machine, not so much. If you

have a surgical system, maybe a little bit more. What are you gonna do if somebody

doesn't finish? Probably the answer shouldn't be lock them in the room until

they do. For a lot of tasks, I recommend deciding in advance what a reasonable

upper band for time is. And you can let participants know that. Another strategy

is that if somebody hits a small stumbling block, you may elect to help them along so

that they can get to the rest of the user interface so that you're not purely

stalled out on one point. And finally, pilot your study. A pilot participant is

somebody that runs through your studies before the actual participants. And the

main goal of a pilot participant is to work out the kinks in the study design.

Also for a usability study, pilots can find really obvious useability bugs that

you wouldn't want to burn real participants on. In fact, I recommend

doing two pilots. One with a colleague who may know the interface well just as a way

of getting all of your materials ready. Figuring out what the tasks are, all that

kind of stuff. Then, do another pilot with one real user and that will help you see a

bunch of these kinks that you didn't see ahead of time. I promise you that if you

run a pilot, you'll discover things that otherwise would have screwed up the study.

And how are you gonna capture the results? There's a number of ways of getting

feedback from people. At the very least I recommend having a paper notebook or a

laptop or other note taking device handy so that you can note down critical

incidents, either aha moments that you have about how to improve the design,

things that work really well, stories that you wanna share with other stakeholders or

problems that you'll need to bring back to the design team to try and figure out how

to fix. You might record video and a great reason to record video is it will let you

grab those moments of either success or real challenge and let you share those

with other people. Depending on what you're doing, screen recording can serve

much the same purpose. It's going to depend on whether you want to gather the

user's facial expression or the crisp contents of the user interface. Are you

going to interrupt participants? If you have a question to ask them about why they

are doing what they are doing, if you want to talk to them about some feature of the

user interface or if you'd like to help them get past a stumbling block so that

they can get onto the next thing. There's no universal answer to this question. It

really depends on what you're hoping to get out of the study. And finally, are you

going to ask users to think aloud while they go through the study? The think aloud

method can be a really powerful way of getting feedback about a user interface.

And here's what I mean by think aloud. Say, for example, you are studying how

somebody changes the staples in a stapler. If I were a participant in that study

[noise] and I were using the think-aloud method, I might start out by saying, well,

I'm changing the staples in the stapler because it, it seems to be out. It's not,

it's not working anymore. So, and see I haven't used this stapler before. I'm

going to, oh, there we go. Okay, so I grabbed the top. I thought that might work

and that opened it up. And I have my staples here. And I'm gonna put those

right in there. And make sure this part goes forward. Oops, that didn't go. There

we go. And then try and close it without getting a staple out. There we go. That's

think aloud. And it helps you get a window into how people are thinking about the

task they're doing. It's not something that most of us are used to doing. And so

you'll need to prompt people at the beginning of the study to think as they're

going. And then you'll need to offer prompts during the study as well. Prodding

people to remember to say what they're thinking or what they're trying to do or

any questions that arrives. Or even what it is they're reading on the screen. So,

remember to prompt people to keep talking. And decide ahead of time what things you

will help on and what things you won't. If you can, write that down. Consistency is h

elpful. Is thinking aloud always going to give you the right answers? No. Among

other things, if you gotta talk while you're doing the task, that may change how

you do the task. In some cases, it may mean that you're more effective because by

trying to bubble up what it is that's the issue, that may help you get through it.

In another case, it may make you less effective because all of a sudden you have

to do two separate things, work the interface, and speak up simultaneously.

And at the very least it's almost certainly going to slow you down. And you

shouldn't take what people say as a veridical representation of what they're

thinking. We can almost always get a rationale or reason for something even if

it isn't at all the actual reason. In studies like this, it's usually not

because people are being malicious or trying to hide anything. It's usually just

because, it's usually just because they're trying to help you out and generate an

answer. Oftentimes those answers are useful for your design process. But they

may or may not be what's actually going on in your doodle. When people arrive, the

first thing that you want to do is greet them. So, the facilitator is going to

welcome the participant, explain what the setup is, explain the overall narrative of

the scenarios that they'll do. And get them setup and trained, if necessary, with

the interface. During the studies, you wanna collect both processed data and

bottom-line data. The processed data tends to be more qualitative and tends to have

real insights to it. The bottom-line data tends to be more of the numbers that the,

you will then use for more quantitative analysis. Bottom-line numbers are great

for things like task completion time. Did my interface speed people up or increase

completion rate? But, don't combine it with think aloud. The reason, think aloud

slows you down and it slows different people down, different and so, the

variance in task completion time, introduced by think aloud means that it

just doesn't make that much sense anymore. So, if yo u care both about the numbers

and about the think aloud data, you may wanna run those on separate studies. At

the end of the study, debrief the participant, bot, so they learn what your

goals were and they can find out more information about what you're trying to

do. And also, so that you can learn more holistically from tham after the

completion of everything, what it is that they're thinking. And now that you're

through this study, in the next video we'll move on to analyzing the data.

In-Person Experiments

Design Principles: an Introduction

与讲师见面