Cross-Device Usability Testing. Any advice?


#1

How would you plan and conduct a usability study for a sequence of related tasks (or subtasks of a single task) across multiple devices?

Example:

DESKTOP
A user, Jim, visits a state government website on his desktop computer in order to download an application form to renew his driver’s license. His plan is to have a friend drive him to the DMV to submit his application.

PHONE
His friend comes and picks him up. While in the car, Jim uses his phone to

  • Return to the website to make sure he has all of the supporting documentation.
  • Verify the hours of that the DMV is open that day.

I would love to hear some lessons learned on running this kind of test. I am really interested in that moment of transition between two interfaces and how a user applies his previous experience to deciphering the UI in a new format.


#2

A couple questions/comments come to mind:

  • Based on what you’re trying to learn from this study, how important is it to take a single person through this entire scenario? Is it important to do the tasks in context (i.e. in the car)?
  • You could have somebody act as “the friend” and actually do the drive to the DMV to simulate the whole activity
  • Question some of your existing assumptions: Why would Jim check the DMV hours once he’s already in the car? Wouldn’t he want to know if they’re open before his friend comes to pick him up? Things like that indicate that you might have more work to do before you’re ready to get the most out of a study like this.

#3

Hey, thanks for your response! I am actually planning to do the cross-device test in a usability lab w/ eye-tracking on both desktop and phone. I should have clarified that. It is going to be a comparative study - so I will need users to try their cross-device task on website A, and then try it again on website B.

For both website A and website B, the user will need to do the same tasks, on the same device/task combo. I am still working out the details of what those tasks will be - but they will be relatively simple. My goal is to evaluate which of the two sites navigation models (A or B) had the best UX from a cross-device perspective for the tasks at hand.

With all of this in mind, do you have any suggestions/best practices for testing tasks cross-device? What I am seeing in the HCI literature re: cross device research, is there seem to be a lot of open-ended field studies. I am not seeing a lot of lab studies. An open-ended field study sounds very interesting - but for this particular project, I am not looking for something that exploratory, my needs for the research are much more focused (and my resources are limited as well).

Any other thoughts?