User Testing for IoT: Challenges and Lessons

Working on IoT projects can present a number of issues that are quite different from traditional software development. This article will discuss some of the challenges and lessons for user testing IoT devices, using a recent Very case study for guidance.

Case Study: Self-Serve Beer through Facial Recognition 

Here at Very, we helped one of our clients build the world’s first self-serve beer technology using artificial intelligence and facial recognition. The client, Las Vegas-based Adult Beverage Co., has launched a series of kiosks called Hop across the Vegas Strip. Before using the Hop kiosks, users need to enter some personal information, scan their legal ID, and provide a few reference images of themselves for facial recognition.

When users first show up at a Hop kiosk, the machine takes their photo and compares it with the provided images, issuing a confidence rating out of 100 percent. The request is then approved or denied by an on-site bartender. Approved users can be automatically recognized during future visits, making the ordering process much shorter.

To learn more about the cutting-edge technology underlying the project, read our full case study.

Problem: Pouring Beer Correctly

One of the issues that we encountered during user testing for the Hop project was how to communicate the right way to use the machine. When people first used the device’s interface, they often held their glass incorrectly under the tap. As any good bartender would tell you, failing to tilt your glass at the right angle produces a lot of foam, which can leave customers disappointed.

Despite an on-screen animation showing people how to tilt the glass and turn on the tap, many customers were still producing too much foam. User testing revealed that there were a number of underlying causes-in particular, a cognitive load that was too high.

Many customers were talking with their friends or the bartender while using the machine, which caused them to pay less attention to the instructions. Because self-serve beer kiosks are a complete novelty, customers were operating on autopilot. They assumed that the system worked like a soda fountain, where they could simply place the glass under the tap and push a button.

In order to make the process more clear, we simplified the machine’s user interface and added a better animation indicating the correct way to pour, which did have a noticeable improvement. Still, as long as customers are given the independence to pour their own drinks, there’s no perfect fix to this issue.

Problem: Remote Kiosk Management

Another challenge for the Hop project was how to best manage the kiosks. We initially began with a cloud-based administration panel that could oversee each kiosk remotely. However, we soon realized that it was difficult to manage physical devices using an interface with a high level of abstraction. For example, if you see that “tap 4” is having problems, how do you know which tap that corresponds to?

Using a diagram was one potential option that we considered. However, it’s difficult to code and design and would need to change as the kiosk’s physical layout also changes.

In the end, we offered both a remote administration panel in the cloud, but also added individual tap controls for each kiosk. Managers can log into the device and make changes while they’re standing in front of the machine. Despite all the hype around the cloud, sometimes you’re not able to make everything an abstraction.

Why Testing is a Challenge for IoT Applications

Real-World User Testing

At the beginning of an IoT project, testing is difficult, because replicating the hardware is impossible until you build everything out. There’s very little way for designers and developers to see how people use both the hardware and software in a 3D environment.

Still, the knowledge and insights that you gain from real-world user testing is invaluable. You learn a lot about how people’s brains work by seeing them physically interact with a device.

The best way that we’ve found to get around this issue is to move as fast as you can in the early stages of the project. Start by creating a small working hardware prototype with minimally functional software that you can use for UX testing, and then progress from there as you develop more features.

Multidisciplinary Teams

Another challenge during the Hop project—and IoT projects in general—is working on a multidisciplinary team of both hardware and software experts. Designers and software developers need to collaborate closely with engineers throughout the entire process.

Even though it’s not their chosen area of expertise, designers and developers need to have a solid understanding of the system’s different hardware components, as well as the capabilities of each component. This will help prevent conflicts when designers have an idea that isn’t physically feasible due to hardware limitations.

Final Thoughts

Working on the Hop kiosks taught us some valuable takeaways about IoT projects. First, everyone on the project should dive into the hardware as soon as possible. Gain a full understanding of the hardware’s abilities and limitations so that you don’t have unrealistic ideas about what it can do.

Second, at the outset you should put focus into building a bare-bones proof of concept or minimum viable product (MVP). Work on establishing a simple user interface with the most basic features enabled, and see how users respond to it during testing. By including user testing throughout the development process, you’ll be much more likely to build a product that satisfies the client and that people actually want to interact with.

Are you thinking about starting an IoT project? Reach out and tell us about your project here; we’d love to help.