IoT App Development: How We Built a Facial Recognition-Powered Beer Kiosk

Then you see a kiosk of beer taps where people are quickly moving through line, pouring their own beer. Instead of pulling out their wallets to show their ID and pay, return customers are identified using facial recognition and are automatically charged only for the beer they pour.

It sounds like science fiction, but it’s not. It’s Hop, an application we built for Las Vegas-based Adult Beverage Company (ABC).

Beer to the People!

Hop’s founders had a simple goal: to share their passion for beer by delivering a unique, convenient experience to fellow beer lovers. But the product they envisioned required complex hardware-software interactions, which can be surprisingly difficult to execute.

In fact, the market is currently flooded with so-called Internet of Things applications that are great in theory, but not-so-great in practice. The crux of the problem is that hardware engineers typically aren’t experts at building software – and software engineers aren’t experts at building hardware. As a result, even the most brilliant IoT application ideas can become fragile, flaw-ridden systems.

And that’s exactly the situation Hop’s founders were in when they engaged us to help. While they had a working product that was up and running, it was difficult to use and incredibly buggy. What’s more: critical portions of their software and hardware were purchased from an inflexible third-party vendor, which meant that it couldn’t be customized to meet their needs.

Bringing Web Development Best Practices to IoT

At Very, we’ve developed a unique approach to building connected devices: by applying web tools and technology to the hardware world (through Elixir and Nerves), we can iterate quickly and bring other innovations, like machine learning, to IoT hardware.

Knowing we’d take this approach with Hop, we began the project with a technical audit of their existing hardware and software to understand what was working and what needed more work.

While the system’s hardware was functioning, we identified a range of improvements that would enhance product performance. For example: The previous system’s compute module controlled only two taps. We decided to quadruple the number of taps, building a module that can control up to eight taps.

On the software side, we quickly built and integrated AWS Rekognition’s facial recognition technology, which was completely absent in the original system. While this certainly contributes to the product’s “cool factor,” it actually wasn’t the most complex piece of the process.

Most of our engineering energy was actually directed toward rebuilding the existing software so that it would better serve ABC’s business objectives. The most critical software component in the system – which tracks how much beer is poured – had been purchased from another company. Because ABC didn’t own the technology, they found themselves in an undesirable situation: not only was the success and scalability of their product dependent on a third party, they didn’t even have the power to properly customize the technology to suit their needs.

So our team focused on developing proprietary technology that is now fully owned by the client. Unlike the previous software, the new software can be configured and operated remotely – an incredibly important feature, as the client’s business model includes plans to white label the system and distribute it to restaurants and other customers.

How It Works

Hop asks first-time users to enter basic personal information, scan an image of their legal ID, take a few reference photos for facial recognition, and swipe their credit card. When pouring a beer, a photo is taken at the tap and compared to the user’s reference images, giving a match rating out of 100%.

A bartender onsite reviews the rating and approves or denies service. On future visits, customers will be automatically recognized — which means all they have to do is approach the kiosk, choose a tap and pour their beer.

Hop makes the beer-buying process fast and seamless for thirsty tourists. For vendors, it eliminates waste (as it charges by the ounce poured, not the glass) and allows for easy inventory analysis. The Vegas Strip was a natural location for a pilot due to its open-container policy and the high volume of foot traffic, but Hop has countless potential applications — like event venues, stadiums, and restaurants — which ABC plans to leverage in the near future.

The Outcome

We worked closely with ABC throughout the process, making every hardware and software decision with the client’s short-term and long-term business objectives in mind. As a result, the new Hop system performs exactly the way its founders envisioned, and — when ABC is ready — it has the potential to scale seamlessly. (Which is key, because Hop has many potential applications that extend far beyond the Strip.)

From the start of the engagement, we made it a priority to fix existing mistakes quickly, ensuring that the original system could continue functioning, with significant improvements, while we built and iterated on the new system.

Ultimately, our unique approach of bringing web development best practices to the hardware world allowed us to deliver quality hardware and software as a package — which, we’ve found, is pretty rare.