Scalable IoT Infrastructure: Designing for Modern Software Development

Executive Summary

Internet of Things (IoT) devices are only as good as their infrastructure.

SurePoint Ag Systems struggled to grow one of its most important product lines due to back end integration issues. Very worked with the SurePoint team to redesign their development pipeline, validate code quality, and deploy a scalable production IoT infrastructure that’s optimized for the cloud. And we delivered it under the proposed budget.

SurePoint Ag Systems: An Agricultural Innovator

Since 2007, SurePoint has designed and built state-of-the-art control and application systems for liquid and dry fertilizer and chemical delivery systems for agricultural applications across the US and Canada. Because of the strong focus on customer service and on being experts in liquid application, they’ve built up a loyal customer base by solving problems based on real-world needs. They use IoT devices to remotely control, manage, and monitor their equipment, providing extra value to their clients.

SurePoint first launched its 100% SurePoint-designed product called Quickdraw in 2014. It’s a fully automated IoT spray tender system that automatically calculates product amounts of fertilizers and/or herbicides at the correct ratios depending on spray rate, gallons, and acreage. And, because it’s remotely controlled, Quickdraw also eliminates all operator exposure to chemicals, protecting the health and safety of workers.

SurePoint continues to design and build innovations like this with an in-house engineering team based in Atwood, KS.

An Emergent Challenge: Restructuring Back End IoT Infrastructure

After releasing their third iteration of the Quickdraw product – Quickdraw 3000 – in 2016, SurePoint had been experiencing delays. While trying to improve the functionality of the QuickDraw Spray system, they encountered back end integration issues between the client-facing web application and the controller on the sprayer. These issues caused downtimes that required code rollbacks.

This back end IoT infrastructure, hosted on DigitalOcean, prevented their front end team from developing new features in a sustainable and efficient way. To avoid disrupting live customer operations, SurePoint needed to solve this problem without redesigning the communication between their IoT devices and the cloud-based controller. The team faced two important constraints:

• The controller had already been manufactured at scale.
• Many customers currently used the product in the market.

SurePoint had a primary software developer but was missing the systems and infrastructure to develop software efficiently. Their team was performing all code updates and rollbacks manually. This time-consuming process led to several downtime issues when bugs were detected and the software needed to be rolled back.

They needed a solution for their developers to deliver code faster, have less downtime, and avoid the undo/redo cycle when issues were found in their deployed code. In short, they needed a partner to guide them through the development of a scalable IoT infrastructure.

Building a Scalable IoT Infrastructure for SurePoint Ag Systems: How We Got There

SurePoint asked Very for recommendations on maturing to a DevOps approach with a continuous integration / continuous delivery (CI/CD) pipeline that would improve how they ship and manage software.

We started, as we always do, with a technical design sprint to define the problem and the current state of the work. Two of Very’s Senior Software Engineers, Jacob Arellano and Mitchel Haan, also walked the SurePoint development team through the IoT maturity model. SurePoint began to see how a few key changes would help them establish a modern, efficient development process.

To make sure the project was a success, we conducted a deep discovery exercise for their current practices. We identified the different user personas to consider and came to a clear, detailed understanding of the needs of the development team. We then documented all of these processes so that SurePoint could maintain continuity within their teams.

After gaining a solid understanding of the users and their needs, we moved to the next step in planning – collaborating with the SurePoint team to create a product roadmap for the necessary software engineering. The first milestone on the roadmap was to increase connectivity between the web application and the IoT controller. We developed a plan to migrate data and de-risk potential connectivity issues while scaling with the current controller infrastructure.

Move Code From Development to Production More Easily

When SurePoint first engaged Very, they knew they needed help, but didn’t know exactly what that should look like. They asked about advanced orchestration tools like Kubernetes, but, after some consideration, the team at Very concluded that those tools would be unnecessarily complex at this stage. When solving problems, Very recommends only what is best for the client – never anything that they don’t need.

We needed to solidify the communication between the controller and back end infrastructure by enabling the infrastructure to handle increased throughput. To accomplish this and streamline code movement, we had to redesign the code path between development and production.

• We worked with SurePoint to containerize the QuickDraw backend application programming interface (API) to create a consistent experience and environment between development and production. This consistency made it easier to spot potential bugs before they pushed to production, and easier to roll back to previous code versions if changes caused problems.
• We established a continuous integration (CI) pipeline for building production containers in Docker and running subsequent tests. By automating this process, we reduced the chance for human error and increased the consistency and quality of new containers. We chose not to automate the deployment at this time. Instead, the team preferred to follow Very’s philosophy to start simply and build complexity as the organization gains maturity.
• Another issue to address was the monolithic nature of SurePoint’s architecture. Because monolithic stacks are more at risk for downtime and data loss if part of the system fails, we restructured a significant portion of the SurePoint stack. We also extracted some key services to separate managed instances that would be more resilient in case of issues.

Clear Documentation Brings Confidence for SurePoint’s Development Team

Throughout the engagement, Jacob and Mitchel trained the SurePoint development team in standardized delivery and implementation practices using these new processes and tools. They carefully documented each process for the SurePoint team so that the steps could be understood and replicated during future deployments. It’s hard to overstate the impact this level of clarity had in giving SurePoint the confidence they needed to successfully manage future code deployments on their own.

The CI system that Very designed is very slick and takes significant time out of development and keeps us from wrangling code and servers as often as before.”

Albert Popp
Chief Engineer

With consistency established, it was time to address code quality issues.

Validate Code Quality

As with any production business, in farming, downtime costs customers money. SurePoint had experienced several downtime events that required code rollback, so we took steps to minimize downtime and post-deployment headaches.

1. We created a staging environment. This staging environment – which sits in between development and production – acts as a replica of the production environment to allow for critical testing before the code moves into customer-facing production.
   
2. We extracted critical services and moved them to an isolated environment. SurePoint was running all of their QuickDraw software – the website, database, and cache – on a single droplet in DigitalOcean. We worked with the team to separate this monolithic architecture into component pieces that could be hosted more robustly. We moved the database and cache into managed instances and placed the website into a droplet by itself. This separation allowed SurePoint to control each function independently and limited the chances that a failure with one of these critical services (redis, postgres, external apis, etc.) would cascade to other critical systems.
   
3. We implemented more modern and robust testing practices. Using GitHub actions, we set up automated tests of new code commits to check for common errors.

We have been significantly more efficient with our resources since working with Very because we do not need to spend as much time focusing on the system admin and software dependencies portion of the project and are now able to move forward with the front end design and controller integration portion of the project.“

Albert Popp
Chief Engineer

Prepare for Future Scale

At Very, we never recommend tools or processes until our clients need them to solve a problem. While SurePoint didn’t yet need advanced orchestration tools, they will need them in the future as they grow. To prepare for this growth – and because of the client’s curiosity – we worked with SurePoint to share knowledge with their development team, and to establish clear next steps for scaling their IoT infrastructure.

1. We implemented IoT infrastructure as code (IaC) using Terraform. At scale, managing all of the different pieces of code can quickly become overwhelming. Terraform allows teams to create reproducible infrastructure by defining the infrastructure in advance with declarative config files. We worked with SurePoint to create multiple orchestrations that could be launched from inside of GitHub to test and deploy new code into production using a blue-green deployment model.
2. We analyzed Kubernetes to containerize all infrastructure parts and manage clusters. Containers can make it easy to deploy applications, but managing all of the currently-running containers can quickly become overwhelming in mid-large scale environments. Kubernetes orchestrates the creation and destruction of containers based on a variety of pre-defined factors to make it easier to practically manage production deployments.After analyzing SurePoint’s medium-term requirements, we helped their development team understand why implementing Kubernetes at this time would be a net resource drain on their lean DevOps team. When you’re working on a limited budget (and let’s be honest, who isn’t?), sometimes the best change to make is no change at all. Instead, we carefully documented the orchestration steps to make them easy to manage and execute consistently.
3. We analyzed cloud providers with more options. SurePoint’s current infrastructure provider, DigitalOcean, has served them well so far. But, as they grow, SurePoint will run into some inherent challenges with DigitalOcean based on its more limited scope. An opinionated cloud service provider, like AWS, will enable them to scale their operations to support expansive growth. Our senior engineers walked their team through the available options and provided guidance so they’ll recognize when the time is right to move.

Working with Very provided the needed industry expertise to help us make sure that we are on a path that will not cause us bottlenecks and delays in the future. “

Albert Popp
Chief Engineer

Modern Deployment With a Future-Proof Education

Very worked with SurePoint Ag Systems to create a modern, efficient development process that could scale sustainably with future growth. We worked efficiently and creatively with the SurePoint team to deliver the entire project under the originally provisioned budget. They started the engagement with an idea of what they wanted to accomplish, but without some of the critical domain-specific expertise that they needed to make it happen. We listened to their concerns, analyzed their needs, and worked together to deliver an IoT infrastructure that SurePoint can brag about.

Today, SurePoint uses a mature DevOps IoT infrastructure to support one of their high-growth products, and they have the tools and understanding to confidently develop future IoT solutions in a more effective way.