IoT QA Testing: How to Ship Quality Products

Why QA Matters for Your IoT Products

At the highest level, QA matters for IoT because everyone wants to make sure they’re shipping products that function as intended in all the environments and conditions where they may be used. Developing a strong QA practice (with a QA lead and, ideally, one QA resource for every four developers) helps development teams reach desirable outcomes in several specific ways.

Your QA lead or team can:

1. Help identify gaps during product planning 

During the planning phase of an IoT development project (at Very, we call this the Technical Design Sprint), having a QA resource present can proactively prevent lengthy testing cycles down the line, when budgets and timelines are more likely to be cut short.

2. Help define acceptance criteria in job stories 

Job stories define a given situation, a user’s motivation for acting within that situation, and the expected outcome of the action. Acceptance criteria are the formalized requirements that help your team bring those job stories to life in an application. By involving your QA professional in the job stories process upfront, they can ensure that your acceptance criteria are high-quality and testable.

3. Help catch regressions when integrating functionality with past iterations

In software testing, a regression happens when a change to the system (like an upgrade or patch) causes a feature to stop functioning properly. When regressions are caught, a QA lead can coordinate with engineers to make sure your automated tests will catch the issue later if it surfaces again.

4. Help catch the bugs your automated tests missed

While automated testing can be great for efficiency, relying on automated test suites alone is not sufficient as a QA strategy. Automated suites do not take usability and unexpected human interactions into account. Additionally, seemingly low-risk code changes in one component can break integrations with other components, and automated tests don’t always catch these regressions.

5. Help you catch “real-world” bugs in your hardware

To ensure that your product will work across various platforms, development teams should practice device compatibility testing to catch the “real-world” bugs that can only be replicated under real-world conditions (as opposed to simulated conditions). At Very, we’ve opened a “device lab” managed by our Hardware QA Specialist to enable this kind of physical device compatibility testing.

6. Help product managers focus more of their efforts on strategy and execution

Product managers are responsible for driving an IoT development project forward, so they shouldn’t be focused on excessive testing or putting out fires for issues discovered late in the process. Your QA personnel can take the reins on testing and quality assurance so that product managers don’t miss other crucial aspects of the development process.

7. Increase overall client/business confidence in your product

Whether you’re an in-house team or a third-party firm, maintaining the confidence and trust of your product owners is vital for success. At Very, we find that investing in exploratory testing, well-defined acceptance criteria, and transparency in pass/fail/blocked results of written test cases elevates our clients’ confidence and makes it easier to deliver fully-functioning, innovative products on time and on budget.

How to Build a Strong QA Practice for IoT

Developing a strong IoT QA testing practice involves at least two central components: hiring the right people, and building a process that empowers those people to make the biggest impact. Here are some guidelines for each:

Qualities to Look For in Your QA Lead

The person you choose to spearhead your QA practice should have the following experience and qualities:

• Experienced in manual testing of web and mobile apps
• If your team must frequently juggle multiple IoT development projects, look for someone with agency experience.
• Familiar with the following tools: Xcode, Android Studio, and Charles Proxy or some other debugging proxy tool
• Good exploratory testing skills
  • This is the recommended focus for extreme programming (XP) regarding manual QA. Extensive regression tests do not constitute a lean approach and are not recommended with XP.
• Experience in refining acceptance criteria for job stories and identifying unhappy paths/edge cases for testing
• Experience in writing high-level test cases for “smoke tests” that cover critical paths/user flows in a product. It’s important to find a person who has the ability to see the bigger picture and doesn’t get lost in the details.
  • Background: I’ve often seen experienced QA professionals that don’t have agency experience spend too much time on less important things. They are used to being able to spend all day, every day, focused on just one product. In an agency work environment, this is not sustainable.
• Nice to have:
  • Test automation experience or interest in learning it (using frameworks like Selenium, Appium, Jest, TestCafe…)
  • Performance testing experience (Load testing, Stress testing, Soak/Endurance testing, Spike testing, Scalability testing)
  • Security testing experience
    • Note: this is a very specialized field and it’s usually difficult to find a person who’s good at this as well as the other items above. To do proper security testing, you may need to hire a security testing specialist.

Where Your QA Lead Should Engage in the Development Process

Experienced, driven individuals are often as successful on your team as you allow them to be. Below is an outline of when and how your QA team should be involved during the product development cycle:

Planning Phase

Your QA team should:

• Identify risks in proposed architecture, user flows, and designs.
• Identify opportunities to build certain debugging functions into the test builds that will speed up testing and save time later (e.g. mock responses, easily switch test environments, simulate certain edge cases).
• Help write/refine detailed acceptance criteria for job stories.

Note: QA doesn’t need to spend a lot of time in the planning phase. Attending planning meetings and spending a handful of hours to review documentation and provide feedback should be enough. The important part is to make sure the QA team is included in the process,

Development Process

The QA team will:

• Create high-level written test cases based on acceptance criteria and designs.
• Test stories as they are delivered and update test cases with Pass/Fail/Blocked results.
• Engage in ongoing exploratory testing, refine test cases as needed when new issues are found, and talk to engineers about including them in automated tests.
• Maintain a lean smoke test suite for delivered features.
• Execute smoke tests from previous iterations at the end of each completed iteration to catch any possible regressions.

Supplemental Practices to Deliver High-Quality Products

Going back to the idea that quality assurance is everyone’s responsibility, it’s important to note that QA works best as a companion to mature development practices that your team is already using. A few examples of these that we employ at Very include:

Pair Programming

As the name suggests, pair programming is a development approach where a pair of programmers collaborate on a single workstation at the same time (which can be done in-person or remotely via screen-sharing collaboration tools). Pair programming is known to improve quality control because each developer can challenge the other’s assumptions and fill in potential knowledge gaps.

Test-Driven-Development (TDD)

With a test-driven development approach, you write your software test before you write the actual production code. You develop with specific functionality in mind, as well as how that functionality will be tested so that the testing itself will be a more seamless process when the time comes.

While these practices will likely increase the quality of your software output, they won’t replace a QA team and practice. To deliver the best products for your internal or external clients, begin with the guidelines outlined in this post.

If you’re looking for an IoT development team committed to quality, reach out to the team at Very today.