Solving Automated Testing Challenges for Embedded Systems DevSecOps

DevSecOps is driving real improvements in quality, security, and efficiency for modern software development organizations, but domain-specific challenges have hampered application of DevSecOps principles in the intelligent edge and embedded systems spaces. This post explores the importance of automated testing in DevSecOps, the domain-specific problems that affect embedded development, and three necessary parts of a solution enabling edge developers to realize the benefits of DevSecOps.

Automated Testing in DevSecOps 

DevSecOps is a methodology that integrates development, security, and operations to enhance software development practices. Testing plays a central role in ensuring the reliability, security, and quality of software, and DevSecOps encourages automating testing processes throughout the development cycle to improve efficiency and repeatability.

DevSecOps often leverages “pipelines” to automate development workflows such as building the code, testing it, and storing important artifacts from pipeline runs for use in other stages or pipelines. These pipelines are often discussed in groups related to their function: continuous integration (CI) pipelines focus on preparing artifacts that are ready to be deployed, continuous deployment (CD) and continuous delivery pipelines focus on deploying artifacts into a testing or production environment, and while both often contain testing stages there are also continuous testing (CT) pipelines focused exclusively on testing.

While cultural changes make it difficult to establish DevSecOps practices in any organization, traditional IT projects have typically been more successful than edge-focused projects.

Why Embedded Systems Struggle with DevSecOps Automation

One of the biggest challenges for edge-focused projects relates to the availability and connectivity of the embedded systems to be tested. This makes it challenging to do any testing of embedded systems, and the testing that is done tends to be late in the development process where it increases rework and negatively impacts schedule and cost.  It also impedes automation because testing techniques often require manual steps or are performed on isolated systems or in specialized labs.

The challenges that testers of embedded systems need to overcome include:

1. Software development happens earlier or faster than hardware development and manufacturing, often creating a lengthy period where software needs to be tested but hardware has not yet been manufactured.
2. Product hardware, especially when designed for the development team with extra resources like debug headers and ports, is often in short supply and expensive.  To avoid bottlenecks that impede testing – impacting quality, schedule, and cost – organizations need to balance the expense of designing and building such hardware with the schedule and accessibility needs of the development team.
3. Development hardware is often isolated on developer desktops or in a testing lab where it is difficult to access from automation pipelines running in the cloud or on other systems.
4. Developers can build automated tests to run in specialized contexts such as on their desktop or in a test bench, but that increases demand for expensive development hardware and requires tests to be run in specialized contexts that impede programmatic administration and access.
5. Automating tests for programmatic pipelines requires a reliable way to access testing resources remotely, which includes managing an inventory of targets, establishing a connection to a specific type of target to run tests, and preventing others from using that target while tests are running. 

Enabling Automated Testing and DevSecOps with Embedded Systems 

To enable DevSecOps in development processes for intelligent edge systems, teams must solve all these problems. From a practical perspective, there are three key parts of the solution: simulation, remote connectivity, and target management.

Leveraging Simulation for Automated Testing 

Hardware is often unavailable while embedded software is being developed, so embedded projects often delay testing until later in the process. This is better than not testing at all, of course, but it creates a lot of extra work since code is older, harder to remember and modify when failures are identified. It also increases schedule risk because teams have less time to address failures.

Simulation technologies offer a way to test software earlier in the process, even before hardware is available, and enable testing on a far greater scale  with faster iteration and shorter feedback loops than would be possible if relying exclusively on hardware-based testing.  Offloading certain testing to simulated targets also enables teams to use their expensive and scarce hardware only when it is needed  – reducing the risk of damage while improving control and flexibility of hardware costs and schedule.

Ultimately, adoption of simulation technology allows for earlier and more scalable testing while minimizing the risk of development hardware overproduction.

Improving Collaboration with Remote Connectivity to Embedded Hardware 

Embedded testers often rely on development boards attached to their development machine, or they move those machines to specialized testing labs to perform tests. This can be convenient from an individual’s point of view, but the use of isolated systems and labs impedes adoption of standardized testing run from CI, CD, and CT pipelines. It also increases the cost of development hardware because each developer or tester needs hardware of their own.

Remote connectivity to development hardware eliminates the need for that hardware to be physically present on developer desktops, allowing developers and testers to share, access and test targets from anywhere. This improves utilization of expensive development hardware, enhances collaboration and accelerates development cycles.

Enabling Test Automation with Embedded Device Target Management 

Connecting pipelines to testing targets is more than simply having remote connectivity, though. Each target system might have a different means of connection; some may have a network interface while others require a serial port or JTAG, and simulated targets may need to be spun up before they can be used. Pipelines require a way to find an appropriate target upon which tests can be run, to ensure that test runs produce consistent results by preventing interference by other users or test runs, and to gracefully wait or fail when a suitable target is not immediately available.

Effective target management, which provides a robust, uniform interface for accessing all targets enables pipelines to execute the same tests across both simulated and physical targets without forcing developers to build separate test logic for each  . It improves collaboration by sharing test configurations and logic across the entire team while enabling everybody with on-demand access to test resources. This increases flexibility and scalability of testing practices while increasing the frequency and coverage of testing.

Achieving DevSecOps with Embedded Systems

Automated testing plays a vital role in ensuring the reliability, security, and quality of embedded systems in the DevSecOps era. By embracing approaches such as simulation, remote connectivity, and effective target management, organizations can overcome the challenges that commonly interrupt the DevSecOps journey when developing for the intelligent edge. By automating tests and seamlessly incorporating them into CI and CD pipelines, developers gain real-time insights into the quality and security of their code. These insights help identify and mitigate potential issues from the early stages of the development process, reducing development cycles and expediting time to market.

Wind River Studio Developer is a full-featured DevSecOps platform for intelligent edge development, purpose-built to solve the challenges that prevent embedded systems teams from adopting DevSecOps. Contact us today to learn how Wind River Studio Virtual Lab helps users accelerate embedded development by establishing predictable, scalable, collaborative testing throughout the embedded development life cycle.

About the author

Jon Jarboe is a Senior Product Marketing Line Manager at Wind River