Delivering high-quality software often requires testing every release for compatibility in varied customer environments. Diverse operating systems, hardware configurations, and software dependencies among customers can lead to unforeseen issues, such as crashes or feature inconsistencies, that might hinder adoption and satisfaction; thorough compatibility testing as part of your continuous integration (CI) processes minimizes these risks, fostering user trust and loyalty while reducing post-release support burdens.
Testing is a key technique in the efficient delivery of high-quality software and tools. From unit testing to end-to-end testing and everything in-between, teams of all sizes rely on different testing techniques to ship often and with high confidence. When building apps that will run in customer-hosted environments, however, testing every software release against every environment it might run in becomes quite difficult to do regularly or in any sort of automated fashion. For this reason, many teams settle for “best effort” testing of their customer hosted software, perhaps testing against only the two most recent editions of Kubernetes, or only regularly testing against their known most-popular K8s distribution (e.g. EKS).
If not tested, customers’ diverse operating systems, hardware configurations, and software dependencies can lead to unforeseen issues, crashes or feature inconsistencies. True compatibility testing for customer-hosted Kubernetes applications means embracing completeness (testing all relevant combinations) and frequency (shifting left and testing as early in the development process as possible). Doing compatibility testing well minimizes the risks in distributing to heterogenous customer environments, fostering user trust and loyalty while reducing post-release support burdens.
Common types of release tests and how they fit in CI methodologies
Developers often combine continuous integration processes with release testing methodologies like compatibility testing, canary testing, smoke tests, and load handling to ensure the quality and reliability of software releases. In the CI pipeline, code changes are automatically integrated into a shared repository several times a day. As part of this process, automated unit tests and smaller-scale integration tests are executed to catch early-stage bugs and prevent integration issues. Once these initial tests pass, the release testing phase begins.
- Compatibility Testing: During compatibility testing, software is tested on various operating systems, browsers, and hardware configurations that customers might use. This is usually automated using virtual environments or cloud-based testing services. CI systems can trigger compatibility tests for different environments and report any compatibility-related issues that arise.
- Canary Testing: Canary testing involves releasing new features or updates to a small subset of users before a wider release. This helps detect issues that might not have been caught in earlier testing stages. Developers integrate canary releases into the CI pipeline, where automated deployment to the canary group is triggered after passing certain quality gates, such as passing unit tests and basic integration tests.
- Smoke Tests: Smoke tests are quick and basic tests that check if the software build is stable enough for further testing. In the CI pipeline, smoke tests are often the first step after deployment, ensuring that the build didn't break during integration. If smoke tests fail, the pipeline halts, preventing faulty builds from progressing further.
- Load Handling: Load testing assesses the software's performance under different levels of user load. Developers integrate load testing tools into the CI pipeline to automatically simulate varying levels of user activity on the application. This helps identify bottlenecks, resource constraints, or performance degradation under load, allowing optimizations before the release.
By incorporating these testing methodologies into the CI process, developers can catch bugs and issues early in the development cycle, leading to more stable and reliable software releases. Automated testing at each stage ensures that code changes are continuously validated, reducing the risk of regressions and providing a solid foundation for more extensive manual testing and user acceptance testing before the final release. We’ve written about this before in the blog on ISV Testing Strategies for Reliability and Compatibility, but now let’s take a closer look at HOW we can help specifically around CI processes.
How Replicated can help automate release compatibility testing into your CI processes
As we discussed in the blog on Testing Releases in Customer-representative Environments, the issue is not that developers are uninterested in testing, it’s the complexity and effort around identifying customer environments, provisioning exponential combinations, and automating testing that causes people to conduct only limited release testing.
The secret power here is how our new Compatibility Matrix can be… ahem… continuously integrated with continuous integration. The primary tool here will be the replicated CLI, which in this case enables developers to programmatically request customer-representative combinations of Kubernetes and operating environments for testing with every release. It’s worth noting here that we can authoritatively identify the customer-representative combinations from our Instance Insights and telemetry notifications.
So once you’ve identified common combinations and provisioned relevant resources, then it’s a very small leap to run the tests you’ve already configured against every possible combination, and do so on every single commit and release (if you so choose.) Our approach is generally adaptable to a wide range of CI tools such as GitHub Actions, Jenkins, CircleCI, TravisCI, and more.
As noted elsewhere, the Compatibility Matrix enables you to create test clusters (often under 2 minutes) for Amazon EKS, GKE, Red Hat OpenShift, kind, K3s, and more K8s environments coming soon. You can spin up representative combinations programmatically (with consolidated per-minute billing!) and a time-to-live (TTL) to shut down resources after testing to minimize your costs.
Note that we don’t provide all the different types of release tests specific to your product, but we empower you to automate them widely for all your releases.
For more information on the specific steps to take, please visit our documentation About Integrating with CI/CD.
Learn more:
Interested in getting access to the Compatibility Matrix and giving us feedback? Sign up now to learn more and join our waitlist.
You can also checks out the Compatibility Matrix docs or watch our short Compatibility Matrix intro video from RepliCon Q2.
Not a Replicated customer, but interested? Schedule a demo with us!