Delivering high-quality software often requires testing every release for compatibility in varied customer environments. Diverse operating systems, hardware configurations, and software dependencies among customers can lead to unforeseen issues, such as crashes or feature inconsistencies, that might hinder adoption and satisfaction; thorough compatibility testing as part of your continuous integration (CI) processes minimizes these risks, fostering user trust and loyalty while reducing post-release support burdens.
Testing is a key technique in the efficient delivery of high-quality software and tools. From unit testing to end-to-end testing and everything in-between, teams of all sizes rely on different testing techniques to ship often and with high confidence. When building apps that will run in customer-hosted environments, however, testing every software release against every environment it might run in becomes quite difficult to do regularly or in any sort of automated fashion. For this reason, many teams settle for “best effort” testing of their customer hosted software, perhaps testing against only the two most recent editions of Kubernetes, or only regularly testing against their known most-popular K8s distribution (e.g. EKS).
If not tested, customers’ diverse operating systems, hardware configurations, and software dependencies can lead to unforeseen issues, crashes or feature inconsistencies. True compatibility testing for customer-hosted Kubernetes applications means embracing completeness (testing all relevant combinations) and frequency (shifting left and testing as early in the development process as possible). Doing compatibility testing well minimizes the risks in distributing to heterogenous customer environments, fostering user trust and loyalty while reducing post-release support burdens.
Developers often combine continuous integration processes with release testing methodologies like compatibility testing, canary testing, smoke tests, and load handling to ensure the quality and reliability of software releases. In the CI pipeline, code changes are automatically integrated into a shared repository several times a day. As part of this process, automated unit tests and smaller-scale integration tests are executed to catch early-stage bugs and prevent integration issues. Once these initial tests pass, the release testing phase begins.
By incorporating these testing methodologies into the CI process, developers can catch bugs and issues early in the development cycle, leading to more stable and reliable software releases. Automated testing at each stage ensures that code changes are continuously validated, reducing the risk of regressions and providing a solid foundation for more extensive manual testing and user acceptance testing before the final release. We’ve written about this before in the blog on ISV Testing Strategies for Reliability and Compatibility, but now let’s take a closer look at HOW we can help specifically around CI processes.
As we discussed in the blog on Testing Releases in Customer-representative Environments, the issue is not that developers are uninterested in testing, it’s the complexity and effort around identifying customer environments, provisioning exponential combinations, and automating testing that causes people to conduct only limited release testing.
The secret power here is how our new Compatibility Matrix can be… ahem… continuously integrated with continuous integration. The primary tool here will be the replicated CLI, which in this case enables developers to programmatically request customer-representative combinations of Kubernetes and operating environments for testing with every release. It’s worth noting here that we can authoritatively identify the customer-representative combinations from our Instance Insights and telemetry notifications.
So once you’ve identified common combinations and provisioned relevant resources, then it’s a very small leap to run the tests you’ve already configured against every possible combination, and do so on every single commit and release (if you so choose.) Our approach is generally adaptable to a wide range of CI tools such as GitHub Actions, Jenkins, CircleCI, TravisCI, and more.
As noted elsewhere, the Compatibility Matrix enables you to create test clusters (often under 2 minutes) for Amazon EKS, GKE, Red Hat OpenShift, kind, K3s, and more K8s environments coming soon. You can spin up representative combinations programmatically (with consolidated per-minute billing!) and a time-to-live (TTL) to shut down resources after testing to minimize your costs.
Note that we don’t provide all the different types of release tests specific to your product, but we empower you to automate them widely for all your releases.
For more information on the specific steps to take, please visit our documentation About Integrating with CI/CD.
Interested in getting access to the Compatibility Matrix and giving us feedback? Sign up now to learn more and join our waitlist.
Not a Replicated customer, but interested? Schedule a demo with us!