Distributing Operators in KOTS

Starting in KOTS 1.13.0, new support for packaging and distributing Kubernetes Operators has been introduced. Operators are a great way to make self-managing applications by automating day-to-day maintenance tasks that run in a “reconciliation loop” alongside the application. Operators alone aren’t great for 3rd-party software delivery because they require manual intervention to deploy or upgrade across different environments, which incurs a customer burden and makes maintenance/support more difficult.

With direct support for operators, KOTS overcomes the above shortcomings by managing operator deployments/upgrades and handling support for diverse environments (even airgapped environments). KOTS ties it all together and takes scalable delivery of 3rd-party applications one step further with it’s new support of Kubernetes Operators.

Why Operators?

Operators are strong at automating “day 2” operations of an application. This is especially true for operations that are technically involved, coupled to the application, and reactive to usage patterns or time. For example:

TLS cert renewal
Garbage collection of intermediate files
Reindexing databases
Persistent Volume maintenance (resizing, reallocating, rebalancing)
Tuning internal application runtime parameters based on usage

These operations help to allow the application to be self-managing, saving the enterprise from manual intervention by codifying day-to-day maintenance tasks into an operator that runs closely alongside the application. These tasks all adhere to the Kubernetes self-correcting reconciliation/control loop pattern, which is a key characteristic of self-maintaining Kubernetes applications.

Operating the Operators: Where KOTS Comes In

With the introduction of operators, KOTS now fully supports all central methods of resource management/delivery in Kubernetes:

Manifests: Deterministic files to directly create resources. Easy to change and understand, but difficult to scale.
Helm Charts: Packages that indirectly cause the deterministic creation and usage of resources. Assuming the chart is portable and can be treated as a component, this helps accelerate development and scale maintenance.
Operators: Code that dynamically (non-deterministically) creates, manages, and uses resources. Much harder to understand and work with than the other methods, but as long as the operator is in the form of a reconciliation loop, this can be an elegant method to create self-maintaining applications.

The problem is that, without KOTS, all the of the above methods are not operationally scalable, especially in the context of 3rd-party application delivery. In-depth technical customization for each and every environment would be required, making it a nightmare for software vendors to support and maintain their application. This is the central problem and solution of KOTS. That is, to operationalize and scale the distribution, management and support of 3rd-party applications on Kubernetes.

With this new version of KOTS, 3rd-party operators can be delivered to enterprise environments (including airgapped networks) in a truly scalable and seamless fashion, without requiring customization or manual intervention. More importantly, this capability enables all three delivery models to be leveraged by the Kubernetes application in harmony. The vendor’s application can benefit from rapid development through charts, elegant self-maintenance through operators, and flexibility/visibility through manifests. The result is a 3rd-party application where installation is seamless, maintenance is easy/automated, and where the level of Kubernetes sophistication required is vastly reduced.

How it works

The challenge of deploying Operators to enterprise and other secure environments is giving the Operator access to the images it needs at runtime in a way that meets the enterprise workflow and/or security requirements. For example, when the Confluent Operator receives a custom resource to deploy a new Kafka instance, it shouldn’t (or can’t) download Kafka container images from DockerHub. Instead, the images need to be made available to the cluster from the enterprise existing internal image registry (i.e. Artifactory or Harbor) where they can be scanned and processed.

Without KOTS, this process is highly involved for initial deployment and for each subsequent update of container images referenced custom resources. Every time this happens, the cluster-operator needs to re-package the image with a new tag on the local registry and update the custom resource to refer to the new local image.
The steps are far too cumbersome to be practical.

With KOTS, the process is simple:

Place all operator CRDs and custom resource manifests within KOTS. Use KOTS template functions for images or pull secrets within resource manifests or environment variables.
Reference the image and version (e.g., ‘elasticsearch:7.6.0’) in the KOTS application. At that point, KOTS does all the work to repackage and re-host the image automatically in a way that custom resources sent to the operator can reference (even in airgapped environments).
For image updates, just change the image reference in the KOTS application (e.g., ‘elasticsearch:7.6.1’) and that’s it!