The next key metric that we’ll dive into in the Instance Insights Series is the rate at which you’re able to successfully complete the installation of an instance, commonly referred to as install success rate. Measuring your install success rate gives you insight into the quality of the packaging of your software and how prepared you and your customers are to begin each install. Most importantly, a strong install success rate can drive a better customer experience and faster trial conversions.
For the purposes of this exploration, we’ll discuss a hypothetical Kubernetes application that end users install in their own environment, either in a shared cluster or in a purpose-built cluster designed to host only that application. We won’t make any assumptions about how the Kubernetes cluster is created and we’ll even include cases where a vendor includes a Kubernetes installer as part of their application. While some of the notes are Kubernetes specific, most of these concepts should apply to any application that’s distributed into customer environments, including as a linux package, standalone binary, OVA, or other package format.
When beginning the process of surveying your install success rate, you can start by thinking about how many attempts it takes before an install is successful. What percentage of installs require more than one attempt? How often are installs successful on the first attempt? This post will explore different ways of defining install success, evaluating and measuring progress, and techniques and technologies to help improve your application’s performance in this area. With any luck, by the end, you’ll be well on your way to understanding how to achieve best-in-class performance.
Based on our (Replicated’s) limited anecdotal experience, best in class vendors have a 90% install success rate, averaging 1.1 installation attempts to get an instance up and running.
When defining this metric, you’ll need to define some instance install statuses and determine what “installation success” means to you.
The example definitions provided below are those we have found to be the most common for describing all possible application states as an installation attempt progresses.
For instances managed by Replicated, these statuses are reported by default for internet-connected (online) instances. If your customer installations don’t have outbound internet access (air gapped), you can still manually record timestamps for these transitions to understand your performance.
While you may define a successful install differently, at Replicated, we define it as an instance that ends up in a ready state and stays there for at least 2 weeks. Defining success in this way ensures that successful installs don’t include instances that become ready but end up in a failed or inactive state soon after. If an instance is deployed but experiences downtime or degradation every 2 or 3 days, this generally signals that additional set-up effort is necessary to stabilize the instance. You should experiment with different values for this readiness threshold and see how it affects the insights your data might surface.
Install success rate is generally represented as a ratio of successful installation to total installation attempts.
Attempts per install is the inverse:
To put some real numbers on this concept, let’s explore an example installation with 3 attempts, 2 failures followed by 1 success.
For this individual customer, the install success rate is 33% or 3 attempts per install.
If the vendor had 9 additional installs that were successful on the first attempt, the total install success rate would be 10 successful installs / 12 total attempts = 83% install success rate, or 1.2 attempts per install.
Knowing, or being able to estimate, your current install success rate is important for being able to set a realistic goal. If you aren’t currently measuring install success rate, but you have a team that assists your customers with installs, they may be able to estimate, on average, how many install attempts it takes before one is successful. While estimation and general experiences aren’t reliable data points, they’re a good place to start if you don’t have data yet.
You can use the information gathered during the evaluation of your current install success rate to establish your goals and target specific areas for improvement.
We recommend setting short and long term goals for install success rate which should be based on current performance. Your ability to identify areas that may be negatively impacting the success of your installs and commitment to addressing and improving these detractors will increase the likelihood of attaining more substantial goals.
Methods of measurement may vary depending on how you define install statuses and success. We’ll be using the definitions from earlier for the examples in this section.
In the example chart below, you’ll see the breakdown of in-progress, failed, and successful install attempts. The install success rate is calculated by identifying the percentage of total install attempts (not including ones still in-progress) that were successful. The in-progress install attempts should be noted, but are not included in the calculation as, at the time of measurement, it is unknown if they will end up as a failure or success.
This example shows a much lower success rate for air gap installs, and this vendor should focus on adjustments to improve the success of those specific install types.
When measuring install success rate over time, you can see the impact that segments with a lower success rate have on your overall performance and how making adjustments to them can help you reach your goals.
The graph below depicts the cumulative flow of customer installation statuses over time. Visualization of the data can help customer support teams conceptualize the progression of installations.
Many teams also choose to measure their overall success rate in weekly, monthly, or quarterly intervals and track how that’s trending over time, especially in relation to their goals:
In-progress installations are generally excluded from the total number of installation attempts. However, you may find it helpful to fit a cumulative distribution function to inspect the time window after which install attempts are unlikely to progress further. That is, plot the percentage of customers that get a successful install as a function of the number of days after a license is issued. License issued is an example—as always, you can choose your own “intent to deploy” event. This technique can help you determine which in-progress installs ought to be counted as failed due to how long they have been open.
Install success rate is impacted by the pre-install steps that come before your customer or user hits enter on the command line. The key to a successful installation is for you and your customers to be prepared for nuances that may occur during the deployment. The adjustments below are focused on being equipped to begin the install and can help you increase chances of success:
Adjustments made to improve your install success rate can also have a significant impact on other metrics, most notably your time to install and trial conversion rate. As you iterate, you’ll want to look at what impact your adjustments had on your performance and take them into account as you go back through the define-evaluate-goal-measure-adjust cycle and continue working towards reaching your goals.
At Replicated, we are making progress on improving metrics and telemetry within our product to help ISVs understand application health and performance across customer instances. In addition to providing information on individual instance install statuses, data is also rolled up into customer and channel overviews.
In addition to measuring success rates, Replicated has resources that can be used to make adjustments to improve your install success rate: