For decades, telemetry has been essential for infrastructure and software teams, offering key insights into software performance and usage. DevOps Research and Assessment (DORA) has innovated software measurement in the last five years, introducing "Four Key Metrics" to define what an effective software team looks like. These metrics provide a benchmark for continuous improvement at all organizational levels, from individual contributors to CTOs. However, applying DORA metrics to SaaS applications is more straightforward than in on-prem or customer-hosted applications, where deployment frequency and lead time are harder to measure. In customer-hosted environments additional complexities arise, such as varying deployment schedules and multiple versions of the software in use simultaneously. These complexities necessitate new approaches and perspectives in understanding software delivery in diverse environments.
Instance Insights: Filling the Gaps
Replicated recognizes the challenges faced by teams in effectively monitoring customer-hosted applications due to build complexity, customer concerns, and/or requirements for air gapped environments. To give folks better visibility into the complexities of distributing applications to customer environments, Replicated has developed Instance Insights, designed to provide critical data for optimizing the development, installation, and upgrade lifecycle of customer-hosted applications. Through this family of features, Replicated aims to give vendors the data to make sense of their biggest challenges and drive continuous improvement through quantifiable goals.
Instance insights enables vendors to start to understand some of the key metrics for on-prem software delivery, including Age of Deployed Software and Time to Install. With the ability to fully export data, vendors can pull raw event data to construct views for more complex metrics like Upgrade Success Rate and Mean Time Between Failures.
In-app reporting data is viewable under the Dashboard tab of the Replicated Vendor Portal:
Software Adoption as a Key Performance Indicator
Visualize Version Adoption Over Time
Unlike with SaaS applications, releasing 100 times per week isn’t realistic for customer-hosted applications. Even the highest performing teams like Hashicorp and GitLab only release once every 4 weeks, with many releasing on up to a 12- or 24-week cadence. In a world where the “definition of done” for a feature or fix should probably include “deployed to at least N% of customer instances”, adoption metrics (e.g. adoption rate and median age of deployed software) are the key to understanding the value throughput and other delivery performance metrics.
As a product manager, I want to see which versions of my product are deployed at customers so that I can follow up with customers on really old versions, determine the right release cadence so we don’t release too often with minimal uptake, and figure out why customers are staying on a specific version--maybe there’s a breaking change or something difficult about the upgrade.
-- Nick Walker, Principal Product Manager, Stormforge
The instance adoption graph offers a visual representation of version adoption over time. It provides a snapshot of which versions of your applications are actively running in customer instances during each week within the reporting period. This high-level view assists with identifying adoption trends and outliers, allowing you to make informed decisions about version management.
Key Metrics for Software Adoption Performance
In addition to visualizing version adoption, we’ve found it can be very valuable to surface certain key single-number metrics to help software teams set goals and know how their performance is trending week over week, month over month, and beyond. To that end, the adoption report defines the following key performance metrics.
- Unique versions: The unique version metric informs you of the number of different versions that are active in customer environments. By reducing this number, you can drive efficiency by minimizing the documentation and support required for more versions.
- Instances on the last three versions: The adoption rate is represented by the percentage of active instances on the last three versions of your software. By monitoring and increasing adoption rate, you ensure that customers can access the latest features, benefit from security patches, and reduce the support burden caused by outdated versions.
- Median relative age: The median relative age of deployed software gives you the middle most age of all active instances deployed in the field. By setting goals around median age, you can understand the overall adoption of your software, regardless of your release cadence.
- Upgrades completed: Total upgrades completed is the count of when a customer instance has a single version change. This simple indicator helps you understand whether customers are adopting new releases more or less frequently over time and enables you to identify potential areas for improvement in your upgrade process.
If we have customers who are staying on old versions - we’re likely not building things they care about, or we need to work on our ideal customer profile.
-- Lance Larsen, Head of Solutions, Opal.dev
Replicated’s adoption performance measures help ISVs make and evaluate data-driven decisions about product roadmaps and resource allocation. For a decade DORA metrics have been a powerful way to analyze the success of SaaS products, and we’re excited that, through our Instance Insights, ISVs now have the ability to gain similar visibility into their customer-managed products. These metrics help provide clarity to everything from feature adoption to upgrade completion, and we believe will become an essential element to any software distribution team.
To see Instance Insights in action and better understand how they can help you and your organization Sign up for a free trial or request a demo anytime.