For decades, telemetry has been essential for infrastructure and software teams, offering key insights into software performance and usage. DevOps Research and Assessment (DORA) has innovated software measurement in the last five years, introducing "Four Key Metrics" to define what an effective software team looks like. These metrics provide a benchmark for continuous improvement at all organizational levels, from individual contributors to CTOs. However, applying DORA metrics to SaaS applications is more straightforward than in on-prem or customer-hosted applications, where deployment frequency and lead time are harder to measure. In customer-hosted environments additional complexities arise, such as varying deployment schedules and multiple versions of the software in use simultaneously. These complexities necessitate new approaches and perspectives in understanding software delivery in diverse environments.
Replicated recognizes the challenges faced by teams in effectively monitoring customer-hosted applications due to build complexity, customer concerns, and/or requirements for air gapped environments. To give folks better visibility into the complexities of distributing applications to customer environments, Replicated has developed Instance Insights, designed to provide critical data for optimizing the development, installation, and upgrade lifecycle of customer-hosted applications. Through this family of features, Replicated aims to give vendors the data to make sense of their biggest challenges and drive continuous improvement through quantifiable goals.
Instance insights enables vendors to start to understand some of the key metrics for on-prem software delivery, including Age of Deployed Software and Time to Install. With the ability to fully export data, vendors can pull raw event data to construct views for more complex metrics like Upgrade Success Rate and Mean Time Between Failures.
In-app reporting data is viewable under the Dashboard tab of the Replicated Vendor Portal:
Unlike with SaaS applications, releasing 100 times per week isn’t realistic for customer-hosted applications. Even the highest performing teams like Hashicorp and GitLab only release once every 4 weeks, with many releasing on up to a 12- or 24-week cadence. In a world where the “definition of done” for a feature or fix should probably include “deployed to at least N% of customer instances”, adoption metrics (e.g. adoption rate and median age of deployed software) are the key to understanding the value throughput and other delivery performance metrics.
As a product manager, I want to see which versions of my product are deployed at customers so that I can follow up with customers on really old versions, determine the right release cadence so we don’t release too often with minimal uptake, and figure out why customers are staying on a specific version--maybe there’s a breaking change or something difficult about the upgrade.
-- Nick Walker, Principal Product Manager, Stormforge
The instance adoption graph offers a visual representation of version adoption over time. It provides a snapshot of which versions of your applications are actively running in customer instances during each week within the reporting period. This high-level view assists with identifying adoption trends and outliers, allowing you to make informed decisions about version management.
In addition to visualizing version adoption, we’ve found it can be very valuable to surface certain key single-number metrics to help software teams set goals and know how their performance is trending week over week, month over month, and beyond. To that end, the adoption report defines the following key performance metrics.
If we have customers who are staying on old versions - we’re likely not building things they care about, or we need to work on our ideal customer profile.
-- Lance Larsen, Head of Solutions, Opal.dev
Replicated’s adoption performance measures help ISVs make and evaluate data-driven decisions about product roadmaps and resource allocation. For a decade DORA metrics have been a powerful way to analyze the success of SaaS products, and we’re excited that, through our Instance Insights, ISVs now have the ability to gain similar visibility into their customer-managed products. These metrics help provide clarity to everything from feature adoption to upgrade completion, and we believe will become an essential element to any software distribution team.