When developing a SaaS product plan, it’s important to recognize two foundational principles. First, SaaS is a business strategy, not a technology strategy. Second, there is no one-size-fits-all SaaS architecture (the second principle is a corollary of the first). The challenge is to build common ground between business and architecture so as to translate business assumptions into critical technical solution inputs.

We begin by taking stock of three foundational dimensions: (1) User Model, (2) Monetization, and (3) Measurement. In this blog series, we explore how these three dimensions figure into key technical recommendations which enable scale in pursuit of SaaS business growth.

Part 3: Measurement

One of the great breakthroughs of SaaS as a business strategy is in how it puts users and subscribers front and center of software development and keeps them there. The recurring revenue user model of SaaS demands deep and broad attention at every step, as it empowers you to provide continued feedback on what you are selling them.

Translating that feedback into actionable insights can make or break the growth potential of your SaaS product offering.

  • SaaS business metrics are not hard to find; we won’t analyze them in-depth here.
  • As it turns out, SaaS operational platform metrics are also abundant. DevOps has largely completed the evolution of systems monitoring from the datacenter/IT worldview to what is commonly referred to as observability.

However, it is still too often the case that business metrics and operations metrics live in parallel universes. The success of a SaaS platform depends on creating a positive feedback loop between business metrics and operational metrics.

Customer-centric software delivery

SaaS sets solving customer problems as its north star. For most customers, the hard work that goes into coding features, bug fixes, and operational configurations is just plain invisible. If you’re lucky, they won’t notice when competitors introduce features that solve a problem better than you (at least for now).

The intensity of competition in the SaaS marketplace puts a premium on new, stable, and reliable features. The winners offer a better and faster way to solve customer problems.

  • Devising new ideas and improvements that comprise your feature roadmap is at the root of your SaaS product strategy (more on that in the next section).
  • Achieving the clarity and focus required to implement those ideas and improvements means that they are sufficiently understood and scoped for execution. That’s what separates brainstorms from business results.

But that’s just the beginning. Until your better-and-faster solution reaches those customers in production, that great idea that’s somewhere in the software development? It makes no difference to anyone but you.

Software delivery is that sequence of steps between completing work on a new feature and when that new feature reaches your customers in production. (By implication, one-off features and changes reduce software delivery efficiency.) We’ve written about this before.

Given that engineering is a core competence in software development, it should come as no surprise that there are plenty of ways to measure. The endpoint of any measurement is when software reaches the people who are paying for it, as measured by what is known as the four keys:

  • Deployment Frequency: How often an organization successfully releases to production
  • Lead Time for Changes: The amount of time it takes a commit to get into production
  • Change Failure Rate: The percentage of deployments causing a failure in production
  • Time to Restore Service: How long it takes an organization to recover from a failure in production

Deployment Frequency and Lead Time for Changes measure velocity, while Change Failure Rate and Time to Restore Service measure how reliably the service reaches your customers. Consider two perspectives on the value of this set of measures.

  • They collectively demonstrate how successfully your SaaS software delivers changes which matter to your customers.
  • It turns out that software delivery velocity has been proven to correlate with greater business success.

Whatever you call software-enabled, cloud-based competitive advantage, understanding the business impact of “digital transformation” is the focus of the massive 8-year study conducted by the DevOps Research Institute. Their conclusion:

Delivering software quickly, reliably, and safely is at the heart of technology transformation and organizational performance. We see continued evidence that software speed, stability, and availability contribute to organizational performance (including profitability, productivity, and customer satisfaction). Our highest performers are twice as likely to meet or exceed their organizational performance goals.State of DevOps Report, DORA Research Institute, 2019

The latest iteration of the study was launched in May 2021. (The results show interesting findings for the critical success factors for recent new distributed & remote work paradigms following the pandemic). Here is a quick gauge (less than 5 minutes!)of software delivery success compared to others in your industry.

Monitoring and Logging

Understanding system health, and what interventions might be required, is not a new problem. Big data has been a game-changer.

  • It has revolutionized tracking systems utilization and health data, by allowing capture of almost any conceivable system behavior.
  • Platforms operations teams had potentially endless evidence of any set of data points describing what happened where and when.
  • Analytics tooling such as ELK means that you can slice and dice insights about any data points you have already captured.

It’s hard to overstate how useful this data can be in the rear view mirror, particularly to operations teams tackling continuous optimization in the face of changing behaviors, system capabilities, market demands, new releases, and on and on. Powerful analytics tooling can help you discover anything that has happened in captured log data, from forensic troubleshooting to endless varieties of visual analytics.

The more distributed the systems that power your SaaS platform, the more vital it is for you to ensure you have a strategy for log aggregation. Log data needs to be in a place that can be accessed in a secure, well-structured fashion.

Using logging to track, compliance, traceability, resource utilization, and system behaviors are a core competence for application of log data. What’s more, this data can be extended by specifying a trigger for an action, where a certain set of inputs triggers an alert, or even an automated change to a configuration. Examples abound:

  • All major database systems have a feature for logging slow queries. use this in forensics or in capacity analysis
  • Log data can be used to prevent, detect, and respond to threats, breaches, endpoint hacks, threat hunting, cloud capacity management, etc.

The endless possibilities have one irreducible prerequisite: data has to be captured in the logs. Lest this seem like a trite observation, the fact is that most data that goes into logs comes directly from bottom-up infrastructure. Developers add specialized logging to track the behavior of a particular function. The specification for that function does not track the big picture needs of a user model or a monetized feature.

The good news is that this does not have to be difficult.

  • If the architecture specifies tracking of user-level activity, it’s not difficult to slice-and-dice the log data to select for a user parameter (such as with a JWT token), or using tags to match user/tenant activity to the per unit billing that AWS itemizes in your monthly cloud bill.
  • Monitoring technology such as Prometheus can do much of this for you, aggregating event and time-series data for actionable monitoring, metrics, trending and diagnostics.

Finally, translating logging into insights requires iteration. It’s tempting to assume there is a metrics model which tells you everything you need to know about the feedback loop of your SaaS strategy. The speed of SaaS development and SaaS market competition suggest that you resist that temptation. The FinOps Foundation has a good approach to a metrics life cycle that extends well beyond monetization:

How’s that feature working for you? and you? and you?

No amount of efficiency in crafting and releasing software code will matter if your customers aren’t spending time and money to use the features in your product. This is true whether you have a large number of small customers or a small number of large customers (or anywhere in between). The more you can find out about how your customers use your SaaS offering, the better.

Depending on the kind of SaaS you offer, there’s a lot to learn from talking to customers about your product. Similarly, there’s a lot to learn from talking to the people who talk to your customers in sales, marketing, and customer success. To get to the next level, you want to know what customers are doing.

The challenge is to ensure that the software is built in a way that enables differentiated and precise measurement. Combining SaaS tenant identity with the activities and features in your SaaS user model creates the necessary conditions for your SaaS measurements to deliver value.

  • The question of who does what in the user model is a great starting point.
  • Taken together, the set of use cases that describe user experience, the intended sequences of tasks, and successful outcomes combine to form inputs to measurement.
  • When tenant context is built in, from the outset of onboarding, you can evaluate users’ actions directly.
  • Well-architected SaaS includes a unique secure identifier to track each user, customer, tenant, and feature. That lets you slice and dice to compare each and every one of them.
  • Tracking users/tenants/customers across each feature interaction shows who is (or is not) using particular features, at various points in time, well after onboarding.

There are limitless possibilities. In our experience, the most valuable measures of user activity are tied to whether they demonstrate something about user value over time. This allows everyone in the company to all understand how they contribute – individually and collectively – to company success.

The more we understand of your user model, monetization strategy, and measurement needs, the better we can recommend which architectural options best support your business strategy.

Aligning your SaaS business strategy with SaaS technical architecture is critical to SaaS success. See more about all 3 M’s in these two companion blog posts, Part 1: User Model and Part 2: Monetization. For a thorough assessment of options for a SaaS business model, download the AWS SaaS Journey Framework whitepaper, and/or take a technical deep dive into the SaaS Lens for the AWS Well-Architected Framework.