Learn how to track and manage your data with our free email course.
Little Data for Better Product
You’ve heard about big data. But how are you using Little Data?
Maybe you wear a FitBit or track your spending. Seeing how many steps you take each day can help you to set incremental goals to help you to get more healthy.
In software development, you can learn a lot by tracking your data, too. If you don’t have a good way to measure and analyze your metrics, you’ll get stuck in an endless cycle of guessing and speculation.
In the Lean Startup, Eric Ries described an ideal process as “Build, Measure, Learn.” But a lot of teams are still relying on a process that is more like, “build, get a vague impression, guess.” That’s because, despite the many sources of data we have at our disposal, getting ongoing context and learning can be a real challenge.
For example, you may be collecting data about your development process (ticketing & bugs) in JIRA, customer feedback in a tool like Zendesk, user interaction behaviours in you own app database and engagement metrics in something like HubSpot. Awesome! You should have everything you need!
Little data: Big headache
If all this is so great, why don’t all companies have a great Build, Measure, Learn approach?
In the past, enterprising CEOs and Product Managers would create custom databases, make elaborate spreadsheets, or just eyeball different metrics in different systems. There could be a veritable flood of PDFs, powerpoints, emails but the information would still be disjointed.
Because many of our software tools measure on a short-term basis, it’s often hard to see how data trends across time. Many of the metrics that do get tracked aren’t useful, because they only tell “what happened” without giving context to make better future decisions.
There is a significant challenge of giving context to data that may be small in each segment, but can grow completely unwieldy if it’s all thrown together without a framework.
You could have been tracking your steps by figuring out the distances you walked and adding it all every day, but that probably would have gotten old quickly. We designed Notion to make tracking your team’s data more intuitive and centralized so you can visualize it and make correlations more easily.
Why do you need little data?
The data that you can get about your product engineering team is crucial in planning your product process and sprints. Collecting information over time about what you can accomplish in a typical cycle lets you estimate how many features you can develop, how much time you need for QA, and what reasonable expectations look like.
Other metrics can tell you about your customer’s satisfaction with what you’re making, and about the financial indicators of your success. These are all numbers that are unique to your product, team and market. There is no giant repository of data that can replace these individual measurements and lead you to better decisions.
There are a ton of things you could measure and sometimes the sheer volume can make selecting key metrics confusing. When we’ve talked to successful Product Managers we’ve discovered that there are some measurements that will benefit almost every company and team, and some that are more specific to your particular set of goals.
We’ve put together a guide of some of the most important metrics for each stage of your company, based on the idea of Dave McClure’s Pirate Metrics for SaaS companies. In addition, our AARRRT! Guide explains some of the key Team Metrics that reflect what your engineering team can accomplish. These are the metrics that most SaaS teams will learn the most from.
At Notion, our focus on product development means that Build metrics like Throughput, DRE and Cycle time can be compared with the metrics that indicate customer happiness. Team Intelligence means understanding how your developers are impacting your business.
Most SaaS companies focus on the process of bringing on new customers in their growth stage, while retention and customer success might be more emphasized with maturity. When your product is in its early stages, your team may be putting all its energy into developing features and increasing velocity, which may turn into a focus on quality and stability as you grow.
In the School of Little Data, we’ll also be covering specific metrics to track for early stage and mature products, for comparing metrics from sales, support and marketing, and for solving specific problems such as Churn.
You don’t need to use Notion to get started with your Measure step. Most of the recipes we include in the AARRRT guide, for example, could be entered into a spreadsheet and will start to give you a picture of what you can Learn after you Build and Measure.
It’s helpful to see all your data in nice-looking graphs, but without acting on the information, you really can’t get anywhere. The point of Build, Measure, Learn is that learning informs your next build process.
For example, you’ve noticed that user churn has been increasing over the last few weeks. Members of your team believe that to resolve the issue the focus should be on delivering new features as fast as possible.
Custom User Churn
As the product owner, it’s your responsibility to understand if this is the best solution. And to do that, the more data you have the better.
To start, you’ll want to look at some of your dev team metrics.
Dev Team Velocity
Velocity has been pretty consistent and going up. But Team Morale — a team poll you set up to track how the team feels about the amount of work they are doing each sprint — is starting to drop, which indicates to you that you can’t push the team much harder. Meaning trying to release new features at a faster rate with the current team isn’t possible.
So you probably need another option to reduce your churn.
Next you look at Throughput. Over the past few sprints you can see that your team is already delivering more features than normal, and as a result fewer bugs are being delivered.
While this could mean that your code quality is improving and there are simply fewer bugs to work on, it could also mean code quality is dropping because the team isn’t spending time there. Let’s look at more data.
Defect Removal Efficiency
Your Defect Removal Efficiency is dropping. This is a way to track where your bugs are being found, and in this case, more are being found in production.
And when you peek at the number of customer support requests, you see that it is increasing as well.
Open Support Tickets
This is all the info you need now to form the hypothesis: Code quality is decreasing, which is resulting in an increase user churn. The team is releasing features too fast already, which is taking time away from fixing existing bugs AND increasing new bugs into the system.
To track this hypothesis, you create a new dashboard and set goals around the following recipes:
- Bug Reaction Time
- Bug Cycle Time
- User Churn
- Team Morale
Communicating your results
One of the most frustrating aspects of data that we encountered before building Notion was that this information was often siloed in specific departments. Developers, the support team, marketing folks and product people didn’t have much of the data that could help them to each make more effective decisions. Despite our common goal across the organization, vital data collected in an app could get stuck and never leave the department that used that software.
We also encountered situations where we needed to communicate results with investors, clients or other stakeholders and there wasn’t a simple way to create a good-looking, meaningful report to convey progress. It’s been great to be able to create custom reports anytime for people who contribute to our success.
For us, the goal of greater communication seems to be as important as collecting and visualizing data so that if you’re making decisions about the product, you’ll have the context and expertise of the whole company to inform you.
What is your experience with Little Data? What do you want to learn more about? Write to us at firstname.lastname@example.org.