🔢 📏Moving Beyond Vanity Metrics

Insights

Imagine this: Your product just hit 1 million downloads. Yay!!!! 🎉 

The team is celebrating, the CEO is thrilled, and the press is buzzing. But a month later, you notice that only 10% of those users are active, and most have already churned.

What went wrong? 😔 
The answer lies in the metrics you’re tracking. Vanity metrics like downloads, page views, or social media likes might look impressive, but they don’t tell the whole story of your product’s success.

In this issue, we’ll explore:

• Vanity Metrics: What & Examples.

• Why Vanity Metrics are misleading.

• Identifying KPI metrics that truly matter.

• Frameworks to guide metric selection.

• A detailed discussion between my AI friends (Must listen!)

Let’s dive in! 🚀 

🔍 Introduction

Vanity metrics are numbers that look impressive on the surface but don’t provide meaningful insights into your product’s performance or user behavior. They’re often easy to measure and can make your product seem successful—think metrics like downloads, page views, social media likes, or total registered users. While these numbers might boost your ego or impress stakeholders, they often fail to answer critical questions:

• Are users engaging with your product?

• Is your product driving tangible business outcomes?

• Are users finding value in what you’ve built?

For example, an e-commerce website might celebrate 100,000 monthly visitors, but if only 1% of those visitors make a purchase, the traffic metric alone doesn’t reflect true success. Vanity metrics can create a false sense of achievement and lead to poor decision-making if they’re the primary focus.

The key to moving beyond vanity metrics is to identify and track actionable metrics that reflect user behavior, align with business goals, and help you make informed decisions. By shifting your focus to metrics like engagement, retention, revenue, and user satisfaction, you can gain a clearer picture of your product’s true success and drive meaningful improvements.

In short, vanity metrics might look good in a headline, but they won’t help you build a product that delivers real value.

Why Vanity Metrics Are Misleading

Vanity metrics often fail to correlate with long-term success or user value. Here’s why they’re problematic:

• They don’t reflect user behavior: A high number of downloads doesn’t mean users are engaging with your product.

• They don’t align with business goals: Page views might be up, but if conversions are down, your product isn’t driving real value.

• They can lead to poor decision-making: Focusing on vanity metrics might cause you to prioritize the wrong features or initiatives.

Identifying Meaningful KPIs

To measure true product success, you need actionable KPIs that reflect user behavior and business outcomes. Here’s how to choose the right ones:

1. Focus on Engagement

   • Metrics: Daily Active Users (DAU), Monthly Active Users (MAU), session duration, feature usage.

   • Why it matters: Engagement metrics show how often and how deeply users interact with your product.

2. Track Retention

   • Metrics: Churn rate, retention rate, cohort analysis.

   • Why it matters: Retention metrics reveal whether users find ongoing value in your product.

3. Measure Revenue

   • Metrics: Customer Lifetime Value (CLTV), Average Revenue Per User (ARPU), conversion rate.

   • Why it matters: Revenue metrics tie product success directly to business outcomes.

4. Assess User Satisfaction

   • Metrics: Net Promoter Score (NPS), Customer Satisfaction (CSAT), app store ratings.

   • Why it matters: Satisfaction metrics help you understand how users feel about your product.

Frameworks to Guide Your KPI Selection

To ensure you’re tracking the right metrics, use these frameworks:

1. HEART Framework

   • Happiness: Measures user satisfaction (e.g., NPS, CSAT).

   • Engagement: Tracks user interaction (e.g., DAU, session duration).

   • Adoption: Measures new user acquisition (e.g., sign-ups, activation rate).

   • Retention: Tracks user retention over time (e.g., churn rate, cohort analysis).

   • Task Success: Measures how well users achieve their goals (e.g., conversion rate, error rate).

2. North Star Metric

Identify the one metric that best reflects the value your product delivers to customers and aligns with business goals, ensuring everyone on the team is rowing in the same direction. For example:

• Airbnb: Nights booked.

• Spotify: Time spent listening.

• Slack: Messages sent.

While the NSM is critical, it’s often a lagging indicator, meaning it shows the result of efforts after they’ve already happened. To proactively influence the NSM, teams need input metrics—leading indicators that predict changes in the North Star. For example:

• If your NSM is Nights Booked (Airbnb), input metrics might include:

  • New host sign-ups (supply growth).

  • Guest search activity (demand growth).

  • Conversion rate from search to booking.

• If your NSM is Time Spent Listening (Spotify), input metrics might include:

  • Number of playlists created.

  • Songs added to libraries.

  • Retention rate of new users.

To learn more, see a deep dive into North Star Metrics

Case Study: Airbnb’s North Star Metric

What Airbnb Tracked

Airbnb’s success is often attributed to its focus on a North Star Metric—a single, overarching metric that reflects the core value the product delivers to customers. For Airbnb, this metric is "Nights Booked."

• Why Nights Booked?

  • It directly ties to Airbnb’s mission of creating a world where anyone can belong anywhere.

  • It reflects both supply (hosts listing properties) and demand (guests booking stays).

  • It aligns with revenue growth, as more nights booked means more income for Airbnb and its hosts.

How AirBnB Moved Beyond Vanity Metrics

Early on, Airbnb could have focused on vanity metrics like website visits or registered users. However, these metrics didn’t capture the true health of the business. For example:

• A high number of website visits didn’t guarantee bookings.

• A large number of registered users didn’t mean those users were actively hosting or traveling.

By focusing on Nights Booked, Airbnb ensured that every team—whether engineering, marketing, or customer support—aligned their efforts around driving this key metric. This focus helped them scale sustainably and build a product that delivered real value to both hosts and guests.

Source: Masters of Scale Podcast, Lenny’s Newsletter, and industry reports.

Closing Thoughts

Vanity metrics might give the team a quick ego boost, but they won’t help you build a successful product. By focusing on actionable KPIs, you can ensure that your product efforts drive real value for users and align with business goals. Remember, the right metrics will evolve as your product grows, so revisit and refine them regularly.

🙋‍♀️ ❓ Questions & Answers

1. Q: How do I know if a metric is a vanity metric?

   • A: Ask yourself: Does this metric directly correlate with user value or business outcomes? If not, it’s likely a vanity metric.

2. Q: What if my stakeholders are fixated on vanity metrics?

   • A: Educate them on the limitations of vanity metrics and present alternative KPIs that better reflect success. Use data and case studies to make your case.

3. Q: How many KPIs should I track?

   • A: Focus on 3-5 key metrics to avoid analysis paralysis. Ensure they cover different aspects of product success (e.g., engagement, retention, revenue).

✅ Take Action

Audit Your Current Metrics:

• Review your existing KPIs and identify which ones are vanity metrics. Replace them with actionable metrics.

If you liked this newsletter, please share it with your friends and colleagues. 👯‍♀️