Product Manager Toolkit

Purpose of this page

As a Product Manager, you constantly have data-related questions. You want to know how your feature is performing, who your top users are, how many Self Managed installations have adopted a specific feature, etc. Plus, you potentially want all of these metrics broken out by product tier, paid vs free, edition, etc.

While we have a lot of data, accessing some of it requires some level of proficiency and comfort with SQL. And even though you can try to copy some of the existing charts to tune them to answer the question you have, it can be time-consuming and you might not be 100% confident that the chart shows accurate data.

The Data program has created a set of different snippets to address your needs. This list is not exhaustive, and is mainly based on the feedback, questions, and tasks we received in the past from PMs. All of these snippets will be explained with detailed examples on how to change them and adapt them to your own needs. These are mainly focused on Self-Managed data from Service Ping.

If you prefer to learn by doing, we provide a documented example Sisense dashboard that you can check out, and most charts include documentation in their SQL edit view to help you customize them. You can look at the code used to generate the charts. The code has some extra guidelines commented out that will help you better understand how the given snippet works and what you can achieve with it.

Overall, these snippets are meant to help any team member who is looking for specific data from Service Ping. The goal of this page is to enable self-service analysis, in addition to building greater confidence in our data and models. We created an issue (internal link) to gather feedback. Please add any feedback on this page and the snippets provided.

The snippets listed below will allow you to:

  • create a visualization with estimated Self Managed uplift for a Service Ping metric (either an xMAU metric or other PI) ([td_xmau])
  • get a monthly count of installations using one of your features ([td_xmau_metrics_instance_count])
  • get a installation adoption rate of your feature ([td_xmau_metrics_instance_adoption])
  • get a list of your Top Users, Top Growing Users, or Churning Users ([td_xmau_metrics_top_users], [td_xmau_metrics_increasing_usage_users], [td_xmau_metrics_decreasing_usage_users])
  • gain more insights into a drop or an increase of your feature usage ([td_xmau_monthly_change_breakdown])

Trusted Data snippets

All these snippets start with a td_ (Trusted Data) prefix and are considered to meet the requirements for the Data team’s Trusted Data Framework.

Trended snippets

The following snippets return usage grouped by reporting period (either week or month). The current reporting period is excluded in order to avoid including incomplete data.

xMAU or PI chart with estimate: [td_xmau]

[td_xmau(user_segment, xmau_type, filter, target, estimation_grain)]

The [td_xmau] snippet includes both recorded and estimated usage. For all-time metrics, this returns the installation-level change from the previous period. To return the original value for all-time metrics, please use the [td_xmau_metrics_original_metric_value_with_total] snippet (discussed below).

This snippet will allow you to:

This is the only snippet that includes estimated usage. All other snippets are limited to recorded data. {: .alert .alert-info}

Usage

The [td_xmau] snippet requires 5 parameters: user_segment, xmau_type, filter, target, and estimation_grain.

  1. user_segment: ‘All’ or ‘Paid’
  2. xmau_type: one of the xMAU types (‘GMAU’, ‘SMAU’, ‘CMAU’, ‘UMAU’) or ‘PI’
  3. filter: the metric(s) you want included in the chart. This value is dependent on the xMAU type. Section, stage, and group names should be lowercase with underscores in the place of spaces.
  • ‘GMAU’ –> ‘group_name’ or ‘All’
  • ‘SMAU’ –> ‘stage_name’ or ‘All’
  • ‘CMAU’ –> ‘section_name’ or ‘All’
  • ‘UMAU’ –> ‘All’
  • ‘PI’ –> ‘metrics_path’
  1. target: the target to draw for this metric
    • If you set a target value in the performance indicator yaml file (as described below), then that value will be used
    • If you provide a number between 0 and 1, the target will increase MoM by that percentage.
      • Ex: 0 will yield a target that equal to the previous month’s total (0% change MoM)
      • Ex: 0.1 will yield a target that is a 10% increase from the previous month’s total
    • If you provide an integer, that number will be the fixed target line
  2. estimation_grain: The estimation methodology you want to use
  • ‘metric/version check - subscription based estimation’ * Use this one! It is our “official” methodology
  • ‘metric/version check - seat based estimation’
  • ‘reported metric - subscription based estimation’
  • ‘reported metric - seat based estimation’
  • More details about the different estimation_grains on the Self-Managed Estimation Algorithm handbook page

Examples

  • Paid Verify SMAU: [td_xmau('Paid', 'SMAU', 'verify', 0.1, 'metric/version check - subscription based estimation')]
  • Total Web IDE Editors: [td_xmau('All', 'PI', 'usage_activity_by_stage_monthly.create.action_monthly_active_users_web_ide_edit', 0, 'metric/version check - subscription based estimation')]
  • Total Plan SMAU: [td_xmau('All', 'SMAU', 'create', 0.1, 'metric/version check - subscription based estimation')]
  • Total Dev CMAU: [td_xmau('All', 'CMAU', 'dev', 5, 'metric/version check - subscription based estimation')]

Recorded monthly metric value

[td_xmau_metrics_recorded_metric_value(metrics_path)]

The [td_xmau_metrics_recorded_metric_value] snippet shows you the recorded (non-estimated) value of any of your metrics implemented in Service Ping. For all-time metrics, this returns the installation-level change from the previous period.

The resulting dataset will enable you to break down recorded usage by:

  • delivery (SaaS vs Self-Managed)
  • edition
  • product_tier

Usage

The [td_xmau_metrics_recorded_metric_value] snippet requires 1 parameter: metrics_path.

  1. metrics_path: the Service Ping metric of interest

You can add a target value to the chart by adding a monthly_estimated_targets key to the performance indicator yaml as described below. If a target value is not provided in the performance indicator yaml file, a target will not appear on the chart.

Example

Recorded monthly metric value change with total

[td_xmau_metrics_recorded_metric_value_with_total(metrics_path)]

This snippet extends [td_xmau_metrics_recorded_metric_value] with a total line.

Usage

The [td_xmau_metrics_recorded_metric_value_with_total] snippet requires 1 parameter: metrics_path.

  1. metrics_path: the Service Ping metric of interest

You can add a target value to the chart by adding a monthly_estimated_targets key to the performance indicator yaml as described below. If a target value is not provided in the performance indicator yaml file, a target will not appear on the chart.

Example

Original metric value

[td_xmau_metrics_original_metric_value_with_total(metrics_path)]

This snippet is a variation of [td_xmau_metrics_recorded_metric_value_with_total] that shows the original value (as it appears in the Service Ping payload), instead of a change from the previous period. For 28-day metrics, this will return the same results as the [td_xmau_metrics_recorded_metric_value_with_total] snippet.

Usage

The [td_xmau_metrics_original_metric_value_with_total] snippet requires 1 parameter: metrics_path.

  1. metrics_path: the Service Ping metric of interest

You can add a target value to the chart by adding a monthly_estimated_targets key to the performance indicator yaml as described below. If a target value is not provided in the performance indicator yaml file, a target will not appear on the chart.

Recorded weekly metric value

[td_xmau_metrics_weekly_recorded_metric_value(metrics_path)]

The [td_xmau_metrics_weekly_recorded_metric_value] snippet shows you the recorded (non-estimated) value of any of your 7-day metrics implemented in Service Ping.

The resulting dataset will enable you to break down recorded usage by:

  • delivery (SaaS vs Self-Managed)
  • edition
  • product_tier

Usage

The [td_xmau_metrics_weekly_recorded_metric_value] snippet requires 1 parameter: metrics_path.

  1. metrics_path: the Service Ping metric of interest

Example

Recorded weekly metric value with total

[td_xmau_metrics_weekly_recorded_metric_value_with_total(metrics_path)]

This snippet extends td_xmau_metrics_weekly_recorded_metric_value with a total line.

Usage

The [td_xmau_metrics_weekly_recorded_metric_value_with_total] snippet requires 1 parameter: metrics_path.

  1. metrics_path: the Service Ping metric of interest

Example

Monthly active installation count

[td_xmau_metrics_instance_count(metrics_path)]

The snippet [td_xmau_metrics_instance_count] shows you how many installations have used your feature (returned a metric_value > 0) in a given month. This snippet does not include any estimations, just recorded data. The count is broken out by:

  • ping_delivery_type (always Self-Managed in this snippet)
  • ping_edition
  • ping_product_tier

Usage

The [td_xmau_metrics_instance_count] snippet requires 1 parameter: metrics_path.

  1. metrics_path: the Service Ping metric of interest

Example

Monthly metric installation adoption

[td_xmau_metrics_instance_adoption(breakdown, metrics_path)]

The snippet [td_xmau_metrics_instance_adoption] can be used to find out installation-level adoption of your features. It provides you the count and share of installations reporting metric_value > 0 for a specific Service Ping metric. It is effectively the same as the td_xmau_metrics_instance_count snippet, but it allows you to specify how you group the data. This snippet does not include any estimations, just recorded data.

Usage

The [td_xmau_metrics_instance_adoption] snippet requires 2 parameters: breakdown and metrics_path.

  1. breakdown: How you want the data grouped. This value should not be put in single quotes. (Ex: use ping_edition, not 'ping_edition'). Here are some example breakdowns (non-exhaustive list):
  • ping_edition
  • ping_product_tier
  • ping_edition_product_tier
  • major_minor_version
  1. metrics_path: the Service Ping metric of interest. This value should be put in single quotes. (Ex: use 'usage_activity_by_stage_monthly.release.deployments', not usage_activity_by_stage_monthly.release.deployments)

Examples

Monthly change breakdown

[td_xmau_monthly_change_breakdown(metrics_path)]

The [td_xmau_monthly_change_breakdown] snippet enables you to better understand the growth/decrease of a specific monthly metric value by breaking usage into 4 buckets:

  1. Decreased Usage: Comparing Month M with M-1, lower usage of a feature (for example, less users opening an issue)
  2. Increased Usage: Comparing Month M with M-1, higher usage of a feature (for example, more users opening an issue)
  3. New tracked installations: We have not seen this installation reporting this feature counter before. That means either it is a new installation, or it just upgraded from an older version that did not have the counter
  4. No data received: An installation stopped reporting (it could be because of expiring subscription, churn, or disabling of Service Ping)

This snippet does not include any estimations, just recorded data.

Usage

The [td_xmau_monthly_change_breakdown] snippet requires 1 parameter: metrics_path.

  1. metrics_path: the Service Ping metric of interest

Example

Monthly breakdown change - Issues: [td_xmau_monthly_change_breakdown('usage_activity_by_stage_monthly.plan.issues')]

Other snippets

The following snippets return usage from a fixed point in time (ex: last month). The data is not trended and is intended to allow PMs to dig deeper into movement in a specific metric.

Top 50 installations last month

[td_xmau_metrics_top_users(metrics_path)]

The [td_xmau_metrics_top_users] snippet returns a list of the Top 50 Self-Managed installations using your specific feature last month.

Usage

The [td_xmau_metrics_top_users] snippet requires 1 parameter: metrics_path.

  1. metrics_path: the Service Ping metric of interest

Example

Installations with biggest MOM usage increase

[td_xmau_metrics_increasing_usage_users(metrics_path)]

The snippet renders a list of Top 50 Self-Managed Installations who have seen their monthly usage grow the most in the last completed month.

Usage

The [td_xmau_metrics_increasing_usage_users] snippet requires 1 parameter: metrics_path.

  1. metrics_path: the Service Ping metric of interest

Example

Installations with biggest MOM usage decrease

[td_xmau_metrics_decreasing_usage_users(metrics_path)]

The [td_xmau_metrics_decreasing_usage_users] snippet returns a list of Top 50 Self-Managed installations who have seen their monthly usage decrease the most in the last completed month.

Usage

The [td_xmau_metrics_decreasing_usage_users] snippet requires 1 parameter: metrics_path.

  1. metrics_path: the Service Ping metric of interest

Example

Other tips and tricks

How to show dynamic targets from PI yaml files

The following snippets support adding dynamic target values from performance indicator yaml files:

  • td_xmau
  • td_xmau_metrics_recorded_metric_value
  • td_xmau_metrics_recorded_metric_value_with_total
  • td_xmau_metrics_original_metric_value

Usage

Add the metric_name and monthly_estimated_targets keys to a performance indicator yaml file.

  • metric_name: The name should equal to the metrics_path used in the snippet
  • monthly_estimated_targets: is a list of "key": value pairs where key is the target date to reach the target value

Example

Imagine the following was included in a PI yaml file

1
2
3
4
5
6
7
8

...
- name: Configure:Configure - SMAU, GMAU - MAU of Denomas Managed Terraform State
  metric_name: redis_hll_counters.terraform.p_terraform_state_api_unique_users_monthly
  monthly_estimated_targets:
    "2023-01-30": 5000
    "2022-10-30": 4000
    "2022-07-30": 2000
...

That would produce the following targets

reporting_month target_value
All previous months 2000
2022-06-01 2000
2022-07-01 2000
2022-08-01 4000
2022-09-01 4000
2022-10-01 4000
2022-11-01 5000
2022-12-01 5000
2023-01-01 5000

Forecasted xMAU

You can leverage the [td_xmau] snippet to forecast your PI growth over time. Check out this video on how to get started.

  • Edit the chart in SiSense.
  • Below the query, expand the dropdown next to Python 3.7 and select Predictive Forecast Plot.
  • Next you will need to edit the snippet to update the names of a couple of fields. You can reference the query used in this chart to forecast Package stage GMAU (also displayed below)
  • Now you are ready to click Run SQL
  • Then scroll down and click Run Python
  • Save the chart.
  • Happy forecasting!
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

WITH forecast AS (

  [td_xmau('All', 'GMAU', 'package', 0.1, 'metric/version check - subscription based estimation')]

)

SELECT
  reporting_month AS ds_month,
  SUM(mau_value) AS y_value
FROM forecast
WHERE mau_value > 0
  AND stage_name = 'All'
GROUP BY 1

Interpreting product_tier/ping_product_tier

Currently in our data models, we use the value Core to represent installations on the free tier. To help align with how we define and talk about product tiers at Denomas, we have an issue to update the logic and use Free instead.

Go further

These key snippets will help you generate some quick charts and key data about your metrics. Of course, this list of snippets is just a glimpse of what can be done with our current data models.

Feedback

If you have ideas for new snippets that can enable data exploration or any other feedback about these snippets or this handbook page, please contribute to this issue (internal link) .

Last modified December 1, 2023: bulk update (176cf9ec)
View page source - Edit this page - please contribute. Creative Commons License