The Before/After analyzer is designed to help you measure the change in a KPI after a specific event or date. It allows you to specify how to aggregate the KPI, how to determine the timeframe and event date, and whether to normalize or group your KPI. It then performs a t-test to let you know if the changes in your KPI are statistically significant.
The analyzer is available as a full-screen page, linked from the side navigation bar, and as a component on a dashboard.
Configuring A New Analysis¶
To run an analysis, you'll first need to configure a few settings. When you start a new analysis, you'll be brought to an empty screen that prompts to you to open the configuration panel.
Selecting the KPI¶
When you click on the "Configure" link or button, you'll be brought to the first step of configuring a new analysis: selecting the column that contains your key metric. In this screen, you'll need to select the column that contains the numerical values that will be used in your analysis. These values can be transformed in a variety of ways in later steps, for example, you can normalize these values by another column, or you can count them instead of adding them up, but for now you'll need to select the column that contains the data that you'll ultimately use.
The next step is to select the column in your dataset, in the same collection/table as your KPI, that holds the date that is associated with your KPI records. For example, if you're analyzing transaction size, you'll need to select the column that's associated with each transaction.
For the system to know how to calculate whether your events happened before or after some specifc point in time, you need to tell it what that point in time is. The Before/After Analyzer allows you to specify whether that point in time should be an absolute date or a relative date, specified by a value in your data.
On the next screen, you'll be able to select and specify this point in time.
Here, if you select an absolute date, you'll see a date picker that will allow you to select the date in time that you want to use for the segmentation date. If you select to use a column in your dataset instead, you'll be asked to select a column that contains date values.
If you select a column in the same table as your KPI, then you'll be all set, but if you select a column in a new dataset, you'll need to tell the system how to connect the dataset that has your KPI with the dataset that has the date, this is called joining datasets.
In order to join datasets, you need to select a column in the first dataset that links to the second dataset. This is usually an ID column.
An example of when you might need to join two tables is if you're trying to analyze the lift in average user transaction size after they receive a gift card. If you have one dataset that holds records of transactions, and another table that holds the date that each user received a gift card, you'll need to join the transactions dataset to the gift card dataset. You might have a column in the transactions dataset that holds a user ID and a column in the gift card dataset that also has a user ID. These would be the linking columns that you would specify during this step.
Once you select the KPI, event date, and segmentation date, the system will have enough information to run an analysis. However, the analysis will reflect the default value of several other settings, which you also have the ability to change. By default, your analysis will compare the average KPI value one month before your segmentation date to the average KPI value value one month after the segmentation date.
In the Advanced Settings section, you can reconfigure these settings to reflect different data aggregation methods, different timeframes, and how to break down your analysis into groups.
To see these advanced settings, click on the button labeled "Show Advanced Options".
By default, the system will compare the average value of your KPI column before and after the segmentation date, however you have the way this data is aggregated and grouped. You can normalize your KPI by taking the average value with respect to another column in the dataset (for example, if you wanted to take the average transaction size by user ID), or by taking the average number of times your KPI occurs with respect to another column (for example, if you wanted to count the average number of transactions by user ID).
On the metric aggregation screen, you can select from the following options:
- Average Value: Compare the average KPI value before and after the segmentation date
- Average with Respect to Column: Compare the average KPI value with respect to another column before and after the segmentation date
- Count with Respect to Column: Compare how often your KPI appears with respect to another column before and after the segmentation date
Example: Average Transaction Size Per User
As an example, let's say you want to compare the average transaction value per user before
and after some point in time. In this screen, you would set the metric aggregation to
Average With Respect to Column, and then you would select the column in your main
dataset that represents the user ID.
By default, the system will compare your KPI one month before or after your segmentation date. If you'd like to change that, cou can choose from a pre-defined list of timeframes on the timeframe selection screen.
Breaking down your metric by group¶
You can optionally specify a column that will be used to group your data. In doing so, you will then receive separate statistics for each group. To specify how to group your data, select the column in your dataset that contains grouping values in the metric break down screen in the advanced options section.
Interpreting the Results¶
See our Before/After Results Page to see how to interpret the results.