How to Evaluate the Cost Efficiency of the Attribution Model

0. A short summary of our previous articles:

For those of you who have joined us recently, or just in case you’ve forgotten it, let’s recap what we talked about in our previous articles:

1. Standard attribution models, such as Last Non-Direct Click or associated conversions, have significant drawbacks.
2. Applying models which consider the mutual impact of channels allows you to  significantly increase the efficiency of online advertising.
3. The attribution problem can be solved by evaluating user sessions on the basis of the probability that a user moves through the funnel.
4. By implementing the Funnel Based attribution model, you can evaluate your advertising campaigns considering their impact on each step of the sales funnel, not just on the completed orders. All the data for the reports is processed in Google BigQuery. This approach takes into account users’ actions, costs for all advertising campaigns, and orders from CRM.

The task now is to ensure that in practice Funnel Based attribution allows you to manage your advertising budget more efficiently. Let’s take a closer look on how to accomplish that goal.

1. Build the report with key performance indicators

First, choose the performance target to evaluate the efficiency of your advertising efforts. Most businesses focus on ROAS ROAS (return on ad spend) = revenue from advertising / costs for advertising * 100% but in your case it can also be ROI ROI (return on investment) = (revenue—investment) / investment * 100% , CPA CPA (cost per action) = the amount of advertising costs / number of target actions or Share of advertising costs share of advertising costs = advertising costs / income from advertising * 100% . The important thing is that when you compare this indicator for the two campaigns, you can understand which of the two is performing better.

Now let’s build a report to compare the values of the chosen KPI. The first value is calculated according to the Funnel Based Attribution model. The other value is calculated according to your current attribution model. If you’re relying on the data found in standard Google Analytics reports, this is most likely the Last Non-Direct Click attribution.

Campaign — the name of the advertising campaign you are evaluating. Please note that, since campaigns within one channel may affect different steps of the funnel, a report with a source/medium-level channel grouping may not be enough. This often happens with ads triggered by product or category search terms.

Costs — the total cost of the given campaign during the reporting period.

Sessions — the number of sessions generated by the given advertising campaign.

All these three values (Campaign, Costs and Sessions), apparently don’t depend on the attribution model.

Revenue —the revenue you get with a certain advertising campaign. This value directly depends on the attribution model. For this reason, it’s better to calculate the revenue separately for both attribution models. The results will be different, since the Last Non-Direct Click model assigns all the values to the last session while the Funnel Based attribution model evaluates each session which helped a user move towards a given order.

When you acquire both Revenue and Costs, you can calculate the ROAS for each of the attribution models:

ROAS, predictably, will be different, since the Revenue is different.

The ROAS Difference shows the relative difference between the ROAS calculated on the basis of the Funnel Based attribution model and the ROAS calculated according to the Last Non-Direct Click one. This difference shows how the value of the advertising campaign would change if, rather than assessing the campaign through orders only, you focus on its contribution to each particular on a user’s journey through the funnel.

The difference in ROAS gives you valuable information, but it does not consider the size of the budget spent on the given campaign. Obviously, with the same difference, the primary interest is in the campaigns where you spend more.

Therefore, add the final column —Costs Difference.

You can then see the difference in the evaluation of advertising campaigns according to the Funnel Based and the Last Non-Direct Click attribution models. Campaign size is still taken into account. Campaigns with the greatest value of  Costs Difference should be analyzed and re-evaluated in the first place.

2. Segment your users for testing

How do you understand if the new attribution model and the ROAS you’ve calculated provide a more objective measurement of your advertising campaigns? It’s quite simple — use them while allocating your advertising budget and compare the results.

One user interacts with multiple campaigns. Therefore, you can’t feasibly split advertising campaigns into two groups for testing. Moreover, the same advertising budget can’t be divided in two different ways at one time, which means you can’t run classic A/B testing. Thus, you need to identify audience segments that do not overlap, yet at the same time are subject to the influence of external factors.

Within segment A, let’s work it out "the old-fashioned way" When allocating the budget in the B segment, let’s consider the ROAS calculated according to the new model.

The only thing remaining to do is to choose how to break down the audience into A and B segments.

Option 1. Break down the segments and campaigns by country.

Option 2. Break down the segments and campaigns by region.

Option 3. Reallocate your budget for the whole project, all at once.

As a result, you should expect to see a graph similar to the one displayed below which shows the change of ROAS in the tested segment (in the third option there’s one segment):

As you can see, there are advantages and limitations for each option. However, testing is useless in the event that the indicator you’re going to measure is unpredictable prior to testing:

This is often the result of incorrect data. A good example would be dips in the uploaded costs for large amounts of money.

3. Reallocate the advertising budget

The report you built in the first step will reveal which campaigns are overvalued — the Costs difference is negative, and which campaigns are undervalued — the Cost Difference is positive. Before the reallocation of the budget, you need to decide which holds the higher priority for you:

• Maximize revenue at the current advertising budget.
• Maximize revenue at the current ROAS.
• Preserve revenue at a lower advertising budget.

The logic of reallocation is slightly different for each of these options:

If you need to maximize revenue at the current advertising budget simply reallocate the advertising budget from the overvalued advertising campaigns to the undervalued ones.

If you need to maximize the revenue at the current ROAS, just increase the budget for the undervalued campaigns.

If you need to preserve revenue at a lower advertising budget, transfer a portion of the revenue from the overvalued campaigns to the undervalued ones.

Before increasing the budget for any of the campaigns, it’s important to factor in an external indicator which can’t be measured by attribution: the amount of traffic a channel can realistically generate. If you conclude that you’ve reached a «growth plateau» in the campaign (a budget increase exponentially increases the average cost per click), this advertising campaign should be the last one to have its budget increased. The «growth plateau» effect is most commonly observed with branded campaigns, specialized platforms, or regional campaigns in small towns. You can assess the capacity for the channel by the impression share you have received, but unfortunately, this data is not provided by all the services. In Google Adwords, you can find this data in the Impression Share metric.

4. Determination of the testing period

The model testing time depends on the conversion window, i.e., the time during which buyers pass through the funnel. This includes from the very first visit to making an order on the website. It is important to consider which advertising campaigns help your customers move through the funnel at its upper steps and require more time to accumulate value. These are the campaigns that are the most likely to be undervalued when using Last Non-Direct Click attribution.

In our experience, the optimal testing period is between 30 and 90 days. The first 75% of time is not generally the period when things are happening; this is when you’re re-allocating the budget according to the new attribution model.

5. The bottom line

When comparing the results of using different attribution models, you need to look at the changes of the overall ROAS in the segment, not the ROAS of any particular advertising campaign. The objective is the team result, with advertising campaigns complementing each other harmoniously to guide users through the funnel. Therefore, it is expected that after the allocation of the advertising budget, some campaigns may acquire a lower value, but the project as a whole will earn more. This result can’t be reached with the Last Non-Direct Click attribution where each campaign is evaluated «as is» and the «assists» are not taken into account.

Please be aware that the attribution can’t measure the undervaluation of channels if you’re not using them. For example, if you are not using media advertising or the RTB, the attribution will have nothing to evaluate. The only way to discover the value of new sources is to test them out.

Most importantly, as you attribute, so shall you receive. If you’re evaluating your advertising channels by the revenue from the transactions recorded in Google Analytics, you will optimize your advertising campaigns to work towards this goal. With that said, it’s more likely that your business goals are to make profits from online customers, with consideration of the orders made via a call center, product margins and transportation expenses. The impact of these factors on your advertising campaigns is uneven. Therefore, if you take this into account, it will also affect the allocation of your advertising budget and allow you to increase ROAS.
A simple example: the more expensive the item, the less likely it is that customers will place an order via the website. Instead, they will contact the call center or come to the store in person. If the revenue from such orders is not attributed to the advertising campaigns, such campaigns will remain undervalued, despite the fact that they may have attracted more customers for expensive items.

What’s next? It’s simple:

• Calculate the potential value  of the Funnel Based attribution.
• Set up Funnel Based attribution model with OWOX BI.
• Improve your existing budget management processes by using the new unbiased indicator to evaluate your ad campaigns.