How to monitor data quality — a detailed guide

Preventing an error in data collection is easier than dealing with its consequences. The sagacity of your business decisions depends on the quality of your data. In this article, we tell you how to check the quality of data at all stages of collection, from the statement of work to completed reports.

Unfortunately, many companies that spend substantial resources storing and processing data still make important decisions based on intuition and their own expectations instead of data.

Why does that happen? Distrust of data is exacerbated by situations where data provides an answer that’s at odds with the expectations of the decision-maker. In addition, if someone has encountered errors in data or reports in the past, they’re inclined to favor intuition. This is understandable, as a decision made on the basis of incorrect data may throw you back rather than move you forward.

Imagine you have a multi-currency project. Your analyst has set up Google Analytics in one currency, and the marketer in charge of contextual advertising has set up cost importing into Google Analytics in another currency. As a result, you have an unrealistic return on ad spend (ROAS) in your advertising campaign reports. If you don’t notice this error in time, you may either disable profitable campaigns or increase the budget on loss-making ones.

In addition, developers are usually very busy, and implementing web analytics is a secondary task for them. While implementing new functionality — for example, a new design for a unit with accessories — developers may forget to check that data is being collected in Google Analytics. As a result, when the time comes to evaluate the effectiveness of the new design, it turns out that the data collection was broken two weeks ago. Surprise.

Imagine you’ve made an error during the specification phase. If you find it and correct it immediately, the fix will be relatively cheap. If the error is revealed after implementation, when building reports, or even when making decisions, the cost of fixing it will be very high.

Data collection typically consists of five key steps:

1. Formulate a business challenge. Say you need to assess the efficiency of an algorithm for selecting goods in a recommendations block.
2. The analyst or person responsible for data collection designs a system of metrics to be tracked on the site.
3. The person sets up Google Analytics and Google Tag Manager.
4. The one sends terms of reference for developers to implement.
5. After the developer implements metrics and sets up data collection, the analyst works with reports.

At almost all of these stages, it’s very important to check your data. It’s necessary to test technical documentation, Google Analytics and Google Tag Manager settings and, of course, the quality of data collected on your site or in your mobile application.

Before you go to each step, let’s take a look at some requirements for data testing:

• You can’t test without tools. At a minimum, you’ll have to work with the developer console in a browser.
• There’s no abstract expected result. You need to know exactly what you should end up with. We always have a certain set of parameters we need to collect for any user interaction with a site. And we know the values these parameters should take.
• Special knowledge is necessary. At a minimum, you need to be familiar with the documentation for the web analytics tools you use, the practice, and the experience of market participants.

As we’ve mentioned, it’s much easier to correct an error if you catch it in the specifications. Therefore, checking documentation starts long before collecting data. Let’s figure out why we need to check your documentation.

Purposes of testing documentation:

• Fix mistakes with little effort. An error in documentation is just an error in written text, so all you have to do is make a quick edit.
• Prevent the need for changes in the future that may affect the site/application architecture.
• Protect the analyst’s reputation. An instrument with errors in development could call into question the competence of the person who drafted it.

Most common errors in specifications:

1. Typos. A developer can copy the name of parameters without reading them. This isn’t about grammatical or spelling errors, but rather about incorrect names of parameters or values that these parameters hold.
2. Ignoring fields when tracking events. For example, an error message may be ignored if a form wasn’t submitted successfully.
3. Invalid field names and a mismatch with the enhanced ecommerce scheme. Implementing enhanced ecommerce with a dataLayer variable requires clear documentation. Therefore, it’s better to check all fields twice when drafting your specifications.
4. You don’t have currency support for a multi-currency site. This problem is relevant to all revenue-related reports.
5. Hit limits aren’t taken into account. For example, say there can be up to 30 different products on a catalog page. If we transfer information about views at the same time for all products, it’s likely that the hit in Google Analytics will not be transferred.

The next step after you check your technical documentation is to check your Google Analytics and Google Tag Manager settings.

Why test Google Analytics and Google Tag Manager settings?

• Ensure that parameters are correctly processed by data collection systems. Google Analytics and Google Tag Manager can be configured in parallel with the implementation of metrics on your site. And until the analyst is done, the data will not appear in Google Analytics.
• Make it easier to test metrics embedded on the site. You’ll only need to concentrate on part of the developer’s work. At the final stage of X, you’ll need to look for the cause of the error directly on the site, not in the platform settings.
• Low repair cost as there’s no need to involve developers.

Most common errors in Google Analytics:

1. A custom variable wasn’t created. This is especially relevant for Google Analytics 360 accounts, which can have up to 200 metrics and 200 parameters. In that case, it’s very easy to miss one.
2. The specified access scope is invalid. You won’t be able to catch this error during the dataLayer review phase or by reviewing the hit you’re sending, but when you create report, you’ll see that the data doesn’t look as expected.
3. You get a duplicate of an existing parameter. This error doesn’t affect the data being sent, but it may cause problems when checking and building reports.

Most common errors in Google Tag Manager:

1. No parameters have been added, such as to the Universal Analytics tag or Google Analytics Settings variable.
2. The index in the tag doesn’t match the parameter in Google Analytics, creating a risk that values will be passed to the wrong parameters. For example, say you specified the index of the number of users parameter in GTM for the item rating parameter. This error is likely to be immediately found when building reports, but you’ll no longer be able to influence the collected data.
3. Invalid variable name specified in the dataLayer. When you create a dataLayer, be sure to specify by which name the variable will be found in the dataLayer array. If you type or write another value, this variable will never be read from the dataLayer.
4. Enhanced ecommerce tracking is not enabled.
5. The start trigger isn’t configured correctly. For example, the regular expression for triggering X is written incorrectly or there’s an error in the event name.

The last stage of testing is testing directly on the site. This stage requires more technical knowledge because you’ll need to watch the code, check how the container is installed, and read the logs. So you need to be savvy and use the right tools.

Why test embedded metrics?

• Check that what’s implemented complies with the specifications and record any errors.
• Check whether the values to be sent are adequate. Verify that the parameters are transmitting the values to be transmitted. For example, the category of goods don’t pass its name instead.
• Give feedback to developers on the quality of implementation. Based on this feedback, developers can make changes to the site.

The most common mistakes:

1. Not all scenarios are covered. For example, say an item can be added to the cart on the product, catalog, promo, or master page — that is, anywhere where there’s a link to the item. With so many entry points, you can miss something.
2. The task isn’t implemented on all pages. That is, for some pages or some partition/directory, data isn’t collected at all or is only partially collected. To prevent such situations, we can draw up a checklist. In some cases, we can have as many as 100 checks for one function.
3. Not all parameters are implemented; that is, the dataLayer is only partially implemented.
4. The dataLayer scheme for enhanced ecommerce is broken. This is especially true for events such as adding items to the cart, moving between checkout steps, and clicking on items. One of the most common errors in implementing enhanced ecommerce is missing square brackets on the Products array.
5. The dataLayer uses an empty string instead of null or undefined to zero the parameter. In this case, Google Analytics reports contain empty lines. If you use null or undefined, this option will not even be included in the hit you’re sending.

Tools we use to test data:

• Google Analytics Debugger Chrome extension
• GTM Debugger, which you can use to enable preview mode in Google Tag Manager
• The dataLayer command in the developer console
• The Network tab in the developer console
• Google Tag Assistant Chrome extension

Let’s take a closer look at these tools.

To get started, you need to install this extension in your browser and enable it. Then open the page ID and go to the Console tab. The information you see is provided by the extension.

This screen shows the parameters that are transmitted with hits and the values that are transmitted for those parameters:

There’s also an extended e-commerce block. You can find it in the console as ec:

In addition, error messages are displayed here, such as for exceeding the hit size limit.

If you need to check the composition of the dataLayer, the easiest way to do this is to type the dataLayer command in the console:

Here are all the parameters that are transmitted. You can study them in detail and verify them. Each action on the site is reflected in the dataLayer. Let’s say you have seven objects. If you click on an empty field and call the dataLayer command again, an eighth object should appear in the console.

To access Google Tag Manager Debugger, open your Google Tag Manager account and click the Preview button:

Then open your site and refresh the page. In the lower pane, a panel should appear that shows all the tags running on that page.

Events that are added to the dataLayer are displayed on the left. By clicking on them, you can check the real-time composition of the dataLayer.

Features of mobile browser testing:

• On smartphones and tablets, sites can be launched in adaptive mode or there can be a separate mobile version of the site. If you run the mobile version of a site on your desktop, it will be different from the same version on your phone.
• In general, extensions cannot be installed in mobile browsers.
• To compensate for this, you must enable Debug mode in the Universal Analytics tag or in the Google Analytics tracking code on the site.

Features of mobile application testing:

• Working with application code requires more technical knowledge.
• You’ll need a local proxy server to intercept hits. In order to keep track of the number of requests a device sends, you can filter requests by the name of the application or the host to which they’re sent.
• All hits are collected in Measurement Protocol format and require additional processing. Once hits have been collected and filtered, they must be copied and parsed into parameters. You can use any convenient tool to do this: Hit Builder, formulas in Google Sheets, or a JavaScript or Python app. It all depends on what’s more convenient for you. Plus, you’ll need knowledge of Measurement Protocol parameters to identify errors in the sent hits.

1. Connect your mobile device to your laptop via USB.
2. Open Google Chrome on your device.
3. In the Chrome developer console, open the Remote Devices report:

1. Confirm the connection to your device by clicking Ok in the dialog box. Then select the tab you want to inspect and click Inspect.
2. Now you can work with the developer console in standard mode, as in the browser. You will have all the familiar tabs: Console, Network, and others.

1. To work with a mobile application, you must install and run a proxy server. We recommend Charles.
2. One your proxy server is installed, check which IP address the application connects to:

3. Then take your device and configure the Wi-Fi connection through the proxy server using port 8888. This is the port Charles uses by default.

4. After that, it’s time to collect hits. Note that in applications, hits are not sent to collect but to batch. Batch is a packaged request that helps you send multiple requests. First, it saves application resources. Second, if there are network problems, the requests will be stored in the application and one common pool will be sent as soon as the network connection is reestablished.

5. Finally, the collected data must be parsed (disassembled) into parameters, checked in order, and checked against the specifications.

This step is the fastest and easiest. At the same time, it makes sure the data collected in Google Analytics makes sense. In your reports, you can check hundreds of different scenarios and look at indicators depending on the device, browser, etc. If you find any anomalies in the data, you can play the script on a specific device and in a specific browser.

You can also use Google Analytics reports to check the completeness of data transferred to the dataLayer. That is, depending on each of the scenarios, the variable is filled, whether there are all parameters in it, whether the parameters take the correct values, etc.

We want to share the most useful reports (in our opinion). You can use them as a data collection checklist:

• Ecommerce reports:
  • Product Performance
  • Product List Performance
  • Internal Promotion
• Behavior — Events — Top events
• Acquisition — Campaigns — Cost Analysis
• Custom reports — For example, one that shows duplicate orders

Let’s see what these reports look like in the interface and which of these reports you need to pay attention to first.

The most valuable tab in this report is Shopping Behavior. It analyzes the completeness of data collection at each stage of enhanced ecommerce. That is, we can see if Google Analytics transfers product list views, clicks, product detail views, addition/deletion of products to/from the basket, and the purchases themselves.

What should we pay attention to here? First, it’s very strange if you have zero values in any of the columns. Second, if you have more values at some stage than at the previous stage, you’re likely to have problems collecting data. For example, say the number of unique purchases of an item is greater than the number of checkouts. That’s weird and it’s worth paying attention to.

You can also switch between other parameters in this report, which should also be sent to Enhanced Ecommerce. For example, if you select Item Category as the main option, you may see there are sales for certain categories of items but there are no views for these items, no adds to the cart, etc.

First of all, it’s necessary to walk through all parameters that are transmitted to Google Analytics and see what values each parameter takes. Usually it’s immediately clear whether everything is okay. More detailed analysis for each of the events can be carried out in custom reports.

Another standard report that can be useful for checking the importing of expense data into Google Analytics is Cost Analysis.

We often see reports where there are expenses for some source or advertising campaign but there are no sessions. This can be caused by problems or errors in UTM tags. Alternatively, filters in Google Analytics may exclude sessions from a particular source. These reports need to be checked from time to time.

We would like to highlight the custom report that allows you to track duplicate transactions. It’s very easy to set up: the parameter must be a transaction ID and the key dimension must be transactions.

Note that when there’s more than one transaction in the report, this means that information about the same order was sent more than once.

If you find a similar problem, read these detailed instructions on how to fix it.

Google Analytics has a very good Custom Alerts tool that allows you to track important changes without viewing reports. For example, if you stop collecting information about Google Analytics sessions, you can receive an email notification.

We recommend that you set up notifications for at least these four metrics:

• Number of sessions
• Bounce rate
• Revenue
• Number of transactions

In our experience, this is the most difficult and time-consuming task — the narrow line where mistakes are the most common.

To avoid problems with dataLayer implementation, checks must be done at least once a week. In general, the frequency should depend on how often you implement changes on the site. Ideally, you need to test the dataLayer after each significant change. It’s time-consuming to do this manually, so we decided to automate the process.

To automate testing, we’ve built a cloud-based solution that enables us to:

• Check whether the dataLayer variable on the site matches the reference value
• Check the availability and functionality of Google Tag Manager code
• Check that data is sent to Google Analytics and OWOX BI
• Collect error reports in Google BigQuery

Advantages of test automation:

• Significantly increase the speed of testing. In our experience, you can test thousands of pages in a few hours.
• Get more accurate results, since the human factor is excluded.
• Lower the cost of testing, as you need fewer specialists.
• Increase the frequency of testing, as you can run tests after each change to the site.

A simplified scheme of the algorithm we use:

When you sign in to our app, you need to specify the pages you want to verify. You can do this by uploading a CSV file, specifying a link to the sitemap, or simply specifying a site URL, in which case the application will find the sitemap itself.

Then it’s important to specify the dataLayer scheme for each scenario to be tested: pages, events, scripts (a sequence of actions, such as for checkout). Then you can use regular expressions to specify that the page types match the URL.

After receiving all this information, our application runs through all pages and events as scheduled, checks each script, and uploads test results to Google BigQuery. Based on this data, we set up email and Slack notifications.

P. S. If you need a full website audit, you can request consulting services from OWOX BI. Sign up for a demo and we’ll discuss the possibilities.