How to monitor data quality — a detailed guide
Preventing an error in data collection is easier than dealing with its consequences. The sagacity of your business decisions depends on the quality of your data. In this article, we tell you how to check the quality of data at all stages of collection, from the statement of work to completed reports.
Want to be sure about the quality of your data? Leave it to OWOX BI. We’ll help you develop metrics and customize web analytics. With OWOX BI, you don’t need to look for connectors and clean up and process data. You’ll get ready data sets in an understandable and easy-to-use structure. Sign up for a demo and we’ll discuss how OWOX can meet your needs.
Table of contents
- The importance of testing in web analytics
- Testing documentation for data collection from the site
- Testing Google Analytics and Google Tag Manager settings
- Testing of the implementation of Google Analytics
- Tools for checking data
- Testing mobile browsers and applications
- Checking data in Google Analytics reports
- Testing automation
The importance of testing in web analytics
Unfortunately, many companies that spend substantial resources storing and processing data still make important decisions based on intuition and their own expectations instead of data.
Why does that happen? Distrust of data is exacerbated by situations where data provides an answer that’s at odds with the expectations of the decision-maker. In addition, if someone has encountered errors in data or reports in the past, they’re inclined to favor intuition. This is understandable, as a decision made on the basis of incorrect data may throw you back rather than move you forward.
Imagine you have a multi-currency project. Your analyst has set up Google Analytics in one currency, and the marketer in charge of contextual advertising has set up cost importing into Google Analytics in another currency. As a result, you have an unrealistic return on ad spend (ROAS) in your advertising campaign reports. If you don’t notice this error in time, you may either disable profitable campaigns or increase the budget on loss-making ones.
In addition, developers are usually very busy, and implementing web analytics is a secondary task for them. While implementing new functionality — for example, a new design for a unit with accessories — developers may forget to check that data is being collected in Google Analytics. As a result, when the time comes to evaluate the effectiveness of the new design, it turns out that the data collection was broken two weeks ago. Surprise.
We recommend testing web analytics data as early and as often as possible to minimize the cost of correcting an error.
Cost of correcting a mistake
Imagine you’ve made an error during the specification phase. If you find it and correct it immediately, the fix will be relatively cheap. If the error is revealed after implementation, when building reports, or even when making decisions, the cost of fixing it will be very high.
How to implement data collection
Data collection typically consists of five key steps:
- Formulate a business challenge. Say you need to assess the efficiency of an algorithm for selecting goods in a recommendations block.
- The analyst or person responsible for data collection designs a system of metrics to be tracked on the site.
- The person sets up Google Analytics and Google Tag Manager.
- The one sends terms of reference for developers to implement.
- After the developer implements metrics and sets up data collection, the analyst works with reports.
At almost all of these stages, it’s very important to check your data. It’s necessary to test technical documentation, Google Analytics and Google Tag Manager settings and, of course, the quality of data collected on your site or in your mobile application.
Features of data collection testing
Before you go to each step, let’s take a look at some requirements for data testing:
- You can’t test without tools. At a minimum, you’ll have to work with the developer console in a browser.
- There’s no abstract expected result. You need to know exactly what you should end up with. We always have a certain set of parameters we need to collect for any user interaction with a site. And we know the values these parameters should take.
- Special knowledge is necessary. At a minimum, you need to be familiar with the documentation for the web analytics tools you use, the practice, and the experience of market participants.
Testing documentation for website data collection
As we’ve mentioned, it’s much easier to correct an error if you catch it in the specifications. Therefore, checking documentation starts long before collecting data. Let’s figure out why we need to check your documentation.
Purposes of testing documentation:
- Fix mistakes with little effort. An error in documentation is just an error in written text, so all you have to do is make a quick edit.
- Prevent the need for changes in the future that may affect the site/application architecture.
- Protect the analyst’s reputation. An instrument with errors in development could call into question the competence of the person who drafted it.
Most common errors in specifications:
- Typos. A developer can copy the name of parameters without reading them. This isn’t about grammatical or spelling errors, but rather about incorrect names of parameters or values that these parameters hold.
- Ignoring fields when tracking events. For example, an error message may be ignored if a form wasn’t submitted successfully.
- Invalid field names and a mismatch with the enhanced ecommerce scheme. Implementing enhanced ecommerce with a dataLayer variable requires clear documentation. Therefore, it’s better to check all fields twice when drafting your specifications.
- You don’t have currency support for a multi-currency site. This problem is relevant to all revenue-related reports.
- Hit limits aren’t taken into account. For example, say there can be up to 30 different products on a catalog page. If we transfer information about views at the same time for all products, it’s likely that the hit in Google Analytics will not be transferred.
Testing Google Analytics and Google Tag Manager settings
The next step after you check your technical documentation is to check your Google Analytics and Google Tag Manager settings.
Why test Google Analytics and Google Tag Manager settings?
- Ensure that parameters are correctly processed by data collection systems. Google Analytics and Google Tag Manager can be configured in parallel with the implementation of metrics on your site. And until the analyst is done, the data will not appear in Google Analytics.
- Make it easier to test metrics embedded on the site. You’ll only need to concentrate on part of the developer’s work. At the final stage of X, you’ll need to look for the cause of the error directly on the site, not in the platform settings.
- Low repair cost as there’s no need to involve developers.
Most common errors in Google Analytics:
- A custom variable wasn’t created. This is especially relevant for Google Analytics 360 accounts, which can have up to 200 metrics and 200 parameters. In that case, it’s very easy to miss one.
- The specified access scope is invalid. You won’t be able to catch this error during the dataLayer review phase or by reviewing the hit you’re sending, but when you create report, you’ll see that the data doesn’t look as expected.
- You get a duplicate of an existing parameter. This error doesn’t affect the data being sent, but it may cause problems when checking and building reports.
Most common errors in Google Tag Manager:
- No parameters have been added, such as to the Universal Analytics tag or Google Analytics Settings variable.
- The index in the tag doesn’t match the parameter in Google Analytics, creating a risk that values will be passed to the wrong parameters. For example, say you specified the index of the number of users parameter in GTM for the item rating parameter. This error is likely to be immediately found when building reports, but you’ll no longer be able to influence the collected data.
- Invalid variable name specified in the dataLayer. When you create a dataLayer, be sure to specify by which name the variable will be found in the dataLayer array. If you type or write another value, this variable will never be read from the dataLayer.
- Enhanced ecommerce tracking is not enabled.
- The start trigger isn’t configured correctly. For example, the regular expression for triggering X is written incorrectly or there’s an error in the event name.
Testing the implementation of Google Analytics
The last stage of testing is testing directly on the site. This stage requires more technical knowledge because you’ll need to watch the code, check how the container is installed, and read the logs. So you need to be savvy and use the right tools.
Why test embedded metrics?
- Check that what’s implemented complies with the specifications and record any errors.
- Check whether the values to be sent are adequate. Verify that the parameters are transmitting the values to be transmitted. For example, the category of goods don’t pass its name instead.
- Give feedback to developers on the quality of implementation. Based on this feedback, developers can make changes to the site.
The most common mistakes:
- Not all scenarios are covered. For example, say an item can be added to the cart on the product, catalog, promo, or master page — that is, anywhere where there’s a link to the item. With so many entry points, you can miss something.
- The task isn’t implemented on all pages. That is, for some pages or some partition/directory, data isn’t collected at all or is only partially collected. To prevent such situations, we can draw up a checklist. In some cases, we can have as many as 100 checks for one function.
- Not all parameters are implemented; that is, the dataLayer is only partially implemented.
- The dataLayer scheme for enhanced ecommerce is broken. This is especially true for events such as adding items to the cart, moving between checkout steps, and clicking on items. One of the most common errors in implementing enhanced ecommerce is missing square brackets on the Products array.
- The dataLayer uses an empty string instead of null or undefined to zero the parameter. In this case, Google Analytics reports contain empty lines. If you use null or undefined, this option will not even be included in the hit you’re sending.
Tools for checking data
Tools we use to test data:
- Google Analytics Debugger Chrome extension
- GTM Debugger, which you can use to enable preview mode in Google Tag Manager
- The dataLayer command in the developer console
- The Network tab in the developer console
- Google Tag Assistant Chrome extension
Let’s take a closer look at these tools.
Google Analytics Debugger
To get started, you need to install this extension in your browser and enable it. Then open the page ID and go to the Console tab. The information you see is provided by the extension.
This screen shows the parameters that are transmitted with hits and the values that are transmitted for those parameters:
There’s also an extended e-commerce block. You can find it in the console as ec:
In addition, error messages are displayed here, such as for exceeding the hit size limit.
If you need to check the composition of the dataLayer, the easiest way to do this is to type the dataLayer command in the console:
Here are all the parameters that are transmitted. You can study them in detail and verify them. Each action on the site is reflected in the dataLayer. Let’s say you have seven objects. If you click on an empty field and call the dataLayer command again, an eighth object should appear in the console.
Google Tag Manager Debugger
To access Google Tag Manager Debugger, open your Google Tag Manager account and click the Preview button:
Then open your site and refresh the page. In the lower pane, a panel should appear that shows all the tags running on that page.
Events that are added to the dataLayer are displayed on the left. By clicking on them, you can check the real-time composition of the dataLayer.
Testing mobile browsers and mobile applications
Features of mobile browser testing:
- On smartphones and tablets, sites can be launched in adaptive mode or there can be a separate mobile version of the site. If you run the mobile version of a site on your desktop, it will be different from the same version on your phone.
- In general, extensions cannot be installed in mobile browsers.
- To compensate for this, you must enable Debug mode in the Universal Analytics tag or in the Google Analytics tracking code on the site.
Features of mobile application testing:
- Working with application code requires more technical knowledge.
- You’ll need a local proxy server to intercept hits. In order to keep track of the number of requests a device sends, you can filter requests by the name of the application or the host to which they’re sent.
How to use your mobile browser
- Connect your mobile device to your laptop via USB.
- Open Google Chrome on your device.
- In the Chrome developer console, open the Remote Devices report:
- Confirm the connection to your device by clicking Ok in the dialog box. Then select the tab you want to inspect and click Inspect.
- Now you can work with the developer console in standard mode, as in the browser. You will have all the familiar tabs: Console, Network, and others.
How to work with a mobile app
- To work with a mobile application, you must install and run a proxy server. We recommend Charles.
- One your proxy server is installed, check which IP address the application connects to:
- Then take your device and configure the Wi-Fi connection through the proxy server using port 8888. This is the port Charles uses by default.
- After that, it’s time to collect hits. Note that in applications, hits are not sent to collect but to batch. Batch is a packaged request that helps you send multiple requests. First, it saves application resources. Second, if there are network problems, the requests will be stored in the application and one common pool will be sent as soon as the network connection is reestablished.
- Finally, the collected data must be parsed (disassembled) into parameters, checked in order, and checked against the specifications.
Checking data in Google Analytics reports
This step is the fastest and easiest. At the same time, it makes sure the data collected in Google Analytics makes sense. In your reports, you can check hundreds of different scenarios and look at indicators depending on the device, browser, etc. If you find any anomalies in the data, you can play the script on a specific device and in a specific browser.
You can also use Google Analytics reports to check the completeness of data transferred to the dataLayer. That is, depending on each of the scenarios, the variable is filled, whether there are all parameters in it, whether the parameters take the correct values, etc.
The most useful reports
We want to share the most useful reports (in our opinion). You can use them as a data collection checklist:
- Ecommerce reports:
- Behavior — Events — Top events
- Acquisition — Campaigns — Cost Analysis
- Custom reports — For example, one that shows duplicate orders
Let’s see what these reports look like in the interface and which of these reports you need to pay attention to first.
Product Performance report
The most valuable tab in this report is Shopping Behavior. It analyzes the completeness of data collection at each stage of enhanced ecommerce. That is, we can see if Google Analytics transfers product list views, clicks, product detail views, addition/deletion of products to/from the basket, and the purchases themselves.
What should we pay attention to here? First, it’s very strange if you have zero values in any of the columns. Second, if you have more values at some stage than at the previous stage, you’re likely to have problems collecting data. For example, say the number of unique purchases of an item is greater than the number of checkouts. That’s weird and it’s worth paying attention to.
You can also switch between other parameters in this report, which should also be sent to Enhanced Ecommerce. For example, if you select Item Category as the main option, you may see there are sales for certain categories of items but there are no views for these items, no adds to the cart, etc.
Top Events report
First of all, it’s necessary to walk through all parameters that are transmitted to Google Analytics and see what values each parameter takes. Usually it’s immediately clear whether everything is okay. More detailed analysis for each of the events can be carried out in custom reports.
Cost Analysis report
Another standard report that can be useful for checking the importing of expense data into Google Analytics is Cost Analysis.
We often see reports where there are expenses for some source or advertising campaign but there are no sessions. This can be caused by problems or errors in UTM tags. Alternatively, filters in Google Analytics may exclude sessions from a particular source. These reports need to be checked from time to time.
We would like to highlight the custom report that allows you to track duplicate transactions. It’s very easy to set up: the parameter must be a transaction ID and the key dimension must be transactions.
Note that when there’s more than one transaction in the report, this means that information about the same order was sent more than once.
If you find a similar problem, read these detailed instructions on how to fix it.
Learn more about what to pay attention to when configuring web analytics and which reports to use for verifying data quality in our post on how to conduct an audit of website analytics.
Automatic email alerts
Google Analytics has a very good Custom Alerts tool that allows you to track important changes without viewing reports. For example, if you stop collecting information about Google Analytics sessions, you can receive an email notification.
We recommend that you set up notifications for at least these four metrics:
- Number of sessions
- Bounce rate
- Number of transactions
To set up notifications, see our post on automating reports in Google Analytics.
In our experience, this is the most difficult and time-consuming task — the narrow line where mistakes are the most common.
To avoid problems with dataLayer implementation, checks must be done at least once a week. In general, the frequency should depend on how often you implement changes on the site. Ideally, you need to test the dataLayer after each significant change. It’s time-consuming to do this manually, so we decided to automate the process.
Why automate testing?
To automate testing, we’ve built a cloud-based solution that enables us to:
- Check whether the dataLayer variable on the site matches the reference value
- Check the availability and functionality of Google Tag Manager code
- Check that data is sent to Google Analytics and OWOX BI
- Collect error reports in Google BigQuery
Advantages of test automation:
- Significantly increase the speed of testing. In our experience, you can test thousands of pages in a few hours.
- Get more accurate results, since the human factor is excluded.
- Lower the cost of testing, as you need fewer specialists.
- Increase the frequency of testing, as you can run tests after each change to the site.
A simplified scheme of the algorithm we use:
When you sign in to our app, you need to specify the pages you want to verify. You can do this by uploading a CSV file, specifying a link to the sitemap, or simply specifying a site URL, in which case the application will find the sitemap itself.
Then it’s important to specify the dataLayer scheme for each scenario to be tested: pages, events, scripts (a sequence of actions, such as for checkout). Then you can use regular expressions to specify that the page types match the URL.
After receiving all this information, our application runs through all pages and events as scheduled, checks each script, and uploads test results to Google BigQuery. Based on this data, we set up email and Slack notifications.
You can read more about how automatic website metric testing works in our case study about OZON.ru.
P. S. If you need a full website audit, you can request consulting services from OWOX BI. Sign up for a demo and we’ll discuss the possibilities.