How not to Ruin All Your Web Metrics and Save 1.5 Burden Hours after Every New Release
Imagine you’ve recently added a new filter to your online store search, and want to measure its efficiency. When it comes to calculations, you realize that you’ve ruined the already existing metrics when implementing the new feature on your site. Sound familiar? Then you definitely know that you’ll now have to double check every single parameter and the data sent to your analytics services, losing your precious time and money.
Our cooperation with OZON.ru started when the company actually faced a similar challenge, realizing that they spend a lot of time and resources each time they release a new feature. OWOX BI suggested automating the process of the web metric testing.
Founded in 1998, OZON.ru is one of the largest Russia’s online stores, selling over 23 categories of products: books, software, home and garden products, clothes, footwear, consumer electronics, sports goods, cosmetics, car and motoring products, pet food, etc.
To order products, OZON’s customers can use the website, the mobile app, choosing the courier delivery option or collecting orders from the pickup points (over 4 thousand in 137 cities). More than 3 million customers buy from OZON.ru annually, and over 1 million people visit the company’s site every day.
With over 5 million items in their product range, OZON.ru regularly updates its website, improving product cards, favorites, and so on. Thus, the company needs to implement new metrics and test user behavior scenarios, to properly measure the efficiency of the recent updates.
Each time a new feature was released, the company’s specialists had to manually test the new metrics and make sure that all the data is properly collected in Google Analytics. Moreover, there was always a risk to ruin the metrics that had been already implemented. For large projects like OZON.ru, all of the aforementioned issues could seriously influence the decisions taken, leading to money and time losses.
Manual tests of the implemented and new metrics consumed too much of time and efforts. Each time a new release came out, developers, testers, and analysts needed to check if everything was okay with the info in dataLayer, and if the transactions or add-to-cart events worked fine. This could take 0.15 to 2 hours, depending on the release.
To reduce risks when adding new functions to the website, the company decided to cut the time and money expenses and implement automatic tests for the website metrics.
Analysts from OWOX BI had already had experience in creating automated tests for site metrics for big Ecommerce projects, so they suggested a three-staged solution for OZON.ru:
- Collect data for testing.
- Set up testing and process data according to the testing results.
- Build reports and use the obtained data.
Let’s give in some more details about each of the stages.
1. Collect information for testing
OZON.ru keeps regularly adding new features to its website, that’s why there are more and more metrics to test. The company’s analysts created a table in Google Sheets with all of the important dimensions and metrics. This way they can update or edit it any time necessary, and let the other colleagues know what KPIs are tracked on the website.
The OZON.ru analysts started the table with adding the dimensions and metrics that are sent to Google Analytics and are used in reports on sales performance and micro conversions. These very dimensions and metrics are used for evaluating the main business KPIs and for exchanging data with ad services, influencing the decisions made by the CMO and analysts.
As a result, OZON.ru got three data sets for testing:
- Page data
- Event data
- User scenario data.
We’ll have a closer look at the info in each of the data sets.
1.Page data set contains the following info:
- Definition — name of the parameter in dataLayer, like clientID, pageType, categoryName, prodName, etc.
- Type — numeric or string parameter.
- False pattern — the format that parameter turns in on the pages where it’s not filled.
- True pattern — the format that parameter turns in on the pages where it’s filled.
- Rules for testing — description of the type of page and user (authenticated or not) the value for which should or shouldn’t be sent. For example, the TotalOrderCount parameter is only sent for authenticated users, and there should be no value sent if a user is not authenticated.
Below is the sample of such a data set:
2. Event data set includes this info:
- Event name — for instance, product view, click on the product, adding a product to the cart.
- Parameter — to test action, list, id, position, name, price, etc.
- Description — for the acceptable parameter values, like any value greater than 0.
- Type — numeric or string parameter.
- Comment — required or not required parameter
- Page Type — for testing if there are any parameters on a certain page.
Here’s an example of such a data set:
3. User scenario data set that contains such info:
- Scenario name, for example, new user — online payment or new user — offline payment.
- Event — a set of events in a scenario.
- Parameters to test, for example: ecommerce.purchase.actionField.action (id, revenue, affiliation), ecommerce.purchase.products.id, ecommerce.purchase.products.name, ecommerce.purchase.products.price, etc.
You can find an example of the aforementioned data set below:
Please note, that the list of dimensions and metrics in all the data sets can be changed or updated at any time.
2. Set up testing and process data
Once the analysts from OZON.ru provided the list of metrics and dimensions, the OWOX BI team created and set up a script to check if the variables in dataLayer on site pages match the ones provided in the list. The testing script can be launched regularly at any time necessary, and OZON.ru decided to run it every day.
Here’s how the script works:
- It imitates user action on the website, like opening a page, clicking on the product, adding it to the cart, and so on.
- Next, it checks if all the received parameters in DataLayer match the ones that are sent to Google Analytics after the user actions.
- Finally, it imports the testing results to Google BigQuery for creating error reports.
The OWOX BI analysts used Google BigQuery to collect data because this cloud warehouse meets the highest security standards. Moreover, with the BigQuery set of connectors and the data export to Google Sheets, OZON.ru can comfortably conjure up the necessary information on the dashboards or in tables.
The testing results are saved in a Google BigQuery table with the following structure:
- Time for testing
- Page for testing
- PageType for the page tested
- Status of testing (Ok or Error)
- Type of data tested (pageview, event or scenario)
- EventName to define the event tested
- ErrorDescription to give more details about the error.
3. Build reports and use the obtained data
To easily use the testing results for work, analysts export data from Google BigQuery to Data Studio. The OWOX BI specialists prepared a dashboard for OZON.ru to view the following info:
- Tested Pages, Events, and Scenarios — the exact number.
- Share of Pages/Events/Scenarios with Error — the percentage of completed tests with errors across pages, events, and scenarios.
Details about errors:
- Internal — script errors
- Redirect — sitemap errors
- Implementation — dataLayer errors.
Below you can see a dashboard with the testing results, with filters to apply to the obtained info. For example, you can find out about the errors on the catalog page, by using such filters: Page Type = catalog list, Test type = pageview, Error Type = Implementation.
From the graph of changes in test launch above, you can see that starting with January 16th there appeared much more tests with the errors due to the redesign of the product card (DetailProductView). The reason for that was that analysts hadn’t considered that the banner “Name1” was deleted in the updated site version. Therefore, when testing the click on the product card banner, there was an error with the text “ValueError — no elements to click.”
Let’s have a look at another dashboard example. We can understand that the parameters “position” and “list” were not sent in the productView event. It happened because the position parameter disappeared from dataLayer, while the list and name parameters were confused. Therefore, the names for the product lists were sent to the wrong parameter.
Now the OZON.ru analysts can check if the important KPIs are tracked properly and at any time they need. This allows analysts to fix errors quickly. The OWOX BI team has also set up automatic emails to provide OZON.ru with short reports on errors across page types and events, on a daily basis.
Here’s the example of such a letter:
If needed, OZON’s analysts send this info to developers for bug fixing.
Developers and analysts of OZON.ru now run automated tests for website metrics every time the site is updated. This solution from OWOX BI allowed the company to:
- Significantly increase testing speed and accuracy, excluding the human factor.
- Cut the testing costs. Thanks to this, OZON.ru runs tests on a daily basis, saving around 1.5 burden hours with every new release.
- Discover the missed out 2,700 deficiencies on 37 types of pages. Some of the recommendation blocks sent the wrong values of product prices or simply sent incorrect values. Reports also showed that the catalogs could place unavailable products on first positions. The redirect functions didn’t work properly, sending users to pages that don’t exist.
Automated tests help us avoid the manual regressive testing of all the parameters and events that were not directly changed in new releases. This also allows us to save 1.5 burden hours after each release. To describe the testing scenario, we set specific parameters that are supposed to send data to Google Analytics but don’t yet, as they were not implemented by our developers. Monitoring allows our IT specialists solve such issues.
Moreover, we start auto tests according to certain scenarios with our QA department to find the errors before we release new features on our website and to prevent possible losses."
We hope that you’re also certain about the accuracy of the data you collect. The OWOX BI analysts prepared a template of the dashboard we described in the success story. You can use it to improve the quality of the data you collect on your website. To download the dashboard, simply fill in a short form, and we’ll send you the link to the dashboard.
Feel free to configure the dashboard as you please, by setting the BigQuery table as the data source. Below is the structure, necessary for the table:
If you need any help with collecting error data, kindly email to email@example.com, and we’ll gladly answer your questions. To get more info about automated testing, you can watch our webinar Better Safe than Sorry: How to Monitor the Quality of Data.