How to avoid sampling and collect complete data for advanced analytics

Have you ever wondered how to avoid sampling and limits in the Google Analytics standard reports? How can you get complete data for marketing reporting? There are several solutions, such as:

• Switch to paid Google Analytics 360 and set up the export of raw data to Google BigQuery (GBQ).
• Use Google Analytics App+WEB. The product is still in beta and allows you to download user behavior data to Google BigQuery for free.
• Use other services that collect data from the website to Google BigQuery.

Each of these solutions has its specialities and limitations. In the article, we explain these features in detail and describe how to get around with OWOX BI.

The main limitation of Google Analytics 360 is its high cost. The most recent user actions appear in the BigQuery tables in more than 15 minutes. To set such data refresh rate, you’ll have to pay additionally: by default, the sessions table is updated three times a day.

In some situations, you have to pay extra for processing requests in the Google BigQuery. The export table from GA 360 contains many nesting levels and fields that can be blank in 80% of cases. The more nesting levels are in the table, the more expensive the queries are. If you have many reports to update daily or weekly, data processing will cause most of your GBQ costs.

App+Web collects data in another structure than standard Google Analytics. If in GA, everything is based on sessions, in App+Web — on users and events. It could be a problem for many analysts and marketers who got used to GA — they have to explore a new data structure, remake the metrics system for the website and rebuild existing reports. Other services that collect data from the website to GBQ in their own structure may also have this problem.

As export from GA 360 and App+Web to Google BigQuery is a representation of data collected into Google Analytics, it doesn’t provide a retrospective update of records. For example, a purchase return will be typically recorded as a separate session that won’t be associated with a payment session. So the correct revenue from this order must be recalculated manually.

None of the mentioned solutions can transfer your ad campaign costs at the session level into the Google BigQuery. It means you won’t know how much you’ve spent attracting a specific segment of users. You cannot calculate ROI and DRR for different cohorts, product groups, and landing pages.

Knowing these limitations, we at OWOX decided to improve raw data processing experience for any kind of business. We conducted hundreds of product interviews and received the following requests:

1. Website data should be completely collected into Google BigQuery in a non-aggregated form.
2. User behavior data must be delivered to Google BigQuery in near real-time mode.
3. Session tables in the Google BigQuery should contain advertising costs data.
4. The table structure must be compact and as full as possible to use data queries optimally.
5. Data must be synchronized by key parameters in the historical period.
6. There must be a possibility to compare GBQ data with GA data if significant discrepancies appear.

We’ve considered all these requests and developed our own website data streaming into Google BigQuery.

Let’s take a closer look at how OWOX BI bypasses restrictions of Google Analytics and standard BigQuery export and how business can benefit from it.

The idea of sampling is to build reports on samples of the data, not the whole volume of available data. If the sample is too small, you can’t trust a report built on such a sample. With OWOX BI, you can collect complete unsampled data to create more accurate reports and make data-informed decisions.

Each user action on the website (page view, click on the button, banner view, or placing order) is transmitted to GA by a separate message — hit. You should consider that there are no restrictions on the number of hits at the moment of data collection.

The GA interface applies a hit limit to data processing and reporting. Thus, you get either data sampling or aggregated tables (row other in the report). Besides, if you want to use a segment or create a manual report, it is mostly displayed in a sampled form.

The service copies the hit content when it departures from your website and then sends it to your project in Google BigQuery. Thus, you get raw unsampled data with unlimited collected and processed hits.

Let’s say you are forced to collect promo page traffic, advertising partners data, or landing page statistics for each brand/manufacturer into separate Google Analytics properties to collect more valuable data. Anyway, to meet the GA limitations, you have to turn off traffic that is valuable to you. When you can collect this data in a single table in Google BigQuery by the means of OWOX BI. It provides you a basis for strategic solutions and a comprehensive vision of the website and advertising campaign effectiveness.

With OWOX BI, you can quickly send a triggered email or detect an anomaly in collected website data in Google BigQuery because user behavioral data appears in your project within 1-5 minutes.

All daily collected data reflects in GA only after 48 hours (in GA 360 — after 4 hours). Real-time reports display a limited number of parameters and metrics and don’t allow you to personalize user actions.

The standard export from GA 360 to GBQ can operate in two modes:

• Upload data once every 8 hours roughly for free (you pay for a GA 360 license and data storage in GBQ).
• Upload data every 15 minutes (you have to pay extra for every terabyte processed).

App+WEB creates two tables:

• Transitional — it contains event records for the current day. The table is updated every 15 minutes and deleted after the full-day table is generated.
• Table with data for the full day — it’s generated once a day.

OWOX BI sends data immediately from the website to your GBQ project. You receive complete non-aggregated user behavioral data within 1-5 minutes after the event occurs.

By monitoring user activity on the website in real time, you can stimulate sales with trigger email campaigns and personal offers. For example, when a user added an item to a shopping cart but didn’t buy anything or didn’t sign the deposit on the website.

It can also help you to detect and correct problems on the website in time: you can get notifications from triggers on website code errors and traffic or conversions decrease.

With OWOX BI, you get a broad picture of user activity on your website, even if they make orders consisting of 15+ items, or browse large lists of items.

There is a limit in GA on the size of the transferred hit — 8 KB. For example, if a transaction message exceeds this limit, it doesn’t get into GA.

In OWOX BI, the maximum size of hit transferred to Google BigQuery is increased to 16 KB.

A hit can exceed 8 KB containing information about a product list, when you’re tracking user’s browsing of catalog pages with Enhanced Ecommerce. It happens if:

• More than 10 items are sent in one hit;
• Each item has an additional parameter.

In this case, packed hit won’t get into GA. However, you can find it in the raw data in GBQ and build a more accurate report on the viewed product lists thanks to OWOX BI.

Let’s say you want to collect more information about the customer and the product itself, along with bank product requests. With OWOX BI, you have more accurate conversion statistics.

OWOX BI calculates the value of each session. It allows you to calculate ROAS for new and returned users. Also, you can do the following actions:

• Compare the profitability of cohorts created of those customers who saw the banner and those who didn’t.
• Find out how much you spend and earn on each of the product groups.
• Evaluate the effectiveness of advertising for different regions, landing pages, mobile versions, and applications.

You can manually load ad costs from different advertising services into GA. However, you don’t know how much you’ve spent attracting a particular customer or user segment because cost data in GA is aggregated and linked to advertising campaigns.

App+WEB also doesn’t allow you to transfer ad campaign costs at the session/user level into the Google BigQuery.

OWOX BI automatically uploads ad costs to Google Analytics and Google BigQuery distributing them across sessions according to UTM tags. This is why you know the cost of each session across 5 main UTM tags (source, medium, campaign, keyword, content), and you can build reports on CPC, CPA, CPO, and CRR on raw data. It improves your report accuracy and helps to allocate your marketing budget more efficiently.

With OWOX BI, you can find out how much you spent on each session, and group costs and revenue by certain user, cohort, or landing page.

For example, you want to understand which groups of goods should be promoted to bring more profit because a product that attracts a user isn’t obligatory the same product that user buys. You need to group your digital ad costs into categories to find the most effective product categories to promote. This can be done with OWOX BI by transeffing cost data from GA to GBQ and distributing costs across all sessions with UTM tags. In this case, both costs and income will be properties of the session.

With OWOX BI, there’s no need for any additional actions to evaluate order redemption, purchase returns, or to find out what a new subscriber has been doing on your site for the last three months. Our service allows you to retrospectively update the data uploaded into Google BigQuery.

In the standard GA, you can only retrospectively update the order status if less than 4 hours have passed from making the order to the payment. Otherwise, the payment is recorded in GA as a new session. For example, if you add a return on April 3 for the transaction made on March 18, in the report for March 18, you would still see the full transaction sum. Accordingly, the distribution of the value across the channels won’t be correct because of this mistake.

Measurement Protocol is currently not available on App+Web. It prevents you from retrospective updating of the order status or downloading user information.

In the tables that OWOX BI fills up with data in Google BigQuery, you can retrospectively update the following information:

• Transaction details — you can enter a return, partial return, or change the sum for a 30 days period.
• User Information — if a user has signed or logged into a website, his User ID will be rewinded and appointed for sessions that appeared during the last 30 days before the date of authentication.
• Cost data for 90 days (in the adCost, attributedAdCost fields), including costs from all advertising services imported into Google Analytics.

If the user with User ID has applied for credit today, this User ID appears in all previous sessions from the same device. You can use these sessions to estimate a user’s path to your order, to attribute revenue from advertising campaigns in a more accurate way, and to predict the conversion of other users.

With OWOX BI, it’s easier for you to consider returns and measure the effectiveness of advertising campaigns —- streaming tables always show you the current order sum. If the return was on another day, you don’t need to search for it in the other day’s tables.

If you plan to build a ROPO (Research Online Purchase Offline) report, it’s easier to make calculations when the User ID is appointed to the maximum number of client’s historical sessions.

As mentioned above, the standard export table from Google Analytics 360 to Google BigQuery has too many nesting and empty fields, which increases the processing cost of this data.

OWOX BI helps you optimize costs of data storage and processing in GBQ. There are fewer blank fields in the streaming tables, fewer nesting levels, so the tables themselves and their queries “weigh” less.

OWOX BI helps you to avoid GA limitations on data collection. If you plan to track all user activity — from viewing the banner to selecting additional services in the order — all additional hits can be sent directly to Google BigQuery.

All information about the user interaction with your website is transmitted to Google Analytics through separate hits — messages to the GA server.

At the moment of hit creation, OWOX BI replaces the address for sending data from the GA resource with the address of your project in Google BigQuery. We can duplicate all GA-addressed hits into GBQ tables or block (fully or partially) all data in GA and send hits only to GBQ tables.

This solution has following advantages:

1. You get unsampled data from the website in the well-known structure of Google Analytics 360, with thousands of SQL queries that have already been written for it. It saves your time and decreases time spent on regular analysts consulting.
2. You can set up data collection without lowering your website’s load speed. It’s possible because OWOX BI duplicates GA hits rather than overloading your GTM with a separate JS counter.
3. We guarantee SLA in the contract and have the necessary functionality to monitor data quality and automatically store information if your Google Analytics and Google Cloud project fails. You can be sure of the data quality for your reports without additional efforts.

With OWOX BI, it’s also possible to collect user behavior data without the Google Analytics JS counter. We can collect hit data from your website into the Google BigQuery with our tracking code.

If you plan to collect sensitive data from the website (customer type, details, order parameters), you should send it directly to the Google BigQuery where the Tier 4 security standard regulates information protection. Any sensitive information may be transmitted in hashed or encrypted form.

Let’s say there are more than 1,000 banners on your site. You plan to track views and clicks for each of them and measure how it affected the purchase in the current or following sessions. If you track the actual views of each banner, the amount of data that is sent to GA will exceed the limit by 20-25%.

With OWOX BI, you can send all banner statistics directly to Google BigQuery (bypassing GA) and get insights about the effectiveness of internal promo campaigns. So you don’t lose any information due to data sampling and aggregation in standard GA, and going beyond the limits in GA 360.

With OWOX BI, you can transfer an unlimited number of user parameters and dimensions from the website to Google BigQuery. It allows you to segment users by any feature and build more detailed reports for in-depth analysis.

Google Analytics also has restrictions on the number of custom metrics collected from the website: 20 for the standard version and 200 for the GA 360. App+Web has quite tight limits too: 500 events, 25 event parameters, and 25 user properties.

OWOX User ID helps you to optimize your promotion and remarketing costs if you have a network of websites with intersecting audiences. You also can evaluate the Post-View effect of banner and media ads.

With the help of third party cookies, OWOX BI puts the same identifier on different websites (it can be your other website on a different domain or media advertising on an independent advertising platform). Then this ID brings together sessions of several websites and you’ll be able to analyze your intersecting audiences.

If your group of companies consists of multiple brands and their websites, OWOX BI can measure the intersection of your audiences, and show you how to optimize the budget for remarketing and outreach campaigns.

In our experience, the accuracy of determining a user’s geographic (Geo) position in OWOX BI is, on average, 20% higher than in similar GA reports.

GA defines a geolocation using the Google API. This method isn’t always reliable because it may not be supported or be blocked in some browsers or by user actions (especially thanks to GDPR).

OWOX BI determines geolocation using another Google service — Cloud Load Balancing. This service specializes in balancing network load and linking geo data within a specific IP address. Accordingly, streaming doesn’t require any user permission; we avoid blocking and thus obtain more accurate geodata.

isTrueDirect is a parameter that specifies whether the session refers to direct website access (direct traffic) or must be attributed to another source.

The isTrueDirect variable in GA is true when:

• It was a direct link to the website (the user typed the website address or used the bookmark).
• If source, channel, campaign parameters are fully matched for two consecutive visits. GA defines the second session as direct traffic, although it’s initiated by advertising.

In the OWOX BI tables, we define isTrueDirect = true only in the case of direct website access. It allows you to evaluate brand awareness accurately and to estimate the cost of a particular session correctly.

Reports with the isTrueDirect parameter help you measure the effectiveness of brand awareness ads. The growth of direct entries at the beginning of the user journey makes it possible to evaluate the contribution of offline and outreach campaigns for your marketing.

Unlike in Google Analytics, you can collect and use PII customer data including email and phone numbers in Google BigQuery. Don’t lose users who didn’t reach the end of the funnel but tried to leave a request on the website. Phone or email in GBQ can be used to segment users and send them feedback.

GA doesn’t allow the transfer of personal data in unencrypted form. Otherwise, your account will be blocked.

In the OWOX BI tables, we use the &tel and &email parameters (they aren’t supported in GA) to transmit the personal data to the BigQuery in an easy-to-use way.

You can collect data about users who tried to apply for a loan or other service on the website, but for some reason, they were unable to submit a form. Then the call center can contact these clients and help them to finish the procedure by the phone.

If you don’t collect User ID within your online and offline systems, then the phone number and email can be used to link online and offline data of your users.

If you don’t have an analyst that works with raw data, use the Report Builder in our Smart Data. You can choose ready templates or create custom dashboards without any SQL knowledge. Smart Data allows you to merge streaming and CRM data to get a reliable base for analysis and decision making.

We regularly add new reports that cannot be found in Google Analytics:

• CPA partners monitoring.
• ROPO analysis.
• User conversion paths.
• Orders redemption in CRM.
• Call-tracking efficiency.

You can try all the features of OWOX BI Smart Data for free.

If you need special reports according to your business needs and accounting system, a team of OWOX BI analysts can help you. Sign up for a demo to request a meeting and find out the details.