How to upload data from ‌Instagram‌ ‌to ‌Google‌ ‌BigQuery‌

Supporting all types of content and armed with advanced visual tricks, Instagram has become not just the world’s largest photo album with cats, selfies, and food. It’s also a great advertising platform that takes visual e-commerce search to the next level.

A huge number of companies reach out to their customers through Instagram on a daily basis, using its emerging trends, new customer engagement, and shopping features. This is no surprise, as Instagram is the largest photo app, used by more than 1.2 billion people a month.

The Instagram team constantly improves the application, adding new features and business opportunities: advanced analytics, scheduled content publishing, and more. Another reason for the popularity of Instagram and Facebook among advertisers is the low price of contact and the ability to accurately target audiences.

You can see Instagram ad statistics in the application itself by looking in Instagram Insights. Information about ad views, coverage, profile views, calls, and links from ads, likes, comments, and other audience engagement options are available here. You can also analyze subscribers by demographic data to determine the percentage of your audience that’s a given gender or in a given age group and even to see where the bulk of your audience is from.

Because Facebook owns Instagram, you can use Facebook Ads Manager to run ads on both platforms. If your goal is to increase sales — not user activity or the number of comments — it’s best to run and analyze ads on Instagram through Ads Manager. Compared to Instagram Insights, there are more technical capabilities and metrics in Facebook Ad Manager both for setting up campaigns and working with data.

By default, Ads Manager provides information about coverage, views, clicks, ad costs, and so on. And if you install the Facebook pixel on your website and set up event tracking, you can find out the behavior of users who go to your website by clicking on an ad.

You can track all kinds of conversions on any device your ad brings to your website and see how much a desired action costs you. Facebook reports show you all clicks, registrations, orders, purchases, etc.

Data stored in Ads Manager can tell you about the effectiveness of only those sources that belong to Facebook. But what if a user comes to you from organic traffic, then returns to your website, clicks on a link on Instagram, and makes a purchase after getting an email? How can you evaluate the effectiveness of the Instagram‌ channel in this case? You won’t know how it interacts with other marketing sources and how they affect the business as a whole until you consolidate your data.

In large companies, marketing is not limited to Instagram, and sales can be online and offline. Therefore, it’s necessary to combine data from Facebook Ads Manager with data from your website, other advertising services, and your CRM so you can see the whole picture: how much money is spent on advertising, which advertising channels are more effective, and the financial results.

By uploading cost data from advertising services into Google BigQuery, you can associate costs with user activities on the website, calls to the call center, emails, and purchased orders from the CRM system. This helps you set up advanced analytics and assess the impact of all marketing efforts, both online and offline, on business performance.

Data from marketing services uploaded to Google BigQuery can be used to calculate your attribution model, and then BigQuery can pass the results to systems for automatic bid management. Also, in Google BigQuery you can create user audiences based on combined data and the results of attribution calculations and then automatically send those audiences to advertising services. Finally, you can use data collected in BigQuery to build any reports you need without restrictions.

The most popular cloud platforms are Amazon Redshift, Google BigQuery, and Microsoft Azure. They have common advantages over a conventional data warehouse:

• You don’t need to maintain servers and connect new ones when the load grows. Cloud storage scales automatically.
• Cloud platforms are faster than traditional storage and automatically redistribute the load.
• You don’t need to install a server on your computer to get into cloud storage. Just open your browser and log in to the cloud.

We’ve chosen Google BigQuery and recommend it to our customers for the following reasons:

• Google is the leader in the number of data sources for marketers: Ads, Analytics, Search Console, YouTube. All these services seamlessly integrate with BigQuery.
• It processes large amounts of data fast.
• It’s easy to get started with BigQuery without the help of database administrators. Just create a Google Cloud account.
• You only pay for the services you use.
• There are ready-made services and solutions for uploading data to Google BigQuery without the help of developers.

Google BigQuery doesn’t have a built-in tool for importing data from Facebook Ads Manager. You can solve this problem in several ways: by uploading data manually, writing your own scripts, or using specialized services such as OWOX BI. Let’s take a closer look at each of these options.

You can upload cost data from Facebook’s advertising profile into separate files and manually upload them through the BigQuery interface. The disadvantages of this method are obvious: a lot of unnecessary work and no automation.

You can also write scripts that will upload the data you need from a marketing service. However, you’ll need to constantly monitor and support these scripts. You’ll also have to spend developer resources to combine data from different accounts and on different dates, check data quality, and quickly respond to possible changes in advertising service APIs. In addition, you’ll need to retrospectively update data if it’s changed in Facebook Ads, convert costs from all advertising sources into a single currency, etc. If you don’t do these things, poor quality data can lead to suboptimal solutions that will cost your business a lot.

With OWOX BI, you can set up automatic cost import from different advertising services into Google Analytics and upload cost data to cloud storage in one data stream. If you use this method, consider the limitations of Google Analytics:

• A maximum of 90 MB can be loaded into a single dataset per day.
• You can download a maximum of 50 files per day per resource.
• Google Analytics may take up to 24 hours to process uploaded data and make it available in reports.
• When uploading a large amount of data, there may be problems with the Google Analytics API.
• You can’t upload more parameters than the cost dataset schema from Google Analytics can contain.

OWOX BI has direct data streams from Facebook, Instagram, and LinkedIn, with which you can upload raw data on all your campaigns into BigQuery. These data schemes contain 80 to 200 parameters, which ensures a high granularity of data for your reports.

OWOX BI offers a complete solution out of the box — you don’t need to do anything manually. The service collects all the data we need in a convenient format, cleans and deduplicates it, checks existing UTM tags in your campaigns, reports possible errors, recognizes dynamic parameters, converts costs into a single currency, and monitors the relevance of data.

If historical data changes in Facebook Ads, OWOX BI updates the data uploaded to BigQuery automatically. In addition, if necessary, you can upload your historical data for the last six months. This will help you evaluate campaign dynamics.

Since cost data importing occurs without the participation of Google Analytics:

1. You get the data faster: Costs are uploaded directly to BigQuery, and you don’t have to wait an extra 24 hours for Google Analytics to process them.
2. Data is downloaded without Google Analytics limits of 90 MB and 50 downloads per day.
3. You can download statistics from multiple ad accounts into a single BigQuery table by simply specifying the same dataset when creating a stream.
4. Data from the advertising source is compiled into a single-party table in Google BigQuery. This greatly simplifies the compilation of queries and analysis of downloaded data.
5. Cost data is uploaded to BigQuery in two currencies. One field stores costs in the original currency from the advertising service. You can specify the other currency to convert cost data to when you create the flow. Converted costs are helpful for reports where data from different sources must be converted to a single currency.

1. Make sure your advertising links have UTM tags.

2. On the OWOX BI home page, click Create Pipeline:

3. As the source, select Instagram:

4. As the destination, select Google BigQuery:

If you don’t have a project in Google BigQuery, learn how to start working with cloud storage and create a dataset.

5. Select a Facebook account (or grant access to a new Facebook account) from which you want to export ad cost data:

Under no circumstances does OWOX BI change the settings of your account or advertising campaigns!

For most advertising platforms, OWOX BI requests access only to read data. This level of access is enough for us to get statistics and information about link markup in your ads. But not all advertising services have such a level of access or work as we expect. For example, to import raw data from Facebook and Instagram into Google BigQuery using OWOX BI, you need ads_management permission.

6. Select a connected account or grant access to a Google BigQuery account where you want to store the exported data:

7. Select a Google BigQuery project and create a dataset you want to upload your data to (or create a new one):

Note! To set up data collection, your Google account must be granted both BigQuery Data Editor and BigQuery User roles for the destination project. Otherwise, BigQuery won’t let you upload data.

To check/grant these permissions, go to the Identity and Access Management page in your Google Cloud Platform project. Read more in the Google documentation.

8. Specify the settings for your pipeline:

• Select the date from which you want to upload cost data. You can set up either a future or past date. If you’re choosing a past date, learn about the limitations on historical data import.

• Choose the source/medium you want to apply to the imported data by default. The default values you specify for the UTM source/channel will be written to the cost data tables only if OWOX BI doesn’t receive the actual UTM parameter values. For more information on why you should specify default values for UTM source/channel in the pipeline settings, refer to the Help.

Important! At any time, you can change the source/medium settings on the pipeline page to any values except google/organic. The specified values will be applied to new data imported after making changes and historical data within the update window.

9. Click Create pipeline.

Done! Data for the previous day will appear in the table in the specified dataset daily at 00:00 (UTC). Depending on the amount of data and the features of the advertising service API, importing data can take up to 24 hours. See this article for the structure of the table.

If the information in an advertising service changes retrospectively, OWOX BI will update all data uploaded to BigQuery within the established update window.

In the Instagram Insights, Facebook advertising profile and web analytics systems, you can analyze the basics of Instagram advertising performance. For example, you can analyze CTR, CPC, CPA, CR, sessions, viewing depth, bounce rate, RPC, and ROAS. For small businesses with a small number of advertising channels, this is enough.

If you have many touchpoints with customers, have offline stores, and want to see the entire path to purchase, however, then you should think about setting up advanced analytics and creating an automatically updated dashboard with all the metrics that interest you. This will allow you to always have up-to-date information at hand, conduct a comprehensive assessment of advertising effectiveness, and make important decisions faster.