How to Upload Data from ‌Instagram‌ ‌to ‌Google‌ ‌BigQuery‌

Supporting all types of content and armed with advanced visual tricks, Instagram has become not just the world’s largest photo album with cats, selfies, and food. But what’s more importantly, it’s also one of the best advertising platforms that takes visual e-commerce search to the next level.

A huge number of companies reach out to their customers through Instagram daily, using its emerging trends, new customer engagement, and shopping features. This is no surprise, as Instagram is the largest photo app, used by more than 2 billion people a month.

Connecting Instagram Business to Google BigQuery is crucial for leveraging analytics and marketing optimization, enabling businesses to own and query massive volumes of data for data-driven decisions and competitive advantage.

The Instagram team constantly improves the application, adding new features and business opportunities: advanced analytics, scheduled content publishing, and more. Another reason for the popularity of Instagram and Facebook among advertisers is the low price of contact and the ability to accurately target audiences.

You can see Instagram ad statistics in the application itself by looking at Instagram Insights. Information about ad views, coverage, profile views, calls, and links from ads, likes, comments, and other audience engagement options are available here. You can also analyze subscribers by demographic data to determine the percentage of your audience that’s a given gender or in a given age group, and even to see where the bulk of your audience is from.

Because Facebook owns Instagram, you can use Facebook Ads Manager to run ads on both platforms. If your goal is to increase sales — not user activity or the number of comments — it’s best to run and analyze ads on Instagram through Ads Manager. Compared to Instagram Insights, there are more technical capabilities and metrics in Facebook Ad Manager, both for setting up campaigns and working with data.

By default, Ads Manager provides information about coverage, views, clicks, ad costs, and so on. If you install the Facebook pixel on your website and set up event tracking, you can find out the behavior of users who go to your website by clicking on an ad.

You can track various types of conversions from any device that interacts with your ad and visits your website. Additionally, you can evaluate the cost of each desired action. Facebook's analytics provide detailed insights into metrics such as, clicks, registrations, orders, purchases, and more.

Data stored in Ads Manager can tell you about the effectiveness of only those sources that belong to Facebook.

• But what if a user initially visits your site through organic search, later returns via an Instagram link, and finally purchases after receiving an email?
• How can you evaluate the effectiveness of the Instagram‌ channel in this case?

To fully understand its impact, it's essential to extract data from Instagram and integrate it with insights from other channels, allowing for a comprehensive analysis of how Instagram contributes to your marketing efforts and interacts with other data sources to affect your business. In large companies, marketing is not limited to Instagram, and sales can be online and offline.

It becomes necessary to integrate data from Facebook Ads Manager with information from your website, other advertising platforms, and your CRM. This comprehensive data integration also allows you to see the complete landscape – tracking advertising expenditure, evaluating the efficacy of different marketing channels, and understanding their financial outcomes.

When you import cost data from advertising platforms into Google BigQuery, it becomes possible to link these costs to various user interactions such as website visits, call center interactions, emails, and CRM system transactions. This integration helps you set up advanced analytics, allowing you to evaluate the effectiveness of your marketing strategies, both online and offline, on overall business performance. Integrating Instagram data into this setup enhances your analytical capabilities by supporting efficient data transfer, data consolidation, and ETL processes.

Moreover, leveraging data from marketing platforms in Google BigQuery enables you to develop your attribution model, which BigQuery can then distribute to automatic bid management systems. In Google BigQuery, you also can create user audience segments based on aggregated data and insights from attribution analysys, and seamlessly transfer these segments to advertising platforms. Additionally, BigQuery allows for the unrestricted creation of various reports using the gathered data.

Analyzing historical data in BigQuery is vital for bypassing Instagram's constraints on retaining historical information, providing a robust solution for long-term trend analysis and insights.

Among the top data warehouse solutions in the cloud, such as Amazon Redshift, Google BigQuery, and Snowflake, there are several universal benefits over traditional data warehouses:

• There's no need for maintaining or connecting new servers as demand increases – cloud storage scales effortlessly.
• Cloud platforms provide faster access and processing times than traditional data storage methods and can automatically balance the load.
• Accessing cloud storage doesn't require installing any server hardware; you simply use your web browser to log in.

We’ve chosen Google BigQuery and recommend it to our customers for the following reasons:

• Google offers the broadest range of native data sources that are crucial for marketers, including Google Ads, Google Analytics, Google Search Console, and YouTube, all of which integrate smoothly with BigQuery.
• BigQuery is user-friendly for beginners and doesn't necessitate database administrators for setup. Setting up is as simple as creating a Google Cloud account.
• Costs are based strictly on the services utilized, ensuring you only pay for what you need.
• Google provides ready-to-use services and solutions that facilitate data uploads to BigQuery without needing developer assistance.

Connecting Instagram to Google BigQuery involves several prerequisites to ensure a smooth and efficient integration process.

Here’s a checklist to get started:

• Instagram Account as a Data Source: You need an Instagram Business or Creator account to access Instagram’s Graph API, which is necessary for extracting data.
• Facebook Page: Your Instagram Business or Creator account must be linked to a Facebook Page, as the Instagram Graph is part of the Facebook Graph API ecosystem.
• Facebook Developer Account: To access the Instagram Graph API, you must first establish a Facebook Developer account and configure an application. This app will manage the API calls to Instagram.
• Google Cloud Project: Set up a Google Cloud Project if you don’t already have one. This project will be used to access BigQuery and potentially other Google Cloud services.
• BigQuery Dataset: Create a dataset in Google BigQuery where the Instagram data will be stored. Plan your schema based on the data you intend to collect.
• Permissions and Roles: Ensure appropriate access in both Facebook and Google Cloud for API and data management.

By meeting these prerequisites, you’ll be well-prepared to connect your Instagram account to Google BigQuery, enabling advanced data analysis and insights.

While Google BigQuery lacks a native tool for importing data directly from Facebook Ads Manager, several effective methods can address this gap:

• manual uploads
• scripting custom solutions
• OWOX BI

Implementing a change data capture (CDC) process can further enhance data consistency and centralize the collection of information from various sources. Here's a detailed examination of each method:

By leveraging data pipelines, businesses can automate the data import process from Instagram to BigQuery, significantly simplifying the task of integrating data from various sources. This automation not only streamlines the process of creating data pipelines but also supports a wide range of data integration possibilities without requiring custom code or engineering resources.

When choosing to load data manually or through specialized services, it's crucial to consider the ease of use and automation these services provide. They facilitate the loading data process by automating the ETL (extract, transform, load) process, enabling businesses to quickly and flexibly synchronize data from Instagram to BigQuery. This includes handling changes in data structure and ensuring frequent data synchronization, thereby enhancing the ability to generate advanced analytics and insights.

Manually uploading cost data from Facebook's advertising platform involves exporting the data into separate files and then uploading these through the BigQuery interface. This method is labor-intensive and lacks automation, making it less suitable for ongoing or large-scale data handling needs.

Alternatively, creating custom scripts to automate data uploads from marketing services can streamline the process.

However, this approach requires continuous monitoring and maintenance of the scripts to ensure they function correctly and adapt to any changes in advertising service APIs. Significant developer resources are also necessary to amalgamate data across different accounts and dates, ensure data quality, and swiftly address any API modifications.

Moreover, if the data in Facebook Ads changes, these scripts must be capable of retrospective updates. They also need to handle currency conversion for costs from various advertising sources to maintain data consistency. Neglecting these aspects can result in low-quality data, potentially leading to costly inefficiencies in business operations.

OWOX BI offers direct data streams from major advertising platforms such as Facebook, Instagram, and LinkedIn, enabling the seamless upload of raw data on all your campaigns directly into BigQuery.

These data schemas are designed to support a high level of detail, containing 80 to 200 parameters per data schema.

This granularity allows for more precise and in-depth reporting, giving you the ability to analyze your advertising campaigns with greater accuracy and insights.

By leveraging these direct connections and tools, you can bypass the complexities of manual data handling and ensure that your data is consistently up-to-date and readily available for analysis in BigQuery.

This method not only saves time but also enhances the reliability and actionable quality of your data analytics efforts.

When you have your ad cost data in BigQuery you can do so many things with that. You can build customer reports in Looker Studio, you can manipulate your data in BigQuery and visualize in Google Sheets, or you can non-Google ROAS report in Google Analytics 4 by uploading cost data from BigQuery into GA4 with OWOX BI.

That’s a smart move, and probably, the easiest one. However, you should be aware of certain limitations on the Google Analytics side:

• You can download a max of 120 files per day.
• When uploading a large amount of data, there may be some limitations with Google Analytics API.
• You can’t upload more parameters than the cost dataset schema from Google Analytics can contain (so your reporting is limited).

OWOX BI offers a complete solution out of the box — you don’t need to do anything manually.

OWOX BI collects all the data you need in a convenient format, cleans and deduplicates it, checks existing UTM tags in your campaigns, reports possible errors, recognizes dynamic parameters, converts costs into a single currency, and monitors the relevance of data.

If there are updates or changes to historical data in Facebook or Instagram Ads, OWOX BI can automatically refresh this historical data analysis in Google BigQuery. This feature ensures that your analysis is always based on the most accurate and current data. Additionally, OWOX BI provides the flexibility to upload historical data for up to the past six months.

This capability is crucial for assessing the performance trends and dynamics of your campaigns over a significant period, allowing for a deeper understanding of campaign effectiveness and strategic adjustments.

1. You get the data faster: Costs are uploaded directly to BigQuery and are available for you right away.
2. You can merge the data from multiple ad accounts (and ad platforms) into a single BigQuery table by using no-code OWOX BI Transformations.
3. You can then merge those costs with Google Analytics tracking data and build comprehensive reports.

Here is just an example of what you can get:

If you have different accounts with different currencies, OWOX BI Has an automated tool to convert all of the currencies into the one suitable for your reporting with everyday fresh exchange rates from openexchangerates.org.

Integrating Instagram data reports into Google BigQuery unlocks a plethora of opportunities for businesses to leverage their social media insights more effectively. From advanced analytics to improved data security, this integration facilitates a range of applications that enhance business needs for data-driven decision-making and strategic planning. Here are a few Use Cases:

BigQuery's powerful data processing capabilities allow for deep analysis of Instagram data. This enables detailed insights that go beyond what Instagram's native analytics can provide.

By integrating Instagram data with BigQuery, businesses can consolidate their data from various sources into one place. This centralization makes it easier to manage and analyze multiple sources of data comprehensively, while also growing businesses by ensuring data consistency through changed data capture processes.

BigQuery offers strong security measures to protect Instagram data. This integration aids in maintaining high standards of data governance and compliance management, safeguarding sensitive information effectively.

BigQuery is quite good at handling lots of data, which makes it perfect for businesses that are getting more and more data over time. This scalability ensures that performance is not compromised, even as data needs expand.

Using BigQuery, companies can apply machine learning models to their Instagram data for advanced applications like predictive analytics and customer segmentation. This opens up new possibilities in data science and machine learning endeavors.

Although Instagram provides its own reporting and data visualization tools, connecting Instagram business to BigQuery allows for the use of advanced data visualization tools like Tableau, Looker (formerly known as Google Data Studio), and Power BI. This enhances the ability to derive business intelligence from Instagram data.

BigQuery enables businesses to store and analyze Instagram data over long periods, overcoming the platform's limitations on historical data retention. This facilitates the study of long-term trends and patterns.

1. Make sure your advertising links have UTM tags.

2. On the OWOX BI home page, click Create Pipeline:

3. As the source, select Facebook & Instagram Ads (Meta Ads):

4. As the destination, select Google BigQuery:

If you haven't yet set up a project in Google BigQuery and are new to working with cloud storage, here's a step-by-step guide on how to start working with cloud storage and create a dataset.

5. Select a Facebook account (or provide access to a new Facebook account) from which you wish to export advertising cost data:

Under no circumstances does OWOX BI change the settings of your account or advertising campaigns!

For most advertising platforms, OWOX BI only requires read access to retrieve statistics and information about link markup in your ads. However, not every advertising service offers this level of access or functions as anticipated. For example, to transfer raw data from Facebook and Instagram to Google BigQuery using OWOX BI, you should have ads_management permission.

6. Choose a linked account or provide access to a Google BigQuery account where you intend to store the exported data:

7. Choose a Google BigQuery project and select a dataset where you want to upload your data (or create a new dataset):

Note! To configure data collection, your Google account needs to have both the BigQuery Data Editor and BigQuery User roles assigned for the target project. Without these permissions, you will not be able to upload data to BigQuery.

To verify or assign these permissions, navigate to the Identity and Access Management page in your Google Cloud Platform project. For further details, consult the Google documentation.

8. Specify the settings for your pipeline:

• Select the date from which you want to upload cost data. You can set up either a future or a past date. If you're choosing a past date, learn about the limitations of historical data import.
• Choose the source/medium you want to apply to the imported data by default. The default values you specify for the UTM source/channel will be written to the cost data tables only if OWOX BI doesn't receive the actual UTM parameter values. For more information on why you should specify default values for UTM source/channel in the pipeline settings, refer to the Help.

Important! You can modify the source/medium settings on the data pipeline page at any time to any values, except for google/organic. The changes will apply to new data imported after the adjustments and to historical data within the specified update window.

9. Click Create pipeline

Done! Data from the previous day will be available in the specified dataset daily at 00:00 (UTC). If you need your data to be collected at another time - just let us know in the chat.

Depending on the volume of data and the specifics of the advertising service API, importing data may take up to 24 hours. For details on the table structure, refer to this article.

If there are retrospective changes in advertising service, OWOX BI will update all data previously uploaded to BigQuery within the defined update window.

Within Instagram Insights, Facebook advertising profiles, and web analytics platforms, you can access fundamental metrics of Instagram advertising performance such as CTR, CPC, CPA, CR, sessions, viewing depth, bounce rate, RPC, and ROAS. This level of analysis suffices for small businesses utilizing a limited number of advertising channels.

However, for businesses engaging in numerous customer touchpoints, possessing offline stores, and needing visibility into the complete purchase pathway, it's advisable to invest in advanced analytics. Setting up an automatically updated dashboard that encompasses historical trends and all relevant metrics can provide you with consistently current data. This setup enables a thorough evaluation of advertising effectiveness, meets business requirements, and aids in making swift, informed decisions.