Collect GitHub data in Google BigQuery
Load repository details, contributors and daily star counts from GitHub into your own BigQuery dataset with OWOX Data Marts, then query engineering activity next to the product and revenue tables you already keep there.
GitHub shows today's star count, not its history, so last quarter's growth is a guess.
Paste this into your AI assistant. It loads GitHub repository statistics into your Google BigQuery dataset daily with OWOX Data Marts, so a star and contributor history builds up from the first run.
The prompt
Read https://www.owox.com/ai-agent-setup.md and follow it to set up OWOX Data Marts for me, starting from /connectors/github-connector-to-google-bigquery.md. The task: load GitHub repository statistics into Google BigQuery every day and build a history of stars and contributors per repository, with the change by month and by week. Tag the links you give me with utm_campaign=ai-task and utm_term=data-foundation. Use what you know about my context and goals
GitHub Endpoints
3 GitHub endpoints you can import. Each lands in its own table in your data warehouse.
Why GitHub data belongs next to your product and revenue tables
Stars, forks and contributors say how alive a project is. They say more when they sit beside sign-ups and revenue in the same warehouse: you can ask what changed on the day a release shipped, in one BigQuery query.
- Line up daily stars with sign-ups in one BigQuery query
- Keep a daily star count that GitHub's repository page does not show
- Report engineering activity next to revenue without a spreadsheet in between
Step 1: Connect GitHub and pick the fields
Create a Data Mart, choose GitHub as the source, enter the repository as owner/repo, add an access token and tick the fields you need.
- Three endpoints: Repository Information, Repository Contributors and Repository Statistics
- Authentication is by access token, which you paste once
Step 2: Load GitHub into a BigQuery table
Pick the dataset and table in your storage; the names are pre-filled and you can change them. Run it and the first load lands in your own BigQuery project.
- The tables are github_repository, github_contributors and github_repository_stats
- The statistics table is partitioned by date, one row per day, so a date filter scans only that period
Step 3: Get a readable output schema
After the first run you get the output schema: every field with a plain description. Your quality checks can read it too.
- full_name is the owner and repository, stargazers_count a number, pushed_at a date
- date is a DATE column, and the statistics table is partitioned on it
Step 4: Link repositories to your product and revenue tables
GitHub tables are small and descriptive. The value comes from joining them to yours: by repository name to your releases and by date to your sign-ups, described once in your model.
- full_name joins to a release table that carries the same repository name
- The statistics date joins to the sign-up date, so stars and sign-ups line up by day
Step 5: Schedule the refresh and start the history
Set it once. A scheduled refresh adds today's row to the statistics table and updates the repository and contributor rows in your Google BigQuery tables.
- Run it daily so the statistics table, partitioned by date, gets a row for every day
- GitHub returns today's counts only, so there is nothing to backfill: your BigQuery history starts with the first run
Step 6: Reuse the same BigQuery tables everywhere
One set of tables feeds your model, , , assistants like Claude and ChatGPT, and scheduled AI insights in Slack or email.
- Every report reads the same numbers, so the totals agree
- Ask an AI assistant how many stars the repository gained last month, from your BigQuery tables
- 0.43.0Parallel reads of a SQL Data Mart on Google BigQuery no longer fail with a rate limit error
- 0.41.0Rotating a warehouse key is one change: Storages share reusable Credentials
- 0.41.0Shared SQL logic stays in one place: build a Data Mart on top of another
- 0.41.0Rotating a connector key is one change: Sources share reusable Credentials
- 0.39.0Reload a year of history in one backfill, not twelve
Why OWOX for GitHub data in BigQuery
A script that calls the GitHub API, or a weekly export to a spreadsheet, gets you numbers. OWOX keeps them in the Google BigQuery tables where your product and revenue data already are.
- You own the data. The data lands in your own BigQuery project, in tables you control and can join to anything.
- History you keep. GitHub's repository page shows the current count. In BigQuery the statistics table adds a row for every day you run it, so the trend stays.
- Ready for a data model. full_name and date are the join columns, so releases and sign-ups are one line in your model.
- Maintained for you. No script to fix when the GitHub API changes.
You are booked. The invitation is on its way to , with the calendar invite and the meeting link.