Collect GitHub data in Databricks
Load repository details, contributors and daily star counts from GitHub into your own Databricks schema with OWOX Data Marts, and join them to product and commercial tables in the same Databricks workspace for one view of delivery and results.
GitHubConnectedRepository Contr…Repository Infor…Repository Stati…
DatabricksLoginIdNodeDaily 09:00Product usage and revenue sit in Databricks, while repository activity is still read off GitHub one page at a time.
Paste this into your AI assistant. It loads GitHub repository details, contributors and daily star counts into your Databricks schema with OWOX Data Marts, joined to your usage and revenue tables.
The prompt
Read https://www.owox.com/ai-agent-setup.md and follow it to set up OWOX Data Marts for me, starting from /connectors/github-connector-to-databricks.md. The task: load GitHub repository stats into Databricks and join them to our product usage and revenue tables, so I get one monthly view of stars, contributors, product usage and revenue. Tag the links you give me with utm_campaign=ai-task and utm_term=data-foundation. Use what you know about my context and goals
GitHub Endpoints
3 GitHub endpoints you can import. Each lands in its own table in your data warehouse.
Why GitHub data belongs in your Databricks workspace
Engineering activity usually lives in GitHub and nowhere else. Loaded into your warehouse on Databricks, it becomes tables you can query with SQL beside the sign-up, usage and revenue data your analysts already work with.
- Query stars and contributors beside usage data in the same schema
- Start a daily star history from your first run
- Report delivery next to revenue in one SQL query
GitHubRepository activityYour data warehouseRepositorySign-upsRevenueJoined by idYour release dateSign-ups and revenue afterOne Databricks query lines up the release date with what came next.Step 1: Connect GitHub and pick the fields
Create a Data Mart, pick GitHub as the source, give the repository as owner/repo, add an access token and tick the fields you want.
- The endpoints are Repository Information, Repository Contributors and Repository Statistics
- One access token authenticates all three
Google AdsGitHub
Facebook AdsAuthorizeConnectedEndpointRepository Inform…Repository Contri…Repository Statis…Repository Inform…full_namestargazers_countforks_countopen_issues_countlanguagepushed_atStep 2: Load GitHub into a Databricks table
Pick the schema in your storage; the catalog is main by default, and the names are pre-filled. Run it and the first load lands in your own Databricks workspace.
- The tables are github_repository, github_contributors and github_repository_stats, in the schema you pick
- The statistics table holds one row per run date, so stars and contributors filter by date like any other table
DestinationCatalogmainSchemagithubTablerepositoryRunTablemain.github.repositoryStep 3: Get a readable output schema
Once the first run finishes you have the output schema, each field with a plain description. Your quality checks can use it too.
- type marks each contributor as User or Bot, so you can leave bots out of a headcount
- stargazers_count, forks_count and open_issues_count are integers
main.github.repositoryOutput schemaFieldDescriptionfull_nameFull repository name (owner/repo)stargazers_countNumber of starsforks_countNumber of forksopen_issues_countNumber of open issueslanguagePrimary programming languagepushed_atRepository last push dateStep 4: Link GitHub tables to usage and revenue in Databricks
The GitHub tables stand on their own, but they answer more beside your usage and revenue tables. Join by repository name and by date, and keep the join in your model.
- full_name joins to the repository column of any table that tracks releases
- date joins to sign-ups or usage by day, in the same schema
Repository Inform…full_namestargazers_countlanguageRepository Statis…datestarsStep 5: Schedule the refresh and begin the history
Set it once. A scheduled refresh adds today's row to the statistics table and updates the repository and contributor rows in your Databricks schema.
- Schedule it daily and the statistics table grows by one row a day
- GitHub returns today's counts only, so there is nothing to backfill: your Databricks history starts with the first run
Step 6: Reuse the same Databricks tables everywhere
One schema feeds your model, , , assistants like Claude and ChatGPT, and scheduled AI insights in Slack or email.
- Every consumer reads the same tables, so figures agree everywhere
- Ask an AI assistant how many stars the repository gained last month, from your Databricks tables
GitHubData MartPublished
Google Sheets
Data Studio dashboards
Claude, ChatGPT
Slack, email
- 0.41.0Rotating a warehouse key is one change: Storages share reusable Credentials
- 0.41.0Shared SQL logic stays in one place: build a Data Mart on top of another
- 0.41.0Rotating a connector key is one change: Sources share reusable Credentials
- 0.39.0Reload a year of history in one backfill, not twelve
- 0.37.0Clearer connector errors, and custom connectors that handle more APIs
Why OWOX for GitHub data in Databricks
A script that calls the GitHub API, or a weekly export to a spreadsheet, gets you numbers. OWOX keeps them in your Databricks schema, where usage and revenue tables already sit.
- You own the data. The data lands in your own Databricks workspace, under the catalog and schema you control.
- History you keep. GitHub's repository page shows the current count. In Databricks the statistics table adds a row for every day you run it, so the trend stays.
- Ready for a data model. full_name and date join to your own tables in the same schema.
- Maintained for you. No script to fix when the GitHub API changes.
You are booked. The invitation is on its way to , with the calendar invite and the meeting link.