The past few weeks I spent building a front-end for an app store crawler that I made. The end goal is just to provide an open front-end for the apps details for the approximate 2.5m apps (already 1/2 those are no longer live) I’ve crawled from the Google & Apple stores. The site is now live on appgoblin.info and hosts a variety of Android and iOS stats and store information. Hopefully it’s something that some marketers may find interesting, and the code is all free on Github.
History
Originally this came out of a curiosity I had about the IAB app-ads.txt standard in 2022. In order to scrape app-ads.txt files I had to first get all the iOS and Android apps + their respective publisher URLs. This ended with a small dashboard I put on https://ads.jamesoclaire.com/dash/ads but I never expanded it further.
While that project, like app-ads.txt itself, went unused, I remained interested in letting the database of apps keep growing.
Then in the fall of 2023 while on hiatus from real work, I started working on an API and UI. Creating the UI gave me exactly what I wanted, which was use cases for displaying the data. Adding one feature led to the next so now there are several sections to browse.
Features
App Details
Each app has it’s own page for all the data pulled from the app store as well as second order of information derived from that data such as installs per day or change in ratings over time.
App Rankings
Since we have each apps history, it’s easy to build the historical ranking charts. In this example on the right, you can see a game quickly climbing multiple charts at once over the past month. This change in rankings is a great way to track an apps popularity over time.
Ranking Charts
No site like this would be without it’s own app rankings page. This is straight from the Google & Apple stores top charts each day for tracking apps overtime.
New Apps
Now this might seem like an obvious one, but it really brought home the slow burn that the app stores barely surface new apps anymore. Long gone from both stores are any structured “new” sections which showcase new apps. The cynical answer here is that those sections actively take away from the ad-revenue generated by the in store advertising that makes up a large percentage of the apps on screen at any given time in each store.
So, why does AppGoblin exist?
Yes, there are a large number of similar offerings like data.ai and sensortower.com which have many more features. For me, I enjoyed learning more front-end and Javascript as well as give me something to do with the scraper which I had built previously (whats the point of a database if you cant use it somehow). For now App Goblin is simply here as a resource free of charge as it’s running costs only $20 a month or so in hosting.
Open Source Tech Stack
First is the underlying data which is collected and maintained from github.com/ddxv/adscrawler The adscrawler uses Python & PostgreSQL to crawl app stores to collect apps and their basic information like names and rating counts. This data is then stored to the PostgreSQL database. I also use some of the original parts of github.com/ddxv/app-ads-dash for monitoring the scraper. Hopefully in the future the parts specific to the scraper are moved to that repository.
The website itself is also open source and can be found at github.com/ddxv/app-store-dash The website is built from a Python backend API based on LiteStar which manages queries to the PostgreSQL database. The API is used by a JavaScript Svelte frontend with Tailwind and Skeleton for CSS. All projects are glued together with Systemd and web sockets where needed. Everything is hosted on AWS in the small EC2 instances.
As always, if you have any interest or questions please feel free to reach out.