What does DIRECT mean for App-Ads.txt?

Recently I started working on a project on appgoblin to estimate the number of client apps for all mobile advertising and data collection companies. In doing this there are currently two sources of data: decompiled SDKs of which I have only done ~20k apps and app-ads.txt which I have scraped closer to ~200k apps’ app-ads.txt files. Both have their blind spots. Some apps I have yet to successfully unzip. My processes for opening / unzipping iOS is much less reliable, also once opened it’s not always a guaranteed I can correctly identify whether an app is using specific SDKs.

Still, as I get more and more data, it was my assumption that the two should converge. Yet I have started noticing something interesting. Advertising networks that I would not have thought of as “DIRECT” have been showing up in the app-ads.txt but I am not sure what the related SDKs would be.

The first example is Rubicon which has a massive 50k apps, but I have yet to map it to a specific SDK. Looking through the evidence I have of SDKs, I can’t find any mention of it. I am thinking that rubiconproject.com does not actually have direct SDKs that apps use, but instead partners with existing players like AppLovin or Unity who do have a large number of apps mediation and monetization SDKs. This is a bit odd, because it obfuscates which ad network the programmatic traffic is being bought on.

This will need further investigation to understand what is the correct meaning of “DIRECT” and whether it is being applied correctly here or what other information I might be missing.

Svelte 5 Data Table Comparisons

While working on a new analytics dashboard I wanted to find a good table tool manage my table like data from the backend. My main goal is to have dynamic group by for the table’s dimensions. While this has been something that I’ve always liked about Python based backends, it definitely wasn’t something I found in any of the following data table packages for Javascript.

In the end, I’ll be building my own break down / group by feature by combining dropdowns and manually massaging the full table itself. I may try node-polars if my lack of JS ability stunts my progress.

For the rest of the post, here’s a quick breakdown of what I thought of the various data table libraries I checked for Svelte 5 compatibility.

AG Grid

https://www.ag-grid.com/
Looks like exactly what I want! But doesn’t have a Svelte library. There was a community project svelte-ag-grid which was built for Svelte 3 but is not well updated (still hasn’t transitioned to Svelte 4 much less Svelte 5).

I went ahead and tried anyways and was able to get ag-grid working with Svelte 5 and actually it seemed to be working pretty good. That is until I tried implementing features I saw on their examples/demo page: AG Grid costs $1k for the features I was looking at. Oops, hadn’t realized that! I didn’t realize the features you see in the demos were not freely available.

TanStack Table

https://tanstack.com/table/latest
This one is much better at Open Source, with a huge and active community. Though the Svelte 5 support isn’t quite out yet, it does look like support is on the way. One other issue I hit, is none of their examples work on Firefox, I think just due to a tool used on their site for embedding, not the TanStack Table I assume.

Vincjo/Datatables

https://github.com/vincjo/datatables/
This works well now with Svelte 5. I’ve also used it with Skeleton.dev but unfortunately that does not yet have support for Svelte 5 but is also right around the corner. As far as using it as a table, my only issue with Datatables is that it has much less features out of the box, which led me to my final answer.

DIY

What I’m looking to do is likely not that complicated, just time for me to roll up my sleeves and learn more Javascript! Either way, it’s exciting to see so much support moving forward for Svelte 5 and learning the overview of the table like packages out there that support the release candidate for Svelte 5.

How to figure out which 3rd Parties might be integrated with an iOS App?

In the past couple months I built out scraping for Android apps which downloads, decompiles, scrapes the `AndroidManifest.xml` then analyzes the Android app for known 3rd party ad networks or trackers.

I enjoyed this a lot, but the other half of the equation, iOS remained elusive. I was surprised how much more difficult this turned out so I’ll document a bit of what I did in case it helps anyone else.

How to tell which 3rd party ad networks or trackers are in Apple iOS apps?

First, what is the most analogous file I could find to AndroidManifest? Turns out this is the Info.plist but the similarities with AndroidManifest don’t go far. In terms of 3rd party integrations, it does not seem like most make an appearance in the Info.plist so I guess I might need to dig further.

Next, my challenge was where to download files from. In the end, I decided to go ahead and try downloading from the iTunes store directly. The downside of this is that I will be using a personal iTunes account, though I don’t use Apple as a daily driver, it’s still a risk that they block/ban the account. I tried searching for whether this was possible, but didn’t find too much.

Downloading ipa files

After some trial and error with other tools, I found the open source ipatool which has a CLI interface. It requires authenticating with the email and password of your account, as well as some 2FA text messages to the related phone number.

The first issue I hit was that despite having entered an email, a phone number AND a credit card already for this account (#ApplePrivacy) I still needed to accept a license agreement via iTunes. Luckily, I was able to find this blog article for how to download IPA files for Windows by using an old version of iTunes (because of course Apple now blocks this in the newer versions). In the end I was able to get that older version of iTunes working on Linux with Wine and was able to accept the license agreement and download IPAs. This then made it so I could also use the CLI ipatool as well.

Now that CLI ipatool is working I am able to download IPA files and start doing more investigation

See which MMPs and Networks Apps and Games are Using

I recently added a free feature to AppGoblin to see which advertising/monetization ad networks and MMP partners apps are using. You can break down the results by category and and group by parent companies.

The data is pulled from the top ~10k Android apps, which I downloaded, de-compiled and examined their AndroidManifest.xml to determine which ad partners they might be using. The list of partners is something I manually created, so if there are any mobile MMPs or ad networks you think are missing, feel free to let me know and I’ll add them in.

My original thesis I wanted to learn about was what percentage of apps use AppsFlyer vs Adjust and the biggest surprise I got was that it was much lower than I expected and Firebase was much much higher than I expected.

Also, I’d love any feedback or questions that popup when people see this. I wouldn’t mind taking this a bit further.

Is this a TikTok security vulnerablity for ad fraud?

I de-compiled TikTok (“com.zhiliaoapp.musically” v33.3.3 from Feb 2nd, 2024) from ApkPure.net and noticed that the way it called AppsFlyer looked a bit different than what I expected and quickly led me to a GitHub issue which makes it seem like they are using an outdated way to collect install information from Google Play which may have security vulnerabilities. Particularly, could a malicious app use this to ‘steal’ TikTok install attributions?

In this setup you see that TikTok is using a receiver com.appsflyer.SingleInstallBroadcastReceiver to listen for the com.android.vending.INSTALL_REFERRER event. This might allow a malicious app to listen for for the INSTALL_REFERRER event. You can see this is not recommended by AppsFlyer in this GitHub issue and is not their recommended installation setup.

<application>
 <receiver android:exported="true" android:name="com.appsflyer.MultipleInstallBroadcastReceiver">
        	<intent-filter>
            	<action android:name="com.android.vending.INSTALL_REFERRER"/>
        	</intent-filter>
    </receiver>
    <receiver android:exported="true" android:name="com.appsflyer.SingleInstallBroadcastReceiver">
        	<intent-filter>
            	<action android:name="com.android.vending.INSTALL_REFERRER"/>
        	</intent-filter>
    </receiver>
</application>

The right way?

This is the standard way that the AppsFlyer setup will be implemented, which in turn will use the google store referrer.

   <queries>
        <intent>
            <action android:name="com.appsflyer.referrer.INSTALL_PROVIDER"/>
        </intent>
   </queries>

I think the potential here might that if this version of TikTok is leaking the INSTALL_REFERRER data then

  • It could provide a malicious app valuable information in attempting to steal information about the source of TikTok’s users.
  • Additionally, a malicious app might be able to perform click jacking given that they know some information about the source of the install very early on.
  • And finally since the INSTALL_REFERRER is broadly scoped the malicious app could then send the a false INSTALL_REFERRER which TikTok may not be able to validate.

Before You Agree: What data does TikTok collect before Terms of Service?

Would you like to see what data is coming out of TikTok when you first open it up? Let’s get to it.

Didn’t this used to be easy to do?

As security for iPhones and Androids increased it continually made viewing the traffic leaving your own device more difficult. This is in stark contrast to a regular web browser where you can easily open the network traffic for any site to see the back and forth of the network calls while you visit that site. For mobile apps, if you want to see the traffic on a computer, you need to do a proxy with a custom CA certificate. For years this was with Charles Proxy, Burpsuite and Wireshark. But apps continued to evolve and many began to put their own custom CA certificates inside the apps, meaning it was no longer enough to just forward the traffic to a proxy, you now had to unpin the SLL certificate within the app. This flow below is still possible with Charles or others, but in the end I ended up using the following tools.

  • Waydroid – Android Emulator: To install the target app
    • + Magisk for root + allowing user certificates
    • + LSPosed for SSL unpinning
  • MITM Proxy: to capture and view the decrypted traffic

If you’d like more detailed instructions for viewing TikTok HTTPS traffic you can find them on GitHub.

Installing TikTok

I got the APK from APKpure.net and installed it into the emulator. We rolled the dice on a lucky day and happened to download TikTok v33.3.3 so you’ll be seeing that number a lot later. Then with the MITM proxy running, I opened the app for the first time.

Launching TikTok

All the request in the rest of the post are fired within 2-3 seconds of launching, so lots of back and forth within the app and on the network.

Keep in mind, this is all traffic from the emulator, so there are other network requests mixed in that are not from TikTok.


The first one that is for sure from TikTok. You you can see the request to tiktokv.com/monitor/appmonitor/v3/settings with some basic settings being recorded.

GET https://mon-va.tiktokv.com/monitor/appmonitor/v3/settings?update_version_code=30107125&os=Android&app_version=3.1.7-rc.75.oversea&device_id=&channel=release&aid=2010&crash=npth HTTP/1.1
Content-Type: application/json; charset=utf-8

Query

update_version_code: 30107125
os:                  Android
app_version:         3.1.7-rc.75.oversea
device_id:           
channel:             release
aid:                 2010
crash:               npth

This is followed shortly after with a Firebase installation:

POST https://firebaseinstallations.googleapis.com/v1/projects/musically-c51f4/installations HTTP/1.1
Content-Type: application/json

{
    "appId": "1:340331662088:android:3c6c52c4762af402",
    "authVersion": "FIS_v2",
    "fid": "cMbnNXy6QCGDpHT_mB3TXN",
    "sdkVersion": "a:17.0.1"
}

User Data

Here we see TikTok trying to figure out the carrier_region and network_sim_region all of which are empty due to Waydroid emulator not having a sim card.

POST https://api-boot.tiktokv.com/region/submit/?sdk_version=1.2.2&ac=mobile&channel=googleplay&aid=1233&app_name=musical_ly&version_code=330303&version_name=33.3.3&device_platform=android&os=android&ab_version=33.3.3&ssmix=a&device_type=WayDroid+x86_64+Device&device_brand=waydroid&language=en&os_api=30&os_version=11&openudid=6b623cdca1a14c25&manifest_version_code=2023303030&resolution=3456*1970&dpi=360&update_version_code=2023303030&_rticket=1706845427021&is_pad=1&app_type=normal&sys_region=US&timezone_name=Asia%2FTaipei&app_language=en&ac2=unknown&uoo=1&op_region=US&timezone_offset=28800&build_number=33.3.3&host_abi=armeabi-v7a&locale=en&region=US&ts=1706845427&cdid=e1907a45-45ae-4508-b5e2-171d7964d125 HTTP/1.1

Content-Type: application/json

JSON
  

{
    "carrier_region": "",
    "locale": "en_US",
    "mcc_mnc": "",
    "network_sim_region": "",
    "system_language": "en",
    "system_region": "US"
}

Now we see the main set of data coming out of the device. This information is repeated in most other requests back and forth. Let’s note some of the interesting ones:

  • op_region/sys_region: US
  • device_type: WayDroid x86_64 Device
  • device_brand: waydroid
  • language: en
  • os_api: 30
  • os_version: 11
  • openudid: 6b623cdca1a14c25
  • manifest_version_code: 2023303030
  • resolution: 3456*1970
  • dpi: 360

These are all accurate to the emulator Waydroid, running on my laptop, note the resolution. The op_region/local US and en are related to the Google Play account I am signed into with on the phone, while Asia/Taipei is the current timezone.

GET https://api-boot.tiktokv.com/passport/account/info/v2/?scene=normal&multi_login=1&account_sdk_source=app&passport-sdk-version=19&ac=mobile&channel=googleplay&aid=1233&app_name=musical_ly&version_code=330303&version_name=33.3.3&device_platform=android&os=android&ab_version=33.3.3&ssmix=a&device_type=WayDroid+x86_64+Device&device_brand=waydroid&language=en&os_api=30&os_version=11&openudid=6b623cdca1a14c25&manifest_version_code=2023303030&resolution=3456*1970&dpi=360&update_version_code=2023303030&_rticket=1706845427440&is_pad=1&app_type=normal&sys_region=US&timezone_name=Asia%2FTaipei&app_language=en&ac2=unknown&uoo=0&op_region=US&timezone_offset=28800&build_number=33.3.3&host_abi=armeabi-v7a&locale=en&region=US&ts=1706845427&cdid=e1907a45-45ae-4508-b5e2-171d7964d125&support_webview=1&okhttp_version=4.2.137.48-tiktok&use_store_region_cookie=1 HTTP/1.1
Accept-Encoding: gzip
x-tt-app-init-region: carrierregion=;mccmnc=;sysregion=US;appregion=US
sdk-version: 2
x-tt-dm-status: login=0;ct=0;rt=7
X-SS-REQ-TICKET: 1706845427445
passport-sdk-version: 19
pns_event_id: 9
x-vc-bdturing-sdk-version: 2.3.5.i18n
User-Agent: com.zhiliaoapp.musically/2023303030 (Linux; U; Android 11; en_US; WayDroid x86_64 Device; Build/RQ3A.211001.001;tt-ok/3.12.13.4-tiktok)
X-Ladon: T3YLBE4LXYRR2+5qAqfacGfZklgSFiMMsQJG+LLwWS/K2KTv
X-Khronos: 1706845427
X-Argus: 7GLcN9rck1opShLax2FYZ6IEse0/+7WsuaKmyIiBDG+6fGK3w28aOXo0P7vp+fn22ye3LeqP7I4Ae7hQ/WjqBfQLsgEqgEj/jXsiIw7KXBcp/70AXu0vGG/KKiwGBTCmPaGv38VAQGTzWfyiwLU1K7IcxJ4QInnEhmQjmzf2A6trku+NcY0FBApGxTeWBHipRBSvOrkMnuzT5hIq/99QXXgTId9ylIiqjilm9iGrk0aU3JnpA9BKsXQonstxlmpQrJqApi3c7FOVdHJs5ZwatAkRNVNXvQP5gVdQZcCrILMDcw==
X-Gorgon: 0404c05b000015eb3032a8aa4fd780a54ff4e00eabc7dc70d487
Host: api-boot.tiktokv.com
Connection: Keep-Alive
Query
  

scene:                   normal
multi_login:             1
account_sdk_source:      app
passport-sdk-version:    19
ac:                      mobile
channel:                 googleplay
aid:                     1233
app_name:                musical_ly
version_code:            330303
version_name:            33.3.3
device_platform:         android
os:                      android
ab_version:              33.3.3
ssmix:                   a
device_type:             WayDroid x86_64 Device
device_brand:            waydroid
language:                en
os_api:                  30
os_version:              11
openudid:                6b623cdca1a14c25
manifest_version_code:   2023303030
resolution:              3456*1970
dpi:                     360
update_version_code:     2023303030
_rticket:                1706845427440
is_pad:                  1
app_type:                normal
sys_region:              US
timezone_name:           Asia/Taipei
app_language:            en
ac2:                     unknown
uoo:                     0
op_region:               US
timezone_offset:         28800
build_number:            33.3.3
host_abi:                armeabi-v7a
locale:                  en
region:                  US
ts:                      1706845427
cdid:                    e1907a45-45ae-4508-b5e2-171d7964d125
support_webview:         1
okhttp_version:          4.2.137.48-tiktok
use_store_region_cookie: 1

Next we see calls for api-boot.tiktokv.com/aweme/v1/compliance which note that I am not a teen or child. Additionally is_new_user=0 which must mean that I’ve before downloaded TikTok with this Google account. Keep in mind, this is all within a second or two of running the app, so I assume this must be coming from Android operating system, and it’s possible last year I had installed TikTok at some point on the emulator, though I don’t remember that.

GET https://api-boot.tiktokv.com/aweme/v1/compliance/settings/?teen_mode_status=0&ftc_child_mode=-1&is_new_user=0&ac=mobile&channel=googleplay&aid=1233&app_name=musical_ly&version_code=330303

Query

    teen_mode_status:      0
    ftc_child_mode:        -1
    is_new_user:           0

Bytes & Encoded data

Now we get to several sets of data that I am unsure how to decode. They generally were mixed in and looked like raw data as well as base64 looking encoded data inside JSONs. These were usually outgoing to log-boot.tiktokv.com/service/2/ . If anyone has some advice for how I can decode those, please let me know, I’d love to try. Below is one of the ones in a json called ‘tt_info’

tt_info: dGMFEAAAlvLESZ997zk5Px-2kAYlx-kHxuOSdiZnulg3tu52u3oUB2f5Q4YZlXJbWNffksbeHYS8
fVPOICA0eRZ_HNph_GdODKIqUG7rHpJXoyo_MaqR8AAtXHFVaHOflwjqxcMO6hq_94eqktKGSD2b
geQOYn4dT9SegkOvBAgp8EK_wvB88CIeiF9nydl1ep-BmZfiz4PTVQTRiLzlQCOlP-Y4zhzSMVip
w5T1oZmsVg0AwkLVMEO0pm56uIxxtKJDwH00px-oSnnf8P5CxtAWxNaVHTRUw27sLRxggc6O5Y0Z
6LZdcORCFTwTEqTEpzbAtxIZmutVjU_IXYenztd5Xqv6OoQBYtDsuB6cHBihG4GUW-lRF3f6z6-C
Mi55QsM9_8EE2pRZr2Z4xNZPWnJHQMWvfHHPGF0hvbaYDls-EhnH1HNK0wZaRGbCBxE0SDd8kDB1
G8buDOg4XsqsaKKZWIQTe9EAZLkhMWRpOja1_mUXxhHFDOcJoZcxo2rErrQRLjTnK40FktrjiD3U
jedjB_6Pq3VNVQO_vuXoC5NsepWQyqWbBIwu7VtAcLQoa_AAlqYt7zn9wvcfii1rhHaS7f2V6E3U
sIouBAgQAjX6MKePQWRlEhOggFBQHoLd74z4ccrRxdu1wj0QmeGVcQdzXY7nyKQ8z8qF5CnohC3F
gSb0ar6vgGuZNmBng-FiLe7N7lY32lyLRZO3B1hZEe04tms_OFta-nDyJocxKjn0vGIaYvFdV4HF
H3JdVClIpd6Iomd4

Here’s another interesting one, maybe it’s reading into it too much, but it goes to passport/device/trust_users and the response trusted_users: null makes me think I’m not yet a trusted user 🤔

POST https://api22-normal-c-useast1a.tiktokv.com/passport/device/trust_users/?iid=7330845107434735365&device_id=7330843323034600966&ac=mobile&channel=googleplay

URLEncoded form

last_sec_user_id:    
d_ticket:            
last_login_way:      -1
last_login_time:     0
last_login_platform: 

RESPONSE:
{
    "data": {
        "trust_users": null
    },
    "message": "success"
}

This one libra was super interesting, in the RESPONSE we see some interesting things. Looks like they are testing AppsFlyer as well as something related to mac address, though unclear what it is:

GET https://libra-va.tiktokv.com/common?
...

RESPONSE:
{
    "client_hash_value": "lCGnf1lHGbMNzL6XcO6aHA==",
    "code": 0,
    "data": {
        "AWERouter_schema_intercept_switch": {
            "type": 2,
            "val": false,
            "vid": 121360057
        },
        "AppLog_sample_switch": {
            "type": 2,
            "val": 1,
            "vid": 121347504
        },
        "AppLog_send_callback_config": {
            "type": 2,
            "val": {
                "ban_header_list": [
                    "mc",
                    "mac_address"
                ],
                "enableDefault": true
            },
            "vid": 121347516
        },
        "AppsFlyRresourceExperiment": {
            "type": 2,
            "val": 1,
            "vid": 121345122
        },
        "BA_policy_number": {
            "type": 2,
            "val": 1,
            "vid": 121376992
        },
...

Setting permissions

Later on in that we also see some lists of whitelisted js (javascript?) services that likely can run in the Webview. Interestingly, there is also a large list for music as well. I tried looking into these services and many seem like likely partners for tools like captcha, payment. faceueditor I couldn’t find though.

        "_jsmanage_tt_js_auth": {
            "type": 2,
            "val": {
                "allowList": [
                    ".sgsnssdk.com",
                    ".isnssdk.com",
                    ".byteoversea.com",
                    ".tiktokcdn.com",
                    ".whizsolve.com",
                    ".snapsolve.com",
                    ".faceueditor.com",
                    ".tiktok.com",
                    ".tiktokv.com",
                    ".ibytedtos.com",
                    ".immers.page",
                    ".capcut.net",
                    ".byteintl.com",
                    ".tiktokcdn-us.com",
                    ".tiktokv-us.com",
                    ".ttwstatic.com",
                    ".soundon.global",
                    ".pipopay.com",
                    ".oneunita.com",
                    ".pipopayment.com",
                    ".g-p-static.com",
                    ".ttlstatic.com",
                    ".tiktokv-eu.com",
                    ".oecstatic.com",
                    ".tiktokv.eu",
                    ".tiktokv.us",
                    ".ttadsmanager.com"
                ],
                "blockList": [],
                "businessLine": 3,
                "geckoUrl": "https://lf16-gecko-source.tiktokcdn.com/obj/tiktok-teko-source-sg/tt/webview/js_manage/tiktok_webview_js_inject_manage/assets/js/tt_js_auth.js",
                "injectTime": "very_beginning",
                "isUseHardCode": false,
                "name": "_jsmanage_tt_js_auth"
            },
            "vid": 121373567
        },
        "_jsmanage_tt_music_perf": {
            "type": 2,
            "val": {
                "allowList": [
                    ".tickets.com",
                    "ticketmaster.",
                    "ticketmaster.at",
                    "ticketmaster.be",
                    "ticketmaster.ch",
                    "ticketmaster.co.nz",
                    "ticketmaster.co.uk",
                    "ticketmaster.com",
                    "ticketmaster.com.au",
                    "ticketmaster.cz",
                    "ticketmaster.ca",
                    "ticketmaster.de",
                    "ticketmaster.dk",
                    "ticketmaster.es",
                    "ticketmaster.fi",
                    "ticketmaster.fr",
                    "ticketmaster.ie",
                    "ticketmaster.it",
                    "ticketmaster.no",
                    "ticketmaster.pl",
                    "ticketmaster.se",
                    "ticketmastergiftcard",
                    "ticketmastergiftcard.com",
                    "ticketmasterpartners.",
                    "tickets.janto.es",
                    "ticketweb.co.uk",
                    "ticketweb.uk",
                    "universe.com",
                    "ticketmaster.evyy.net",
                    ".ticketmaster.",
                    ".ticketmaster.net",
                    ".admission.",
                    ".altitudetickets.",
                    ".amptickets.",
                    ".artsiowa.",
                    ".austintheatre.org.",
                    ".axs.",
                    ".baltimoresoundstage.",
                    ".banknhpavilion.",
                    ".broadwaycenter.org.",
                    ".carolinatix.org.",
                    ".centurylinkarenaboise.",
                    ".chastainseries.",
                    ".cirk.me.",
                    ".cirquedusoliel.",
                    ".cubs.",
                    ".etix.",
                    ".eventbrite.",
                    ".evenue.",
                    ".evenue.net.",
                    ".fgtix.",
                    ".fgtix.to.",
                    ".frontgatetickets.",
                    ".gracelandlive.",
                    ".houstontoyotacenter.",
                    ".ictickets.",
                    ".indians.",
                    ".insomniacshop.",
                    ".instagram.",
                    ".kcstarlight.",
                    ".knoxvilletickets.",
                    ".legendsinconcert.",
                    ".livemu.sc.",
                    ".livenation.",
                    ".livenationpremiumseats.",
                    ".livenationpremiumtickets.",
                    ".lnpromos.com.",
                    ".megaticket.",
                    ".metrapark.",
                    ".mets.",
                    ".mlb.",
                    ".mobilitus.net",
                    ".moshtix.co.nz",
                    ".moshtix.com.au",
                    ".mountbakertheatre.",
                    ".nationals.",
                    ".oceanicentertainment.",
                    ".osheaga.",
                    ".pabsttheater.org.",
                    ".playhousesquare.org.",
                    ".redsox.",
                    ".rutheckerdhall.",
                    ".seetickets.us",
                    ".selectaseat.",
                    ".selectaseatlubbock.",
                    ".shops.ticketmasterpartners.",
                    ".smithstix.",
                    ".spokanearena.",
                    ".strazcenter.org.",
                    ".stubwire.",
                    ".theqarena.",
                    ".ticketalternative.",
                    ".ticketexchangebyticketmaster.",
                    ".ticketf.ly.",
                    ".ticketfly.",
                    ".ticketingcentral.",
                    ".ticketomaha.",
                    ".tickets.",
                    ".ticketsnow.",
                    ".ticketstoday.",
                    ".ticketweb.",
                    ".tigers.",
                    ".toyotacenter.",
                    ".twitch.",
                    ".universe.",
                    ".unlvtickets.",
                    ".upstreammusicfestival.",
                    ".veeps.",
                    ".vendini.",
                    ".vipnation.",
                    ".visulite.",
                    ".waneefestival.",
                    ".wellsfargocenterphilly.",
                    ".wolsteincenter.",
                    ".youtube.",
                    "fgtix.to.",
                    "found.ee.",
                    ".tmdev.co",
                    ".tmtickets.se",
                    "billetnet.dk",
                    "downloadfestival.co.uk",
                    "eticketing.co.uk",
                    "frontgatetickets.com",
                    "moshtix.co.nz",
                    "moshtix.com.au",
                    "rlwc2021.com",
                    "shops.ticketmasterpartners.com"
                ],
                "blockList": [],
                "businessLine": 4,
                "geckoUrl": "https://lf16-gecko-source.tiktokcdn.com/obj/tiktok-teko-source-sg/tt/webview/js_manage/tiktok_webview_js_inject_manage/assets/js/tt_music_perf.js",
                "injectTime": "custom_manual",
                "isUseHardCode": true,
                "name": "_jsmanage_tt_music_perf"
            },
            "vid": 121377589
        },

Advertising

And finally we see the first indications of ads with the Google Advertising ID gaid from the Waydroid device.

GET https://api-boot.tiktokv.com/tiktok/ug/landing/ads/dest/get/v1/

Query
gaid:                  d5f9e1fb-d518-4efe-86d7-250787a2a1a7
ac:                    mobile
channel:               googleplay
aid:                   1233
app_name:              musical_ly
...

Followed by the install conversion event for AppsFlyer, the contents of which are just a raw hex dump I’m not sure what to do with.

POST https://conversions.appsflyer.com/api/v6.4/androidevent?app_id=com.zhiliaoapp.musically&buildnumber=6.4.0 HTTP/1.1

0000000000 b9 ce ac 61 47 19 b4 f7 4f e0 2e 56 6f 9c 0b 82   ...aG...O..Vo...
0000000010 3f 72 47 4f 50 f4 14 6e 46 f6 d6 82 37 6f 55 0a   ?rGOP..nF...7oU.
0000000020 b7 91 d0 93 7f 08 4a fb d5 93 51 cb 23 32 56 52   ......J...Q.#2VR
0000000030 7a b9 f2 49 ac f8 d4 b2 6b 40 bf 7b 52 ed 61 c9   z..I....k@.{R.a.
0000000040 18 07 29 ac 61 37 9b 4c 8a 51 7a 8d 62 3d eb ed   ..).a7.L.Qz.b=..
...

The response from AppsFlyer shortly after correctly identifies me as an organic (I didn’t arrive from an ad) user:

{
    "af_message": "organic install",
    "af_status": "Organic",
    "install_time": "2024-02-02 03:43:50.386"
}

Content & Terms of Service

And now we’re getting to some content, like this still JPEG:


So, now it’s 3 seconds later, and let’s see where we are in the app experience:

{
    "data": {
        "/aweme/v1/compliance/settings/": {
            "body": {
                "about_privacy_policy_url": "https://www.tiktok.com/legal/privacy-policy-row",
                "ad_personality_settings": {
                    "att_status": 255,
                    "is_follow_sys_config": false,
                    "is_np_user": 1,
                    "is_show_settings": false,
                    "limit_ad_tracking": false,
                    "mode": 0,
                    "need_pop_up": false,
                    "pa_revising_switch": false,
                    "pers_ad_data_received_partner_mode": 0,
                    "pers_ad_show_data_received_partner": false,
                    "pers_ad_show_third_party_networks": false,
                    "pers_ad_third_party_networks_mode": 0,
                    "unified_mode": 0
                },
                "age_gate_info": {
                    "age_gate_action": 0,
                    "age_gate_post_action": 0,
                    "register_age_gate_action": 0
                },
                "cmpl_enc": "UNKNOWN",
                "commercial_content_library_url": "https://www.tiktok.com/adlibrary",
                "device_limit_register_expired": true,
                "extra": {
                    "fatal_item_ids": [],
                    "logid": "20240202034347AC84497362B5DDABA59B",
                    "now": 1706845428000
                },
                "idfa_popup_allow": false,
                "interface_control_settings": "{\"rules\":null,\"use_new_control\":true,\"user_type\":\"-1\",\"version\":\"11\"}",
                "log_pb": {
                    "impr_id": "20240202034347AC84497362B5DDABA59B"
                },
                "parental_guardian_name": "Family Pairing",
                "policy_info_list": [
                    {
                        "policy_key": "privacy-policy",
                        "policy_url": "https://www.tiktok.com/legal/privacy-policy"
                    },
                    {
                        "policy_key": "terms-of-service",
                        "policy_url": "https://www.tiktok.com/legal/terms-of-service"
                    },
                    {
                        "policy_key": "tiktok-shoutouts-user-terms-of-service",
                        "policy_url": "https://www.tiktok.com/legal/tiktok-shoutouts-user-terms-of-service"
                    },
                    {
                        "policy_key": "cookie-policy",
                        "policy_url": "https://www.tiktok.com/legal/cookie-policy"
                    },
                    {
                        "policy_key": "virtual-items",
                        "policy_url": "https://www.tiktok.com/legal/virtual-items"
                    },
                    {
                        "policy_key": "rewards-policy-eea",
                        "policy_url": "https://www.tiktok.com/legal/rewards-policy-eea"
                    },
                    {
                        "policy_key": "privacy-policy-for-younger-users",
                        "policy_url": "https://www.tiktok.com/legal/privacy-policy-for-younger-users"
                    },
                    {
                        "policy_key": "copyright-policy",
                        "policy_url": "https://www.tiktok.com/legal/copyright-policy"
                    },
                    {
                        "policy_key": "changes-to-personalised-advertising-in-the-eea",
                        "policy_url": "https://www.tiktok.com/legal/changes-to-personalised-advertising-in-the-eea"
                    }
                ],
                "policy_notice_enable": true,
                "status_code": 0,
                "terms_consent_for_register_info_new_users": {
                    "checkbox_agree_all_terms": "Agree to all",
                    "checkbox_privacy_policy": "Consent to the collection and use of personal information (Required)",
                    "checkbox_terms_of_use": "Consent to the Terms of Service (Required)",
                    "checkbox_tr_notification_subtitle": "Get notifications about trending videos and promotions on TikTok. You can review and edit your settings at any time. Not allowing this type of notification does not limit your use of the TikTok service.",
                    "checkbox_tr_notification_title": "Consent to the receipt of trending content and promotional notifications (Optional)",
                    "tiktok_privacy_policy_url": "https://www.tiktok.com/legal/terms-and-conditions-kr?lang=ko-KR",
                    "tiktok_terms_of_use_url": "https://www.tiktok.com/legal/page/row/terms-of-service/ko-KR",
                    "title": "Terms and conditions"
                }
            },
...

Let’s tap agree and see what happens.

POST https://api22-normal-c-useast1a.tiktokv.com/consent/api/record/create/v1?...

entity_keys:   conditions-policy-device-consent
business_flow: consent_box
status:        1

And an updated set of compliance, similar to the one from above.

GET https://api22-normal-c-useast1a.tiktokv.com/aweme/v1/compliance/settings/?teen_mode_status=0&ftc_child_mode=-1&is_new_user=0
...

RESPONSE:
{
    "about_privacy_policy_url": "https://www.tiktok.com/legal/privacy-policy-row",
    "ad_personality_settings": {
        "ad_free_subscription": {
            "subscription_mode": 0
        },
        "att_status": 255,
        "description": "With this setting, the ads you see on TikTok can be more tailored to your interests based on data that advertising partners share with us about your activity on their apps and websites.\nYou will always see ads on TikTok based on what you do on TikTok or other data described in our privacy policy.",
        "disable_att_overwrite_pa": 1,
        "enable_toggle_decoupling": true,
        "is_follow_sys_config": false,
        "is_new_user": 1,
        "is_np_user": 0,
        "is_show_3p_data_control": false,
        "is_show_reset_entry": false,
        "is_show_settings": true,
        "is_teenager_mode": 0,
        "limit_ad_tracking": false,
        "mode": 1,
        "need_pop_up": false,
        "pa_revising_switch": false,
        "pers_ad_data_received_partner_mode": 0,
        "pers_ad_main_mode_title": "Using Off-TikTok activity for ad targeting",
        "pers_ad_show_data_received_partner": false,
        "pers_ad_show_interest_label": true,
        "pers_ad_show_third_part_measurement": false,
        "pers_ad_show_third_party_networks": false,
        "pers_ad_third_party_networks_mode": 0,
        "show_advertiser_settings": true,
        "unified_mode": 0,
        "use_new_interests": 1
    },
    "age_gate_info": {
        "age_gate_action": 0,
        "age_gate_post_action": 0,
        "register_age_gate_action": 2
    },

After Terms Of Service: And we’re off!

The app shows the video stream and the API requests are flying back and forth now, many a second, so there’s lots more to look at next time. Overall the data seems pretty standard with a few interesting things to look into like the still encrypted data, the connected services and of course, everything after you agree to those Terms of Service.

ClickHouse: Refreshing Take on Materialized Views

Working on my Open Source MMP I have been seeing how much of it will work with ClickHouse. Unfortunately combining multiple streaming data sources in ClickHouse was proving difficult as data would semi randomly not join correctly.

This led to a potential fix by using the very recent ClickHouse refreshable materialized views. These refreshable views work much closer to how materialized views work in other databases like PostgreSQL but come with additional benefits like setting the refreshes inside the materialized view logic itself.

Official Documentation:
https://clickhouse.com/docs/en/sql-reference/statements/create/view#refreshable-materialized-view

At the time of writing, refreshable views are new enough that they require setting this to enable support. Since this is needed per session in which a table is created, you will need to use this setting. This shouldn’t be necessary after Q2 2024.

SET allow_experimental_refreshable_materialized_view = 1;

Then to make the view refreshable you just add REFRESH EVERY {INT} {TIME} like this:

CREATE MATERIALIZED VIEW attribute_impressions_mv
REFRESH EVERY 5 SECOND
TO attributed_impressions -- Specify the destination table
AS
WITH
merged_impression_event AS (
    -- Ranked rows by impression time
    SELECT 
            app.event_time AS app_event_time,
            app.store_id AS store_id,
            app.event_id,
            ...

What I then discovered though was that once a refreshable view is used, it will replace the entire contents of the destination table. This means that they likely will need further management in the future as the data grows. Additionally, they cannot be combined with the columnar data inserts I was using previously with AggregatingMergeTree.

So, once you add a refreshable view, all higher level aggregations will likely all have to be refreshable as well.

In the end, I was able to use refreshable views to solve my problem, but I am not certain they wont cause delays later due to the way they seem to refresh across all data. Perhaps I’ll have to find a way to narrow the range they aggregate.

appgoblin.info

AppGoblin: Free App Stats & Info

The past few weeks I spent building a front-end for an app store crawler that I made. The end goal is just to provide an open front-end for the apps details for the approximate 2.5m apps (already 1/2 those are no longer live) I’ve crawled from the Google & Apple stores. The site is now live on appgoblin.info and hosts a variety of Android and iOS stats and store information. Hopefully it’s something that some marketers may find interesting, and the code is all free on Github.

History

app-ads.txt dash

Originally this came out of a curiosity I had about the IAB app-ads.txt standard in 2022. In order to scrape app-ads.txt files I had to first get all the iOS and Android apps + their respective publisher URLs. This ended with a small dashboard I put on https://ads.jamesoclaire.com/dash/ads but I never expanded it further.

While that project, like app-ads.txt itself, went unused, I remained interested in letting the database of apps keep growing.

Then in the fall of 2023 while on hiatus from real work, I started working on an API and UI. Creating the UI gave me exactly what I wanted, which was use cases for displaying the data. Adding one feature led to the next so now there are several sections to browse.

Features

App Details

Each app has it’s own page for all the data pulled from the app store as well as second order of information derived from that data such as installs per day or change in ratings over time.


App Rankings

Since we have each apps history, it’s easy to build the historical ranking charts. In this example on the right, you can see a game quickly climbing multiple charts at once over the past month. This change in rankings is a great way to track an apps popularity over time.


Ranking Charts

No site like this would be without it’s own app rankings page. This is straight from the Google & Apple stores top charts each day for tracking apps overtime.


New Apps

Now this might seem like an obvious one, but it really brought home the slow burn that the app stores barely surface new apps anymore. Long gone from both stores are any structured “new” sections which showcase new apps. The cynical answer here is that those sections actively take away from the ad-revenue generated by the in store advertising that makes up a large percentage of the apps on screen at any given time in each store.


So, why does AppGoblin exist?

Yes, there are a large number of similar offerings like data.ai and sensortower.com which have many more features. For me, I enjoyed learning more front-end and Javascript as well as give me something to do with the scraper which I had built previously (whats the point of a database if you cant use it somehow). For now App Goblin is simply here as a resource free of charge as it’s running costs only $20 a month or so in hosting.

Open Source Tech Stack

First is the underlying data which is collected and maintained from github.com/ddxv/adscrawler The adscrawler uses Python & PostgreSQL to crawl app stores to collect apps and their basic information like names and rating counts. This data is then stored to the PostgreSQL database. I also use some of the original parts of github.com/ddxv/app-ads-dash for monitoring the scraper. Hopefully in the future the parts specific to the scraper are moved to that repository.

The website itself is also open source and can be found at github.com/ddxv/app-store-dash The website is built from a Python backend API based on LiteStar which manages queries to the PostgreSQL database. The API is used by a JavaScript Svelte frontend with Tailwind and Skeleton for CSS. All projects are glued together with Systemd and web sockets where needed. Everything is hosted on AWS in the small EC2 instances.

As always, if you have any interest or questions please feel free to reach out.

Keeping Mobile Gamers Engaged with App Widgets

I love using widgets. I probably have too many of them on my phone’s home screen. News apps, weather apps, stocks, stats. I love being able to glance through them without opening the corresponding apps. This led me to realize though that I have seen very few widgets for mobile games. As I started working on making widgets, I realized you could actually make simple games inside of widgets.

Many apps use widgets already, but I think due to poor tooling, not many games have embraced widgets as a part of their game play loop. Recently I started looking into what some of the possibilities around making a simple game which could be played inside of a widget.

Widget Limitations:

  1. Very limited dragging or other complex touches. This means no drag/swipe functionality. You can implement drag for lists, but picking up an item and dragging it around seems out of the question.
  2. Animations are quite limited. I don’t think implementing most game animations would be possible.
  3. Computations are subject to being cut off at any point. The widget must essentially be at rest at all times, this definitely limits what you can do, but still fits well for some gaming uses.

So, let’s try a Tic-Tac-Toe Widget

I chose Tic-Tac-Toe since it was the game I could think of with the least logic. My best guess was that implementing the game logic might be tricky or troublesome. Additionally, the turn based logic was perfect for falling into the background when the game is not being played.

This turned out to be pretty easy. I was able to get a prototype up and running within a day or so. One surprise I had was having trouble using a delay. In the end I gave up trying to implement a half second delay to simulate the computer ‘thinking’. I’m not sure if that is something that could be addressed later or not.

The game is mostly limited by the fact that it’s just tic-tac-toe that you’re playing, which gets old fast.

What about a more difficult puzzle game?

Next I went with a simple sliding puzzle game. The goal is to slide the pieces to slowly sort them into the correct order. These could be a puzzle image or as you see on the side just pure numbers. This would definitely be a bit flashier if I used images, and I think that is also doable for the future.

Though this is barely more complex than Tic-Tac-Toe the difficulties became apparent quickly. The biggest issue I faced was that the Android operating system will randomly shut down the widget’s thread which would cause the game to lose anything currently stored in memory.

This means that the game state can’t be stored in memory, but needs to be backed up to disk each time, then recovered. So each ‘turn’ is played by the user, then the game state is saved to disk, and when the widget is interacted with a second time, the game state is read back from disk first.

This means the game is quite solid and recovers from it’s process being ended without the player knowing, but it is just a bit laggy when ‘playing’ quickly as you can see in the gif.

That’s cool, but what about non puzzle games?

This is where I started getting more excited. While I think you could recreate many simple puzzle games as widgets, what is the point for the developer? While the player might enjoy some play outside the app any of the more pleasant animations or clever game play mechanics would remain out of reach for the game widget.

But what the widget does give the developer is a large real estate for the the player’s home screen. This could be a place for managing notifications to entice the player to open the game. A few examples:

  1. Connection to in game functions like responding to events or setting a status
  2. Notification that an in game timer is finished
  3. Countdown to live events
  4. Button to collect daily reward and open game to boost game re-engagement
  5. Community Bulletin board

I think the ideas here are pretty basic, but could really help to reconnect games with their existing user bases.

If anyone out there has any interest in trying out the idea of building a widget for a game I’d be happy to help.

If you’re curious about either of the above examples you can get both on Google play and all code on GitHub:

Tic-Tac-Toe Widget on Google Play and Tic-Tac-Toe Android Code on GitHub

Number Puzzle Widget on Google Play and Number Puzzle Code on GitHub

Stay up to date with the latest tech new with Hacker News Widget & App

Hacker News is one of my go to sites to read the best takes on the latest tech news.

Probably the reason I keep going back to it is the honest discussions and supportive community. But you probably already know this, since Hacker News is one of the most popular tech news sources out there.

And though there are dozens of high(er) quality Hacker News apps out there, I’ve always wished they supported widgets. In fact most apps I use I wish they supported Widgets.

I love waking up in the morning and checking the latest news from my home screen, rather than opening up web pages.
This is why I decided to use Hacker News to make my first Android app and widget.

The whole experience was absolutely amazing compared to my recent attempt to use Unity. Android Studio + Kotlin and Jetpack Compose were an absolute dream to use. I rarely felt frustrated and was able to envision a few more interesting projects using widgets coming up next week.

In the meantime, please feel free to use my app or code:

On Google Play

Or Clone it On GitHub