Is this a TikTok security vulnerablity for ad fraud?

I de-compiled TikTok (“com.zhiliaoapp.musically” v33.3.3 from Feb 2nd, 2024) from ApkPure.net and noticed that the way it called AppsFlyer looked a bit different than what I expected and quickly led me to a GitHub issue which makes it seem like they are using an outdated way to collect install information from Google Play which may have security vulnerabilities. Particularly, could a malicious app use this to ‘steal’ TikTok install attributions?

In this setup you see that TikTok is using a receiver com.appsflyer.SingleInstallBroadcastReceiver to listen for the com.android.vending.INSTALL_REFERRER event. This might allow a malicious app to listen for for the INSTALL_REFERRER event. You can see this is not recommended by AppsFlyer in this GitHub issue and is not their recommended installation setup.

<application>
 <receiver android:exported="true" android:name="com.appsflyer.MultipleInstallBroadcastReceiver">
        	<intent-filter>
            	<action android:name="com.android.vending.INSTALL_REFERRER"/>
        	</intent-filter>
    </receiver>
    <receiver android:exported="true" android:name="com.appsflyer.SingleInstallBroadcastReceiver">
        	<intent-filter>
            	<action android:name="com.android.vending.INSTALL_REFERRER"/>
        	</intent-filter>
    </receiver>
</application>

The right way?

This is the standard way that the AppsFlyer setup will be implemented, which in turn will use the google store referrer.

   <queries>
        <intent>
            <action android:name="com.appsflyer.referrer.INSTALL_PROVIDER"/>
        </intent>
   </queries>

I think the potential here might that if this version of TikTok is leaking the INSTALL_REFERRER data then

  • It could provide a malicious app valuable information in attempting to steal information about the source of TikTok’s users.
  • Additionally, a malicious app might be able to perform click jacking given that they know some information about the source of the install very early on.
  • And finally since the INSTALL_REFERRER is broadly scoped the malicious app could then send the a false INSTALL_REFERRER which TikTok may not be able to validate.

Before You Agree: What data does TikTok collect before Terms of Service?

Would you like to see what data is coming out of TikTok when you first open it up? Let’s get to it.

Didn’t this used to be easy to do?

As security for iPhones and Androids increased it continually made viewing the traffic leaving your own device more difficult. This is in stark contrast to a regular web browser where you can easily open the network traffic for any site to see the back and forth of the network calls while you visit that site. For mobile apps, if you want to see the traffic on a computer, you need to do a proxy with a custom CA certificate. For years this was with Charles Proxy, Burpsuite and Wireshark. But apps continued to evolve and many began to put their own custom CA certificates inside the apps, meaning it was no longer enough to just forward the traffic to a proxy, you now had to unpin the SLL certificate within the app. This flow below is still possible with Charles or others, but in the end I ended up using the following tools.

  • Waydroid – Android Emulator: To install the target app
    • + Magisk for root + allowing user certificates
    • + LSPosed for SSL unpinning
  • MITM Proxy: to capture and view the decrypted traffic

If you’d like more detailed instructions for viewing TikTok HTTPS traffic you can find them on GitHub.

Installing TikTok

I got the APK from APKpure.net and installed it into the emulator. We rolled the dice on a lucky day and happened to download TikTok v33.3.3 so you’ll be seeing that number a lot later. Then with the MITM proxy running, I opened the app for the first time.

Launching TikTok

All the request in the rest of the post are fired within 2-3 seconds of launching, so lots of back and forth within the app and on the network.

Keep in mind, this is all traffic from the emulator, so there are other network requests mixed in that are not from TikTok.


The first one that is for sure from TikTok. You you can see the request to tiktokv.com/monitor/appmonitor/v3/settings with some basic settings being recorded.

GET https://mon-va.tiktokv.com/monitor/appmonitor/v3/settings?update_version_code=30107125&os=Android&app_version=3.1.7-rc.75.oversea&device_id=&channel=release&aid=2010&crash=npth HTTP/1.1
Content-Type: application/json; charset=utf-8

Query

update_version_code: 30107125
os:                  Android
app_version:         3.1.7-rc.75.oversea
device_id:           
channel:             release
aid:                 2010
crash:               npth

This is followed shortly after with a Firebase installation:

POST https://firebaseinstallations.googleapis.com/v1/projects/musically-c51f4/installations HTTP/1.1
Content-Type: application/json

{
    "appId": "1:340331662088:android:3c6c52c4762af402",
    "authVersion": "FIS_v2",
    "fid": "cMbnNXy6QCGDpHT_mB3TXN",
    "sdkVersion": "a:17.0.1"
}

User Data

Here we see TikTok trying to figure out the carrier_region and network_sim_region all of which are empty due to Waydroid emulator not having a sim card.

POST https://api-boot.tiktokv.com/region/submit/?sdk_version=1.2.2&ac=mobile&channel=googleplay&aid=1233&app_name=musical_ly&version_code=330303&version_name=33.3.3&device_platform=android&os=android&ab_version=33.3.3&ssmix=a&device_type=WayDroid+x86_64+Device&device_brand=waydroid&language=en&os_api=30&os_version=11&openudid=6b623cdca1a14c25&manifest_version_code=2023303030&resolution=3456*1970&dpi=360&update_version_code=2023303030&_rticket=1706845427021&is_pad=1&app_type=normal&sys_region=US&timezone_name=Asia%2FTaipei&app_language=en&ac2=unknown&uoo=1&op_region=US&timezone_offset=28800&build_number=33.3.3&host_abi=armeabi-v7a&locale=en&region=US&ts=1706845427&cdid=e1907a45-45ae-4508-b5e2-171d7964d125 HTTP/1.1

Content-Type: application/json

JSON
  

{
    "carrier_region": "",
    "locale": "en_US",
    "mcc_mnc": "",
    "network_sim_region": "",
    "system_language": "en",
    "system_region": "US"
}

Now we see the main set of data coming out of the device. This information is repeated in most other requests back and forth. Let’s note some of the interesting ones:

  • op_region/sys_region: US
  • device_type: WayDroid x86_64 Device
  • device_brand: waydroid
  • language: en
  • os_api: 30
  • os_version: 11
  • openudid: 6b623cdca1a14c25
  • manifest_version_code: 2023303030
  • resolution: 3456*1970
  • dpi: 360

These are all accurate to the emulator Waydroid, running on my laptop, note the resolution. The op_region/local US and en are related to the Google Play account I am signed into with on the phone, while Asia/Taipei is the current timezone.

GET https://api-boot.tiktokv.com/passport/account/info/v2/?scene=normal&multi_login=1&account_sdk_source=app&passport-sdk-version=19&ac=mobile&channel=googleplay&aid=1233&app_name=musical_ly&version_code=330303&version_name=33.3.3&device_platform=android&os=android&ab_version=33.3.3&ssmix=a&device_type=WayDroid+x86_64+Device&device_brand=waydroid&language=en&os_api=30&os_version=11&openudid=6b623cdca1a14c25&manifest_version_code=2023303030&resolution=3456*1970&dpi=360&update_version_code=2023303030&_rticket=1706845427440&is_pad=1&app_type=normal&sys_region=US&timezone_name=Asia%2FTaipei&app_language=en&ac2=unknown&uoo=0&op_region=US&timezone_offset=28800&build_number=33.3.3&host_abi=armeabi-v7a&locale=en&region=US&ts=1706845427&cdid=e1907a45-45ae-4508-b5e2-171d7964d125&support_webview=1&okhttp_version=4.2.137.48-tiktok&use_store_region_cookie=1 HTTP/1.1
Accept-Encoding: gzip
x-tt-app-init-region: carrierregion=;mccmnc=;sysregion=US;appregion=US
sdk-version: 2
x-tt-dm-status: login=0;ct=0;rt=7
X-SS-REQ-TICKET: 1706845427445
passport-sdk-version: 19
pns_event_id: 9
x-vc-bdturing-sdk-version: 2.3.5.i18n
User-Agent: com.zhiliaoapp.musically/2023303030 (Linux; U; Android 11; en_US; WayDroid x86_64 Device; Build/RQ3A.211001.001;tt-ok/3.12.13.4-tiktok)
X-Ladon: T3YLBE4LXYRR2+5qAqfacGfZklgSFiMMsQJG+LLwWS/K2KTv
X-Khronos: 1706845427
X-Argus: 7GLcN9rck1opShLax2FYZ6IEse0/+7WsuaKmyIiBDG+6fGK3w28aOXo0P7vp+fn22ye3LeqP7I4Ae7hQ/WjqBfQLsgEqgEj/jXsiIw7KXBcp/70AXu0vGG/KKiwGBTCmPaGv38VAQGTzWfyiwLU1K7IcxJ4QInnEhmQjmzf2A6trku+NcY0FBApGxTeWBHipRBSvOrkMnuzT5hIq/99QXXgTId9ylIiqjilm9iGrk0aU3JnpA9BKsXQonstxlmpQrJqApi3c7FOVdHJs5ZwatAkRNVNXvQP5gVdQZcCrILMDcw==
X-Gorgon: 0404c05b000015eb3032a8aa4fd780a54ff4e00eabc7dc70d487
Host: api-boot.tiktokv.com
Connection: Keep-Alive
Query
  

scene:                   normal
multi_login:             1
account_sdk_source:      app
passport-sdk-version:    19
ac:                      mobile
channel:                 googleplay
aid:                     1233
app_name:                musical_ly
version_code:            330303
version_name:            33.3.3
device_platform:         android
os:                      android
ab_version:              33.3.3
ssmix:                   a
device_type:             WayDroid x86_64 Device
device_brand:            waydroid
language:                en
os_api:                  30
os_version:              11
openudid:                6b623cdca1a14c25
manifest_version_code:   2023303030
resolution:              3456*1970
dpi:                     360
update_version_code:     2023303030
_rticket:                1706845427440
is_pad:                  1
app_type:                normal
sys_region:              US
timezone_name:           Asia/Taipei
app_language:            en
ac2:                     unknown
uoo:                     0
op_region:               US
timezone_offset:         28800
build_number:            33.3.3
host_abi:                armeabi-v7a
locale:                  en
region:                  US
ts:                      1706845427
cdid:                    e1907a45-45ae-4508-b5e2-171d7964d125
support_webview:         1
okhttp_version:          4.2.137.48-tiktok
use_store_region_cookie: 1

Next we see calls for api-boot.tiktokv.com/aweme/v1/compliance which note that I am not a teen or child. Additionally is_new_user=0 which must mean that I’ve before downloaded TikTok with this Google account. Keep in mind, this is all within a second or two of running the app, so I assume this must be coming from Android operating system, and it’s possible last year I had installed TikTok at some point on the emulator, though I don’t remember that.

GET https://api-boot.tiktokv.com/aweme/v1/compliance/settings/?teen_mode_status=0&ftc_child_mode=-1&is_new_user=0&ac=mobile&channel=googleplay&aid=1233&app_name=musical_ly&version_code=330303

Query

    teen_mode_status:      0
    ftc_child_mode:        -1
    is_new_user:           0

Bytes & Encoded data

Now we get to several sets of data that I am unsure how to decode. They generally were mixed in and looked like raw data as well as base64 looking encoded data inside JSONs. These were usually outgoing to log-boot.tiktokv.com/service/2/ . If anyone has some advice for how I can decode those, please let me know, I’d love to try. Below is one of the ones in a json called ‘tt_info’

tt_info: dGMFEAAAlvLESZ997zk5Px-2kAYlx-kHxuOSdiZnulg3tu52u3oUB2f5Q4YZlXJbWNffksbeHYS8
fVPOICA0eRZ_HNph_GdODKIqUG7rHpJXoyo_MaqR8AAtXHFVaHOflwjqxcMO6hq_94eqktKGSD2b
geQOYn4dT9SegkOvBAgp8EK_wvB88CIeiF9nydl1ep-BmZfiz4PTVQTRiLzlQCOlP-Y4zhzSMVip
w5T1oZmsVg0AwkLVMEO0pm56uIxxtKJDwH00px-oSnnf8P5CxtAWxNaVHTRUw27sLRxggc6O5Y0Z
6LZdcORCFTwTEqTEpzbAtxIZmutVjU_IXYenztd5Xqv6OoQBYtDsuB6cHBihG4GUW-lRF3f6z6-C
Mi55QsM9_8EE2pRZr2Z4xNZPWnJHQMWvfHHPGF0hvbaYDls-EhnH1HNK0wZaRGbCBxE0SDd8kDB1
G8buDOg4XsqsaKKZWIQTe9EAZLkhMWRpOja1_mUXxhHFDOcJoZcxo2rErrQRLjTnK40FktrjiD3U
jedjB_6Pq3VNVQO_vuXoC5NsepWQyqWbBIwu7VtAcLQoa_AAlqYt7zn9wvcfii1rhHaS7f2V6E3U
sIouBAgQAjX6MKePQWRlEhOggFBQHoLd74z4ccrRxdu1wj0QmeGVcQdzXY7nyKQ8z8qF5CnohC3F
gSb0ar6vgGuZNmBng-FiLe7N7lY32lyLRZO3B1hZEe04tms_OFta-nDyJocxKjn0vGIaYvFdV4HF
H3JdVClIpd6Iomd4

Here’s another interesting one, maybe it’s reading into it too much, but it goes to passport/device/trust_users and the response trusted_users: null makes me think I’m not yet a trusted user 🤔

POST https://api22-normal-c-useast1a.tiktokv.com/passport/device/trust_users/?iid=7330845107434735365&device_id=7330843323034600966&ac=mobile&channel=googleplay

URLEncoded form

last_sec_user_id:    
d_ticket:            
last_login_way:      -1
last_login_time:     0
last_login_platform: 

RESPONSE:
{
    "data": {
        "trust_users": null
    },
    "message": "success"
}

This one libra was super interesting, in the RESPONSE we see some interesting things. Looks like they are testing AppsFlyer as well as something related to mac address, though unclear what it is:

GET https://libra-va.tiktokv.com/common?
...

RESPONSE:
{
    "client_hash_value": "lCGnf1lHGbMNzL6XcO6aHA==",
    "code": 0,
    "data": {
        "AWERouter_schema_intercept_switch": {
            "type": 2,
            "val": false,
            "vid": 121360057
        },
        "AppLog_sample_switch": {
            "type": 2,
            "val": 1,
            "vid": 121347504
        },
        "AppLog_send_callback_config": {
            "type": 2,
            "val": {
                "ban_header_list": [
                    "mc",
                    "mac_address"
                ],
                "enableDefault": true
            },
            "vid": 121347516
        },
        "AppsFlyRresourceExperiment": {
            "type": 2,
            "val": 1,
            "vid": 121345122
        },
        "BA_policy_number": {
            "type": 2,
            "val": 1,
            "vid": 121376992
        },
...

Setting permissions

Later on in that we also see some lists of whitelisted js (javascript?) services that likely can run in the Webview. Interestingly, there is also a large list for music as well. I tried looking into these services and many seem like likely partners for tools like captcha, payment. faceueditor I couldn’t find though.

        "_jsmanage_tt_js_auth": {
            "type": 2,
            "val": {
                "allowList": [
                    ".sgsnssdk.com",
                    ".isnssdk.com",
                    ".byteoversea.com",
                    ".tiktokcdn.com",
                    ".whizsolve.com",
                    ".snapsolve.com",
                    ".faceueditor.com",
                    ".tiktok.com",
                    ".tiktokv.com",
                    ".ibytedtos.com",
                    ".immers.page",
                    ".capcut.net",
                    ".byteintl.com",
                    ".tiktokcdn-us.com",
                    ".tiktokv-us.com",
                    ".ttwstatic.com",
                    ".soundon.global",
                    ".pipopay.com",
                    ".oneunita.com",
                    ".pipopayment.com",
                    ".g-p-static.com",
                    ".ttlstatic.com",
                    ".tiktokv-eu.com",
                    ".oecstatic.com",
                    ".tiktokv.eu",
                    ".tiktokv.us",
                    ".ttadsmanager.com"
                ],
                "blockList": [],
                "businessLine": 3,
                "geckoUrl": "https://lf16-gecko-source.tiktokcdn.com/obj/tiktok-teko-source-sg/tt/webview/js_manage/tiktok_webview_js_inject_manage/assets/js/tt_js_auth.js",
                "injectTime": "very_beginning",
                "isUseHardCode": false,
                "name": "_jsmanage_tt_js_auth"
            },
            "vid": 121373567
        },
        "_jsmanage_tt_music_perf": {
            "type": 2,
            "val": {
                "allowList": [
                    ".tickets.com",
                    "ticketmaster.",
                    "ticketmaster.at",
                    "ticketmaster.be",
                    "ticketmaster.ch",
                    "ticketmaster.co.nz",
                    "ticketmaster.co.uk",
                    "ticketmaster.com",
                    "ticketmaster.com.au",
                    "ticketmaster.cz",
                    "ticketmaster.ca",
                    "ticketmaster.de",
                    "ticketmaster.dk",
                    "ticketmaster.es",
                    "ticketmaster.fi",
                    "ticketmaster.fr",
                    "ticketmaster.ie",
                    "ticketmaster.it",
                    "ticketmaster.no",
                    "ticketmaster.pl",
                    "ticketmaster.se",
                    "ticketmastergiftcard",
                    "ticketmastergiftcard.com",
                    "ticketmasterpartners.",
                    "tickets.janto.es",
                    "ticketweb.co.uk",
                    "ticketweb.uk",
                    "universe.com",
                    "ticketmaster.evyy.net",
                    ".ticketmaster.",
                    ".ticketmaster.net",
                    ".admission.",
                    ".altitudetickets.",
                    ".amptickets.",
                    ".artsiowa.",
                    ".austintheatre.org.",
                    ".axs.",
                    ".baltimoresoundstage.",
                    ".banknhpavilion.",
                    ".broadwaycenter.org.",
                    ".carolinatix.org.",
                    ".centurylinkarenaboise.",
                    ".chastainseries.",
                    ".cirk.me.",
                    ".cirquedusoliel.",
                    ".cubs.",
                    ".etix.",
                    ".eventbrite.",
                    ".evenue.",
                    ".evenue.net.",
                    ".fgtix.",
                    ".fgtix.to.",
                    ".frontgatetickets.",
                    ".gracelandlive.",
                    ".houstontoyotacenter.",
                    ".ictickets.",
                    ".indians.",
                    ".insomniacshop.",
                    ".instagram.",
                    ".kcstarlight.",
                    ".knoxvilletickets.",
                    ".legendsinconcert.",
                    ".livemu.sc.",
                    ".livenation.",
                    ".livenationpremiumseats.",
                    ".livenationpremiumtickets.",
                    ".lnpromos.com.",
                    ".megaticket.",
                    ".metrapark.",
                    ".mets.",
                    ".mlb.",
                    ".mobilitus.net",
                    ".moshtix.co.nz",
                    ".moshtix.com.au",
                    ".mountbakertheatre.",
                    ".nationals.",
                    ".oceanicentertainment.",
                    ".osheaga.",
                    ".pabsttheater.org.",
                    ".playhousesquare.org.",
                    ".redsox.",
                    ".rutheckerdhall.",
                    ".seetickets.us",
                    ".selectaseat.",
                    ".selectaseatlubbock.",
                    ".shops.ticketmasterpartners.",
                    ".smithstix.",
                    ".spokanearena.",
                    ".strazcenter.org.",
                    ".stubwire.",
                    ".theqarena.",
                    ".ticketalternative.",
                    ".ticketexchangebyticketmaster.",
                    ".ticketf.ly.",
                    ".ticketfly.",
                    ".ticketingcentral.",
                    ".ticketomaha.",
                    ".tickets.",
                    ".ticketsnow.",
                    ".ticketstoday.",
                    ".ticketweb.",
                    ".tigers.",
                    ".toyotacenter.",
                    ".twitch.",
                    ".universe.",
                    ".unlvtickets.",
                    ".upstreammusicfestival.",
                    ".veeps.",
                    ".vendini.",
                    ".vipnation.",
                    ".visulite.",
                    ".waneefestival.",
                    ".wellsfargocenterphilly.",
                    ".wolsteincenter.",
                    ".youtube.",
                    "fgtix.to.",
                    "found.ee.",
                    ".tmdev.co",
                    ".tmtickets.se",
                    "billetnet.dk",
                    "downloadfestival.co.uk",
                    "eticketing.co.uk",
                    "frontgatetickets.com",
                    "moshtix.co.nz",
                    "moshtix.com.au",
                    "rlwc2021.com",
                    "shops.ticketmasterpartners.com"
                ],
                "blockList": [],
                "businessLine": 4,
                "geckoUrl": "https://lf16-gecko-source.tiktokcdn.com/obj/tiktok-teko-source-sg/tt/webview/js_manage/tiktok_webview_js_inject_manage/assets/js/tt_music_perf.js",
                "injectTime": "custom_manual",
                "isUseHardCode": true,
                "name": "_jsmanage_tt_music_perf"
            },
            "vid": 121377589
        },

Advertising

And finally we see the first indications of ads with the Google Advertising ID gaid from the Waydroid device.

GET https://api-boot.tiktokv.com/tiktok/ug/landing/ads/dest/get/v1/

Query
gaid:                  d5f9e1fb-d518-4efe-86d7-250787a2a1a7
ac:                    mobile
channel:               googleplay
aid:                   1233
app_name:              musical_ly
...

Followed by the install conversion event for AppsFlyer, the contents of which are just a raw hex dump I’m not sure what to do with.

POST https://conversions.appsflyer.com/api/v6.4/androidevent?app_id=com.zhiliaoapp.musically&buildnumber=6.4.0 HTTP/1.1

0000000000 b9 ce ac 61 47 19 b4 f7 4f e0 2e 56 6f 9c 0b 82   ...aG...O..Vo...
0000000010 3f 72 47 4f 50 f4 14 6e 46 f6 d6 82 37 6f 55 0a   ?rGOP..nF...7oU.
0000000020 b7 91 d0 93 7f 08 4a fb d5 93 51 cb 23 32 56 52   ......J...Q.#2VR
0000000030 7a b9 f2 49 ac f8 d4 b2 6b 40 bf 7b 52 ed 61 c9   z..I....k@.{R.a.
0000000040 18 07 29 ac 61 37 9b 4c 8a 51 7a 8d 62 3d eb ed   ..).a7.L.Qz.b=..
...

The response from AppsFlyer shortly after correctly identifies me as an organic (I didn’t arrive from an ad) user:

{
    "af_message": "organic install",
    "af_status": "Organic",
    "install_time": "2024-02-02 03:43:50.386"
}

Content & Terms of Service

And now we’re getting to some content, like this still JPEG:


So, now it’s 3 seconds later, and let’s see where we are in the app experience:

{
    "data": {
        "/aweme/v1/compliance/settings/": {
            "body": {
                "about_privacy_policy_url": "https://www.tiktok.com/legal/privacy-policy-row",
                "ad_personality_settings": {
                    "att_status": 255,
                    "is_follow_sys_config": false,
                    "is_np_user": 1,
                    "is_show_settings": false,
                    "limit_ad_tracking": false,
                    "mode": 0,
                    "need_pop_up": false,
                    "pa_revising_switch": false,
                    "pers_ad_data_received_partner_mode": 0,
                    "pers_ad_show_data_received_partner": false,
                    "pers_ad_show_third_party_networks": false,
                    "pers_ad_third_party_networks_mode": 0,
                    "unified_mode": 0
                },
                "age_gate_info": {
                    "age_gate_action": 0,
                    "age_gate_post_action": 0,
                    "register_age_gate_action": 0
                },
                "cmpl_enc": "UNKNOWN",
                "commercial_content_library_url": "https://www.tiktok.com/adlibrary",
                "device_limit_register_expired": true,
                "extra": {
                    "fatal_item_ids": [],
                    "logid": "20240202034347AC84497362B5DDABA59B",
                    "now": 1706845428000
                },
                "idfa_popup_allow": false,
                "interface_control_settings": "{\"rules\":null,\"use_new_control\":true,\"user_type\":\"-1\",\"version\":\"11\"}",
                "log_pb": {
                    "impr_id": "20240202034347AC84497362B5DDABA59B"
                },
                "parental_guardian_name": "Family Pairing",
                "policy_info_list": [
                    {
                        "policy_key": "privacy-policy",
                        "policy_url": "https://www.tiktok.com/legal/privacy-policy"
                    },
                    {
                        "policy_key": "terms-of-service",
                        "policy_url": "https://www.tiktok.com/legal/terms-of-service"
                    },
                    {
                        "policy_key": "tiktok-shoutouts-user-terms-of-service",
                        "policy_url": "https://www.tiktok.com/legal/tiktok-shoutouts-user-terms-of-service"
                    },
                    {
                        "policy_key": "cookie-policy",
                        "policy_url": "https://www.tiktok.com/legal/cookie-policy"
                    },
                    {
                        "policy_key": "virtual-items",
                        "policy_url": "https://www.tiktok.com/legal/virtual-items"
                    },
                    {
                        "policy_key": "rewards-policy-eea",
                        "policy_url": "https://www.tiktok.com/legal/rewards-policy-eea"
                    },
                    {
                        "policy_key": "privacy-policy-for-younger-users",
                        "policy_url": "https://www.tiktok.com/legal/privacy-policy-for-younger-users"
                    },
                    {
                        "policy_key": "copyright-policy",
                        "policy_url": "https://www.tiktok.com/legal/copyright-policy"
                    },
                    {
                        "policy_key": "changes-to-personalised-advertising-in-the-eea",
                        "policy_url": "https://www.tiktok.com/legal/changes-to-personalised-advertising-in-the-eea"
                    }
                ],
                "policy_notice_enable": true,
                "status_code": 0,
                "terms_consent_for_register_info_new_users": {
                    "checkbox_agree_all_terms": "Agree to all",
                    "checkbox_privacy_policy": "Consent to the collection and use of personal information (Required)",
                    "checkbox_terms_of_use": "Consent to the Terms of Service (Required)",
                    "checkbox_tr_notification_subtitle": "Get notifications about trending videos and promotions on TikTok. You can review and edit your settings at any time. Not allowing this type of notification does not limit your use of the TikTok service.",
                    "checkbox_tr_notification_title": "Consent to the receipt of trending content and promotional notifications (Optional)",
                    "tiktok_privacy_policy_url": "https://www.tiktok.com/legal/terms-and-conditions-kr?lang=ko-KR",
                    "tiktok_terms_of_use_url": "https://www.tiktok.com/legal/page/row/terms-of-service/ko-KR",
                    "title": "Terms and conditions"
                }
            },
...

Let’s tap agree and see what happens.

POST https://api22-normal-c-useast1a.tiktokv.com/consent/api/record/create/v1?...

entity_keys:   conditions-policy-device-consent
business_flow: consent_box
status:        1

And an updated set of compliance, similar to the one from above.

GET https://api22-normal-c-useast1a.tiktokv.com/aweme/v1/compliance/settings/?teen_mode_status=0&ftc_child_mode=-1&is_new_user=0
...

RESPONSE:
{
    "about_privacy_policy_url": "https://www.tiktok.com/legal/privacy-policy-row",
    "ad_personality_settings": {
        "ad_free_subscription": {
            "subscription_mode": 0
        },
        "att_status": 255,
        "description": "With this setting, the ads you see on TikTok can be more tailored to your interests based on data that advertising partners share with us about your activity on their apps and websites.\nYou will always see ads on TikTok based on what you do on TikTok or other data described in our privacy policy.",
        "disable_att_overwrite_pa": 1,
        "enable_toggle_decoupling": true,
        "is_follow_sys_config": false,
        "is_new_user": 1,
        "is_np_user": 0,
        "is_show_3p_data_control": false,
        "is_show_reset_entry": false,
        "is_show_settings": true,
        "is_teenager_mode": 0,
        "limit_ad_tracking": false,
        "mode": 1,
        "need_pop_up": false,
        "pa_revising_switch": false,
        "pers_ad_data_received_partner_mode": 0,
        "pers_ad_main_mode_title": "Using Off-TikTok activity for ad targeting",
        "pers_ad_show_data_received_partner": false,
        "pers_ad_show_interest_label": true,
        "pers_ad_show_third_part_measurement": false,
        "pers_ad_show_third_party_networks": false,
        "pers_ad_third_party_networks_mode": 0,
        "show_advertiser_settings": true,
        "unified_mode": 0,
        "use_new_interests": 1
    },
    "age_gate_info": {
        "age_gate_action": 0,
        "age_gate_post_action": 0,
        "register_age_gate_action": 2
    },

After Terms Of Service: And we’re off!

The app shows the video stream and the API requests are flying back and forth now, many a second, so there’s lots more to look at next time. Overall the data seems pretty standard with a few interesting things to look into like the still encrypted data, the connected services and of course, everything after you agree to those Terms of Service.

ClickHouse: Refreshing Take on Materialized Views

Working on my Open Source MMP I have been seeing how much of it will work with ClickHouse. Unfortunately combining multiple streaming data sources in ClickHouse was proving difficult as data would semi randomly not join correctly.

This led to a potential fix by using the very recent ClickHouse refreshable materialized views. These refreshable views work much closer to how materialized views work in other databases like PostgreSQL but come with additional benefits like setting the refreshes inside the materialized view logic itself.

Official Documentation:
https://clickhouse.com/docs/en/sql-reference/statements/create/view#refreshable-materialized-view

At the time of writing, refreshable views are new enough that they require setting this to enable support. Since this is needed per session in which a table is created, you will need to use this setting. This shouldn’t be necessary after Q2 2024.

SET allow_experimental_refreshable_materialized_view = 1;

Then to make the view refreshable you just add REFRESH EVERY {INT} {TIME} like this:

CREATE MATERIALIZED VIEW attribute_impressions_mv
REFRESH EVERY 5 SECOND
TO attributed_impressions -- Specify the destination table
AS
WITH
merged_impression_event AS (
    -- Ranked rows by impression time
    SELECT 
            app.event_time AS app_event_time,
            app.store_id AS store_id,
            app.event_id,
            ...

What I then discovered though was that once a refreshable view is used, it will replace the entire contents of the destination table. This means that they likely will need further management in the future as the data grows. Additionally, they cannot be combined with the columnar data inserts I was using previously with AggregatingMergeTree.

So, once you add a refreshable view, all higher level aggregations will likely all have to be refreshable as well.

In the end, I was able to use refreshable views to solve my problem, but I am not certain they wont cause delays later due to the way they seem to refresh across all data. Perhaps I’ll have to find a way to narrow the range they aggregate.

appgoblin.info

AppGoblin: Free App Stats & Info

The past few weeks I spent building a front-end for an app store crawler that I made. The end goal is just to provide an open front-end for the apps details for the approximate 2.5m apps (already 1/2 those are no longer live) I’ve crawled from the Google & Apple stores. The site is now live on appgoblin.info and hosts a variety of Android and iOS stats and store information. Hopefully it’s something that some marketers may find interesting, and the code is all free on Github.

History

app-ads.txt dash

Originally this came out of a curiosity I had about the IAB app-ads.txt standard in 2022. In order to scrape app-ads.txt files I had to first get all the iOS and Android apps + their respective publisher URLs. This ended with a small dashboard I put on https://ads.jamesoclaire.com/dash/ads but I never expanded it further.

While that project, like app-ads.txt itself, went unused, I remained interested in letting the database of apps keep growing.

Then in the fall of 2023 while on hiatus from real work, I started working on an API and UI. Creating the UI gave me exactly what I wanted, which was use cases for displaying the data. Adding one feature led to the next so now there are several sections to browse.

Features

App Details

Each app has it’s own page for all the data pulled from the app store as well as second order of information derived from that data such as installs per day or change in ratings over time.


App Rankings

Since we have each apps history, it’s easy to build the historical ranking charts. In this example on the right, you can see a game quickly climbing multiple charts at once over the past month. This change in rankings is a great way to track an apps popularity over time.


Ranking Charts

No site like this would be without it’s own app rankings page. This is straight from the Google & Apple stores top charts each day for tracking apps overtime.


New Apps

Now this might seem like an obvious one, but it really brought home the slow burn that the app stores barely surface new apps anymore. Long gone from both stores are any structured “new” sections which showcase new apps. The cynical answer here is that those sections actively take away from the ad-revenue generated by the in store advertising that makes up a large percentage of the apps on screen at any given time in each store.


So, why does AppGoblin exist?

Yes, there are a large number of similar offerings like data.ai and sensortower.com which have many more features. For me, I enjoyed learning more front-end and Javascript as well as give me something to do with the scraper which I had built previously (whats the point of a database if you cant use it somehow). For now App Goblin is simply here as a resource free of charge as it’s running costs only $20 a month or so in hosting.

Open Source Tech Stack

First is the underlying data which is collected and maintained from github.com/ddxv/adscrawler The adscrawler uses Python & PostgreSQL to crawl app stores to collect apps and their basic information like names and rating counts. This data is then stored to the PostgreSQL database. I also use some of the original parts of github.com/ddxv/app-ads-dash for monitoring the scraper. Hopefully in the future the parts specific to the scraper are moved to that repository.

The website itself is also open source and can be found at github.com/ddxv/app-store-dash The website is built from a Python backend API based on LiteStar which manages queries to the PostgreSQL database. The API is used by a JavaScript Svelte frontend with Tailwind and Skeleton for CSS. All projects are glued together with Systemd and web sockets where needed. Everything is hosted on AWS in the small EC2 instances.

As always, if you have any interest or questions please feel free to reach out.

Keeping Mobile Gamers Engaged with App Widgets

I love using widgets. I probably have too many of them on my phone’s home screen. News apps, weather apps, stocks, stats. I love being able to glance through them without opening the corresponding apps. This led me to realize though that I have seen very few widgets for mobile games. As I started working on making widgets, I realized you could actually make simple games inside of widgets.

Many apps use widgets already, but I think due to poor tooling, not many games have embraced widgets as a part of their game play loop. Recently I started looking into what some of the possibilities around making a simple game which could be played inside of a widget.

Widget Limitations:

  1. Very limited dragging or other complex touches. This means no drag/swipe functionality. You can implement drag for lists, but picking up an item and dragging it around seems out of the question.
  2. Animations are quite limited. I don’t think implementing most game animations would be possible.
  3. Computations are subject to being cut off at any point. The widget must essentially be at rest at all times, this definitely limits what you can do, but still fits well for some gaming uses.

So, let’s try a Tic-Tac-Toe Widget

I chose Tic-Tac-Toe since it was the game I could think of with the least logic. My best guess was that implementing the game logic might be tricky or troublesome. Additionally, the turn based logic was perfect for falling into the background when the game is not being played.

This turned out to be pretty easy. I was able to get a prototype up and running within a day or so. One surprise I had was having trouble using a delay. In the end I gave up trying to implement a half second delay to simulate the computer ‘thinking’. I’m not sure if that is something that could be addressed later or not.

The game is mostly limited by the fact that it’s just tic-tac-toe that you’re playing, which gets old fast.

What about a more difficult puzzle game?

Next I went with a simple sliding puzzle game. The goal is to slide the pieces to slowly sort them into the correct order. These could be a puzzle image or as you see on the side just pure numbers. This would definitely be a bit flashier if I used images, and I think that is also doable for the future.

Though this is barely more complex than Tic-Tac-Toe the difficulties became apparent quickly. The biggest issue I faced was that the Android operating system will randomly shut down the widget’s thread which would cause the game to lose anything currently stored in memory.

This means that the game state can’t be stored in memory, but needs to be backed up to disk each time, then recovered. So each ‘turn’ is played by the user, then the game state is saved to disk, and when the widget is interacted with a second time, the game state is read back from disk first.

This means the game is quite solid and recovers from it’s process being ended without the player knowing, but it is just a bit laggy when ‘playing’ quickly as you can see in the gif.

That’s cool, but what about non puzzle games?

This is where I started getting more excited. While I think you could recreate many simple puzzle games as widgets, what is the point for the developer? While the player might enjoy some play outside the app any of the more pleasant animations or clever game play mechanics would remain out of reach for the game widget.

But what the widget does give the developer is a large real estate for the the player’s home screen. This could be a place for managing notifications to entice the player to open the game. A few examples:

  1. Connection to in game functions like responding to events or setting a status
  2. Notification that an in game timer is finished
  3. Countdown to live events
  4. Button to collect daily reward and open game to boost game re-engagement
  5. Community Bulletin board

I think the ideas here are pretty basic, but could really help to reconnect games with their existing user bases.

If anyone out there has any interest in trying out the idea of building a widget for a game I’d be happy to help.

If you’re curious about either of the above examples you can get both on Google play and all code on GitHub:

Tic-Tac-Toe Widget on Google Play and Tic-Tac-Toe Android Code on GitHub

Number Puzzle Widget on Google Play and Number Puzzle Code on GitHub

Stay up to date with the latest tech new with Hacker News Widget & App

Hacker News is one of my go to sites to read the best takes on the latest tech news.

Probably the reason I keep going back to it is the honest discussions and supportive community. But you probably already know this, since Hacker News is one of the most popular tech news sources out there.

And though there are dozens of high(er) quality Hacker News apps out there, I’ve always wished they supported widgets. In fact most apps I use I wish they supported Widgets.

I love waking up in the morning and checking the latest news from my home screen, rather than opening up web pages.
This is why I decided to use Hacker News to make my first Android app and widget.

The whole experience was absolutely amazing compared to my recent attempt to use Unity. Android Studio + Kotlin and Jetpack Compose were an absolute dream to use. I rarely felt frustrated and was able to envision a few more interesting projects using widgets coming up next week.

In the meantime, please feel free to use my app or code:

On Google Play

Or Clone it On GitHub

I built my own very simple strategy game

It certainly isn’t anything amazing, but it was something that I really wanted to do for myself.

After recently stepping away from my position at Bubbleye and taking a month or two to enjoy some camping, family time and generally enjoying life I finally got back to Taipei and realized I had a unique opportunity to work on side projects that I’ve always wanted to do. Top of that list was to make a simple strategy game to better understand game design and the monetization side of games. To that end I built my own mini-RTS game using the game engine Unity.

Though Unity is the most popular choice for indie devs, I had an another motive: to learn more about Unity’s ads monetization side, especially as things change through the acquisitions of ironSource and TapJoy (originally acquired by ironSource).

How to make a good game? Be good at everything

I consider myself a generalist, competent at many things and flexible when learning new skills, but not necessarily skilled in all of the things it takes to make a game. There are so many integral parts for games like game graphics, game design, computer science and UI/UX, all of which need to be put together with good game design. It was a taxing job to work on build all these parts, and in the end also try to make a game design loop that was fun. I think this was something that surprised, because as I said I thought my strength would be that I can wear many hats, but making a game you REALLY need to do everything well, and missing even little parts of the loop made it glaringly obvious… the game wasn’t that good. As someone who enjoys the technical problems, it was easy to skip working on the game play loop itself, so I gained respect for game designers who do have the tenacity to tweak their game again and again to find out what would make a fun game.

So what did I accomplish? I learned a lot about Unity

Well, I learned quite a bit of how difficult Unity has become to use. Unity is bloated and was very painful to use. Many times, difficult problems would come down to conflicting settings within Unity itself. Warring Unity features which seem at odds with each other, and the only way to find out was to tap or untap another checkbox. The UI settings and features in Unity feel like a huge spaghetti of new features from last month and legacy support for systems from 10 years ago. I think what was frustrating is that unlike a true IDE, your code barely matters, and the Unity UI components often have little warning or information for what they might do.

This past month news broke that Unity’s pricing model was changing and the uproar was everywhere. This is fueled by a general distrust of Unity. I think it’s interesting, because if I look back, I certainly didn’t feel much trust in Unity’s product. Additionally, the code that is written, is so tightly bound to Unity it is impossible to move it to another Engine without simply rebuilding the entire game. Finally, I think that generally the state of mobile ecosystem is such that another middleman, no matter how respected, trying to assert it’s position to take a revenue share struck a nerve with many people.

Next

Having my first game in some ways did make me excited to make another. Parts were quite fun, but using Unity was a serious pain. I realized this when after the game I moved on and made a couple Android Apps in the weeks after using Android Studio. It was amazingly easy. The contrast of easily building apps in Android studio vs Unity’s spaghetti mess of code, buttons and packages that only maybe co-exist with each other was incredible.

Still, having my own apps is teaching me more about recent changes in the ad monetization landscape.

Also, having my own game gives me a good chance to work on more man in the middle attacks to watch mobile HTTPS traffic in and out of app and app SDKs. This is something I’m quite interested in, and being able to have a better understanding of how apps are built in 2023 is very useful.

How to see traffic from your Android Device

NOTE: This is a snapshot of how I did this in early 2023. There may be a more up to date version on github for newer versions of Android:

https://github.com/ddxv/mobile-network-traffic/

How to Sniff Mobile HTTPs from Apps

These are my notes as I try to find a reliable way to collect HTTPS traffic from mobile apps on my phone or VM. Still have some questions as to which works best. This is all very much just a WIP / notes, but feel free to add or use. This took quite a bit of trial and error with a number of not working solutions, until I found this recent comment which worked perfect. Incase this helps anyone else I’m just writing my process here. Feel free to comment or let me know if you have other advice!

=====================

Waydroid VM & mitmproxy Setup Notes

Unity Bid request

Waydroid and mitmproxy are the two main tools you will use. Waydroid is an emulator for Android on Linux and will need a variety of custom software installed in it to make it work.

Inital Waydroid & MITM Installation

  1. Waydroid Transparent Proxy
  2. Ensure iptables is working. I was able to solve this by explicitly add /lib/x86_64-linux-gnu/xtables to /etc/ld.so.conf.d/x86_64-linux-gnu.conf and rebooting. This was reported working for Debian 11 and Ubuntu 20+
  3. Setup iptables. I put the necessary iptable additions into a script as I ran them quite often, and can sometimes pause your local or Waydroid connection and needed to be cleared after using. You can use this script or copy paste the code block below and adapt as needed.
  4. Run ./proxysetup.sh 8080 -w for waydroid. Use -l if mitm for other device
    1. NOTE: proxysetup.sh runs sudo iptables -t nat -F at end to clear out iptables. This is because some of the iptable settings depending on proxy type can cause your connection to be blocked. But be warned, this will clear all custom iptables on your nat table you may have added.
  5. proxysetup.sh runs the following commands, so feel free to run them yourself: ```#!/bin/bash sudo iptables -t nat -A PREROUTING -i waydroid0 -p tcp --dport 80 -j REDIRECT --to-port $port sudo iptables -t nat -A PREROUTING -i waydroid0 -p tcp --dport 443 -j REDIRECT --to-port $port sudo ip6tables -t nat -A PREROUTING -i waydroid0 -p tcp --dport 80 -j REDIRECT --to-port $port sudo ip6tables -t nat -A PREROUTING -i waydroid0 -p tcp --dport 443 -j REDIRECT --to-port $port sudo sysctl -w net.ipv4.ip_forward=1 sudo sysctl -w net.ipv6.conf.all.forwarding=1 sudo sysctl -w net.ipv4.conf.all.send_redirects=0 mitmweb --mode transparent --showhost --set block_global=false ```
  6. Start Waydroid service: waydroid session start
  7. Start Waydroid UI: waydroid show-full-ui and check internet
    1. Inside Waydroid, if internet can’t connect try sudo waydroid shell and check ip link and ip addr to see if firewall blocking. More info at: ArchWiki Waydroid Networking
    2. To install GApps I used CasualSnek’s Waydroid Script Installer
  8. Install certs (first time per device) in waydroid
    1. On target device, using a browser, navigate to mitm.it
    2. follow ALL instructions on mitm.it after downloading a cert file.
    3. NOTE: Firefox has SEPARTE certs from OS certs
  9. In order to run two certificate tools (MagiskTrustUserCerts & SSLUnpinning) these framework tools installed into Waydroid
    1. Magisk Magisk is a suite of open source software for customizing Android, supporting devices higher than Android 5.0.
      1. To install in waydroid I used CasualSnek’s Waydroid Script Installer
    2. LSPosed Zygisk module trying to provide an ART hooking framework which delivers consistent APIs with the OG Xposed, leveraging LSPlant hooking framework. We will use this to install SSLUnpinning in a future step.
      1. Installation of this is quite easy, but has a few steps, I found this YouTube Video helpful to watch
  10. Finally, we can install our two custom tools into Magisk & LSPosed respectively:
    1. MagiskTrustUserCerts This Magisk module will take your user CA certs and move them to system or ‘root’ CA certifications which more apps will trust.
    2. SSLUnpinning This LSPosed module helps to unpin apps during runtime.
  11. Once everything is installed, shut down and open Waydroid and mitmproxy one more time. After this you should be able to see clear text HTTPS requests from your Waydroid VM.

========

Other Tools

Emulator vs Phone

This is the first question and probably the most dependent on what you want to achieve. Working on a real device gives more space between your device and the proxy which makes things easier. The extra space is costly in other ways. For example, I would prefer to have a single instance running on the computer to collect information, but using a phone is easier but has the physical requirement of a device connected to the network.

Phone

Physical separation allows for clearer testing. Fully functional device means your input and output work as expected.

Emulator – Waydroid

Emulator running on the same computer causes more complicated networking to ensure you don’t block your own traffic. Troubleshooting is trickier as it’s more difficult to easily access parts of the emulator that a phone is easy to access. For example, I spent much more time than I would have expected to move a VPN configuration file from my computer to the virtual machine emulator than I would have ever expected. Adding the same configuration to the phone was a simple QR code scan.

Emulator running in a virtual machine allows for a future use case of running the whole thing in the cloud without a physical device.

Proxies

As far as I know, the only way to capture the HTTPS traffic is to use a proxy. This is in the form of an application running on a separate (virtual or physical as mentioned above) device. The hardest part here is the Certificate Authority which signs the HTTPS traffic when it leaves the app. More sophisticated apps, to prevent fraud, do a variety of actions to prevent the user or 3rd parties from capturing the data in each HTTPS request.

mitmproxy

open source, link

I tried this first as it comes with Python library which would make capturing data for later analysis much easier. Mitmproxy has a few different modes, and ultimately I found that mitmproxy --mode wireguard which runs via VPN captured a good amount of traffic, but still had target SDK traffic unable to be opened. Mitmproxy has a built in tool to help installing the certificate in Android as a user certificate. This will capture some HTTPs traffic, but for some apps and many SDKs this does not capture their traffic. Traffic can be captured in several ways: CLI tool for analysis of live traffic in memory, CLI dump to file and in memory live in browser of choice.

Charles Proxy

free for 30 days, shareware, link

I first used Charles nearly 10 years ago, and it doesn’t feel like it’s changed much, but is actively maintained. When I first started using Charles it was a breeze to use, CA was less of a problem. But as Android changed it also now has the problems of CA needing to be installed, and helps the user by providing it’s own signed certificate which can be installed as a user certificate. Charles is a standalone program that you run and as such it does have a fair amount of issues on my linux environment related to it’s display sizes.

Burp Suite – Community Edition

paid/free, link

Community edition that is free to use. Runs in browser and comes with it’s own CA tool.

Android Certificate Authority

These are the certificates used to sign HTTPS traffic to keep it secure. In Android there are three levels: User, System (root) and App Pinned Certificates. In Android settings you can add a CA which will be considered “user”. Apps can choose whether to ignore this certificate. System CAs can only be set by a root user. While a user can install user CA’s, apps do not have to use these. CAs can be set by users as root certificates. I believe this must be set regardless of device or VM. The majority of the certificates provided by the proxies don’t seem to open a lot of HTTPS traffic. This is likely because Android N (API level 24) certificate pinning was introduced in 2016 and at this point most SDKs and Apps use this for transferring traffic.

JustTrustMe

open source, link

This is installed on a device or emulator. An Xposed addon that can be installed to force apps to use root authorities and prevent them from pinning their own CA.

apk-mitm

open source, link

This can be installed in a separate linux environment and is used to modify an app’s apk before being installed into a VM emultator or phone. It attempts to get around the app’s certificate pinning by patching the APK to disable certificate pinning.

Still can’t unpin? Frida and Objection

Blog for getting around Google pinning using pentesting tools
Objection

Is Someone Else Using Your Game’s Monetization IDs?

Looking around app-ads.txt files one of the first things I was excited to check was whether DIRECT publisher IDs show up on other unrelated apps. To do this I ignored any apps that shared the same developer contact URL or Developer IDs. This doesn’t catch 100% of the legitimate publishers I checked, as the examples to follow show, but does provide a good rough estimate of which networks seem to have publishers whose IDs are more mixed.

DIRECT traffic should mean that your app controls the publisher ID listed. But sometimes when looking through ads-txt there are publisher IDs that appear many time on unrelated apps. This could mean that an ad buyer could be buying ads from more than just the app they were intending, which means both good and bad quality could be mixed together.

The worst possible use case is a bad actor ad network mixing unrelated publishers together, then asking each of them to use the DIRECT tag. It turns out this is much less obvious than I first assumed, but it does seem like certain networks do have a higher prevalence of mixed IDs. Let’s look at some examples.

Number 1: Quiz Game(s)

One of the first examples I looked at was Blackpink Quiz Game who’s Unity, Google and other publisher IDs were shared across hundreds of other apps like this Family Guy-Quiz. As you can see, the two titles visually look quite similar. Each of these hundreds of apps belonged to different Google Play developers each with their own developer ids.

In addition to being visually similar, they also the same top level domain hosting their developer page url. For example Blackpink: http://pub-nbzaw-b7jg6.quickappninja.com/ and Family Guy-Quiz: http://pub-9j4jn-zu5k3.quickappninja.com/

Each link does host it’s own app-ads.txt file, but they were nearly identical across each app, containing the same DIRECT and RESELLER publisher IDs. Looking at the top level domain we see that QuickApp is an online tool that lets you create quize games from your browser. When you export the APK from the site it would likely all contain the same ad monetization accounts, controlled by QuickApp. QuickApp does later share some of the ad revenue back to the original creator.

To sum up, while that is an interesting business model, it certainly is not the kind of ad fraud I was originally looking for, and for now I’ll leave this as undecided for how to handle it.

Number 2: Crazy Marvin

Next try found another Unity ID that showed across a hundred plus apps. Checking the apps I didn’t immediately see a pattern this time. One a Dragon Ball Z game by Yodo1, a large game publisher, while the others were smaller other games with different styles and developers. Most of the developers did have several games.

com.PoisonGames.ROEQ
rocks.poopjournal.pimplepopper
com.jk.us.bullet.train.simulator

I found it really interesting that the oddest of the titles, Pimple Popper, had a GitHub linked with it’s game open sourced with a group of people working on it. So I reached out to Crazy-Marvin the dev of the Pimple Popper game to see if he could lend some insight.

Marvin was very helpful and wrote back immediately to let me know he didn’t know much about ads, but that he had been contacted by Yodo1 and per their request put the relevant ad SDKs and app-ads.txt file on his site.

Again, this seems to be a monetization partner who runs ads on behalf of the client, but without taking over their developer page or showing any other obvious links.

Number 3: rubiconproject.com, 24400, DIRECT

https://famcast.co.za/app-ads.txt

rubiconproject.com, 24400, DIRECT, 0bfd66d529a55807
rubiconproject.com, 24400, RESELLER, 0bfd66d529a55807

Finally I found one that looks more suspicious than the others. This ID 24400 is located in 500+ separate developer sites meaning 1000s of apps are pointed to this publisher ID as DIRECT traffic. Oddly, it is almost always paired with a RESELLER line with the same ID. RESELLER here would be fine, as it is being resold, but DIRECT seems highly unlikely. Looking through the apps again they all appear to be unrelated.

Compared to the first and second attempt, Rubicon is not a SDK ad network, but a programmatic SSP.

Outcome: Inconclusive, but a picture is starting to form

While there is a lot of different ways we can put together our monetization accounts, there do seem to be a higher prevalence of mixed/shared IDs on the programmatic only traffic such as Rubicon, PubMatic, IndexExchange. Looking again at our plot from the beginning, highlighting the traditional networks that do both buying and selling, we do see that they tend to rank higher than the pure programmatic SSPs.

Next?

I think that the results are pretty inconclusive, but do show some directions to move forward. I learned that there are quite a few situations where shared DIRECT publisher IDs is a natural outcome of some company’s revenue setup, but still some situations appear to be incorrectly labeled publisher IDs. The networks that do have these shared publisher IDs seem to be more prominently programmatic SSPs.

Do you have any specific examples you’d like me to check? Feel free to comment or send me your developer URL and I can help look into your IDs. Do you think there is interest in a tool to help check your app-ads.txt for network quality?

What A Mobile Ad Monopoly Looks Like

Google’s market share in in-app advertising is unmistakably dominant. The data above was scraped from 100k+ Google Play Store & Apple App Store apps (with a slight emphasis on games) who use advertising for monetization. This public data is made possible by adoption of IAB’s app-ads.txt standard which allows buyers and sellers to cross verify their ad buying is coming from legitimate direct sources by checking public txt files.

So, Why is Google So Dominant?

First, let’s confirm that the data is somewhat accurate. This data can be correlated to similar reporting by Braedon, creator of well-known.dev, a repository of public data from app-ads.txt files (and much more). At 92% his percentage for Google lined up perfectly with the data here, despite likely slightly different app populations.

His tweets were also picked up a few weeks ago by AdExchanger, who pointed out that as you look closer at the long tail of small advertisers, the app-ads.txt files often contain only one single line: Google.com

Why is Google dominant in mobile? As the operating system for Android, they are the easiest way to integrate an advertising SDK for a small app. This means that the first ads shown, are often through Google.com and if and when the app grows, they will only ever add more competing ad networks, likely never removing the original option. Thus Google is the default advertising option on the device, with other ad networks barely getting a second thought.

Is There No Competing?

Let’s zoom in a bit on the data, using only the top 5% of apps ranked by install/review count, and we see a slightly more competitive world. Here Google is still on nearly every app, but the next closest ad networks are on nearly a 1/3, a healthy increase.

This seems to indicate that once an app a big enough monetization team to manage more than one network, they can much more easily expand to other networks so as to increase their eCPMs. Adding the first ad SDK is a pain, and Google’s moat around their advertising castle is that adding and managing the second SDK takes serious commitment and cost.

What else are we missing?

Let’s start with a caveat: this is programmatic traffic, measured only by those who are hosting their app-ads.txt file. Some ad networks are not so transparent.

For example, Apple Search Ads, with it’s growing stranglehold over the iOS advertising ecosystem is entirely absent in app-ads.txt. This is because Apple does not compete at all in the programmatic markets, but instead uses it’s elevated role as the owner of the App Store to manage it’s own ads.

Another missing piece is that the vast majority of app-ads.txt are polluted now by traffic marked as resellers. Resellers are much more competitive market, fighting over the scraps of what is left after the top networks take their pick of traffic. Unfortunately, there are so many resellers, it is quite hard to understand how app-ads.txt helps with verification at all, as I have seen many app-ads.txt files with hundreds of resellers and only 1 direct account, usually just google.com.

Closing thoughts

Despite the doom and gloom, the most important thing is that this information is publicly available. Advertising is a difficult to parse business and this helps us to have some transparency on the marketplace we all share.