Hacker News is one of my go to sites to read the best takes on the latest tech news.
Probably the reason I keep going back to it is the honest discussions and supportive community. But you probably already know this, since Hacker News is one of the most popular tech news sources out there.
And though there are dozens of high(er) quality Hacker News apps out there, I’ve always wished they supported widgets. In fact most apps I use I wish they supported Widgets.
I love waking up in the morning and checking the latest news from my home screen, rather than opening up web pages. This is why I decided to use Hacker News to make my first Android app and widget.
The whole experience was absolutely amazing compared to my recent attempt to use Unity. Android Studio + Kotlin and Jetpack Compose were an absolute dream to use. I rarely felt frustrated and was able to envision a few more interesting projects using widgets coming up next week.
In the meantime, please feel free to use my app or code:
It certainly isn’t anything amazing, but it was something that I really wanted to do for myself.
After recently stepping away from my position at Bubbleye and taking a month or two to enjoy some camping, family time and generally enjoying life I finally got back to Taipei and realized I had a unique opportunity to work on side projects that I’ve always wanted to do. Top of that list was to make a simple strategy game to better understand game design and the monetization side of games. To that end I built my own mini-RTS game using the game engine Unity.
Though Unity is the most popular choice for indie devs, I had an another motive: to learn more about Unity’s ads monetization side, especially as things change through the acquisitions of ironSource and TapJoy (originally acquired by ironSource).
How to make a good game? Be good at everything
I consider myself a generalist, competent at many things and flexible when learning new skills, but not necessarily skilled in all of the things it takes to make a game. There are so many integral parts for games like game graphics, game design, computer science and UI/UX, all of which need to be put together with good game design. It was a taxing job to work on build all these parts, and in the end also try to make a game design loop that was fun. I think this was something that surprised, because as I said I thought my strength would be that I can wear many hats, but making a game you REALLY need to do everything well, and missing even little parts of the loop made it glaringly obvious… the game wasn’t that good. As someone who enjoys the technical problems, it was easy to skip working on the game play loop itself, so I gained respect for game designers who do have the tenacity to tweak their game again and again to find out what would make a fun game.
So what did I accomplish? I learned a lot about Unity
Well, I learned quite a bit of how difficult Unity has become to use. Unity is bloated and was very painful to use. Many times, difficult problems would come down to conflicting settings within Unity itself. Warring Unity features which seem at odds with each other, and the only way to find out was to tap or untap another checkbox. The UI settings and features in Unity feel like a huge spaghetti of new features from last month and legacy support for systems from 10 years ago. I think what was frustrating is that unlike a true IDE, your code barely matters, and the Unity UI components often have little warning or information for what they might do.
This past month news broke that Unity’s pricing model was changing and the uproar was everywhere. This is fueled by a general distrust of Unity. I think it’s interesting, because if I look back, I certainly didn’t feel much trust in Unity’s product. Additionally, the code that is written, is so tightly bound to Unity it is impossible to move it to another Engine without simply rebuilding the entire game. Finally, I think that generally the state of mobile ecosystem is such that another middleman, no matter how respected, trying to assert it’s position to take a revenue share struck a nerve with many people.
Next
Having my first game in some ways did make me excited to make another. Parts were quite fun, but using Unity was a serious pain. I realized this when after the game I moved on and made a couple Android Apps in the weeks after using Android Studio. It was amazingly easy. The contrast of easily building apps in Android studio vs Unity’s spaghetti mess of code, buttons and packages that only maybe co-exist with each other was incredible.
Still, having my own apps is teaching me more about recent changes in the ad monetization landscape.
Also, having my own game gives me a good chance to work on more man in the middle attacks to watch mobile HTTPS traffic in and out of app and app SDKs. This is something I’m quite interested in, and being able to have a better understanding of how apps are built in 2023 is very useful.
These are my notes as I try to find a reliable way to collect HTTPS traffic from mobile apps on my phone or VM. Still have some questions as to which works best. This is all very much just a WIP / notes, but feel free to add or use. This took quite a bit of trial and error with a number of not working solutions, until I found this recent comment which worked perfect. Incase this helps anyone else I’m just writing my process here. Feel free to comment or let me know if you have other advice!
=====================
Waydroid VM & mitmproxy Setup Notes
Waydroid and mitmproxy are the two main tools you will use. Waydroid is an emulator for Android on Linux and will need a variety of custom software installed in it to make it work.
Ensure iptables is working. I was able to solve this by explicitly add /lib/x86_64-linux-gnu/xtables to /etc/ld.so.conf.d/x86_64-linux-gnu.conf and rebooting. This was reported working for Debian 11 and Ubuntu 20+
Setup iptables. I put the necessary iptable additions into a script as I ran them quite often, and can sometimes pause your local or Waydroid connection and needed to be cleared after using. You can use this script or copy paste the code block below and adapt as needed.
Run ./proxysetup.sh 8080 -w for waydroid. Use -l if mitm for other device
NOTE: proxysetup.sh runs sudo iptables -t nat -F at end to clear out iptables. This is because some of the iptable settings depending on proxy type can cause your connection to be blocked. But be warned, this will clear all custom iptables on your nat table you may have added.
proxysetup.sh runs the following commands, so feel free to run them yourself: ```#!/bin/bash sudo iptables -t nat -A PREROUTING -i waydroid0 -p tcp --dport 80 -j REDIRECT --to-port $port sudo iptables -t nat -A PREROUTING -i waydroid0 -p tcp --dport 443 -j REDIRECT --to-port $port sudo ip6tables -t nat -A PREROUTING -i waydroid0 -p tcp --dport 80 -j REDIRECT --to-port $port sudo ip6tables -t nat -A PREROUTING -i waydroid0 -p tcp --dport 443 -j REDIRECT --to-port $port sudo sysctl -w net.ipv4.ip_forward=1 sudo sysctl -w net.ipv6.conf.all.forwarding=1 sudo sysctl -w net.ipv4.conf.all.send_redirects=0 mitmweb --mode transparent --showhost --set block_global=false ```
Start Waydroid service: waydroid session start
Start Waydroid UI: waydroid show-full-ui and check internet
Inside Waydroid, if internet can’t connect try sudo waydroid shell and check ip link and ip addr to see if firewall blocking. More info at: ArchWiki Waydroid Networking
LSPosed Zygisk module trying to provide an ART hooking framework which delivers consistent APIs with the OG Xposed, leveraging LSPlant hooking framework. We will use this to install SSLUnpinning in a future step.
Finally, we can install our two custom tools into Magisk & LSPosed respectively:
MagiskTrustUserCerts This Magisk module will take your user CA certs and move them to system or ‘root’ CA certifications which more apps will trust.
SSLUnpinning This LSPosed module helps to unpin apps during runtime.
Once everything is installed, shut down and open Waydroid and mitmproxy one more time. After this you should be able to see clear text HTTPS requests from your Waydroid VM.
========
Other Tools
Emulator vs Phone
This is the first question and probably the most dependent on what you want to achieve. Working on a real device gives more space between your device and the proxy which makes things easier. The extra space is costly in other ways. For example, I would prefer to have a single instance running on the computer to collect information, but using a phone is easier but has the physical requirement of a device connected to the network.
Phone
Physical separation allows for clearer testing. Fully functional device means your input and output work as expected.
Emulator – Waydroid
Emulator running on the same computer causes more complicated networking to ensure you don’t block your own traffic. Troubleshooting is trickier as it’s more difficult to easily access parts of the emulator that a phone is easy to access. For example, I spent much more time than I would have expected to move a VPN configuration file from my computer to the virtual machine emulator than I would have ever expected. Adding the same configuration to the phone was a simple QR code scan.
Emulator running in a virtual machine allows for a future use case of running the whole thing in the cloud without a physical device.
Proxies
As far as I know, the only way to capture the HTTPS traffic is to use a proxy. This is in the form of an application running on a separate (virtual or physical as mentioned above) device. The hardest part here is the Certificate Authority which signs the HTTPS traffic when it leaves the app. More sophisticated apps, to prevent fraud, do a variety of actions to prevent the user or 3rd parties from capturing the data in each HTTPS request.
I tried this first as it comes with Python library which would make capturing data for later analysis much easier. Mitmproxy has a few different modes, and ultimately I found that mitmproxy --mode wireguard which runs via VPN captured a good amount of traffic, but still had target SDK traffic unable to be opened. Mitmproxy has a built in tool to help installing the certificate in Android as a user certificate. This will capture some HTTPs traffic, but for some apps and many SDKs this does not capture their traffic. Traffic can be captured in several ways: CLI tool for analysis of live traffic in memory, CLI dump to file and in memory live in browser of choice.
I first used Charles nearly 10 years ago, and it doesn’t feel like it’s changed much, but is actively maintained. When I first started using Charles it was a breeze to use, CA was less of a problem. But as Android changed it also now has the problems of CA needing to be installed, and helps the user by providing it’s own signed certificate which can be installed as a user certificate. Charles is a standalone program that you run and as such it does have a fair amount of issues on my linux environment related to it’s display sizes.
Community edition that is free to use. Runs in browser and comes with it’s own CA tool.
Android Certificate Authority
These are the certificates used to sign HTTPS traffic to keep it secure. In Android there are three levels: User, System (root) and App Pinned Certificates. In Android settings you can add a CA which will be considered “user”. Apps can choose whether to ignore this certificate. System CAs can only be set by a root user. While a user can install user CA’s, apps do not have to use these. CAs can be set by users as root certificates. I believe this must be set regardless of device or VM. The majority of the certificates provided by the proxies don’t seem to open a lot of HTTPS traffic. This is likely because Android N (API level 24) certificate pinning was introduced in 2016 and at this point most SDKs and Apps use this for transferring traffic.
This is installed on a device or emulator. An Xposed addon that can be installed to force apps to use root authorities and prevent them from pinning their own CA.
This can be installed in a separate linux environment and is used to modify an app’s apk before being installed into a VM emultator or phone. It attempts to get around the app’s certificate pinning by patching the APK to disable certificate pinning.
