๐ฆ๐๐ป๐ฑ๐ฎ๐ ๐๐ผ๐ณ๐ณ๐ฒ๐ฒ & ๐๐ผ๐ฑ๐ฒ: ๐๐๐ฟ๐ป๐ถ๐ป๐ด ๐บ๐ ๐๐ผ๐ผ๐ด๐น๐ฒ ๐ฃ๐ถ๐ ๐ฒ๐น ๐ฝ๐ต๐ผ๐ป๐ฒ ๐ถ๐ป๐๐ผ ๐ฎ๐ป ๐๐๐ ๐๐ฒ๐ฟ๐๐ฒ๐ฟ
A week or ago ago Brandon Laur (The White Hatter) mentioned he was curious about the local language models Iโve been running on my phone in LM Playground. That offhand comment sent me down a rabbit hole, and this weekโs SC&C is what came out the other end.
LM Playground is a neat Android app that runs GGUF models fully on-device and offline. Download a model, load it, chat. No cloud, no API keys. Great for privacy, and a genuinely interesting sandbox for the sort of small models I keep writing about.
But I wanted more than a chat window. If the model is already running on the phone, ๐ฐ๐ผ๐๐น๐ฑ ๐ ๐ฟ๐ฒ๐ฎ๐ฐ๐ต ๐ถ๐ ๐ณ๐ฟ๐ผ๐บ ๐ผ๐ป๐ฒ ๐ผ๐ณ ๐บ๐ ๐๐๐๐๐ฒ๐บ๐ ๐ฎ๐ ๐ต๐ผ๐บ๐ฒ? So I forked the repo and sat down with Claude Code to add an OpenAI-compatible API server that binds to the phoneโs WiFi interface. ๐ง๐ต๐ฒ ๐ถ๐ฑ๐ฒ๐ฎ: expose the loaded model at the standard /v1/chat/completions endpoint, protected by a generated bearer key, reachable from any other device on the same network.
A morning of pair-programming later, it works and ๐บ๐ ๐ฝ๐ต๐ผ๐ป๐ฒ ๐ถ๐ ๐ป๐ผ๐ ๐ฎ ๐น๐ถ๐๐๐น๐ฒ ๐ถ๐ป๐ณ๐ฒ๐ฟ๐ฒ๐ป๐ฐ๐ฒ ๐๐ฒ๐ฟ๐๐ฒ๐ฟ. I can point the openai Python SDK at http://
OK, what do you do with a pocket-sized LLM endpoint? Test it, of course - Iโve wired it up to my prompt-injection testing harness, and as I write this itโs grinding through a 500-prompt run, throwing injection attempts at the on-device model and logging how it responds. Maybe the kind of thing Brandonโs world cares about: if people are going to run private models on their phones, how robust are they?
The build itself was a proper yak-shave. Learned a lot about Android Studio, NDK, CMake version pins, a Vulkan backend that wanted a shader compiler I didnโt have. I learned more about the ggml build system than I strictly meant, or wanted to. In the end I turned Vulkan off and the build went clean.
All in all, a perfect Sunday C&C project, fun, learned something new, did some more testing (๐ข๐ฏ๐ฅ ๐ข๐ญ๐ด๐ฐ ๐๐ญ๐ข๐ถ๐ฅ๐ฆ ๐๐ฐ๐ฅ๐ฆ ๐ฅ๐ฐ๐ฆ๐ด ๐ข ๐ฃ๐ณ๐ช๐ญ๐ญ๐ช๐ข๐ฏ๐ต ๐ซ๐ฐ๐ฃ ๐ฐ๐ฏ ๐๐ฏ๐ฅ๐ณ๐ฐ๐ช๐ฅ ๐ข๐ฑ๐ฑ๐ด) and also that on-device models are more accessible than most people realise
Iโll share the results of the injection run in my prompt injection testing repo. For now the coffeeโs gone cold and the testโs still running.
๐ฅ๐ฒ๐ฝ๐ผโ๐