An on-device browser agent
shippedA vision-language model inside a Chrome extension: local inference connected to real browser actions. Built at RunAnywhere.
The problem
An agent that operates a browser needs to connect what it sees with what it can do. Running the model inside the browser adds another constraint: the entire interaction loop has to fit within a Chrome extension's execution and security model.
At RunAnywhere, I worked on an on-device browser agent using vision-language models, Transformers.js, and ONNX.
My contribution
I added VLM support for screenshot-based navigation and refactored the agent toward a state-machine-first architecture. I also contributed the on-device browser playground to the RunAnywhere SDK, alongside the Swift starter app.
The important constraint
Restricted pages and Manifest V3 Content Security Policy are part of the environment the agent has to handle. Model inference alone is not enough: the application needs an explicit way to represent where execution is, what action is possible, and when it cannot continue.
The state-machine-first structure makes that control flow explicit. Transformers.js and ONNX supply the local inference layer; the extension connects model output to browser actions.
Inspect the work
- My contributions to the browser-agent repository
- SDK Playground contribution — PR #297
- Related Android Use Agent contribution — PR #336
Related work
I also worked on Android agent support, x86_64 emulator compatibility, and the cross-platform SDK Playground. See my full experience.