Hackaday just covered an AI agent controlling an iPhone through an ESP32-S3 acting as a USB-HID device. The agent sees the screen via AirPlay and UxPlay, then sends keystrokes and mouse moves through the ESP32-S3. iOS AssistiveTouch bridges the last gap. The result: full input control without an Apple Developer account, Xcode, or a Mac. Read the Hackaday piece
The interesting part isn't the AI. It's the input path.
Why USB-HID changes the constraints
Most remote-control tools need an agent on the target machine. RDP, VNC, SSH, proprietary KVM-over-IP, they all require software installation, admin rights, network configuration, and often specific OS support. If you can't install software (locked-down corporate laptop, kiosk, mobile device, headless server in BIOS), you're stuck.
USB-HID sidesteps all of that. The target sees a standard keyboard and mouse. No drivers. No network stack. No permissions beyond "this USB device is allowed." It works in BIOS, at the login screen, during OS install, on iOS with AssistiveTouch, on Linux without X11 or Wayland, on Windows before the network stack loads.
The Hackaday project uses an ESP32-S3 directly. That's the same microcontroller InputInfinity builds on.
What the ESP32-S3 actually does
The ESP32-S3 has a native USB peripheral. It can enumerate as a composite HID device, keyboard, mouse, consumer control, sysrq, all at once. No bit-banging. No external PHY. The silicon handles the USB protocol; firmware just pushes report descriptors and data.
When you plug it into a target, the target enumerates a keyboard and a mouse. That's it. The target has no idea the keystrokes originate from a browser tab three networks away.
The native USB peripheral is the key. Older ESP32s needed a separate USB-to-serial chip or bit-banged USB on GPIOs. The S3 integrates a full-speed USB OTG controller with its own DMA and PHY. That means the firmware can present standard HID report descriptors at boot, and the host OS loads the generic HID driver automatically. No vendor ID conflicts, no custom driver inf files, no Windows "unknown device" warnings.
Where the browser fits
Raw USB-HID is powerful but low-level. You don't want to hand-craft HID reports for every keystroke. InputInfinity runs a small web server on the ESP32-S3 (or on a paired gateway) that translates browser events, keydown, keyup, mousemove, click, wheel, into HID reports in real time. The browser becomes the operator console.
No client install on the operator side either. Open a tab, connect, type. Works from Chrome, Firefox, Safari, Edge. Works from a phone browser. Works from a tablet.
The web server serves a single-page application over HTTP. Once loaded, the UI opens an encrypted WebSocket back to the device. Keystrokes and pointer events are serialized as compact binary frames, not JSON, to keep latency low and the ESP32-S3's RAM usage predictable. A typical key press round-trip from browser to target USB endpoint lands well under 20 ms on a local network.
The multi-device reality
One ESP32-S3 controls one target at a time. That's a hardware constraint: one USB device port, one host connection. But most people don't need simultaneous control of five machines. They need to switch between a desktop, a laptop, a home server, and maybe a phone.
We support up to three devices on the free tier. You move the USB cable (or use a USB switch) and select the target in the browser. The ESP32-S3 re-enumerates. The target sees a keyboard and mouse appear. You're controlling it.
A powered USB 2.0 switch (the kind with a button on top) costs $15 and lets you keep four targets cabled permanently. Select target A in the browser, press the switch button, you're on target B. No unplugging, no re-enumeration delay beyond the host's normal USB settle time.
What AssistiveTouch adds on iOS
The Hackaday piece mentions AssistiveTouch. On iOS, a standard USB-HID mouse doesn't get a visible cursor by default. AssistiveTouch enables the on-screen pointer and maps mouse events to touch actions. Keyboard input works natively. Together they give full control, tap, swipe, drag, type, through the same USB-HID channel.
This is a configuration change on the iPhone. No app install. No profile. No MDM. Just Settings → Accessibility → Touch → AssistiveTouch.
Once enabled, a small floating button appears on screen. That button is the cursor. Left-click maps to tap. Right-click maps to long-press. Scroll wheel maps to vertical swipe. The on-screen keyboard hides when a hardware keyboard is detected, so you get full screen real estate while typing.
Where this fits in a real workflow
You have a Mac Mini in a closet. A Windows laptop on the desk. A Linux box in the rack. An iPhone on a stand for mobile testing. You sit at one keyboard and mouse (your actual physical ones) and control all of them from a browser tab.
Switch targets in two clicks. No KVM cables. No Synergy/Barrier configuration. No firewall rules. No "wait, which machine has the focus?"
The ESP32-S3 sits on the target side. Your browser sits on the operator side. The connection between them is encrypted WebSocket. The target side sees only USB-HID.
I keep an ESP32-S3 plugged into my rack-mounted Proxmox host. When the management network goes down, I open the browser tab, type root credentials at the console, and fix the firewall rule. No crash cart. No IPMI. No Java Web Start KVM viewer that hasn't worked since Java 8.
What you need to build it
- One ESP32-S3 development board (any with native USB: ESP32-S3-DevKitC-1, Seeed Xiao ESP32S3, Adafruit QT Py ESP32-S3, etc.)
- A USB cable appropriate for the target (USB-A, USB-C, micro-USB with OTG)
- The InputInfinity firmware (open source, flashable via web serial)
- A browser on your operator machine
That's the bill of materials. No server to host. No domain to register. No certificates to manage.
Flashing takes about 30 seconds. Open the firmware page in Chrome or Edge, plug the board in, click "Connect," select the serial port, click "Flash." The board reboots, starts its access point, you join the AP, enter your Wi-Fi credentials, and it connects to your network. Next browser tab: https://inputinfinity.com, sign in, the device appears. Done.
What it doesn't do
It doesn't stream video. The Hackaday project uses AirPlay + UxPlay + GStreamer for that. InputInfinity is input only. Pair it with your preferred screen-sharing tool, VNC, RDP, Sunshine/Moonlight, Parsec, a simple HDMI capture card, or just look at the screen if it's on your desk.
It doesn't do file transfer. It doesn't do clipboard sync (though browser copy/paste into the web UI works for text). It doesn't replace a full remote desktop. It replaces the keyboard and mouse.
If you need clipboard, copy on the operator side, paste into the web UI's text input field, and the firmware types it out as HID keystrokes on the target. Slow for large blocks, fine for passwords and commands.
The free tier covers most personal use
Three devices. No time limit. No feature gating. Flash the firmware, open the browser, start controlling. If you need more devices or team features, paid tiers exist. The core capability, ESP32-S3 as a browser-controlled USB-HID device, is free.
Try it on the hardware you already have
If you have an ESP32-S3 board in a drawer, flash the firmware today. Plug it into the machine you want to control. Open the browser. Type. Move the mouse. See it work on the target with zero software installed there.
That's the same architecture the Hackaday project uses for AI-driven iPhone control. We just packaged it so you don't have to write the firmware, the web server, the WebSocket layer, and the HID report generator yourself.
Start with InputInfinity, flash, plug, control.
Discussion
No comments yet. Be the first.