Quileg HID

Quileg HID lets an AI assistant actually operate an iPhone: the phone's built-in Screen Mirroring brings the picture to the Mac so the assistant can see it, and an ESP32-S3 board sends real Bluetooth keyboard and mouse events so it can tap and type. No jailbreak, no developer mode, and nothing to install on the phone.

Seeing and doing are handled by three parts working together: mirroring shows the AI the screen, the Mac commands the board over Wi-Fi, and the board acts as a standard Bluetooth keyboard and mouse. The AI calls a local control service through MCP — 32 tools covering status, screenshots, taps, typing, scrolling, dragging, key presses and calibration. The tools work one action at a time and the screen is checked again after every action, so a missed tap is noticed and corrected.

The control service only listens on port 8890 on the local machine, so the picture and the commands never leave the computer; with the Mac and the board on the same local network it also works offline. iPhone and iPad use AirPlay mirroring; Android, Windows and macOS need a UVC capture card, and the key layout follows the controlled platform.

It needs an ESP32-S3 board (16MB flash, roughly CN¥30–60). Seeed XIAO ESP32S3 Plus and third-party boards built on the ESP32-S3-WROOM-1-N16R8 module have been verified. Going from zero to working takes about 30–40 minutes, roughly 13 of them the first calibration; it only has to be redone when the phone, mirroring resolution, pointer size, video source or orientation changes — not when the board is swapped.

This is the third generation of the Quileg line: the first (July 2024) was a wired keyboard and mouse simulator built on STM32, the second (July 2025) added multi-device chaining, and neither was publicly released. The third moves to ESP32-S3 over Bluetooth, turns the target from computers to phones, and is published on GitHub as a free download.

The app is free to download and use, with closed source. The board-side control service and the AirPlay receiver are based on esp32-iphone-agent (MIT) and UxPlay (GPL-3.0), with the full licence texts shipped in the installer. Please use it only on devices and accounts you are authorised to operate, and follow the terms of the platforms involved.

Details

• Works with apps that have no API: anything you can tap by hand can be tapped
• Hands repetitive tapping, filling in and paging over to it
• Look, act, look again: the screen is checked after every action
• Full keyboard, with Chinese entered by pinyin and candidate taps
• AirPlay for iPhone and iPad; a UVC capture card for Android, Windows and macOS
• The picture and the commands stay on the local machine

An ESP32-S3 board (16MB flash) is required; without it the app cannot control the phone.