Actions reference
Endpoints
| Endpoint | Method | What it does |
|---|---|---|
/ | POST | Run one action. The body is a JSON object with action and its parameters. |
/screenshot/desktop | GET | The whole screen as PNG. See screenshot sizes. |
/screenshot/<name> | GET | A screenshot a plugin registered. An unknown name returns 404. |
/state | GET | JSON with status, surface, window_offset, resolution, handoff_active, plugins, and whatever plugins add. |
/events | GET | Server-sent events. Query after_id and repeated type. |
/health | GET | ok, or 503 with {"ok": false, "failures": [...]} when a plugin health check fails. Never needs the token. |
/mcp/ | POST | MCP Streamable HTTP; see MCP and agents. |
With AUTH_TOKEN set, every endpoint except /health needs Authorization: Bearer <token> and answers 401 with {"success": false, "error": "Unauthorized"} without it. A request that puts auth_token in the query string is refused.
Requests run one at a time, from HTTP and MCP alike, so only one action drives the X server at once.
Every response carries X-Request-Id and X-Trace-Id headers. Send your own X-Request-Id (1 to 64 characters of A-Za-z0-9._-) to find the request in the JSON logs; anything else is replaced with a random ID.
Screenshot sizes
GET /screenshot/desktop and the desktop_screenshot action take the same size parameters. Without any of them the PNG is full size.
| Parameter | Effect |
|---|---|
whLargest | Scale so the longer side is this many pixels, keeping the aspect ratio. Wins over width and height. |
width or height alone | Scale to that size, keeping the aspect ratio. |
width and height | Scale to exactly that size, which stretches the image if the ratio differs. |
Each value is an integer from 1 to 20000 and can also enlarge the image. Any other value returns 400 over GET and a failed response from the action.
Events stream
GET /events streams events as they happen, each as id, event (the type) and data (the event as JSON). With no events it sends a : keepalive comment every 15 seconds. after_id resumes after an event you already have. See events and handoff for the event shape.
Responses
{
"success": true,
"timestamp": 1794929420.83,
"data": {
"exit_code": 0,
"stdout": "hi\n"
}
}
A failure has "success": false and an error string. Some failures also carry data, such as the output of a command that timed out. Invalid JSON, a body that is not an object, a non-string action and an unknown action all fail this way with HTTP 200.
Coordinates
Input coordinates are screen pixels from the top-left corner. desktop_list_windows returns each window’s geometry in the same space, so a point inside a window is its x and y plus an offset. Derived images can change the coordinate origin; see plugins.
Mouse and keyboard
All input goes through XTest with human timing: curved paths with overshoot and tremor, Fitts’s-law durations, lognormal click holds, key-pair dependent typing rhythm with overlapping keys, and spaced wheel notches. Each container samples one typist and one mouse profile at startup and keeps them. Humanized input explains the model, the defaults and every tuning option.
x and y must be given together. Numbers may be sent as JSON numbers or numeric strings, except for human_pause, which takes JSON numbers only.
| Action | Parameters | Returns |
|---|---|---|
system_click | x, y required; w, h, duration and the mouse overrides | system_clicked, landed_at. With w and h, the click lands at a human offset inside that target. |
system_double_click | as system_click | system_double_clicked, landed_at. Both clicks fall inside the 400 ms double-click time. |
system_right_click | as system_click | system_right_clicked, landed_at |
mouse_click | optional x, y; w, h, duration and the mouse overrides | clicked_at ("current" without coordinates), landed_at. Without coordinates, left-clicks where the pointer is. |
mouse_move, system_hover | x, y required; w, h, duration, speed, curvature, tremor | moved_to, landed_at. Moves without clicking. |
system_mouse_down | button (left default, right, middle); optional x, y with w, h, duration and the mouse overrides | button, down_at. Moves first when x and y are given, then presses. |
system_mouse_up | button (left default); optional x, y | button, up_at. Moves first when x and y are given, then releases. |
system_drag | source_x, source_y, target_x, target_y required; the mouse overrides | source, target, landed_at. Presses at the source, moves along a human path and releases at the target. |
scroll | amount (non-zero integer, default -3, negative scrolls down); optional x, y; notch_gap | scrolled. Moves to x, y first when given, then sends that many wheel notches. |
system_type | text required; interval, typos, typo_rate, key_hold, rollover | typed_len, typos. Types into the focused window. See entering text. |
send_key | key required; key_hold, instant | send_key. See below. |
human_pause | duration; x, y, w, h; drift, drift_interval, drift_distance; speed, curvature, tremor | duration, elapsed, drifts, parked, landed_at. See below. |
paste_text | text required; x, y, w, h and the mouse overrides; replace | See entering text. |
w, h and duration must be greater than 0. duration scales the movement to that many seconds.
The mouse overrides change the session’s mouse profile for one request:
| Override | Range | Meaning |
|---|---|---|
speed | 0.1 to 10 | Multiplier on the profile’s movement speed. |
curvature | 0 to 5 | How much the main stroke bends; 0 is straight. |
tremor | 0 to 5 | Hand tremor amplitude in pixels. |
click_hold | 0.01 to 5 | Median seconds the button is held. |
click_delay | 0 to 10 | Median seconds between arriving and pressing. |
send_key
key is a key name or a combo joined with +, in PyAutoGUI names: enter, esc, tab, backspace, ctrl+a, alt+f4, win+left, ctrl+shift+t. Modifiers go down in order with short human gaps, the last key is held, then all are released in reverse order.
key_hold(0.01 to 2): median seconds the last key is held.instant(defaultfalse): press and release every key at once, with no human timing. It cannot be combined withkey_hold.
Window manager shortcuts are pressed with send_key. Keyboard shortcuts lists them.
human_pause
A visible pause, such as reading time, where the pointer sits still most of the time and makes a few irregular small moves.
| Parameter | Default | Range | Meaning |
|---|---|---|---|
duration | 1 | 0.1 to 300 | Seconds to pause. |
x, y | none | Centre of a region to park the pointer in first. Without them the pointer stays in the screen. | |
w, h | 80 | 4 to 10000 | Size of that region in pixels. Needs x and y. The region must overlap the screen. |
drift | true | Allow the small moves at all. | |
drift_interval | 12 | 1 to 120 | Median seconds between chances to move. |
drift_distance | 60 | 0 to 500 | Largest move in pixels. 0 turns drift off. |
speed, curvature and tremor apply to the parking move and the drift. A pointer outside the screen is first moved onto it.
Entering text
People paste far more than they type. paste_text enters text the way a person pastes it: it puts the text on the clipboard, clicks the field when x and y are given, then presses ctrl+a, backspace and ctrl+v through the humanized send_key, with irregular pauses between the keys and before the first key after the click.
| Parameter | Default | Meaning |
|---|---|---|
text | required | Up to 1 MiB of UTF-8. |
x, y, w, h and the mouse overrides | none | Click the field first, as system_click does. |
replace | true | false skips select-all and clear and pastes at the caret. |
It returns pasted_len and replaced, plus clicked_at and landed_at when it clicked. The text stays on the clipboard afterwards.
paste_text is meant for GUI text fields (browsers, GTK, Qt), where ctrl+a selects all and ctrl+v pastes. Terminals bind those keys differently (in xterm and shells ctrl+a moves to the line start), so type into a terminal with system_type or run the command with run_command.
Keep system_type for a few characters: a word, a short search, a small edit, a one-line command, or a field that blocks pasting. Typing a paragraph key by key is slow and is not what a person does. \n presses Enter and \t presses Tab. Characters outside the US layout are typed through their keysyms.
system_type parameter | Default | Range | Meaning |
|---|---|---|---|
text | required | The text to type. | |
interval | the profile’s | greater than 0 | Median seconds between keystrokes. |
key_hold | the profile’s | 0.01 to 2 | Median seconds each key is held. |
rollover | the profile’s | 0 to 1 | Probability that one key is still down when the next goes down. |
typos | false, or true when typo_rate is above 0 | Make occasional typos and correct them with Backspace. | |
typo_rate | 0.02 when typos is on | 0 to 0.2 | Typos per character. A rate above 0 with typos: false is an error. |
Windows
Each window action except desktop_list_windows and desktop_keybindings picks one window by exactly one of these selectors. The first match wins.
| Selector | Matches |
|---|---|
id | The window ID, such as 0x01200004, in hex or decimal. |
title | A case-insensitive substring of the title. |
wm_class | A case-insensitive substring of WM_CLASS, such as xterm. |
pid | The owning process ID. Only programs that publish _NET_WM_PID have one. |
| Action | Parameters | Returns |
|---|---|---|
desktop_list_windows | none | windows and count. Each window has id, title, wm_class, pid (null when unknown), x, y, width, height, desktop and active. |
desktop_focus_window | selector | window. Raises and focuses it. |
desktop_move_resize_window | selector; at least one of x, y, width, height (integers; width and height at least 1) | window, requested. Restores a maximized window first. Omitted values stay. |
desktop_set_window_state | selector; state | window, state. state is maximize, unmaximize, fullscreen, unfullscreen, minimize, unminimize, above or unabove. |
desktop_close_window | selector | window. Asks the window to close, as its close button would. |
desktop_keybindings | none | The window manager shortcuts; see keyboard shortcuts. |
The window in each return is the window as it was before the action. Geometry comes from wmctrl and includes the frame offset Openbox adds, so a window placed at 100,100 can report a slightly different x and y.
Programs and shell commands
run_command runs any command it is given and launch_app starts any program, both as the desktop user with DISPLAY set. Both are refused when DESK_EXEC_ENABLED=false. See security.
cwd must be an existing directory. env is an object of up to 200 string values added to the container’s environment.
| Action | Parameters | Returns |
|---|---|---|
run_command | command required: a string run with bash -c, or an argv list run directly; cwd, env, stdin (a string up to 1 MiB), timeout_s (default 30, greater than 0, up to 600) | exit_code, stdout, stderr, stdout_truncated, stderr_truncated, timed_out, duration |
launch_app | command required (argv list); cwd, env | id, pid, argv, running, exit_code, started_at, log_path |
list_launched | output (true adds output_tail, the last 16 KiB of each program’s output); forget_finished (true drops programs that exited before listing) | launched and count |
kill_launched | id required (from launch_app); signal (TERM default, INT, KILL) | The program’s entry after the signal. |
run_command keeps the first 1 MiB of stdout and of stderr and sets the _truncated flag when it cut one. It returns when the command exits; any process the command left in the background is killed then, so start long-running programs with launch_app. At the timeout the whole process group is killed, the action fails with command timed out, and data still holds the output so far. A non-zero exit code is still a successful action; check exit_code. Logs record only the program name, never its arguments.
launch_app starts the program in its own session with stdin closed and writes its stdout and stderr to log_path, under DESK_LAUNCH_LOG_DIR. IDs count up from 1. kill_launched signals the program’s whole process group and waits up to 2 seconds for it to exit. Programs still running when the server stops get SIGTERM.
Screen
| Action | Parameters | Returns |
|---|---|---|
get_resolution | none | width, height |
desktop_screenshot | width, height or whLargest (see screenshot sizes); path to also write the PNG inside the container | type, size (bytes) and path. Over POST / the image itself is not in the response; use GET /screenshot/desktop. MCP returns the image, and a script collects it with output_id. |
start_recording | mode (window default, desktop, viewport); fps (1 to 60, default 15); show_cursor (default true) | recording_id, mode, fps, show_cursor, tmp_path, capture_size |
stop_recording | slug required ([a-zA-Z0-9][a-zA-Z0-9_-]{0,62}) | recording_id, mode, slug, path, duration_s, size_bytes |
recording_status | none | active, and while recording recording_id, mode, started_at, elapsed_s, tmp_path |
Recording uses ffmpeg x11grab with H.264 into /recordings, which must be writable. Only one recording runs at a time. In this image window, desktop and viewport all capture the whole screen; viewport crops to the active surface, which a plugin can move. stop_recording writes /recordings/<slug>.mp4, and an existing name gets a -2, -3 suffix. Leftover temporary files older than an hour are removed at startup.
Clipboard
| Action | Parameters | Returns |
|---|---|---|
set_clipboard | text; or format with its value: html with text, image/png with content_base64, files with files | length and bytes for text; format and bytes otherwise |
get_clipboard | format: text (default), html, image/png, files | text, length, bytes for text and HTML; content_base64 for PNG; files (a list of file:// URIs) for files |
Text and HTML are limited to 1 MiB of UTF-8, and a PNG to 1 MiB after decoding. files takes up to 32 absolute paths to existing regular files, not symlinks, totalling at most 1 GiB, and offers them as text/uri-list. The clipboard stays set until something replaces it. Logs never record clipboard contents.
Events and handoff
The server keeps the last 1000 events in memory. Each has id, type, timestamp and data. The base publishes action_failure (with the action name), handoff_started and handoff_ended (with duration and activity); plugins can publish more. When older events were dropped, a read starting before them gets one gap event first, with oldest_available_id.
| Action | Parameters | Returns |
|---|---|---|
get_events | after_id (default 0), types (list), limit (1 to 1000, default 100) | events, count |
wait_for_event | after_id (default 0), types, timeout (greater than 0, up to 300, default 30) | events and count as soon as one matches, or an empty list at the timeout |
begin_handoff | none | active, started_at. Blocks every action that changes something until end_handoff, so a person can work in noVNC. |
handoff_status | none | active, started_at, activity |
end_handoff | none | active, duration in seconds, activity |
activity is null unless a plugin counts human input during the handoff.
During a handoff only these actions run: ping, list_actions, sleep, get_events, wait_for_event, handoff_status, end_handoff, recording_status, get_resolution, desktop_screenshot, desktop_list_windows, desktop_keybindings and list_launched. Everything else fails with human handoff is active.
Scripts and utility
| Action | Parameters | Returns |
|---|---|---|
run_script | steps (list) with optional name (default api_script) and on_error (stop default, or continue); or yaml, a whole script document as a string | The script result; see script mode. Screenshots collected with output_id come back as data URIs. |
list_actions | none | actions: every registered action name, plugins included |
ping | none | message: pong |
sleep | duration in seconds, 0 to 3600, default 1 | slept |
close | none | message: closing. Stops the server, which stops the container. |