Computer Use with Claude

Enable Claude to interact with a computer through screenshots, mouse, and keyboard. This is an advanced experimental feature that allows Claude to see your screen and control your computer to accomplish tasks.

Usage

claude_computer(
  prompt,
  screenshot_fn,
  action_fn,
  display_size = c(1920L, 1080L),
  ml = NULL,
  max_iter = 20L,
  sys_prompt = NULL,
  echo = "all"
)

Arguments

prompt

Character. Instruction for Claude (e.g., "Open the calculator app").

screenshot_fn

Function. Takes no arguments, returns path to current screenshot. Must capture the current screen state. Called initially and after each action.

action_fn

Function. Executes actions. Receives a list with:

• action: The action type (see Actions below)

• coordinate: c(x, y) for mouse actions (optional)

• text: Text to type or key to press (optional)

display_size

Integer vector of length 2. Screen dimensions c(width, height). Default c(1920, 1080). Must match actual display for accurate clicking.

ml

Character. Model ID. Default uses Opus 4.5, which works best for computer use.

max_iter

Integer. Maximum action iterations before stopping. Default 20. Prevents runaway loops.

sys_prompt

Character. Optional system prompt. Default NULL.

echo

Character. Output mode: "none", "output", or "all". Default "all" to observe actions in real-time.

Value

List with:

• response: Claude's final response

• actions: List of all actions executed

• iterations: Number of iterations completed

• chat: The chat object (for continuing the session)

Details

How Computer Use Works

Computer use follows an agentic loop:

1. Your screenshot_fn captures the current screen

2. Claude receives the screenshot and analyzes the UI

3. Claude decides on an action and calls the computer tool

4. Your action_fn executes the action (mouse click, typing, etc.)

5. New screenshot is taken

6. Repeat until task complete or max_iter reached

Available Actions

Claude can request these actions through the computer tool:

Action	Description	Requires
screenshot	Take a new screenshot	-
mouse_move	Move mouse to position	coordinate
left_click	Left mouse click	coordinate
right_click	Right mouse click	coordinate
double_click	Double left click	coordinate
type	Type text	text
key	Press a key	text (key name)
scroll_up	Scroll up	coordinate (optional)
scroll_down	Scroll down	coordinate (optional)

Safety Considerations

Computer use gives Claude control of your system. Follow these guidelines:

• Isolation: Run in a VM or container when possible

• Network: Limit network access to prevent data exfiltration

• Sensitive data: Don't expose passwords, API keys, or personal info

• Monitoring: Watch the session and be ready to interrupt

• Permissions: Run with minimal system permissions

Requirements

• Beta header: beta_headers$BETA_COMPUTER_USE

• Screenshot capability (e.g., magick, screenshot package, or system tools)

• Action execution (e.g., RSelenium, reticulate with pyautogui, or system tools)

See also

beta_headers for the required beta header constant

Other claude-api: claude_new()

Examples

if (FALSE) { # \dontrun{
# Example implementation (conceptual - replace with real functions)

# Screenshot function using system tools
take_screenshot <- function() {
  path <- tempfile(fileext = ".png")
  # On macOS:
  system2("screencapture", c("-x", path))
  # On Linux:
  # system2("scrot", path)
  # On Windows:
  # Use magick or screenshot package
  path
}

# Action function using pyautogui via reticulate
execute_action <- function(action) {
  pyautogui <- reticulate::import("pyautogui")

  switch(action$action,
    "mouse_move" = pyautogui$moveTo(action$coordinate[1], action$coordinate[2]),
    "left_click" = pyautogui$click(action$coordinate[1], action$coordinate[2]),
    "right_click" = pyautogui$rightClick(action$coordinate[1], action$coordinate[2]),
    "double_click" = pyautogui$doubleClick(action$coordinate[1], action$coordinate[2]),
    "type" = pyautogui$write(action$text),
    "key" = pyautogui$press(action$text),
    "scroll_up" = pyautogui$scroll(3),
    "scroll_down" = pyautogui$scroll(-3)
  )
}

# Run computer use
result <- claude_computer(
  prompt = "Open the calculator app and compute 123 * 456",
  screenshot_fn = take_screenshot,
  action_fn = execute_action,
  display_size = c(1920, 1080)
)

# Check results
print(result$response)
print(length(result$actions))
} # }