A browser next to the agent, inside agent-manager

[YoanWai]

A lot of my day is a browser next to an agent. Checking a mockup it wrote, clicking through localhost after a change, reading docs, or showing it the exact button that looks wrong. Today that means jumping from the terminal to Chrome and back, and pasting what I saw by hand. I'd like the browser to live in agent-manager, next to the session that needs it, with every agent able to open and drive it.

cmux and Superset already ship a browser pane agents can script, and Claude desktop, Cursor, VS Code and Codex all let you point at an element and send it to the agent. This would bring that into the terminal, for every CLI agent-manager supports.

What it could look like

• An agent runs agent-manager browser open localhost:3000. A browser row appears nested under that session, the same way terminals do today. I see it in the preview, and I can focus it and click around.
• The agent drives that same tab. Each browser gets its own Chrome DevTools Protocol (CDP) port on 127.0.0.1, so agent-browser, Playwright MCP (--cdp-endpoint) or Chrome DevTools MCP (--browser-url) can attach to it. agent-manager hands out the address and leaves the automation to those tools.
• I point at something on the page and send it to the agent. One key starts a picker. I click an element and the agent's prompt gets the selector, a trimmed outerHTML, the text, and a cropped screenshot as a file path. On a React dev build it can also get the component and source line, the way react-grab does. It lands as a paste without Enter, so I can add "make this bolder" before sending.
• MCP and the CLI get the same few verbs: open_browser, list_browsers, close_browser, and maybe read_browser for a quick text snapshot. A short line in the launch prompt tells agents the browser exists.

Examples I'd use it for

1. Mockups. "Write three versions of the pricing card in one HTML file and open it next to you." I pick the one I like and send it back.
2. Frontend fixes. The agent opens the dev server, I select the broken header, it gets the selector and the source file, fixes it and reloads.
3. Repro. "Log in as the test user and reproduce feat: publish Homebrew as a cask #123." I watch it click through in the preview and step in when it gets stuck.
4. Reading. Docs, a PR or a design doc open beside the session that's using it.
5. Worktrees. Two sessions on two branches each get their own browser, port and profile, so cookies and dev servers stay apart.

What I tried

I ran Carbonyl (Chromium drawn as coloured half-block characters) inside an isolated tmux server.

• capture-pane -e kept the page, 24-bit colours included. So this style of rendering shows in our preview and focus view as they are today.
• agent-browser attached to its CDP port and got a clean snapshot (heading "Hello agent-manager", button "Click me").
• I injected a small picker over CDP, then sent a mouse click into the pane the way focus mode forwards one. It came back with html > body > button#go, the outerHTML and the element's box.
• A 120x40 pane gives the page a 634x413 viewport. --zoom=50 doubles that to 1268x826.

Carbonyl itself is out. Its last release was February 2023 and it runs Chromium 111.

The hard part is drawing it

Terminal browsers come in two kinds.

• Text cells, like Carbonyl and Browsh. They work in every terminal, in our preview and over SSH. Small text gets blurry.
• Real pixels through the Kitty graphics protocol, like terminal-browser (MIT, very active). Sharp, but our preview reads panes with capture-pane, which returns text only, so the image never reaches it. It would show only in a full attach (A), and only in terminals with Kitty Unicode placeholders (Ghostty, kitty, recent iTerm2, not yet WezTerm, Warp or the VS Code terminal).

terminal-browser already does a lot of what I described. It has an element picker that sends to the agent (built on react-grab), an agent-browser CLI on its CDP port, and two ways for a host app to integrate. One is a small socket API (split, listPanes, sendText, pasteKey, focusPane) that maps almost one to one onto how agent-manager creates terminals and pastes into sessions. The other is an embed mode where the host draws the image placeholders itself, which could put the real browser inside our preview box.

The options I see:

• A. Integrate terminal-browser. Implement its socket API so a browser opens as a row under the session and its picker pastes into the right agent. Then try embed mode for the preview. Sharpest result. It needs a Kitty-placeholder terminal, it's 130 to 140 MB per platform, and it sends usage telemetry unless turned off. Its built-in tmux mode sets server-wide options (extended-keys, focus-events) that would change key handling for every session on our server, so the socket API is the way in.
• B. Our own small CDP renderer. Use the Chrome people already have, or chrome-headless-shell, take CDP screencast frames and draw them as half-blocks in a pane. Works everywhere agent-manager runs today, preview included. Lower fidelity, and more code for us.
• C. Both. Same browser row, same CDP port, same picker. Pixels when the terminal can show them, cells when it can't.

I lean towards C, starting with A on terminals that support it.

Still open

• Security. A CDP port is full control of that browser, cookies included. I'd bind it to 127.0.0.1, give each browser its own profile, and never attach to someone's everyday Chrome profile. Should an agent open a browser without asking, the same as a terminal today?
• Hover. Focus mode doesn't forward plain mouse moves yet, so a picker can't highlight what's under the pointer.
• What should a pick send? Selector, HTML and a screenshot path, or also computed styles and the source location?

Which terminal do you run agent-manager in? And what would you open in a browser next to your agents?