Agent Replay

The Problem

AI agents debug by screenshot. That's expensive, slow, and lossy.

Screenshots → Vision Model

Can't see console errors
Can't see network failures
Can't see timing or state
Expensive vision model per frame
Slow round-trip per screenshot

Agent Replay → Structured Data

Every console log and error
Every network request + response
Full DOM state via rrweb
Cheap — just files on disk
Instant — read a JSONL file

Features

Everything that happens in the browser, captured and structured.

🎥

DOM Recording

rrweb-powered. Captures every mutation, scroll, resize, and interaction. Full page reconstruction from events.

📊

Console Capture

Logs, warnings, errors with full stack traces. Separate console.jsonl so agents can scan quickly.

🌐

Network Capture

Every fetch and XHR — URL, method, status, timing, request/response bodies. Spot API failures instantly.

⚛️

React Provider

Drop in <AgentReplayProvider> and you're done. Auto-disabled in production. Zero config.

🤖

Agent-Optimized

Structured JSONL files, separate per signal. LLM-friendly summary.md. No context exhaustion.

🧩

Chrome Extension Soon

Zero-config recording for any site. No code changes needed. Perfect for testing third-party apps.

Architecture

Browser captures events → sidecar writes files → agent reads what it needs.

  ┌─────────────────────────────────┐
  │  Browser (localhost:3000)        │
  │                                 │
  │  AgentReplayProvider captures:  │
  │  • DOM mutations  (rrweb)       │
  │  • Console logs   (plugin)     │
  │  • Network reqs   (plugin)     │
  │  • Errors         (window)     │
  └────────────────┬────────────────┘
                   │
                   ▼
  ┌─────────────────────────────────┐
  │  Sidecar / API Route             │
  │  Receives events via POST/WS    │
  └────────────────┬────────────────┘
                   │
                   ▼
  ┌─────────────────────────────────┐
  │  .agent-replay/                  │
  │  sessions/<timestamp>/          │
  │    events.jsonl                 │
  │    console.jsonl                │
  │    network.jsonl                │
  │    errors.jsonl                 │
  │    summary.md                   │
  └────────────────┬────────────────┘
                   │
          ┌────────┴────────┐
          ▼                 ▼
  🤖 Agent reads      👀 Human replays
  structured files    video playback

Quick Start

Add the provider. That's it.

app/layout.tsx

// Drop into your root layout
import { AgentReplayProvider }
  from "@boe-ventures/agent-replay/react";

export default function Layout({ children }) {
  return (
    <html>
      <body>
        <AgentReplayProvider>
          {children}
        </AgentReplayProvider>
      </body>
    </html>
  );
}

// Auto-disabled in production.
// Zero runtime cost when off.

Agent reads the output

# Check for errors (highest signal)
cat .agent-replay/latest/errors.jsonl

# Get a text summary
npx agent-replay summary

# Check network failures
npx agent-replay network --failures

# Watch events in real-time
npx agent-replay watch

# Output structure:
# .agent-replay/latest/
#   errors.jsonl   ← read this first
#   console.jsonl  ← logs + warnings
#   network.jsonl  ← API calls
#   summary.md     ← LLM-friendly

How Agents Use It

Complete observability when paired with browser automation.

Step 1

✍️

Agent writes code

Makes changes to your codebase

Step 2

🔄

Dev server reloads

Agent Replay starts capturing

Step 3

🤖

Agent tests with browser

agent-browser clicks + navigates

Step 4

📊

Agent reads results

errors.jsonl → fix → repeat

agent-browser + agent-replay workflow

# agent-browser: the outside view (what's on screen)
agent-browser open http://localhost:3000
agent-browser snapshot   # accessibility tree + element refs
agent-browser click @e3  # interact with elements

# agent-replay: the inside view (what happened under the hood)
cat .agent-replay/latest/errors.jsonl    # "TypeError at checkout.tsx:47"
cat .agent-replay/latest/network.jsonl   # "POST /api/checkout → 500"
cat .agent-replay/latest/console.jsonl   # "Warning: cart.items is empty"

# Together: complete observability without screenshots or vision models.

Output Structure

Separate files per signal. Agent picks what it needs.

.agent-replay/
├── sessions/
│   └── 2026-04-25T1430Z/
│       ├── events.jsonl       ← Full rrweb event stream
│       ├── console.jsonl      ← Console logs, warnings, errors
│       ├── network.jsonl      ← Fetch/XHR with timing + bodies
│       ├── errors.jsonl       ← Errors only (read this first!)
│       ├── react-tree.json    ← Component tree snapshot
│       └── summary.md         ← LLM-friendly text summary
└── latest → sessions/2026-04-25T1430Z/