All subsystems

2 of 9 subsystem areas need active recovery and 6 are degraded: 12 sessions are waiting at a usage limit or stalled goal, and 25 of 176 GitHub workflow jobs still target paid runners. 0 decisions are waiting on you; dedicated agents (marked with the green Agent working icon badge) are actively handling all 6 engineering repairs.

77%of last 100 Claude starts worked · holding steady

24 h all launchers: 76% of 1,745 starts · Improving

← 24 h agorecent steady windownow →

Agent working: launcher self-tests 8/8Codex 81% · Gemini 100%

14%of 103 tries (24 h)

12stuck now (6 limit · 6 goal)

Self-test: 8 of 9 pass · 59% turned away earlier · Improving

← 24 h agopost-fix windownow →

Agent working: moving capped sessions✓ 15 goals resumed

52/261rescued (20% · 24 h)

68smove · 7.0m detect

Today’s 457 timed rescues · 28 active goals · Intermittent

← 24 h agored = blocked · amber = partial · green = rescuednow →

Agent working: clearing resume-gate blocks✓ 15 goals resumed

102×of 415× weekly power left (25%) · Budget 59.3×/day

24h burn: 35.8× vs 59.3×/d (60%) · 2/9 Claude · 6/25 Codex

← 10-22dashed line = 59.3×/day max sustainable averagetoday →

Agent working: relogin monitor active● Claude 2/9 · Codex 6/25

37 of 38deputies running · 28 active goals

1 recovering (Ophelia) · 4 missing deputy goals auto-set

38 managed deputies (hover any dot)28 active goals

Agent working: Goal Steward every 2m✓ Signatures fresh

4.8%disk I/O wait

293CPU load (high)

32 threads: <40 normal · 40–250 busy · >250 high (15m: 330)

/home

95% full · 262 GB free

93% full · 11 GB free

Agent working: disk & worktree cleanup✓ Governor policy active

100%of 6,739 messages delivered · last 24 h

Last 100 calls: 100% · 0 failed deliveries · Steady

← 24 h ago100% all daynow →

✓ Unread queues mappedScratch inboxes pruning

58ready (1–2/acct)

102in use · 239 retiring

Release observer healthy · 5 of 12 paths on older commit

■ 58 ready (1–2/acct)■ 102 in use■ 239 awaiting cleanup

Agent working: retiring 239 old workspaces● 5 paths drifting

33/40busy (7 idle · 0 off)

25paid GitHub jobs

38 always-on + 2 of 12 on-demand VMs in use · API: 173/5,000

■ 151 of 176 jobs on Ellie (86%)■ 25 on paid GitHub (14%)

Agent working: moving 25 paid jobs to Ellie✓ 40/40 runners online

Needs you (0) & Web Instructions

No decisions are blocking work right now. You can send an instruction or note from anywhere on this page; drafts are preserved across refreshes and screened by the server safety gate.

Healthy

0 pending decisions — your Launch Governor policy is now active

Applied policy: Anything that collects or stores data/history, interactive sessions, and deputy repairs is never paused. Genuinely minor, non-urgent temporary tasks (temporary probes, minor canary checks, non-urgent batch checks that do not store history) pause briefly during heavy CPU/disk spikes and tell the caller to retry in 60s or pass --priority high.
Have a new instruction, priority change, or question for the fleet? Click Send instruction / note to message the coordinating agents directly from this page (with automatic draft saving and server-side safety verification).

Agents fixing (6)

Active engineering and stewardship sessions already running on Ellie. Paused goals are resumed automatically every 2 minutes.

Healthy

1. Resuming & moving sessions — Fleet Engineering & Launcher Session

Shipped and installed the central-history reconciliation, role preservation, and live-timing fixes into /opt/expertly-agents/bin/cc. Recent session resumes are succeeding cleanly as pre-fix refusals age out of the 24-hour window.

Agent working Active this hour See progress ›

Healthy

2. Rescue supervisor, watchdog & Goal Steward — Hot Watchdog, 15m Backup Rescuer & Goal Steward

Over the last 32 backup-rescuer passes (24h), 52 of 261 stuck-session rescue attempts succeeded (20%) with 209 blocked or failed by the resume-gate safety check. The 2-minute Goal Steward resumed 15 paused goals in the latest pass (28 active goals running now).

Agent working Checked this minute See progress ›

Healthy

3. CI runners, on-demand VMs & migrating remaining paid GitHub jobs — CI Runner & Guest-Agent Investigation (AZ9B5MAA)

All 40 Ellie self-hosted runners are online (33 busy right now, 7 idle, 0 offline). Already migrated 17 workflow jobs to Ellie (151 of 176 jobs on Ellie, 86%); actively migrating the remaining 25 eligible Linux jobs off paid GitHub runners.

Agent working Active this hour See progress ›

Healthy

4. AI account capacity & relogins — Token Manager & Relogins Steward

Tracking 9 of 35 ready automatic accounts right now (101.8× of 415.2× weekly power remaining: 2 of 9 Claude, 6 of 25 Codex, 1 of 1 Gemini) and recording 5-minute usage snapshots across all accounts.

Agent working Checked every 60s See progress ›

Healthy

5. Host disk cleanup & retiring finished workspaces — Ellie Disk & Infrastructure Steward

Maintaining 58 warm spare git workspaces (1–2 per account) with 102 workspaces actively in use and 239 finished workspaces being verified and retired. Launch Governor policy is active (never pauses data/history collectors; pauses only minor temporary probes during heavy disk/CPU spikes).

Agent working Active today See progress ›

Healthy

6. Supervised deputy liveness & goal recovery — Deputy Recovery Steward & Goal Steward

All 38 configured deputies are tracked every 60 seconds (37 running, 1 restarting/recovering). The Goal Steward automatically resumed the Deputy Recovery session and 10 deputy goals in this pass.

Agent working Checked every 60s See progress ›

Focus areas (9)

Click any capsule above or any row below to inspect full charts, tables, and technical session details.

Degraded 1 Starting sessions 77% of last 100 Claude starts worked · 76% across all 1,745 starts in 24 h

The session launchers (cc, cx, cg, and agent start) pick a healthy provider account, allocate a clean git workspace, and open an interactive or background coding session. When recent starts hold steady at a much higher rate than the 24-hour average after a repair, the card highlights the current window first.

Status	Launcher	Last 100 starts	24-hour rate	Miss breakdown	Typical start time
Degraded	Claude launcher (cc)	77% of last 100	72.9% (1,076 of 1,477)	196 turned away · 39 failed	0s typical (p95: 22.8m)
Healthy	Codex launcher (cx)	94% of last 100	88.9% (128 of 144)	1 turned away · 5 failed	12.9m typical (p95: 51.8m)
Broken	Gemini launcher (cg)	44% of last 100	96.4% (27 of 28)	0 turned away · 1 failed	1.3h typical (p95: 2.8h)
Healthy	Durable agent launcher (agent start)	98% of last 100	97.9% (94 of 96)	0 turned away · 0 failed	1s typical (p95: 2.8m)

Broken 2 Resuming and moving sessions 12 sessions stuck right now · 14% of 103 tries worked over 24 h (8 of 9 self-tests passing now)

Reopening or moving a conversation across accounts requires verifying that no other terminal holds the conversation file open, transferring session history, and confirming the reopened prompt is ready. Below, call outcomes and typical duration are shown on two separate stacked charts sharing one 24-hour time axis.

Agent working

Fleet Engineering & Launcher Session Active this hour

Technical details (session ID, workspace & attach command)

Session identifier: 0c152644-7d8d-4a04-b15e-72c936cf10d7 / restore-01a11448-11c4
Model: Gemini 4 Argon High & GPT-6 Astra High
Workspace location: Protected launcher workspace (/srv/agents/state/peaches-launcher)
Verify / attach command: tmux -S /tmp/tmux-1000/cc-protected attach -t restore-01a11448-11c4

12Sessions stuck right now

6At usage limit / token cap

6Goal paused or stalled

15Paused goals auto-resumed this pass

28Active goals running now

Reopen and move tries by outcome (count per bucket)

Worked Turned away by safety check Failed Unfinished

Typical time to finish a try, in seconds (median per bucket)

Median duration (seconds)

Hover any bar or point for exact counts and median seconds. Both charts share the same horizontal 24-hour timeline.

Status	Tool	Last hour (worked & speed)	24-hour rate	24-hour misses	Speed (typical & p95)
Broken	Codex resume (cxr)	0% (0 of 1 worked · 64s)	26.7% (8 of 30)	11 turned away · 6 failed · 3 unfinished	4.0m typical (p95: 16.8m)
Broken	Cross-account session mover (ccmove)	0 tries in last hour	0.0% (0 of 12)	6 turned away · 2 failed · 4 unfinished	75s typical (p95: 11.6m)
Broken	Claude resume (ccr)	0 tries in last hour	9.8% (6 of 61)	44 turned away · 7 failed · 2 unfinished	85s typical (p95: 13.8m)
Unmeasured	Gemini resume (cgr)	No calls in last hour	No calls in 24 h (0 tries)	—	—

Broken 3 Rescue supervisor & watchdog 52 of 261 stuck sessions rescued (24 h) · Typical move: 68s · Typical detect: 7.0m (today's 457 runs)

Three automated layers keep stuck sessions moving:

Hot watchdog (ccsentinel): watches active terminals continuously and moves sessions that hit a usage limit.
15-minute backup rescuer (ccsentinel-rescue-backstop): sweeps every 15 minutes in case the main watchdog misses a session. Red bars on the card indicate passes where a tool failed or candidate sessions were blocked by the resume-gate safety check.
2-minute Goal Steward (ccsentinel-goal-steward): automatically types /goal resume whenever a goal pauses and sets default goals on deputies.

Agent working

Hot Watchdog, 15m Backup Rescuer & Goal Steward Checked this minute

Technical details (session ID, workspace & attach command)

Session identifier: agent-ccsentinel / ccsentinel-goal-steward
Model: GPT-6 Astra High + Automated Timers
Workspace location: Managed deputy session (agent-ccsentinel) & systemd timers
Verify / attach command: tmux -S /tmp/tmux-1000/ccstartup attach -t agent-ccsentinel

68sTypical move time (today, p95: 1.9m)

7.0mTypical detection time (today, p95: 12.8m)

52 / 261Sessions rescued vs blocked/failed over last 32 passes (20%)

15Paused goals resumed by Goal Steward

Degraded 4 AI account capacity & usage burn rate 101.8× of 415.2× weekly power left (25%) · Max sustainable average: 59.3×/day (2.47×/hr) · Last 24h burn: 35.8× (60% of budget)

By recording remaining weekly usage per account every 5 minutes and subtracting consecutive readings (adjusting automatically when an account's weekly limit resets — 5 Claude resets and 0 Codex resets adjusted in window), we compare our actual usage burn against our maximum sustainable average with current accounts: 59.32×/day (or 2.47×/hour). Note: 5-minute snapshot tracking began Oct 22 for Claude and Oct 25 (10:58 PM EDT) for Codex, so Oct 22–24 bars compare Claude usage against the Claude sustainable budget (10.57×/day), while Oct 25–26 include both Claude and Codex.

Agent working

Token Manager & Relogins Steward Checked every 60s

Technical details (session ID, workspace & attach command)

Session identifier: AYJJJSAA (01a0e5fd-4ed2-7423-be91-125996ec7201)
Model: GPT-6 Astra High
Workspace location: Protected relogins workspace (/srv/agents/state/peaches-relogins)
Verify / attach command: cctoken check all --providers

Steady-State Capacity Budget vs. Actual Usage Burn (From 5-Minute Snapshot Subtractions)

Provider Pool	Weekly Power (100% Cap)	Remaining Right Now	Max Sustainable Average	Actual Burn (Last 24h)	Steady-State Verdict (More / Same / Less)
Claude (Anthropic)	74.0× (10 accts)	19.6× (26%)	10.57×/day (0.44×/hr)	12.76× (5 resets adj.)	Using more (121% of budget) — +2.2×/day over steady state (~0.8 extra 20× accounts needed at this pace)
Codex (OpenAI)	341.25× (28 accts)	82.2× (24%)	48.75×/day (2.03×/hr)	23.04× (0 resets adj.)	Within steady state (47% of budget) — sustainable with current accounts
Combined Fleet	415.25×	101.8× (25%)	59.32×/day (2.47×/hr)	35.80×	Within steady state (60% of sustainable budget)

Live Account Readiness Breakdown

Status	Provider	Automatic accounts ready	At usage limit	Needs login	Reserve accounts (manual only)
Degraded	Claude (Anthropic)	2 of 9 ready	6	1	1 reserve
Degraded	Codex (OpenAI)	6 of 25 ready	19	0	4 reserve
Healthy	Gemini (Google)	1 of 1 ready	0	0	0 reserve

Degraded 5 Supervised deputies 37 of 38 deputies running · 28 active goals · 4 missing deputy goals automatically set

Deputies are persistent role-based sessions (such as Commodore, Dorian, Posey, Relogins, and Sentinel). Every 2 minutes, ccsentinel-goal-steward verifies each deputy has an active /goal and automatically sends /goal resume whenever a goal is paused or stalled.

Agent working

Deputy Recovery Steward & Goal Steward Checked every 60s

All 38 configured deputies are tracked every 60 seconds (37 running, 1 restarting/recovering). The Goal Steward automatically resumed the Deputy Recovery session and 10 deputy goals in this pass.

Technical details (session ID, workspace & attach command)

Session identifier: restore-01a10c0b-d5ae
Model: GPT-6 Astra Medium
Workspace location: Protected deputy recovery workspace (restore-01a10c0b-d5ae)
Verify / attach command: tmux -S /tmp/tmux-1000/cc-protected attach -t restore-01a10c0b-d5ae

Degraded 6 Host and disks: Ellie Disk I/O wait 4.8% · Processor load 293 (high on 32 threads) · /home 95% (262 GB free) · / 93% (11 GB free)

Ellie is a 32-thread Linux server hosting all agent sessions, warm git worktrees, and Incus CI virtual machines. Processor load under 40 is normal, 40–250 is busy, and above 250 indicates heavy queueing across concurrent agent and test processes. Per your policy decision, the Launch Governor never pauses anything that collects or stores data/history, interactive sessions, or deputy repairs, and only pauses genuinely minor temporary probes during heavy spikes.

Agent working

Ellie Disk & Infrastructure Steward Active today

Technical details (session ID, workspace & attach command)

Session identifier: c24c0191 / restore-ba8e52a2-52d8
Model: Gemini 4 Argon High & GPT-6 Astra High
Workspace location: Protected infrastructure workspace (restore-ba8e52a2-52d8)
Verify / attach command: tmux -S /tmp/tmux-1000/cc-protected attach -t restore-ba8e52a2-52d8

Healthy 7 Agent messaging 100% of 6,739 inter-agent messages delivered over 24 h · 0 failed deliveries

Agents coordinate via agent tell and durable inbox queues under /srv/agents/agent-hub/inboxes and /tmp/claude-agents. Delivery has remained at 100% across all 24 hours.

Agent working

Expertly Health & Fleet Collector Session Active this hour

Message delivery (agent-tell) is 100% reliable. Per-deputy unread message counts are mapped across both /srv/agents/agent-hub/inboxes and /tmp/claude-agents/<agent_id>/inbox.

Technical details (session ID, workspace & attach command)

Session identifier: A2RGRQAA / 0c152644
Model: Gemini 4 Argon High
Workspace location: Active Fleet Collector
Verify / attach command: /opt/expertly-agents/bin/agent who

Degraded 8 Watchers & workspace pool 58 warm spare workspaces ready (1–2 per account) · 102 in use by active sessions · 239 finished workspaces awaiting cleanup

Instead of keeping 200 idle workspaces, the warm workspace pool maintains 1 to 2 ready spare git worktrees per Unix account (currently 58 ready across 29 accounts) so new sessions start quickly without wasting disk space. Right now 102 workspaces are actively in use by live sessions, and 239 finished workspaces are in the retirement queue being verified and cleaned up.

Agent working

Ellie Disk & Infrastructure Steward Active today

Technical details (session ID, workspace & attach command)

Session identifier: c24c0191 / restore-ba8e52a2-52d8
Model: Gemini 4 Argon High & GPT-6 Astra High
Workspace location: Protected infrastructure workspace (restore-ba8e52a2-52d8)
Verify / attach command: tmux -S /tmp/tmux-1000/cc-protected attach -t restore-ba8e52a2-52d8

58Warm spare workspaces ready now (target: 1–2 per account)

102Workspaces currently in use by active sessions

239Finished workspaces awaiting cleanup (held_for_retirement)

5 of 12Secondary toolkit checkouts on older commit

Degraded 9 CI testing and deployments 33 of 40 Ellie runners busy right now (7 idle, 0 offline) · 151 of 176 workflow jobs on Ellie (86%) · 25 on paid GitHub

How Ellie's CI runners work (Always-On Docker Runners + On-Demand Isolated VMs):

38 Always-On Docker Runners + 2 of 12 On-Demand Isolated VMs = 40 Online Runners (33 busy right now, 7 idle, 0 offline): Ellie runs 38 persistent Docker container runners across our 3 repositories, plus a pool of up to 12 on-demand isolated Incus virtual machines. Why are only 2 of 12 on-demand VMs active right now? Because those VMs are created on demand only when a workflow job specifically requests an isolated disposable VM (the ephemeral label); standard jobs run on the always-on Docker runners.
Why 25 of 176 workflow jobs still target GitHub (and why that number is red): Paid GitHub jobs should be close to zero. We already migrated 17 jobs to Ellie in PR #20452, bringing Ellie to 151 of 176 workflow jobs (86%). Of the remaining 25 paid jobs: 8 Linux test/contract jobs (ci.yml, ellie-sessions.yml, ivibecode-source-contracts.yml) are being moved to Ellie right now; 14 security & secret-scanner jobs run on untrusted external pull requests and stay off Ellie's shared network until sandboxed VM routing is complete; and 3 OS-specific jobs require macOS or Windows.
What happens when new workflows are created (automatic Ellie routing + fallback): Workflows use runs-on: ${{ fromJSON(vars.CI_RUNNER || '["self-hosted", "linux"]') }} so every Linux job runs on Ellie by default and automatically falls back to GitHub if vars.CI_RUNNER is switched during an Ellie outage. In addition, hosted-runner-guard.yml automatically checks every pull request so new workflows cannot accidentally hardcode ubuntu-latest.

Agent working

CI Runner & Guest-Agent Investigation (AZ9B5MAA) Active this hour

Technical details (session ID, workspace & attach command)

Session identifier: AZ9B5MAA / restore-8a37da34-140f
Model: Claude Opus & GPT-6 Astra
Workspace location: Protected CI recovery workspace
Verify / attach command: tmux -S /tmp/tmux-1000/cc-protected attach -t restore-8a37da34-140f

33 / 40Ellie runners busy right now (7 idle · 0 offline)

2 of 12On-demand isolated Incus VMs in use now (created when jobs request ephemeral)

151 of 176Workflow jobs routed to Ellie self-hosted (86%)

25Jobs still on paid GitHub runners (11 files — target: near 0)

● 173 / 5,000GitHub API used this hour (4,827 left)

Ellie Self-Hosted Runner Pools (Live Busy vs. Idle Status)

Status	Runner Pool	Target Repository	Online / Offline	Active Load Right Now	Purpose
Healthy	`expertly-runner-1..16 (+ 2 temp VMs)`	`expertly-develop`	18 online (0 offline)	17 busy · 1 idle	Application test & build jobs (inside ci-runners VM + on-demand VMs)
Healthy	`expertly-coding-runner-1..16`	`expertly-coding-agents`	16 online (0 offline)	16 busy · 0 idle	Coding-agent PR & suite jobs (running on Ellie host Docker, 80 GB free)
Healthy	`expertly-agents-runner-1..6`	`expertly-agents`	6 online (0 offline)	0 busy · 6 idle	Core agent CLI & daemon jobs (running on Ellie host Docker, 80 GB free)

Remaining 25 Workflow Jobs in `expertly-develop` Still Using Paid GitHub Runners

Status	Workflow File	Paid GitHub Jobs	Jobs Already on Ellie	Why It Is Still on GitHub & Next Step
Broken	`.github/workflows/ci.yml`	6 paid jobs	58 jobs on Ellie	6 Linux authority/test jobs — being migrated to Ellie on-demand isolated VMs now
Degraded	`.github/workflows/discover-security-scan.yml`	5 paid jobs	0 jobs on Ellie	5 security scan jobs — runs on untrusted PR branches; kept isolated off shared host
Degraded	`.github/workflows/discover-secret-scan.yml`	4 paid jobs	0 jobs on Ellie	4 secret scan jobs — runs on untrusted PR branches; kept isolated off shared host
Degraded	`.github/workflows/discover-codeql.yml`	2 paid jobs	0 jobs on Ellie	2 CodeQL static analysis jobs — heavy untrusted PR scanner
Degraded	`.github/workflows/scanner-protection.yml`	2 paid jobs	0 jobs on Ellie	2 external scanner-trap verification probes
Broken	`.github/workflows/ellie-sessions.yml`	1 paid jobs	0 jobs on Ellie	1 Linux test job — being migrated to Ellie self-hosted runner now
Degraded	`.github/workflows/ivibecode-mac-app-compile.yml`	1 paid jobs	1 jobs on Ellie	1 macOS Xcode compile job — requires macOS runner (Ellie is Linux)
Broken	`.github/workflows/ivibecode-source-contracts.yml`	1 paid jobs	1 jobs on Ellie	1 Linux contract check — being migrated to Ellie self-hosted runner now
Degraded	`.github/workflows/scanner-protection-audit.yml`	1 paid jobs	0 jobs on Ellie	1 external scanner-trap audit probe
Degraded	`.github/workflows/test-mac-install.yml`	1 paid jobs	0 jobs on Ellie	1 macOS installer test — requires macOS runner (Ellie is Linux)
Degraded	`.github/workflows/test-windows-install.yml`	1 paid jobs	0 jobs on Ellie	1 Windows installer test — requires Windows runner (Ellie is Linux)

Data sources & freshness

Refreshed every 60 seconds by fleet-health-collector and published to Expertly Health and https://fleet-health.ai.devintensive.com.

Status	Data Source	What It Measures	Update Cadence
Healthy	`/var/log/runlog/*.jsonl`	Session starts, resumes, moves, and message delivery outcomes & durations	Live stream (60s aggregation)
Healthy	`rescue-backstop.jsonl` & `ccsentinel-goal-steward`	Today’s live session rescue & detection timings, stuck pane census, and auto-resumed goals	Every 2m (Goal Steward) & 15m (Backstop)
Healthy	`cctoken` / `cxtoken` / `cgtoken`	Provider account readiness and 5-minute snapshot usage burn subtractions	Every 60s (5m burn snapshots)
Healthy	`deputy-status.json` & `deputy-liveness-signatures.json`	All 38 configured deputies and stall signatures	Every 60s
Healthy	Local `.github/workflows/*.yml` & GitHub runner API	Live busy/idle/online state across 40 Ellie runners + YAML-parsed workflow routing	Every 60s

All subsystems

Starting sessions

Resuming and moving sessions

Rescue supervisor & watchdog

AI account capacity

Supervised deputies

Host and disks: Ellie

Agent messaging

Watchers & workspace pool

CI testing and deployments

Needs you (0) & Web Instructions

Agents fixing (6)

Focus areas (9)

Reopen and move tries by outcome (count per bucket)

Typical time to finish a try, in seconds (median per bucket)

Reopen and move tries by outcome (count per bucket)

Typical time to finish a try, in seconds (median per bucket)

Steady-State Capacity Budget vs. Actual Usage Burn (From 5-Minute Snapshot Subtractions)

Live Account Readiness Breakdown

Ellie Self-Hosted Runner Pools (Live Busy vs. Idle Status)

Remaining 25 Workflow Jobs in expertly-develop Still Using Paid GitHub Runners

Data sources & freshness

Remaining 25 Workflow Jobs in `expertly-develop` Still Using Paid GitHub Runners