Compare commits
17 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 8321239461 | |||
| 22cc758b1a | |||
| 0b7ec51d0a | |||
| 4f1a9d7aef | |||
| 503309d244 | |||
| 1c76587ce2 | |||
| 24c70ec974 | |||
| 2d993cd0d5 | |||
| babe5070e2 | |||
| 009f3799cd | |||
| 95237a90a7 | |||
| e6cdc543f3 | |||
| f092bafe88 | |||
| 3f1fffeb6d | |||
| c6083a474c | |||
| 10ec856ff2 | |||
| 0f6c9b3874 |
@@ -1,3 +1,8 @@
|
||||
## Priorities
|
||||
|
||||
- Prioritise, in this order: stability, simplicity, performance.
|
||||
- Before changing session or timeline code, record a production benchmark baseline and compare it after the change.
|
||||
|
||||
## Debugging
|
||||
|
||||
- NEVER try to restart the app, or the server process, EVER.
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
- Prioritize stability, then simplicity, then measurement overhead.
|
||||
- Use Playwright for scenario control, isolation, and completion checks.
|
||||
- Use Chrome Performance traces for generic browser profiling.
|
||||
- Use Electron `contentTracing` for packaged multi-process profiling.
|
||||
- Keep custom probes only for product-specific measurements.
|
||||
- Do not duplicate measurements across the harness, probes, and traces.
|
||||
- Run benchmarks serially to avoid cross-test contention.
|
||||
- Run benchmarks against production builds.
|
||||
- Keep detailed profiling opt-in when it changes workload behavior.
|
||||
- Preserve raw diagnostic data or use lossless representations.
|
||||
- Do not enforce machine-dependent performance thresholds.
|
||||
- Assert scenario completion and metric collection only.
|
||||
- Keep normal test discovery free of manual benchmarks.
|
||||
@@ -0,0 +1,77 @@
|
||||
# Manual app performance suite
|
||||
|
||||
The app's high-volume performance diagnostics live under `packages/app/e2e/performance` and are excluded from normal local and CI Playwright discovery. The benchmark config builds the app and serves the production bundle before running scenarios serially.
|
||||
|
||||
Run the suite explicitly from `packages/app`:
|
||||
|
||||
```sh
|
||||
bun run test:bench
|
||||
```
|
||||
|
||||
PowerShell:
|
||||
|
||||
```powershell
|
||||
$env:PLAYWRIGHT_WORKERS = "1"
|
||||
bun run test:bench
|
||||
```
|
||||
|
||||
The suite contains:
|
||||
|
||||
- cold and hot session-tab timing
|
||||
- cached session repaint and mutation tracing
|
||||
- streaming timeline throughput, RAF-gap, long-task, geometry, and remount diagnostics
|
||||
|
||||
All benchmarks import the shared `benchmark` fixture. Pages created through Playwright's `page` fixture automatically capture main-frame navigation history and emit a Chrome trace when `OPENCODE_PERFORMANCE_TRACE_DIR` is set. Benchmarks that need isolated browser contexts use `withBenchmarkPage`, which owns the context and the same diagnostics lifecycle.
|
||||
|
||||
New benchmarks should look like normal Playwright tests:
|
||||
|
||||
```ts
|
||||
import { benchmark, expect } from "../benchmark"
|
||||
|
||||
benchmark("measures one interaction", async ({ page, report }) => {
|
||||
// Only scenario-specific setup and interaction belong here.
|
||||
report({ durationMs: 42 })
|
||||
})
|
||||
```
|
||||
|
||||
The fixture requires every benchmark to call `report()`, automatically names and closes traces, captures navigation history, attaches that history when a test fails, and emits metrics as a consistent `BENCHMARK` JSON line.
|
||||
|
||||
```text
|
||||
BENCHMARK {"name":"...","context":{"project":"chromium","platform":"darwin"},"metrics":{...}}
|
||||
```
|
||||
|
||||
Every observed page also emits `BENCHMARK_PAGE` with the same run ID, navigation history, and optional trace path before the final status-bearing `BENCHMARK` record. Chrome traces are browser-wide page-lifetime diagnostics; scenario metrics use narrower explicitly named observation windows.
|
||||
|
||||
This follows the stack's own guidance: [Electron recommends repeated Chrome DevTools and Chrome Tracing measurement](https://www.electronjs.org/docs/latest/tutorial/performance), [Chrome DevTools recommends Performance recordings for runtime work](https://developer.chrome.com/docs/devtools/performance), and [Playwright uses traces for test debugging rather than renderer profiling](https://playwright.dev/docs/trace-viewer).
|
||||
|
||||
These Playwright benchmarks profile the shared app renderer in Chromium. A future packaged Electron benchmark that needs main-process and multi-process attribution should use Electron's official [`contentTracing`](https://www.electronjs.org/docs/latest/api/content-tracing/) API rather than extending this renderer harness with bespoke process instrumentation.
|
||||
|
||||
CPU and high-volume visual profiling are disabled by default. Set `TIMELINE_CPU_PROFILE=1` to enable both, or additionally set `TIMELINE_VISUAL_PROFILE=0` for CPU-only profiling.
|
||||
|
||||
The streaming scenario's 30x CPU throttle is a deterministic stress profile, not a simulated end-user device.
|
||||
|
||||
Benchmarks do not assert machine-dependent performance budgets. Streaming processes 160 deltas by default and reports renderer-observed completion time, throughput, RAF callback-gap distributions, frame-budget equivalents, and long tasks through final geometry settlement. Delta count and delivery batch are included in result context when overridden. These are main-thread callback diagnostics, not compositor presentation or dropped-frame measurements. Visual-only and geometry metrics are `null` when their probes are disabled. Tab metrics describe sampled DOM observations. Assertions verify scenario and metric collection completion. Repeated repaint states are run-length grouped, but every original observation timestamp is retained alongside raw mutation batches and layout shifts.
|
||||
|
||||
Committed smoke and regression tests continue to own correctness coverage for pagination, tab paint, context resize, collapse state, and composer spacing.
|
||||
|
||||
## Chrome traces
|
||||
|
||||
Set `OPENCODE_PERFORMANCE_TRACE_DIR` to emit a standard Chrome DevTools trace for every benchmark page automatically:
|
||||
|
||||
```sh
|
||||
OPENCODE_PERFORMANCE_TRACE_DIR=/tmp/opencode-performance-traces \
|
||||
bunx playwright test --config e2e/performance/playwright.config.ts \
|
||||
timeline/session-tab-switch-benchmark.spec.ts
|
||||
```
|
||||
|
||||
The emitted JSON is a standard Chrome trace and can be loaded directly into the Chrome DevTools Performance panel. `devtools-tracing` can optionally inspect it from the command line without adding package scripts or dependencies:
|
||||
|
||||
Trace capture mirrors [Puppeteer's official tracing defaults and lifecycle](https://pptr.dev/api/puppeteer.tracing), using Chrome's `ReturnAsStream` transfer mode and failing when Chromium reports trace data loss.
|
||||
|
||||
```sh
|
||||
bunx devtools-tracing stats <trace-path-from-BENCHMARK_PAGE>
|
||||
```
|
||||
|
||||
INP analysis requires a trace with a supported navigation/interaction insight. Selector statistics require a trace captured with `OPENCODE_PERFORMANCE_SELECTOR_TRACE=1`.
|
||||
|
||||
`e2e/performance/playwright.uncapped.config.ts` disables Chromium frame-rate limiting for explicit uncapped diagnostics. Native product benchmarks should use the default Playwright configuration.
|
||||
@@ -0,0 +1,144 @@
|
||||
import { expect, test as base, type Browser, type Page, type TestInfo } from "@playwright/test"
|
||||
import { startChromeTrace } from "./chrome-trace"
|
||||
|
||||
type BenchmarkFixtures = {
|
||||
report: (metrics: Record<string, unknown>, context?: Record<string, unknown>) => void
|
||||
reportState: { payload?: { metrics: Record<string, unknown>; context: Record<string, unknown> } }
|
||||
benchmarkResult: void
|
||||
}
|
||||
|
||||
export type PerformancePageDiagnostics = {
|
||||
navigations: string[]
|
||||
stop: () => Promise<string | undefined>
|
||||
}
|
||||
|
||||
const pages = new WeakMap<Page, PerformancePageDiagnostics>()
|
||||
|
||||
export const benchmark = base.extend<BenchmarkFixtures>({
|
||||
reportState: async ({}, use) => use({}),
|
||||
report: async ({ reportState }, use) => {
|
||||
await use((metrics, context = {}) => {
|
||||
if (reportState.payload) throw new Error("Benchmark reported metrics more than once")
|
||||
reportState.payload = { metrics, context }
|
||||
})
|
||||
},
|
||||
benchmarkResult: [
|
||||
async ({ reportState }, use, testInfo) => {
|
||||
await use()
|
||||
const missing = !reportState.payload
|
||||
console.log(
|
||||
`BENCHMARK ${JSON.stringify({
|
||||
schemaVersion: 2,
|
||||
runID: process.env.OPENCODE_PERFORMANCE_RUN_ID,
|
||||
name: benchmarkName(testInfo),
|
||||
status: missing ? "failed" : testInfo.status,
|
||||
expectedStatus: testInfo.expectedStatus,
|
||||
retry: testInfo.retry,
|
||||
repeatEachIndex: testInfo.repeatEachIndex,
|
||||
context: {
|
||||
project: testInfo.project.name,
|
||||
platform: process.platform,
|
||||
...reportState.payload?.context,
|
||||
},
|
||||
metrics: reportState.payload?.metrics ?? null,
|
||||
error: missing ? "Benchmark did not report metrics" : undefined,
|
||||
})}`,
|
||||
)
|
||||
if (missing && testInfo.status === testInfo.expectedStatus)
|
||||
throw new Error(`Benchmark did not report metrics: ${benchmarkName(testInfo)}`)
|
||||
},
|
||||
{ auto: true },
|
||||
],
|
||||
page: async ({ page }, use, testInfo) => {
|
||||
const name = benchmarkName(testInfo)
|
||||
const diagnostics = await observePerformancePage(page, name)
|
||||
try {
|
||||
await use(page)
|
||||
} finally {
|
||||
try {
|
||||
await reportPerformancePage(name, diagnostics, testInfo)
|
||||
} finally {
|
||||
if (testInfo.status !== testInfo.expectedStatus) {
|
||||
await testInfo.attach("performance-navigations", {
|
||||
body: JSON.stringify(diagnostics.navigations, null, 2),
|
||||
contentType: "application/json",
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
})
|
||||
|
||||
function benchmarkName(testInfo: TestInfo) {
|
||||
return testInfo.titlePath.slice(1).join(" > ")
|
||||
}
|
||||
|
||||
export { expect }
|
||||
|
||||
async function observePerformancePage(page: Page, name: string) {
|
||||
const navigations: string[] = []
|
||||
const onNavigation = (frame: ReturnType<Page["mainFrame"]>) => {
|
||||
if (frame === page.mainFrame()) navigations.push(frame.url())
|
||||
}
|
||||
page.on("framenavigated", onNavigation)
|
||||
const stopTrace = await startChromeTrace(page, name).catch((error) => {
|
||||
page.off("framenavigated", onNavigation)
|
||||
throw error
|
||||
})
|
||||
let stopping: Promise<string | undefined> | undefined
|
||||
const diagnostics: PerformancePageDiagnostics = {
|
||||
navigations,
|
||||
stop() {
|
||||
page.off("framenavigated", onNavigation)
|
||||
return (stopping ??= stopTrace?.() ?? Promise.resolve(undefined))
|
||||
},
|
||||
}
|
||||
pages.set(page, diagnostics)
|
||||
return diagnostics
|
||||
}
|
||||
|
||||
export async function withBenchmarkPage<T>(
|
||||
browser: Browser,
|
||||
name: string,
|
||||
run: (page: Page) => Promise<T>,
|
||||
testInfo?: TestInfo,
|
||||
) {
|
||||
const context = await browser.newContext()
|
||||
try {
|
||||
const page = await context.newPage()
|
||||
const diagnostics = await observePerformancePage(page, name)
|
||||
try {
|
||||
return await run(page)
|
||||
} finally {
|
||||
await reportPerformancePage(name, diagnostics, testInfo)
|
||||
}
|
||||
} finally {
|
||||
await context.close()
|
||||
}
|
||||
}
|
||||
|
||||
async function reportPerformancePage(name: string, diagnostics: PerformancePageDiagnostics, testInfo?: TestInfo) {
|
||||
const trace = await diagnostics.stop()
|
||||
console.log(
|
||||
`BENCHMARK_PAGE ${JSON.stringify({
|
||||
schemaVersion: 2,
|
||||
runID: process.env.OPENCODE_PERFORMANCE_RUN_ID,
|
||||
name,
|
||||
test: testInfo ? benchmarkName(testInfo) : undefined,
|
||||
retry: testInfo?.retry,
|
||||
repeatEachIndex: testInfo?.repeatEachIndex,
|
||||
context: {
|
||||
platform: process.platform,
|
||||
trace,
|
||||
selectorTrace: process.env.OPENCODE_PERFORMANCE_SELECTOR_TRACE === "1",
|
||||
},
|
||||
navigations: diagnostics.navigations,
|
||||
})}`,
|
||||
)
|
||||
}
|
||||
|
||||
export function benchmarkDiagnostics(page: Page) {
|
||||
const diagnostics = pages.get(page)
|
||||
if (!diagnostics) throw new Error("Performance diagnostics are not installed for this page")
|
||||
return diagnostics
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
import type { CDPSession, Page } from "@playwright/test"
|
||||
import path from "node:path"
|
||||
import { mkdir, open, rename } from "node:fs/promises"
|
||||
import { Buffer } from "node:buffer"
|
||||
import { createHash, randomUUID } from "node:crypto"
|
||||
|
||||
const categories = [
|
||||
"-*",
|
||||
"devtools.timeline",
|
||||
"v8.execute",
|
||||
"disabled-by-default-devtools.timeline",
|
||||
"disabled-by-default-devtools.timeline.frame",
|
||||
"toplevel",
|
||||
"blink.console",
|
||||
"blink.user_timing",
|
||||
"latencyInfo",
|
||||
"disabled-by-default-devtools.timeline.stack",
|
||||
"disabled-by-default-v8.cpu_profiler",
|
||||
]
|
||||
|
||||
export async function startChromeTrace(page: Page, name: string) {
|
||||
const directory = process.env.OPENCODE_PERFORMANCE_TRACE_DIR
|
||||
if (!directory) return
|
||||
|
||||
const selectors = process.env.OPENCODE_PERFORMANCE_SELECTOR_TRACE === "1"
|
||||
const file = await prepareChromeTrace(directory, name, selectors)
|
||||
const session = await page.context().newCDPSession(page)
|
||||
try {
|
||||
await session.send("Tracing.start", {
|
||||
transferMode: "ReturnAsStream",
|
||||
traceConfig: {
|
||||
excludedCategories: categories
|
||||
.filter((category) => category.startsWith("-"))
|
||||
.map((category) => category.slice(1)),
|
||||
includedCategories: [
|
||||
...categories.filter((category) => !category.startsWith("-")),
|
||||
...(selectors
|
||||
? ["disabled-by-default-blink.debug", "disabled-by-default-devtools.timeline.invalidationTracking"]
|
||||
: []),
|
||||
],
|
||||
},
|
||||
})
|
||||
} catch (error) {
|
||||
await Promise.allSettled([session.detach()])
|
||||
throw error
|
||||
}
|
||||
let stopping: Promise<string> | undefined
|
||||
|
||||
return () =>
|
||||
(stopping ??= (async () => {
|
||||
try {
|
||||
const complete = new Promise<{ stream?: string; dataLossOccurred: boolean }>((resolve) =>
|
||||
session.once("Tracing.tracingComplete", resolve),
|
||||
)
|
||||
await session.send("Tracing.end")
|
||||
const result = await complete
|
||||
if (!result.stream) throw new Error(`Chrome trace stream missing: ${file}`)
|
||||
const partial = `${file}.partial`
|
||||
await writeProtocolStream(session, result.stream, partial)
|
||||
if (result.dataLossOccurred) throw new Error(`Chrome trace lost data; partial capture retained: ${partial}`)
|
||||
await rename(partial, file)
|
||||
return file
|
||||
} finally {
|
||||
await Promise.allSettled([session.detach()])
|
||||
}
|
||||
})())
|
||||
}
|
||||
|
||||
export async function prepareChromeTrace(
|
||||
directory: string,
|
||||
name: string,
|
||||
selectors: boolean,
|
||||
nonce = randomUUID().slice(0, 8),
|
||||
) {
|
||||
await mkdir(directory, { recursive: true })
|
||||
const run = process.env.OPENCODE_PERFORMANCE_RUN_ID ?? "manual"
|
||||
const hash = createHash("sha256").update(name).digest("hex").slice(0, 8)
|
||||
return path.join(
|
||||
directory,
|
||||
`${run}-${name.replace(/[^a-zA-Z0-9_-]/g, "-")}-${hash}-${nonce}${selectors ? "-selectors" : ""}.json`,
|
||||
)
|
||||
}
|
||||
|
||||
async function writeProtocolStream(session: CDPSession, handle: string, file: string) {
|
||||
const output = await open(file, "wx")
|
||||
try {
|
||||
while (true) {
|
||||
const chunk = await session.send("IO.read", { handle })
|
||||
await output.write(chunk.base64Encoded ? Buffer.from(chunk.data, "base64") : chunk.data)
|
||||
if (chunk.eof) break
|
||||
}
|
||||
} finally {
|
||||
await Promise.allSettled([output.close(), session.send("IO.close", { handle })])
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
import config from "../../playwright.config"
|
||||
|
||||
const port = Number(process.env.PLAYWRIGHT_PORT ?? 3000)
|
||||
process.env.PLAYWRIGHT_SERVER_PORT = String(port)
|
||||
process.env.OPENCODE_PERFORMANCE_RUN_ID ??= `${new Date().toISOString().replace(/[:.]/g, "-")}-${process.pid}`
|
||||
|
||||
export default {
|
||||
...config,
|
||||
testDir: ".",
|
||||
testIgnore: "unit/**",
|
||||
outputDir: "../test-results/performance",
|
||||
fullyParallel: false,
|
||||
workers: 1,
|
||||
reporter: [["html", { outputFolder: "../playwright-report/performance", open: "never" }], ["line"]],
|
||||
webServer: {
|
||||
...config.webServer,
|
||||
command: `bun run build && bun run serve -- --host 0.0.0.0 --port ${port} --strictPort`,
|
||||
reuseExistingServer: false,
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
import config from "./playwright.config"
|
||||
|
||||
export default {
|
||||
...config,
|
||||
outputDir: "../test-results/performance-uncapped",
|
||||
reporter: [["html", { outputFolder: "../playwright-report/performance-uncapped", open: "never" }], ["line"]],
|
||||
use: {
|
||||
...config.use,
|
||||
launchOptions: {
|
||||
args: ["--disable-frame-rate-limit", "--disable-gpu-vsync"],
|
||||
},
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
import { benchmark, expect } from "../benchmark"
|
||||
import { expectSessionTitle } from "../../utils/waits"
|
||||
import { fixture } from "./session-timeline-stress.fixture"
|
||||
import {
|
||||
collectCachedRepaintTrace,
|
||||
compressCachedRepaintTrace,
|
||||
installCachedRepaintProbe,
|
||||
waitForCachedRepaintWindow,
|
||||
} from "./session-tab-repaint-probe"
|
||||
import { waitForStableTimeline } from "./session-tab-switch-probe"
|
||||
import {
|
||||
installStressSessionTabs,
|
||||
installTimelineSettings,
|
||||
mockStressTimeline,
|
||||
stressSessionHref,
|
||||
} from "./timeline-test-helpers"
|
||||
|
||||
benchmark("samples cached session repaint after the click", async ({ page, report }) => {
|
||||
benchmark.setTimeout(120_000)
|
||||
await mockStressTimeline(page)
|
||||
await installStressSessionTabs(page)
|
||||
await installTimelineSettings(page)
|
||||
await page.goto(stressSessionHref(fixture.targetID))
|
||||
await expectSessionTitle(page, fixture.expected.targetTitle)
|
||||
await waitForStableTimeline(page, fixture.expected.targetMessageIDs.at(-1)!)
|
||||
await page
|
||||
.locator(`[data-slot="titlebar-tabs"] a[href="${stressSessionHref(fixture.sourceID)}"]`)
|
||||
.first()
|
||||
.click()
|
||||
await expectSessionTitle(page, fixture.expected.sourceTitle)
|
||||
await waitForStableTimeline(page, fixture.expected.sourceMessageIDs.at(-1)!)
|
||||
|
||||
await installCachedRepaintProbe(page, {
|
||||
targetHref: stressSessionHref(fixture.targetID),
|
||||
destination: fixture.messages[fixture.targetID].map((message) => message.info.id),
|
||||
source: fixture.messages[fixture.sourceID].map((message) => message.info.id),
|
||||
last: fixture.expected.targetMessageIDs.at(-1)!,
|
||||
windowMs: 1_000,
|
||||
})
|
||||
|
||||
await page
|
||||
.locator(`[data-slot="titlebar-tabs"] a[href="${stressSessionHref(fixture.targetID)}"]`)
|
||||
.first()
|
||||
.click()
|
||||
await Promise.all([expectSessionTitle(page, fixture.expected.targetTitle), waitForCachedRepaintWindow(page, 1_000)])
|
||||
const result = await collectCachedRepaintTrace(page)
|
||||
report(compressCachedRepaintTrace(result))
|
||||
expect(result.samples.length).toBeGreaterThan(0)
|
||||
})
|
||||
@@ -0,0 +1,251 @@
|
||||
import type { Page } from "@playwright/test"
|
||||
|
||||
type CachedRepaintTrace = {
|
||||
timeOriginEpochMs: number
|
||||
startedAtPerformanceMs: number
|
||||
samples: {
|
||||
observedAtMs: number
|
||||
root: number | undefined
|
||||
scrollTop: number
|
||||
scrollHeight: number
|
||||
bottomErrorPx: number | undefined
|
||||
last: boolean
|
||||
rows: { key: string | undefined; node: number; top: number; bottom: number }[]
|
||||
mounted: number
|
||||
center: string | undefined
|
||||
destination: string[]
|
||||
source: string[]
|
||||
}[]
|
||||
mutations: { observedAtMs: number; changed: { type: string; node: number }[] }[]
|
||||
shifts: { occurredAtMs: number; value: number }[]
|
||||
windowMs: number
|
||||
running: boolean
|
||||
stop: () => void
|
||||
}
|
||||
|
||||
export async function installCachedRepaintProbe(
|
||||
page: Page,
|
||||
input: { targetHref: string; destination: string[]; source: string[]; last: string; windowMs: number },
|
||||
) {
|
||||
await page.evaluate(({ targetHref, destination, source, last, windowMs }) => {
|
||||
const destinationIDs = new Set(destination)
|
||||
const sourceIDs = new Set(source)
|
||||
const nodeIDs = new WeakMap<Node, number>()
|
||||
let nextNodeID = 1
|
||||
const id = (node: Node) => {
|
||||
const current = nodeIDs.get(node)
|
||||
if (current) return current
|
||||
nodeIDs.set(node, nextNodeID)
|
||||
return nextNodeID++
|
||||
}
|
||||
const state: CachedRepaintTrace = {
|
||||
timeOriginEpochMs: performance.timeOrigin,
|
||||
startedAtPerformanceMs: 0,
|
||||
samples: [],
|
||||
mutations: [],
|
||||
shifts: [],
|
||||
windowMs,
|
||||
running: false,
|
||||
stop: () => {},
|
||||
}
|
||||
const recordShifts = (entries: PerformanceEntry[]) => {
|
||||
if (!state.running) return
|
||||
state.shifts.push(
|
||||
...entries
|
||||
.map((entry) => {
|
||||
if (
|
||||
entry.startTime < state.startedAtPerformanceMs ||
|
||||
entry.startTime > state.startedAtPerformanceMs + state.windowMs
|
||||
)
|
||||
return
|
||||
return {
|
||||
occurredAtMs: entry.startTime - state.startedAtPerformanceMs,
|
||||
value: (entry as PerformanceEntry & { value: number }).value,
|
||||
}
|
||||
})
|
||||
.filter((entry): entry is { occurredAtMs: number; value: number } => entry !== undefined),
|
||||
)
|
||||
}
|
||||
const shiftObserver = new PerformanceObserver((entries) => recordShifts(entries.getEntries()))
|
||||
shiftObserver.observe({ type: "layout-shift" })
|
||||
const recordMutations = (entries: MutationRecord[]) => {
|
||||
if (!state.running) return
|
||||
const observedAtMs = performance.now() - state.startedAtPerformanceMs
|
||||
if (observedAtMs > state.windowMs) return
|
||||
const changed = entries.flatMap((entry) => [
|
||||
...[...entry.addedNodes].map((node) => ({ type: "add", node: id(node) })),
|
||||
...[...entry.removedNodes].map((node) => ({ type: "remove", node: id(node) })),
|
||||
])
|
||||
if (changed.length) state.mutations.push({ observedAtMs, changed })
|
||||
}
|
||||
const mutationObserver = new MutationObserver(recordMutations)
|
||||
mutationObserver.observe(document.documentElement, { childList: true, subtree: true })
|
||||
state.stop = () => {
|
||||
recordShifts(shiftObserver.takeRecords())
|
||||
recordMutations(mutationObserver.takeRecords())
|
||||
state.running = false
|
||||
shiftObserver.disconnect()
|
||||
mutationObserver.disconnect()
|
||||
}
|
||||
const sample = () => {
|
||||
if (!state.running) return
|
||||
setTimeout(() => {
|
||||
if (!state.running) return
|
||||
const observedAtMs = performance.now() - state.startedAtPerformanceMs
|
||||
if (observedAtMs > state.windowMs) return
|
||||
const root = [...document.querySelectorAll<HTMLElement>(".scroll-view__viewport")].find((element) =>
|
||||
element.querySelector("[data-timeline-row]"),
|
||||
)
|
||||
if (root) {
|
||||
const view = root.getBoundingClientRect()
|
||||
const rows = [...root.querySelectorAll<HTMLElement>("[data-timeline-key]")]
|
||||
.map((element) => ({
|
||||
key: element.dataset.timelineKey,
|
||||
node: id(element),
|
||||
rect: element.getBoundingClientRect(),
|
||||
}))
|
||||
.filter((item) => item.rect.bottom > view.top && item.rect.top < view.bottom)
|
||||
.map((item) => ({
|
||||
key: item.key,
|
||||
node: item.node,
|
||||
top: item.rect.top - view.top,
|
||||
bottom: item.rect.bottom - view.top,
|
||||
}))
|
||||
const messages = [...root.querySelectorAll<HTMLElement>("[data-message-id]")]
|
||||
.filter((element) => {
|
||||
const rect = element.getBoundingClientRect()
|
||||
return rect.bottom > view.top && rect.top < view.bottom
|
||||
})
|
||||
.map((element) => element.dataset.messageId!)
|
||||
const spacer = root.querySelector<HTMLElement>('[data-timeline-row="bottom-spacer"]')?.getBoundingClientRect()
|
||||
state.samples.push({
|
||||
observedAtMs,
|
||||
root: id(root),
|
||||
scrollTop: root.scrollTop,
|
||||
scrollHeight: root.scrollHeight,
|
||||
bottomErrorPx: spacer ? spacer.bottom - view.bottom : undefined,
|
||||
last: messages.includes(last),
|
||||
rows,
|
||||
mounted: root.querySelectorAll("[data-timeline-key]").length,
|
||||
center: document
|
||||
.elementFromPoint(view.left + view.width / 2, view.top + view.height / 2)
|
||||
?.textContent?.slice(0, 80),
|
||||
destination: messages.filter((messageID) => destinationIDs.has(messageID)),
|
||||
source: messages.filter((messageID) => sourceIDs.has(messageID)),
|
||||
})
|
||||
} else {
|
||||
state.samples.push({
|
||||
observedAtMs,
|
||||
root: undefined,
|
||||
scrollTop: 0,
|
||||
scrollHeight: 0,
|
||||
bottomErrorPx: undefined,
|
||||
last: false,
|
||||
rows: [],
|
||||
mounted: 0,
|
||||
center: document.elementFromPoint(innerWidth / 2, innerHeight / 2)?.textContent?.slice(0, 80),
|
||||
destination: [],
|
||||
source: [],
|
||||
})
|
||||
}
|
||||
requestAnimationFrame(sample)
|
||||
}, 0)
|
||||
}
|
||||
document.addEventListener(
|
||||
"click",
|
||||
(event) => {
|
||||
const link = event.target instanceof Element ? event.target.closest("a") : undefined
|
||||
if (link?.getAttribute("href") !== targetHref) return
|
||||
state.startedAtPerformanceMs = performance.now()
|
||||
state.running = true
|
||||
requestAnimationFrame(sample)
|
||||
},
|
||||
{ capture: true, once: true },
|
||||
)
|
||||
;(window as Window & { __cachedFlash?: CachedRepaintTrace }).__cachedFlash = state
|
||||
}, input)
|
||||
}
|
||||
|
||||
export function layoutShiftSample(entry: Pick<PerformanceEntry, "startTime"> & { value: number }, started: number) {
|
||||
if (entry.startTime < started) return
|
||||
return { occurredAtMs: entry.startTime - started, value: entry.value }
|
||||
}
|
||||
|
||||
export async function waitForCachedRepaintWindow(page: Page, durationMs: number) {
|
||||
await page.waitForFunction((durationMs) => {
|
||||
const state = (window as Window & { __cachedFlash?: CachedRepaintTrace }).__cachedFlash
|
||||
return !!state?.running && performance.now() - state.startedAtPerformanceMs >= durationMs
|
||||
}, durationMs)
|
||||
}
|
||||
|
||||
export async function collectCachedRepaintTrace(page: Page) {
|
||||
return page.evaluate(() => {
|
||||
const state = (window as Window & { __cachedFlash?: CachedRepaintTrace }).__cachedFlash!
|
||||
state.stop()
|
||||
return state
|
||||
})
|
||||
}
|
||||
|
||||
export function summarizeCachedRepaintTrace(trace: CachedRepaintTrace) {
|
||||
const roots = trace.samples.map((sample) => sample.root)
|
||||
const bottomErrors = trace.samples.flatMap((sample) =>
|
||||
sample.bottomErrorPx === undefined ? [] : [Math.abs(sample.bottomErrorPx)],
|
||||
)
|
||||
const category = (sample: CachedRepaintTrace["samples"][number]) => {
|
||||
if (sample.source.length) return "source"
|
||||
if (sample.root === undefined || sample.rows.length === 0) return "blank"
|
||||
if (!sample.destination.length) return "unknown"
|
||||
if (sample.last && Math.abs(sample.bottomErrorPx ?? Infinity) <= 1) return "correct"
|
||||
return "wrongDestination"
|
||||
}
|
||||
return {
|
||||
samples: trace.samples.length,
|
||||
durationMs: trace.samples.at(-1)?.observedAtMs ?? 0,
|
||||
firstSampleObservedMs: trace.samples[0]?.observedAtMs,
|
||||
firstSampleCorrect: trace.samples[0] ? category(trace.samples[0]) === "correct" : false,
|
||||
blankSamples: trace.samples.filter((sample) => category(sample) === "blank").length,
|
||||
sourceSamples: trace.samples.filter((sample) => category(sample) === "source").length,
|
||||
wrongDestinationSamples: trace.samples.filter((sample) => category(sample) === "wrongDestination").length,
|
||||
unknownSamples: trace.samples.filter((sample) => category(sample) === "unknown").length,
|
||||
rootChanges: roots.slice(1).filter((root, index) => root !== roots[index]).length,
|
||||
mountedMin: trace.samples.length ? Math.min(...trace.samples.map((sample) => sample.mounted)) : 0,
|
||||
mountedMax: Math.max(...trace.samples.map((sample) => sample.mounted)),
|
||||
maxBottomErrorPx: Math.max(0, ...bottomErrors),
|
||||
mutationBatches: trace.mutations.length,
|
||||
addedNodes: trace.mutations.reduce(
|
||||
(sum, batch) => sum + batch.changed.filter((change) => change.type === "add").length,
|
||||
0,
|
||||
),
|
||||
removedNodes: trace.mutations.reduce(
|
||||
(sum, batch) => sum + batch.changed.filter((change) => change.type === "remove").length,
|
||||
0,
|
||||
),
|
||||
layoutShiftValueSum: trace.shifts.reduce((sum, shift) => sum + shift.value, 0),
|
||||
maxLayoutShiftValue: Math.max(0, ...trace.shifts.map((shift) => shift.value)),
|
||||
}
|
||||
}
|
||||
|
||||
export function compressCachedRepaintTrace(trace: CachedRepaintTrace) {
|
||||
const samples: {
|
||||
observedAtMs: number[]
|
||||
state: Omit<CachedRepaintTrace["samples"][number], "observedAtMs">
|
||||
}[] = []
|
||||
for (const sample of trace.samples) {
|
||||
const { observedAtMs, ...state } = sample
|
||||
const previous = samples.at(-1)
|
||||
if (previous && JSON.stringify(previous.state) === JSON.stringify(state)) {
|
||||
previous.observedAtMs.push(observedAtMs)
|
||||
continue
|
||||
}
|
||||
samples.push({ observedAtMs: [observedAtMs], state })
|
||||
}
|
||||
return {
|
||||
timeOriginEpochMs: trace.timeOriginEpochMs,
|
||||
startedAtPerformanceMs: trace.startedAtPerformanceMs,
|
||||
windowMs: trace.windowMs,
|
||||
summary: summarizeCachedRepaintTrace(trace),
|
||||
samples,
|
||||
mutations: trace.mutations,
|
||||
shifts: trace.shifts,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
import type { Page } from "@playwright/test"
|
||||
import { expectSessionTitle } from "../../utils/waits"
|
||||
import { benchmark, expect, withBenchmarkPage } from "../benchmark"
|
||||
import { fixture } from "./session-timeline-stress.fixture"
|
||||
import { installStressSessionTabs, mockStressTimeline, stressSessionHref } from "./timeline-test-helpers"
|
||||
import { measureSessionSwitch, waitForStableTimeline } from "./session-tab-switch-probe"
|
||||
|
||||
type Result = Awaited<ReturnType<typeof measureSessionSwitch>>
|
||||
|
||||
benchmark("benchmarks cold and hot session tab switching", async ({ browser, report }, testInfo) => {
|
||||
benchmark.setTimeout(180_000)
|
||||
const results = { cold: [] as Result[], hot: [] as Result[] }
|
||||
for (const mode of ["cold", "hot"] as const) {
|
||||
for (let run = 0; run < 5; run++) {
|
||||
results[mode].push(
|
||||
await withBenchmarkPage(browser, `session-tab-switch-${mode}-${run}`, (page) => trial(page, mode), testInfo),
|
||||
)
|
||||
}
|
||||
}
|
||||
report({ results, summary: summarize(results) })
|
||||
})
|
||||
|
||||
async function trial(page: Page, mode: "cold" | "hot") {
|
||||
await mockStressTimeline(page)
|
||||
await installStressSessionTabs(page)
|
||||
if (mode === "hot") {
|
||||
await page.goto(stressSessionHref(fixture.targetID))
|
||||
await expectSessionTitle(page, fixture.expected.targetTitle)
|
||||
await waitForStableTimeline(page, fixture.expected.targetMessageIDs.at(-1)!)
|
||||
await switchSession(page, fixture.sourceID, fixture.expected.sourceTitle)
|
||||
} else {
|
||||
await page.goto(stressSessionHref(fixture.sourceID))
|
||||
await expectSessionTitle(page, fixture.expected.sourceTitle)
|
||||
}
|
||||
await waitForStableTimeline(page, fixture.expected.sourceMessageIDs.at(-1)!)
|
||||
|
||||
const destinationIDs = fixture.messages[fixture.targetID].map((message) => message.info.id)
|
||||
const sourceIDs = fixture.messages[fixture.sourceID].map((message) => message.info.id)
|
||||
const lastID = fixture.expected.targetMessageIDs.at(-1)!
|
||||
const href = stressSessionHref(fixture.targetID)
|
||||
const result = await measureSessionSwitch(page, {
|
||||
destinationIDs,
|
||||
sourceIDs,
|
||||
lastID,
|
||||
href,
|
||||
switch: () => switchSession(page, fixture.targetID, fixture.expected.targetTitle),
|
||||
})
|
||||
return result
|
||||
}
|
||||
|
||||
function summarize(results: Record<"cold" | "hot", Result[]>) {
|
||||
const stats = (values: (number | null)[]) => {
|
||||
const sorted = values.filter((value): value is number => value !== null).sort((a, b) => a - b)
|
||||
return {
|
||||
min: sorted[0] ?? null,
|
||||
median: sorted[Math.floor(sorted.length / 2)] ?? null,
|
||||
max: sorted.at(-1) ?? null,
|
||||
missing: values.length - sorted.length,
|
||||
}
|
||||
}
|
||||
return Object.fromEntries(
|
||||
Object.entries(results).map(([mode, values]) => [
|
||||
mode,
|
||||
{
|
||||
firstDestinationObservedMs: stats(values.map((value) => value.firstDestinationObservedMs)),
|
||||
firstCorrectObservedMs: stats(values.map((value) => value.firstCorrectObservedMs)),
|
||||
stableObservedMs: stats(values.map((value) => value.stableObservedMs)),
|
||||
},
|
||||
]),
|
||||
)
|
||||
}
|
||||
|
||||
async function switchSession(page: Page, sessionID: string, title: string) {
|
||||
const href = stressSessionHref(sessionID)
|
||||
const tab = page.locator(`[data-slot="titlebar-tabs"] a[href="${href}"]`).first()
|
||||
await expect(tab).toBeVisible()
|
||||
await tab.click()
|
||||
await expectSessionTitle(page, title)
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
export type SessionSwitchSample = {
|
||||
observedAtMs: number
|
||||
destination: string[]
|
||||
source: string[]
|
||||
hasVisibleRows: boolean
|
||||
last: boolean
|
||||
bottomErrorPx?: number
|
||||
}
|
||||
|
||||
export function classifySessionSwitch(samples: SessionSwitchSample[]) {
|
||||
const firstDestination = samples.findIndex((sample) => sample.destination.length > 0)
|
||||
const firstCorrect = samples.findIndex(isCorrectDestination)
|
||||
const stable = samples.findIndex((_, index) => isStableSessionSwitch(samples.slice(index, index + 3)))
|
||||
return {
|
||||
firstDestinationObservedMs: samples[firstDestination]?.observedAtMs ?? null,
|
||||
firstCorrectObservedMs: samples[firstCorrect]?.observedAtMs ?? null,
|
||||
stableObservedMs: samples[stable + 2]?.observedAtMs ?? null,
|
||||
wrongDestinationSamples: samples
|
||||
.slice(firstDestination)
|
||||
.filter((sample) => sample.destination.length > 0 && !sample.last).length,
|
||||
blankSamples: samples.filter((sample) => !sample.hasVisibleRows).length,
|
||||
unknownSamples: samples.filter(
|
||||
(sample) => sample.hasVisibleRows && sample.destination.length === 0 && sample.source.length === 0,
|
||||
).length,
|
||||
sourceSamples: samples.filter((sample) => sample.source.length > 0).length,
|
||||
}
|
||||
}
|
||||
|
||||
export function isCorrectDestination(sample: SessionSwitchSample) {
|
||||
return (
|
||||
sample.destination.length > 0 &&
|
||||
sample.source.length === 0 &&
|
||||
sample.last &&
|
||||
Math.abs(sample.bottomErrorPx ?? Infinity) <= 1
|
||||
)
|
||||
}
|
||||
|
||||
export function isStableSessionSwitch(samples: SessionSwitchSample[]) {
|
||||
return samples.length === 3 && samples.every(isCorrectDestination)
|
||||
}
|
||||
|
||||
export function isStableDestination(samples: Pick<SessionSwitchSample, "last" | "bottomErrorPx">[]) {
|
||||
return (
|
||||
samples.length === 3 && samples.every((sample) => sample.last && Math.abs(sample.bottomErrorPx ?? Infinity) <= 1)
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,152 @@
|
||||
import { expect, type Page } from "@playwright/test"
|
||||
import { classifySessionSwitch, isStableDestination, type SessionSwitchSample } from "./session-tab-switch-metrics"
|
||||
|
||||
type SessionSwitchProbe = {
|
||||
samples: SessionSwitchSample[]
|
||||
stop: () => void
|
||||
}
|
||||
|
||||
async function installSessionSwitchProbe(
|
||||
page: Page,
|
||||
input: { destinationIDs: string[]; sourceIDs: string[]; lastID: string; href: string },
|
||||
) {
|
||||
await page.evaluate(({ destinationIDs, sourceIDs, lastID, href }) => {
|
||||
const destination = new Set(destinationIDs)
|
||||
const source = new Set(sourceIDs)
|
||||
const samples: SessionSwitchSample[] = []
|
||||
let started: number | undefined
|
||||
let running = true
|
||||
const sample = () => {
|
||||
if (!running || started === undefined) return
|
||||
setTimeout(() => {
|
||||
if (!running || started === undefined) return
|
||||
const observedAtMs = performance.now() - started
|
||||
const root = [...document.querySelectorAll<HTMLElement>(".scroll-view__viewport")].find((element) =>
|
||||
element.querySelector("[data-timeline-row]"),
|
||||
)
|
||||
if (root) {
|
||||
const view = root.getBoundingClientRect()
|
||||
const visible = [...root.querySelectorAll<HTMLElement>("[data-message-id]")]
|
||||
.filter((element) => {
|
||||
const rect = element.getBoundingClientRect()
|
||||
return rect.bottom > view.top && rect.top < view.bottom
|
||||
})
|
||||
.map((element) => element.dataset.messageId!)
|
||||
const hasVisibleRows = [...root.querySelectorAll<HTMLElement>("[data-timeline-key]")].some((element) => {
|
||||
const rect = element.getBoundingClientRect()
|
||||
return rect.bottom > view.top && rect.top < view.bottom
|
||||
})
|
||||
const spacer = root.querySelector<HTMLElement>('[data-timeline-row="bottom-spacer"]')?.getBoundingClientRect()
|
||||
samples.push({
|
||||
observedAtMs,
|
||||
destination: visible.filter((id) => destination.has(id)),
|
||||
source: visible.filter((id) => source.has(id)),
|
||||
hasVisibleRows,
|
||||
last: visible.includes(lastID),
|
||||
bottomErrorPx: spacer ? spacer.bottom - view.bottom : undefined,
|
||||
})
|
||||
} else {
|
||||
samples.push({ observedAtMs, destination: [], source: [], hasVisibleRows: false, last: false })
|
||||
}
|
||||
requestAnimationFrame(sample)
|
||||
}, 0)
|
||||
}
|
||||
document.addEventListener(
|
||||
"click",
|
||||
(event) => {
|
||||
const link = event.target instanceof Element ? event.target.closest("a") : undefined
|
||||
if (link?.getAttribute("href") !== href) return
|
||||
started = performance.now()
|
||||
requestAnimationFrame(sample)
|
||||
},
|
||||
{ capture: true, once: true },
|
||||
)
|
||||
;(window as Window & { __sessionSwitchProbe?: SessionSwitchProbe }).__sessionSwitchProbe = {
|
||||
samples,
|
||||
stop: () => {
|
||||
running = false
|
||||
},
|
||||
}
|
||||
}, input)
|
||||
}
|
||||
|
||||
async function waitForStableSessionSwitch(page: Page) {
|
||||
await page.waitForFunction(() => {
|
||||
const samples = (window as Window & { __sessionSwitchProbe?: SessionSwitchProbe }).__sessionSwitchProbe?.samples
|
||||
if (!samples) return false
|
||||
return samples.some((_, index) => {
|
||||
const stable = samples.slice(index, index + 3)
|
||||
return (
|
||||
stable.length === 3 &&
|
||||
stable.every(
|
||||
(sample) =>
|
||||
sample.destination.length > 0 &&
|
||||
sample.source.length === 0 &&
|
||||
sample.last &&
|
||||
Math.abs(sample.bottomErrorPx ?? Infinity) <= 1,
|
||||
)
|
||||
)
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
async function collectSessionSwitchResult(page: Page) {
|
||||
const samples = await page.evaluate(() => {
|
||||
const probe = (window as Window & { __sessionSwitchProbe?: SessionSwitchProbe }).__sessionSwitchProbe!
|
||||
probe.stop()
|
||||
return probe.samples
|
||||
})
|
||||
return classifySessionSwitch(samples)
|
||||
}
|
||||
|
||||
export async function measureSessionSwitch(
|
||||
page: Page,
|
||||
input: { destinationIDs: string[]; sourceIDs: string[]; lastID: string; href: string; switch: () => Promise<void> },
|
||||
) {
|
||||
const { switch: run, ...probe } = input
|
||||
await installSessionSwitchProbe(page, probe)
|
||||
await run()
|
||||
await waitForStableSessionSwitch(page)
|
||||
return collectSessionSwitchResult(page)
|
||||
}
|
||||
|
||||
export async function waitForStableTimeline(page: Page, lastID: string) {
|
||||
const samples: Pick<SessionSwitchSample, "last" | "bottomErrorPx">[] = []
|
||||
await expect
|
||||
.poll(
|
||||
async () => {
|
||||
samples.push(
|
||||
await page.evaluate(
|
||||
(lastID) =>
|
||||
new Promise<Pick<SessionSwitchSample, "last" | "bottomErrorPx">>((resolve) => {
|
||||
requestAnimationFrame(() =>
|
||||
setTimeout(() => {
|
||||
const root = [...document.querySelectorAll<HTMLElement>(".scroll-view__viewport")].find((element) =>
|
||||
element.querySelector("[data-timeline-row]"),
|
||||
)
|
||||
if (!root) {
|
||||
resolve({ last: false })
|
||||
return
|
||||
}
|
||||
const view = root.getBoundingClientRect()
|
||||
const last = [...root.querySelectorAll<HTMLElement>("[data-message-id]")].some((element) => {
|
||||
if (element.dataset.messageId !== lastID) return false
|
||||
const rect = element.getBoundingClientRect()
|
||||
return rect.bottom > view.top && rect.top < view.bottom
|
||||
})
|
||||
const spacer = root
|
||||
.querySelector<HTMLElement>('[data-timeline-row="bottom-spacer"]')
|
||||
?.getBoundingClientRect()
|
||||
resolve({ last, bottomErrorPx: spacer ? spacer.bottom - view.bottom : undefined })
|
||||
}, 0),
|
||||
)
|
||||
}),
|
||||
lastID,
|
||||
),
|
||||
)
|
||||
return isStableDestination(samples.slice(-3))
|
||||
},
|
||||
{ timeout: 30_000, intervals: [0] },
|
||||
)
|
||||
.toBe(true)
|
||||
}
|
||||
@@ -0,0 +1,488 @@
|
||||
import { base64Encode } from "@opencode-ai/core/util/encode"
|
||||
import type { Page } from "@playwright/test"
|
||||
import { mockOpenCodeServer } from "../../utils/mock-server"
|
||||
import { expectAppVisible, expectSessionTitle } from "../../utils/waits"
|
||||
import { expect } from "../benchmark"
|
||||
|
||||
const directory = "C:/OpenCode/TimelineStateRegression"
|
||||
const projectID = "proj_timeline_state_regression"
|
||||
const sessionID = "ses_timeline_state_regression"
|
||||
const userMessageID = "msg_user_regression"
|
||||
const assistantMessageID = "msg_assistant_regression"
|
||||
const editPartID = "prt_0001_edit"
|
||||
export const textPartID = "prt_9999_text"
|
||||
const title = "Timeline collapse state regression"
|
||||
const model = { providerID: "opencode", modelID: "claude-opus-4-6", variant: "max" }
|
||||
|
||||
type EventPayload = {
|
||||
directory: string
|
||||
payload: Record<string, unknown>
|
||||
}
|
||||
|
||||
const userMessage = {
|
||||
info: {
|
||||
id: userMessageID,
|
||||
sessionID,
|
||||
role: "user",
|
||||
time: { created: 1700000000000 },
|
||||
summary: { diffs: [] },
|
||||
agent: "build",
|
||||
model,
|
||||
},
|
||||
parts: [
|
||||
{
|
||||
id: "prt_user_text",
|
||||
sessionID,
|
||||
messageID: userMessageID,
|
||||
type: "text",
|
||||
text: "Please edit the file.",
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
const editPart = {
|
||||
id: editPartID,
|
||||
sessionID,
|
||||
messageID: assistantMessageID,
|
||||
type: "tool",
|
||||
callID: "call_edit_regression",
|
||||
tool: "edit",
|
||||
state: {
|
||||
status: "completed",
|
||||
input: { filePath: "src/regression.ts" },
|
||||
output: "Edited src/regression.ts",
|
||||
title: "src/regression.ts",
|
||||
metadata: {
|
||||
filediff: {
|
||||
file: "src/regression.ts",
|
||||
additions: 1,
|
||||
deletions: 1,
|
||||
before: "export const value = 'before'\n",
|
||||
after: "export const value = 'after'\n",
|
||||
},
|
||||
diff: "diff --git a/src/regression.ts b/src/regression.ts\n-export const value = 'before'\n+export const value = 'after'\n",
|
||||
},
|
||||
time: { start: 1700000001000, end: 1700000002000 },
|
||||
},
|
||||
}
|
||||
|
||||
const streamedTextPart = {
|
||||
id: textPartID,
|
||||
sessionID,
|
||||
messageID: assistantMessageID,
|
||||
type: "text",
|
||||
text: "Streaming added a later assistant text part.",
|
||||
}
|
||||
|
||||
const assistantMessage = {
|
||||
info: {
|
||||
id: assistantMessageID,
|
||||
sessionID,
|
||||
role: "assistant",
|
||||
time: { created: 1700000001000 },
|
||||
parentID: userMessageID,
|
||||
modelID: model.modelID,
|
||||
providerID: model.providerID,
|
||||
mode: "build",
|
||||
agent: "build",
|
||||
path: { cwd: directory, root: directory },
|
||||
cost: 0.01,
|
||||
tokens: { input: 100, output: 200, reasoning: 0, cache: { read: 0, write: 0 } },
|
||||
variant: "max",
|
||||
},
|
||||
parts: [editPart],
|
||||
}
|
||||
|
||||
export async function setupTimelineBenchmark(page: Page, options: { historyTurns: number; eventBatch: number }) {
|
||||
const events: EventPayload[] = []
|
||||
let eventBatch = options.eventBatch
|
||||
await mockOpenCodeServer(page, {
|
||||
directory,
|
||||
project: project(),
|
||||
provider: provider(),
|
||||
sessions: [session()],
|
||||
pageMessages: () => ({
|
||||
items: [
|
||||
...Array.from({ length: options.historyTurns }, (_, index) => performanceTurn(index)).flat(),
|
||||
userMessage,
|
||||
assistantMessage,
|
||||
],
|
||||
}),
|
||||
events: () => events.splice(0, eventBatch),
|
||||
eventRetry: 16,
|
||||
})
|
||||
await page.addInitScript(() => {
|
||||
localStorage.setItem(
|
||||
"settings.v3",
|
||||
JSON.stringify({
|
||||
general: {
|
||||
editToolPartsExpanded: true,
|
||||
shellToolPartsExpanded: true,
|
||||
showReasoningSummaries: true,
|
||||
showSessionProgressBar: true,
|
||||
},
|
||||
}),
|
||||
)
|
||||
})
|
||||
await page.setViewportSize({ width: 1366, height: 768 })
|
||||
const scroller = page.locator(".scroll-view__viewport", { has: page.locator("[data-timeline-row]") })
|
||||
const text = page.locator(`[data-timeline-part-id="${textPartID}"]`).first()
|
||||
await page.goto(`/${base64Encode(directory)}/session/${sessionID}`)
|
||||
await expectSessionTitle(page, title)
|
||||
await expectAppVisible(scroller)
|
||||
return {
|
||||
scroller,
|
||||
text,
|
||||
transport: {
|
||||
enqueue(payload: EventPayload | EventPayload[]) {
|
||||
events.push(...(Array.isArray(payload) ? payload : [payload]))
|
||||
},
|
||||
pendingCount() {
|
||||
return events.length
|
||||
},
|
||||
releaseAll() {
|
||||
eventBatch = events.length
|
||||
},
|
||||
},
|
||||
async scrollToBottom() {
|
||||
await scroller.evaluate((element) => {
|
||||
element.scrollTop = element.scrollHeight
|
||||
})
|
||||
},
|
||||
async waitForStableGeometry() {
|
||||
await expect
|
||||
.poll(() => scroller.evaluate((element) => element.scrollHeight - element.clientHeight - element.scrollTop))
|
||||
.toBeLessThanOrEqual(1)
|
||||
await page.waitForFunction((partID) => {
|
||||
const root = [...document.querySelectorAll<HTMLElement>(".scroll-view__viewport")].find((element) =>
|
||||
element.querySelector(`[data-timeline-part-id="${partID}"]`),
|
||||
)
|
||||
if (!root) return false
|
||||
return new Promise<boolean>((resolve) => {
|
||||
const height = root.scrollHeight
|
||||
requestAnimationFrame(() =>
|
||||
requestAnimationFrame(() =>
|
||||
resolve(root.scrollHeight === height && root.scrollHeight - root.clientHeight - root.scrollTop <= 1),
|
||||
),
|
||||
)
|
||||
})
|
||||
}, textPartID)
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
export function buildInitialStreamEvent(deltaCount: number): EventPayload {
|
||||
return {
|
||||
directory,
|
||||
payload: {
|
||||
type: "message.part.updated",
|
||||
properties: {
|
||||
part: {
|
||||
...streamedTextPart,
|
||||
text: `Streaming${streamChunk(0, deltaCount + 1)}\n\n\`\`\`ts\nconst initial = true\n\`\`\``,
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
export function buildStreamDeltaEvents(deltaCount: number): EventPayload[] {
|
||||
return Array.from({ length: deltaCount }, (_, index) => ({
|
||||
directory,
|
||||
payload: {
|
||||
type: "message.part.delta",
|
||||
properties: {
|
||||
messageID: assistantMessageID,
|
||||
partID: textPartID,
|
||||
field: "text",
|
||||
delta: streamChunk(index + 1, deltaCount + 1),
|
||||
},
|
||||
},
|
||||
}))
|
||||
}
|
||||
|
||||
function performanceTurn(index: number) {
|
||||
const suffix = String(index).padStart(4, "0")
|
||||
const userID = `msg_0000_${suffix}_a_user`
|
||||
const assistantID = `msg_0000_${suffix}_b_assistant`
|
||||
const before = historicalSource(index, false)
|
||||
const after = historicalSource(index, true)
|
||||
const parts = [
|
||||
...(index % 5 === 0
|
||||
? [
|
||||
{
|
||||
id: `prt_0000_${suffix}_reasoning`,
|
||||
sessionID,
|
||||
messageID: assistantID,
|
||||
type: "reasoning",
|
||||
text: `Reviewing the existing implementation. ${"constraint analysis ".repeat(20)}`,
|
||||
time: { start: 1690000001000 + index * 2_000, end: 1690000001200 + index * 2_000 },
|
||||
},
|
||||
]
|
||||
: []),
|
||||
{
|
||||
id: `prt_0000_${suffix}_assistant`,
|
||||
sessionID,
|
||||
messageID: assistantID,
|
||||
type: "text",
|
||||
text: historicalMarkdown(index),
|
||||
},
|
||||
...(index % 8 === 0
|
||||
? [
|
||||
{
|
||||
id: `prt_0000_${suffix}_edit`,
|
||||
sessionID,
|
||||
messageID: assistantID,
|
||||
type: "tool",
|
||||
callID: `call_0000_${suffix}_edit`,
|
||||
tool: "edit",
|
||||
state: {
|
||||
status: "completed",
|
||||
input: { filePath: `src/history-${index}.ts` },
|
||||
output: `Edited src/history-${index}.ts`,
|
||||
title: `src/history-${index}.ts`,
|
||||
metadata: {
|
||||
filediff: { file: `src/history-${index}.ts`, additions: 48, deletions: 48, before, after },
|
||||
},
|
||||
time: { start: 1690000001200 + index * 2_000, end: 1690000001400 + index * 2_000 },
|
||||
},
|
||||
},
|
||||
]
|
||||
: []),
|
||||
...(index % 12 === 0
|
||||
? [
|
||||
{
|
||||
id: `prt_0000_${suffix}_write`,
|
||||
sessionID,
|
||||
messageID: assistantID,
|
||||
type: "tool",
|
||||
callID: `call_0000_${suffix}_write`,
|
||||
tool: "write",
|
||||
state: {
|
||||
status: "completed",
|
||||
input: { filePath: `src/generated-${index}.tsx`, content: after },
|
||||
output: `Wrote src/generated-${index}.tsx`,
|
||||
title: `src/generated-${index}.tsx`,
|
||||
metadata: {
|
||||
filediff: { file: `src/generated-${index}.tsx`, additions: 32, deletions: 0, before: "", after },
|
||||
},
|
||||
time: { start: 1690000001400 + index * 2_000, end: 1690000001500 + index * 2_000 },
|
||||
},
|
||||
},
|
||||
]
|
||||
: []),
|
||||
...(index % 16 === 0
|
||||
? [
|
||||
{
|
||||
id: `prt_0000_${suffix}_patch`,
|
||||
sessionID,
|
||||
messageID: assistantID,
|
||||
type: "tool",
|
||||
callID: `call_0000_${suffix}_patch`,
|
||||
tool: "apply_patch",
|
||||
state: {
|
||||
status: "completed",
|
||||
input: { patchText: realisticPatch(index) },
|
||||
output: "Success. Updated src/components/SessionCard.tsx",
|
||||
title: "src/components/SessionCard.tsx",
|
||||
metadata: {
|
||||
files: [
|
||||
{
|
||||
filePath: "src/components/SessionCard.tsx",
|
||||
relativePath: "src/components/SessionCard.tsx",
|
||||
type: "update",
|
||||
additions: 8,
|
||||
deletions: 3,
|
||||
patch: realisticPatch(index),
|
||||
before,
|
||||
after,
|
||||
},
|
||||
],
|
||||
},
|
||||
time: { start: 1690000001500 + index * 2_000, end: 1690000001700 + index * 2_000 },
|
||||
},
|
||||
},
|
||||
]
|
||||
: []),
|
||||
]
|
||||
return [
|
||||
{
|
||||
info: {
|
||||
id: userID,
|
||||
sessionID,
|
||||
role: "user",
|
||||
time: { created: 1690000000000 + index * 2_000 },
|
||||
summary: { diffs: [] },
|
||||
agent: "build",
|
||||
model,
|
||||
},
|
||||
parts: [
|
||||
{
|
||||
id: `prt_0000_${suffix}_user`,
|
||||
sessionID,
|
||||
messageID: userID,
|
||||
type: "text",
|
||||
text: `Historical prompt ${index}`,
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
info: {
|
||||
id: assistantID,
|
||||
sessionID,
|
||||
role: "assistant",
|
||||
time: { created: 1690000001000 + index * 2_000, completed: 1690000001500 + index * 2_000 },
|
||||
parentID: userID,
|
||||
modelID: model.modelID,
|
||||
providerID: model.providerID,
|
||||
mode: "build",
|
||||
agent: "build",
|
||||
path: { cwd: directory, root: directory },
|
||||
cost: 0.01,
|
||||
tokens: { input: 100, output: 200, reasoning: 0, cache: { read: 0, write: 0 } },
|
||||
variant: "max",
|
||||
finish: "stop",
|
||||
},
|
||||
parts,
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
function historicalMarkdown(index: number) {
|
||||
const code = `import { For, Show, createSignal } from "solid-js"
|
||||
|
||||
type SessionRow = { id: string; title: string; active: boolean }
|
||||
|
||||
export function SessionList(props: { rows: SessionRow[] }) {
|
||||
const [selected, setSelected] = createSignal<string>()
|
||||
return (
|
||||
<section aria-label="Sessions">
|
||||
<For each={props.rows}>{(row) => (
|
||||
<button classList={{ active: row.active }} onClick={() => setSelected(row.id)}>
|
||||
<Show when={selected() === row.id} fallback={row.title}>{row.title.toUpperCase()}</Show>
|
||||
</button>
|
||||
)}</For>
|
||||
</section>
|
||||
)
|
||||
}`
|
||||
return `## Session renderer review ${index}
|
||||
|
||||
The active session keeps **semantic row identity** while reconciling measured content. See [Solid documentation](https://docs.solidjs.com/) and the inline \`measureElement(node)\` call.
|
||||
|
||||
| Concern | Current behavior | Verification |
|
||||
| --- | --- | --- |
|
||||
| streaming | appends Markdown blocks | painted frames |
|
||||
| geometry | anchors visible rows | DOM coordinates |
|
||||
| tools | preserves expanded state | keyed remount probe |
|
||||
|
||||
> Long sessions combine Markdown, syntax highlighting, tool output, and asynchronously rendered diffs.
|
||||
|
||||
${index % 4 === 0 ? `\`\`\`tsx\n${code}\n\`\`\`\n\n\`\`\`bash\nbun typecheck\nbun test --preload ./happydom.ts ./src/pages/session\ngit diff --check\n\`\`\`` : "- preserve the viewport anchor\n- avoid replacing stable Markdown nodes\n- process provider deltas without blocking input"}`
|
||||
}
|
||||
|
||||
function historicalSource(index: number, updated: boolean) {
|
||||
const method = updated ? "toLocaleUpperCase(props.locale)" : "toUpperCase()"
|
||||
const limit = updated ? 24 : 20
|
||||
return `import { createMemo, For } from "solid-js"
|
||||
|
||||
type Message = {
|
||||
id: string
|
||||
role: "user" | "assistant"
|
||||
text: string
|
||||
tokens: { input: number; output: number }
|
||||
}
|
||||
|
||||
export function MessageSummary(props: { messages: Message[]; locale: string }) {
|
||||
const visible = createMemo(() => props.messages.filter((message) => message.text.trim()).slice(-${limit}))
|
||||
const total = createMemo(() => visible().reduce((sum, message) => sum + message.tokens.output, 0))
|
||||
return (
|
||||
<article data-session-index="${index}">
|
||||
<header>{total().toLocaleString(props.locale)} output tokens</header>
|
||||
<For each={visible()}>{(message) => <p data-role={message.role}>{message.text.${method}}</p>}</For>
|
||||
</article>
|
||||
)
|
||||
}
|
||||
`
|
||||
}
|
||||
|
||||
function realisticPatch(index: number) {
|
||||
return `*** Begin Patch
|
||||
*** Update File: src/components/SessionCard.tsx
|
||||
@@
|
||||
-const title = props.session.title.toUpperCase()
|
||||
-const messages = props.messages.slice(-20)
|
||||
+const title = props.session.title.toLocaleUpperCase(props.locale)
|
||||
+const messages = props.messages.filter((message) => message.text.trim()).slice(-24)
|
||||
+const outputTokens = messages.reduce((sum, message) => sum + message.tokens.output, 0)
|
||||
@@
|
||||
- <h2>{title}</h2>
|
||||
+ <h2 data-session-index="${index}">{title}</h2>
|
||||
+ <span>{outputTokens.toLocaleString(props.locale)} output tokens</span>
|
||||
*** End Patch`
|
||||
}
|
||||
|
||||
export function streamChunk(index: number, count: number) {
|
||||
if (index === 0) return `\n\n## Implementation plan\n\nStreaming **bold analysis`
|
||||
if (index === count - 1)
|
||||
return `\n\`\`\`\n\n## Verification\n\n- **Typecheck:** passed\n- **Timeline geometry:** stable\n- **Streaming output:** benchmark-complete <!-- stream-${index} -->`
|
||||
|
||||
const section = Math.floor(index / 18) + 1
|
||||
const fragments = [
|
||||
` continues across three`,
|
||||
` or four word`,
|
||||
` provider deltas and`,
|
||||
` closes in this fragment**. <!-- stream-${index} -->\n\n`,
|
||||
`| Concern | State`,
|
||||
` | Verification |\n|`,
|
||||
` --- | ---`,
|
||||
` | --- |\n|`,
|
||||
` markdown | incremental |`,
|
||||
` painted frames | <!-- stream-${index} -->\n\n`,
|
||||
`\`\`\`tsx\nconst row: SessionRow`,
|
||||
` = rows[index] ??`,
|
||||
` fallback\nconst title =`,
|
||||
` row.title.toLocaleUpperCase(locale)\n`,
|
||||
`const selected = createMemo(()`,
|
||||
` => row.id ===`,
|
||||
` activeID()) // stream-${index}\n`,
|
||||
`// stream-${index}\n\`\`\`\n\n### Iteration ${section}\n\nStreaming **bold analysis`,
|
||||
]
|
||||
return fragments[(index - 1) % fragments.length]!
|
||||
}
|
||||
|
||||
function project() {
|
||||
return {
|
||||
id: projectID,
|
||||
worktree: directory,
|
||||
vcs: "git",
|
||||
name: "timeline-state-regression",
|
||||
time: { created: 1700000000000, updated: 1700000000000 },
|
||||
sandboxes: [],
|
||||
}
|
||||
}
|
||||
|
||||
function session() {
|
||||
return {
|
||||
id: sessionID,
|
||||
slug: "timeline-state-regression",
|
||||
projectID,
|
||||
directory,
|
||||
title,
|
||||
version: "dev",
|
||||
time: { created: 1700000000000, updated: 1700000000000 },
|
||||
}
|
||||
}
|
||||
|
||||
function provider() {
|
||||
return {
|
||||
all: [
|
||||
{
|
||||
id: "opencode",
|
||||
name: "OpenCode",
|
||||
models: { "claude-opus-4-6": { id: "claude-opus-4-6", name: "Claude Opus 4.6", limit: { context: 200_000 } } },
|
||||
},
|
||||
],
|
||||
connected: ["opencode"],
|
||||
default: { providerID: "opencode", modelID: "claude-opus-4-6" },
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
import { benchmark, benchmarkDiagnostics, expect } from "../benchmark"
|
||||
import {
|
||||
buildInitialStreamEvent,
|
||||
buildStreamDeltaEvents,
|
||||
setupTimelineBenchmark,
|
||||
textPartID,
|
||||
} from "./session-timeline-benchmark.fixture"
|
||||
import { startTimelineProfile } from "./session-timeline-profile"
|
||||
import {
|
||||
collectTimelineStreamMetrics,
|
||||
installTimelineStreamProbe,
|
||||
startTimelineStreamProbe,
|
||||
} from "./session-timeline-stream-probe"
|
||||
|
||||
benchmark.describe("performance: session timeline streaming", () => {
|
||||
benchmark("streams assistant text without remounting or oscillating", async ({ page, report }) => {
|
||||
benchmark.setTimeout(480_000)
|
||||
const cpuThrottle = Number(process.env.TIMELINE_CPU_THROTTLE ?? 30)
|
||||
const deltaCount = Number(process.env.TIMELINE_DELTA_COUNT ?? 160)
|
||||
const historyTurns = Number(process.env.TIMELINE_HISTORY_TURNS ?? 320)
|
||||
const eventBatch = Number(process.env.TIMELINE_EVENT_BATCH ?? 1)
|
||||
const minimal = process.env.TIMELINE_MINIMAL === "1"
|
||||
const profileCPU = process.env.TIMELINE_CPU_PROFILE === "1"
|
||||
const profileVisual = !minimal && profileCPU && process.env.TIMELINE_VISUAL_PROFILE !== "0"
|
||||
const fixture = await setupTimelineBenchmark(page, {
|
||||
historyTurns,
|
||||
eventBatch,
|
||||
})
|
||||
|
||||
fixture.transport.enqueue(buildInitialStreamEvent(deltaCount))
|
||||
const contentStart = performance.now()
|
||||
await expect(fixture.text).toBeVisible()
|
||||
await expect(fixture.text).toContainText("Implementation plan")
|
||||
const initialContentObservedMs = performance.now() - contentStart
|
||||
await fixture.scrollToBottom()
|
||||
await fixture.waitForStableGeometry()
|
||||
|
||||
const profile = await startTimelineProfile(page, { cpuThrottle, profileCPU })
|
||||
await installTimelineStreamProbe(page, { textPartID, finalIndex: deltaCount, profileVisual, minimal })
|
||||
const deltas = buildStreamDeltaEvents(deltaCount)
|
||||
await startTimelineStreamProbe(page)
|
||||
fixture.transport.enqueue(deltas)
|
||||
|
||||
await page.waitForFunction(
|
||||
(finalIndex) =>
|
||||
(
|
||||
window as Window & {
|
||||
__timelineStreamBenchmark?: { applied: { index: number }[] }
|
||||
}
|
||||
).__timelineStreamBenchmark?.applied.some((value) => value.index === finalIndex),
|
||||
deltaCount,
|
||||
{ timeout: 420_000 },
|
||||
)
|
||||
await expect(fixture.text).toContainText("benchmark-complete")
|
||||
await expect(fixture.text).toContainText("Streaming")
|
||||
await fixture.waitForStableGeometry()
|
||||
const metrics = await collectTimelineStreamMetrics(page, {
|
||||
textPartID,
|
||||
finalIndex: deltaCount,
|
||||
navigations: benchmarkDiagnostics(page).navigations,
|
||||
})
|
||||
const delivered = deltas.length - fixture.transport.pendingCount()
|
||||
await profile.stop()
|
||||
|
||||
report(
|
||||
{
|
||||
endToEndInitialContentObservedMs: initialContentObservedMs,
|
||||
...metrics,
|
||||
deliveredDeltas: delivered,
|
||||
pendingDeltas: fixture.transport.pendingCount(),
|
||||
},
|
||||
{
|
||||
cpuThrottle,
|
||||
profileCPU,
|
||||
profileVisual,
|
||||
minimal,
|
||||
queuedDeltas: deltas.length,
|
||||
historyTurns,
|
||||
eventBatch,
|
||||
},
|
||||
)
|
||||
|
||||
await profile.reset()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,40 @@
|
||||
import type { CDPSession, Page } from "@playwright/test"
|
||||
|
||||
export async function startTimelineProfile(page: Page, options: { cpuThrottle: number; profileCPU: boolean }) {
|
||||
const cdp = await page.context().newCDPSession(page)
|
||||
if (options.cpuThrottle > 1) await cdp.send("Emulation.setCPUThrottlingRate", { rate: options.cpuThrottle })
|
||||
if (options.profileCPU) {
|
||||
await cdp.send("Profiler.enable")
|
||||
await cdp.send("Profiler.setSamplingInterval", { interval: 100 })
|
||||
await cdp.send("Profiler.start")
|
||||
}
|
||||
return {
|
||||
async stop() {
|
||||
if (!options.profileCPU) return
|
||||
const result = await cdp.send("Profiler.stop")
|
||||
const self = new Map<number, number>()
|
||||
result.profile.samples?.forEach((id, index) => {
|
||||
const duration = (result.profile.timeDeltas?.[index] ?? 0) / 1_000
|
||||
self.set(id, (self.get(id) ?? 0) + duration)
|
||||
})
|
||||
console.log(
|
||||
"timeline cpu profile",
|
||||
JSON.stringify(
|
||||
result.profile.nodes
|
||||
.map((node) => ({
|
||||
function: node.callFrame.functionName || "(anonymous)",
|
||||
url: node.callFrame.url,
|
||||
line: node.callFrame.lineNumber + 1,
|
||||
selfMs: self.get(node.id) ?? 0,
|
||||
}))
|
||||
.filter((node) => node.selfMs > 1)
|
||||
.sort((a, b) => b.selfMs - a.selfMs)
|
||||
.slice(0, 40),
|
||||
),
|
||||
)
|
||||
},
|
||||
async reset() {
|
||||
if (options.cpuThrottle > 1) await cdp.send("Emulation.setCPUThrottlingRate", { rate: 1 })
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,547 @@
|
||||
import type { Page } from "@playwright/test"
|
||||
|
||||
const STREAM_MARKER_PATTERN = "stream-(\\d+)"
|
||||
const STREAM_FRAGMENT_COUNT = 18
|
||||
|
||||
type TimelineProbeState = {
|
||||
started: number
|
||||
ended: number
|
||||
profileVisual: boolean
|
||||
minimal: boolean
|
||||
frames: number[]
|
||||
frameAt: number[]
|
||||
applied: { at: number; index: number }[]
|
||||
geometry: {
|
||||
scrollTop: number
|
||||
scrollHeight: number
|
||||
clientHeight: number
|
||||
distance: number
|
||||
virtualHeight: number
|
||||
headerHeight: number
|
||||
}[]
|
||||
blanks: number
|
||||
longTasks: number[]
|
||||
layoutShifts: number[]
|
||||
visibleMounts: number
|
||||
visibleUnmounts: number
|
||||
visibleRows: Set<Element>
|
||||
visibleSubtreeMounts: string[]
|
||||
visibleSubtreeUnmounts: string[]
|
||||
visibleSubtreeReplacements: number
|
||||
visibleSubtreeDropouts: string[]
|
||||
visibleSubtrees: Map<string, Element>
|
||||
subtreeKeys: WeakMap<Element, string>
|
||||
maxOverlap: number
|
||||
maxGap: number
|
||||
maxPartTopMovement: number
|
||||
previousPartTop: number
|
||||
slowFrames: {
|
||||
duration: number
|
||||
index: number
|
||||
phase: "stream" | "boundary" | "complete" | "unknown"
|
||||
tokenSpans: number
|
||||
blocks: number
|
||||
codeBlocks: number
|
||||
height: number
|
||||
distance: number
|
||||
}[]
|
||||
scroll: {
|
||||
calls: number
|
||||
callNoops: number
|
||||
sameFrameCalls: number
|
||||
assignments: number
|
||||
assignmentNoops: number
|
||||
lastCallFrame: number
|
||||
frame: number
|
||||
}
|
||||
row: HTMLElement
|
||||
markdown: HTMLElement
|
||||
running: boolean
|
||||
previous: number
|
||||
cleanup: () => void
|
||||
start: () => void
|
||||
}
|
||||
|
||||
export async function installTimelineStreamProbe(
|
||||
page: Page,
|
||||
options: { textPartID: string; finalIndex: number; profileVisual: boolean; minimal: boolean },
|
||||
) {
|
||||
await page.evaluate(
|
||||
({ textPartID, finalIndex, profileVisual, minimal, markerPattern, fragmentCount }) => {
|
||||
const part = document.querySelector<HTMLElement>(`[data-timeline-part-id="${textPartID}"]`)
|
||||
const row = part?.closest<HTMLElement>("[data-timeline-row]")
|
||||
const markdown = part?.querySelector<HTMLElement>('[data-component="markdown"]')
|
||||
const root = part?.closest<HTMLElement>(".scroll-view__viewport")
|
||||
if (!part || !row || !markdown || !root) throw new Error("missing streaming benchmark nodes")
|
||||
const viewport = root.getBoundingClientRect()
|
||||
const state: TimelineProbeState = {
|
||||
started: 0,
|
||||
ended: Infinity,
|
||||
profileVisual,
|
||||
minimal,
|
||||
frames: [],
|
||||
frameAt: [],
|
||||
applied: [],
|
||||
geometry: [],
|
||||
blanks: 0,
|
||||
longTasks: [],
|
||||
layoutShifts: [],
|
||||
visibleMounts: 0,
|
||||
visibleUnmounts: 0,
|
||||
visibleRows: new Set(
|
||||
[...root.querySelectorAll("[data-timeline-key]")].filter((element) => {
|
||||
const rect = element.getBoundingClientRect()
|
||||
return rect.bottom > viewport.top && rect.top < viewport.bottom
|
||||
}),
|
||||
),
|
||||
visibleSubtreeMounts: [],
|
||||
visibleSubtreeUnmounts: [],
|
||||
visibleSubtreeReplacements: 0,
|
||||
visibleSubtreeDropouts: [],
|
||||
visibleSubtrees: new Map<string, Element>(),
|
||||
subtreeKeys: new WeakMap<Element, string>(),
|
||||
maxOverlap: 0,
|
||||
maxGap: 0,
|
||||
maxPartTopMovement: 0,
|
||||
previousPartTop: part.getBoundingClientRect().top,
|
||||
slowFrames: [],
|
||||
scroll: {
|
||||
calls: 0,
|
||||
callNoops: 0,
|
||||
sameFrameCalls: 0,
|
||||
assignments: 0,
|
||||
assignmentNoops: 0,
|
||||
lastCallFrame: -1,
|
||||
frame: 0,
|
||||
},
|
||||
row,
|
||||
markdown,
|
||||
running: false,
|
||||
previous: 0,
|
||||
cleanup: () => {},
|
||||
start: () => {},
|
||||
}
|
||||
;(window as Window & { __timelineStreamBenchmark?: TimelineProbeState }).__timelineStreamBenchmark = state
|
||||
const scrollTo = Element.prototype.scrollTo
|
||||
const scrollTop = Object.getOwnPropertyDescriptor(Element.prototype, "scrollTop")!
|
||||
if (profileVisual) {
|
||||
Element.prototype.scrollTo = function (...args) {
|
||||
state.scroll.calls += 1
|
||||
const top = typeof args[0] === "object" ? args[0]?.top : args[1]
|
||||
if (typeof top === "number") {
|
||||
const target = Math.min(top, this.scrollHeight - this.clientHeight)
|
||||
if (Math.abs(this.scrollTop - target) < 1) state.scroll.callNoops += 1
|
||||
}
|
||||
if (state.scroll.lastCallFrame === state.scroll.frame) state.scroll.sameFrameCalls += 1
|
||||
state.scroll.lastCallFrame = state.scroll.frame
|
||||
return scrollTo.apply(this, args)
|
||||
}
|
||||
Object.defineProperty(Element.prototype, "scrollTop", {
|
||||
configurable: true,
|
||||
get: scrollTop.get,
|
||||
set(value) {
|
||||
state.scroll.assignments += 1
|
||||
if (Math.abs(this.scrollTop - value) < 1) state.scroll.assignmentNoops += 1
|
||||
scrollTop.set!.call(this, value)
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
const recordLongTasks = (entries: PerformanceEntry[]) => {
|
||||
if (!state.running) return
|
||||
state.longTasks.push(
|
||||
...entries
|
||||
.filter((entry) => entry.startTime >= state.started && entry.startTime <= state.ended)
|
||||
.map((entry) => entry.duration),
|
||||
)
|
||||
}
|
||||
const longTaskObserver = new PerformanceObserver((list) => recordLongTasks(list.getEntries()))
|
||||
longTaskObserver.observe({ type: "longtask" })
|
||||
const recordLayoutShifts = (entries: PerformanceEntry[]) => {
|
||||
if (!state.running) return
|
||||
state.layoutShifts.push(
|
||||
...entries
|
||||
.map((entry) => {
|
||||
const shift = entry as LayoutShiftEntry
|
||||
if (shift.startTime < state.started || shift.hadRecentInput) return
|
||||
return shift.value
|
||||
})
|
||||
.filter((value): value is number => value !== undefined),
|
||||
)
|
||||
}
|
||||
const layoutShiftObserver = profileVisual
|
||||
? new PerformanceObserver((list) => recordLayoutShifts(list.getEntries()))
|
||||
: undefined
|
||||
layoutShiftObserver?.observe({ type: "layout-shift", buffered: true })
|
||||
|
||||
const visible = (element: Element) => {
|
||||
const rect = element.getBoundingClientRect()
|
||||
const viewport = root.getBoundingClientRect()
|
||||
const style = getComputedStyle(element)
|
||||
return (
|
||||
element.isConnected &&
|
||||
rect.width > 0 &&
|
||||
rect.height > 0 &&
|
||||
rect.bottom > viewport.top &&
|
||||
rect.top < viewport.bottom &&
|
||||
style.display !== "none" &&
|
||||
style.visibility !== "hidden" &&
|
||||
Number(style.opacity) > 0
|
||||
)
|
||||
}
|
||||
const critical = [
|
||||
"[data-timeline-part-id]",
|
||||
'[data-component="edit-content"]',
|
||||
'[data-component="apply-patch-file-diff"]',
|
||||
'[data-component="file"]',
|
||||
'[data-component="markdown-code"]',
|
||||
"[data-markdown-block]",
|
||||
].join(",")
|
||||
const describe = (element: Element) => {
|
||||
const cached = state.subtreeKeys.get(element)
|
||||
if (!element.isConnected && cached) return cached
|
||||
const part = element.closest<HTMLElement>("[data-timeline-part-id]")?.dataset.timelinePartId ?? "unknown"
|
||||
const block = element
|
||||
.closest<HTMLElement>("[data-markdown-key]")
|
||||
?.dataset.markdownKey?.replace(/:(?:code|full|live)$/, "")
|
||||
const component =
|
||||
element.getAttribute("data-component") ?? element.getAttribute("data-markdown-block") ?? element.tagName
|
||||
const key = `${part}:${block ?? "root"}:${component}`
|
||||
state.subtreeKeys.set(element, key)
|
||||
return key
|
||||
}
|
||||
const recordMutations = (records: MutationRecord[]) => {
|
||||
if (!state.running) return
|
||||
records.forEach((record) => {
|
||||
record.addedNodes.forEach((node) => {
|
||||
if (node instanceof HTMLElement && node.matches("[data-timeline-key]") && visible(node)) {
|
||||
state.visibleMounts += 1
|
||||
state.visibleRows.add(node)
|
||||
}
|
||||
if (!(node instanceof Element)) return
|
||||
const added = [node, ...node.querySelectorAll(critical)].filter((element) => element.matches(critical))
|
||||
added.forEach((element) => {
|
||||
if (visible(element)) state.visibleSubtreeMounts.push(describe(element))
|
||||
})
|
||||
})
|
||||
record.removedNodes.forEach((node) => {
|
||||
if (node instanceof HTMLElement && node.matches("[data-timeline-key]") && state.visibleRows.delete(node))
|
||||
state.visibleUnmounts += 1
|
||||
if (!(node instanceof Element)) return
|
||||
const removed = [node, ...node.querySelectorAll(critical)].filter((element) => element.matches(critical))
|
||||
removed.forEach((element) => {
|
||||
const key = describe(element)
|
||||
if (state.visibleSubtrees.get(key) === element) state.visibleSubtreeUnmounts.push(key)
|
||||
})
|
||||
})
|
||||
})
|
||||
}
|
||||
const mutationObserver = profileVisual ? new MutationObserver(recordMutations) : undefined
|
||||
mutationObserver?.observe(root, { childList: true, subtree: true })
|
||||
const currentPart = () => root.querySelector<HTMLElement>(`[data-timeline-part-id="${textPartID}"]`)
|
||||
const observeProgress = (at: number) => {
|
||||
if (!state.running) return
|
||||
const content = currentPart()?.textContent ?? ""
|
||||
const index = content.includes("benchmark-complete")
|
||||
? finalIndex
|
||||
: Number(content.match(new RegExp(markerPattern, "g"))?.at(-1)?.match(/\d+/)?.[0] ?? -1)
|
||||
if (index >= 0 && index !== state.applied.at(-1)?.index) state.applied.push({ at, index })
|
||||
}
|
||||
const progressObserver = new MutationObserver(() => observeProgress(performance.now()))
|
||||
progressObserver.observe(root, { characterData: true, childList: true, subtree: true })
|
||||
state.cleanup = () => {
|
||||
recordLongTasks(longTaskObserver.takeRecords())
|
||||
recordLayoutShifts(layoutShiftObserver?.takeRecords() ?? [])
|
||||
recordMutations(mutationObserver?.takeRecords() ?? [])
|
||||
if (progressObserver.takeRecords().length) observeProgress(performance.now())
|
||||
longTaskObserver.disconnect()
|
||||
layoutShiftObserver?.disconnect()
|
||||
mutationObserver?.disconnect()
|
||||
progressObserver.disconnect()
|
||||
if (!profileVisual) return
|
||||
Element.prototype.scrollTo = scrollTo
|
||||
Object.defineProperty(Element.prototype, "scrollTop", scrollTop)
|
||||
}
|
||||
|
||||
const sample = (now: number) => {
|
||||
if (!state.running) return
|
||||
state.frameAt.push(now)
|
||||
observeProgress(now)
|
||||
if (minimal) {
|
||||
state.frames.push(now - state.previous)
|
||||
state.previous = now
|
||||
requestAnimationFrame(sample)
|
||||
return
|
||||
}
|
||||
setTimeout(() => {
|
||||
if (!state.running) return
|
||||
state.scroll.frame += 1
|
||||
const duration = now - state.previous
|
||||
state.frames.push(duration)
|
||||
state.previous = now
|
||||
const virtualRoot = root.querySelector<HTMLElement>("[data-timeline-virtual-content]")
|
||||
const header = root.querySelector<HTMLElement>("[data-session-title]")
|
||||
state.geometry.push({
|
||||
scrollTop: root.scrollTop,
|
||||
scrollHeight: root.scrollHeight,
|
||||
clientHeight: root.clientHeight,
|
||||
distance: root.scrollHeight - root.clientHeight - root.scrollTop,
|
||||
virtualHeight: virtualRoot?.getBoundingClientRect().height ?? 0,
|
||||
headerHeight: header?.getBoundingClientRect().height ?? 0,
|
||||
})
|
||||
const viewport = root.getBoundingClientRect()
|
||||
if (profileVisual) {
|
||||
const visibleRows = [...root.querySelectorAll<HTMLElement>("[data-timeline-key]")]
|
||||
.map((element) => ({ element, rect: element.getBoundingClientRect() }))
|
||||
.filter((item) => item.rect.bottom > viewport.top && item.rect.top < viewport.bottom)
|
||||
.sort((a, b) => a.rect.top - b.rect.top)
|
||||
state.visibleRows = new Set(visibleRows.map((item) => item.element))
|
||||
const rows = visibleRows.map((item) => item.rect)
|
||||
rows.slice(1).forEach((rect, index) => {
|
||||
const previous = rows[index]!
|
||||
state.maxOverlap = Math.max(state.maxOverlap, previous.bottom - rect.top)
|
||||
state.maxGap = Math.max(state.maxGap, rect.top - previous.bottom)
|
||||
})
|
||||
const partTop = part.getBoundingClientRect().top
|
||||
state.maxPartTopMovement = Math.max(state.maxPartTopMovement, Math.abs(partTop - state.previousPartTop))
|
||||
state.previousPartTop = partTop
|
||||
}
|
||||
const visibleRow = [...root.querySelectorAll<HTMLElement>("[data-timeline-row]")].some((element) => {
|
||||
const rect = element.getBoundingClientRect()
|
||||
return rect.bottom > viewport.top && rect.top < viewport.bottom
|
||||
})
|
||||
if (!visibleRow) state.blanks += 1
|
||||
if (profileVisual) {
|
||||
const subtrees = new Map<string, { element: Element; rendered: boolean }>()
|
||||
const visibleSubtrees = new Map<string, Element>()
|
||||
root.querySelectorAll(critical).forEach((element) => {
|
||||
const key = describe(element)
|
||||
const rect = element.getBoundingClientRect()
|
||||
const style = getComputedStyle(element)
|
||||
const rendered =
|
||||
element.isConnected &&
|
||||
rect.width > 0 &&
|
||||
rect.height > 0 &&
|
||||
style.display !== "none" &&
|
||||
style.visibility !== "hidden" &&
|
||||
Number(style.opacity) > 0
|
||||
subtrees.set(key, { element, rendered })
|
||||
if (rendered && rect.bottom > viewport.top && rect.top < viewport.bottom) {
|
||||
const previous = state.visibleSubtrees.get(key)
|
||||
if (previous && previous !== element && key.startsWith(`${textPartID}:`))
|
||||
state.visibleSubtreeReplacements += 1
|
||||
visibleSubtrees.set(key, element)
|
||||
}
|
||||
})
|
||||
state.visibleSubtrees.forEach((element, key) => {
|
||||
const current = subtrees.get(key)
|
||||
if (key.startsWith(`${textPartID}:`) && !current?.rendered) {
|
||||
const markdown = part.querySelector<HTMLElement>('[data-component="markdown"]')
|
||||
state.visibleSubtreeDropouts.push(
|
||||
`${key}:projection=${markdown?.dataset.markdownProjectionLength}/${markdown?.dataset.markdownProjectionBlocks}:result=${markdown?.dataset.markdownResultLength}/${markdown?.dataset.markdownResultBlocks}:applied=${markdown?.dataset.markdownAppliedBlocks}:dom=${markdown?.children.length}`,
|
||||
)
|
||||
}
|
||||
if (element.matches('[data-component="file"]')) {
|
||||
const hadLines = element.hasAttribute("data-profiler-had-lines")
|
||||
const hasLines = element.shadowRoot?.querySelector("[data-line]") != null
|
||||
if (hasLines) element.setAttribute("data-profiler-had-lines", "")
|
||||
if (hadLines && !hasLines) state.visibleSubtreeDropouts.push(`${key}:shadow-lines`)
|
||||
}
|
||||
})
|
||||
state.visibleSubtrees = visibleSubtrees
|
||||
}
|
||||
if (profileVisual && duration > 33.34) {
|
||||
const livePart = currentPart()
|
||||
const content = livePart?.textContent ?? ""
|
||||
const complete = content.includes("benchmark-complete")
|
||||
const index = complete
|
||||
? finalIndex
|
||||
: Number(content.match(new RegExp(markerPattern, "g"))?.at(-1)?.match(/\d+/)?.[0] ?? -1)
|
||||
state.slowFrames.push({
|
||||
duration,
|
||||
index,
|
||||
phase: complete
|
||||
? "complete"
|
||||
: index >= 0 && index % fragmentCount === 0
|
||||
? "boundary"
|
||||
: index >= 0
|
||||
? "stream"
|
||||
: "unknown",
|
||||
tokenSpans: livePart?.querySelectorAll(".shiki span").length ?? 0,
|
||||
blocks: livePart?.querySelectorAll("[data-markdown-block]").length ?? 0,
|
||||
codeBlocks: livePart?.querySelectorAll('[data-component="markdown-code"]').length ?? 0,
|
||||
height: livePart?.getBoundingClientRect().height ?? 0,
|
||||
distance: root.scrollHeight - root.clientHeight - root.scrollTop,
|
||||
})
|
||||
}
|
||||
requestAnimationFrame(sample)
|
||||
}, 0)
|
||||
}
|
||||
state.start = () => {
|
||||
state.started = performance.now()
|
||||
state.previous = state.started
|
||||
state.running = true
|
||||
requestAnimationFrame(sample)
|
||||
}
|
||||
},
|
||||
{ ...options, markerPattern: STREAM_MARKER_PATTERN, fragmentCount: STREAM_FRAGMENT_COUNT },
|
||||
)
|
||||
}
|
||||
|
||||
export function startTimelineStreamProbe(page: Page) {
|
||||
return page.evaluate(() => {
|
||||
const state = (window as Window & { __timelineStreamBenchmark?: TimelineProbeState }).__timelineStreamBenchmark
|
||||
if (!state) throw new Error("missing streaming benchmark state")
|
||||
state.start()
|
||||
})
|
||||
}
|
||||
|
||||
type LayoutShiftEntry = PerformanceEntry & { value: number; hadRecentInput?: boolean }
|
||||
|
||||
export function layoutShiftValue(
|
||||
entry: Pick<LayoutShiftEntry, "startTime" | "value" | "hadRecentInput">,
|
||||
start: number,
|
||||
) {
|
||||
if (entry.startTime < start || entry.hadRecentInput) return
|
||||
return entry.value
|
||||
}
|
||||
|
||||
export function removeVisibleRow<T>(visible: Set<T>, row: T) {
|
||||
return visible.delete(row)
|
||||
}
|
||||
|
||||
export function streamProgress(content: string) {
|
||||
const index = Number(content.match(new RegExp(STREAM_MARKER_PATTERN, "g"))?.at(-1)?.match(/\d+/)?.[0] ?? -1)
|
||||
return {
|
||||
index,
|
||||
phase: content.includes("benchmark-complete")
|
||||
? ("complete" as const)
|
||||
: index >= 0 && index % STREAM_FRAGMENT_COUNT === 0
|
||||
? ("boundary" as const)
|
||||
: index >= 0
|
||||
? ("stream" as const)
|
||||
: ("unknown" as const),
|
||||
}
|
||||
}
|
||||
|
||||
export async function collectTimelineStreamMetrics(
|
||||
page: Page,
|
||||
options: { textPartID: string; finalIndex: number; navigations: string[] },
|
||||
) {
|
||||
return page.evaluate(({ textPartID, finalIndex, navigations }) => {
|
||||
const state = (window as Window & { __timelineStreamBenchmark?: TimelineProbeState }).__timelineStreamBenchmark
|
||||
if (!state) throw new Error(`missing streaming benchmark state after navigation: ${JSON.stringify(navigations)}`)
|
||||
state.ended = performance.now()
|
||||
state.cleanup()
|
||||
state.running = false
|
||||
const part = document.querySelector<HTMLElement>(`[data-timeline-part-id="${textPartID}"]`)
|
||||
const row = part?.closest<HTMLElement>("[data-timeline-row]")
|
||||
const markdown = part?.querySelector<HTMLElement>('[data-component="markdown"]')
|
||||
const sorted = state.frames.slice().sort((a, b) => a - b)
|
||||
const duration = state.frames.reduce((sum, value) => sum + value, 0)
|
||||
const longestSlowStreak = state.frames.reduce(
|
||||
(result, value) => {
|
||||
const current = value > 33.34 ? result.current + 1 : 0
|
||||
return { current, longest: Math.max(result.longest, current) }
|
||||
},
|
||||
{ current: 0, longest: 0 },
|
||||
).longest
|
||||
const busyStart = state.applied.at(0)?.at
|
||||
const completion = state.applied.find((value) => value.index === finalIndex)
|
||||
const busyEnd = completion?.at
|
||||
const busyFrames =
|
||||
busyStart === undefined || busyEnd === undefined
|
||||
? []
|
||||
: state.frames.filter((_, index) => state.frameAt[index]! >= busyStart && state.frameAt[index]! <= busyEnd)
|
||||
const busySorted = busyFrames.slice().sort((a, b) => a - b)
|
||||
const busyDuration = busyFrames.reduce((sum, value) => sum + value, 0)
|
||||
const completionObservedMs = (completion?.at ?? NaN) - state.started
|
||||
const visual = state.profileVisual
|
||||
? {
|
||||
layoutShiftValueSum: state.layoutShifts.reduce((sum, value) => sum + value, 0),
|
||||
maxLayoutShiftValue: Math.max(0, ...state.layoutShifts),
|
||||
visibleMounts: state.visibleMounts,
|
||||
visibleUnmounts: state.visibleUnmounts,
|
||||
visibleSubtreeMounts: state.visibleSubtreeMounts,
|
||||
visibleSubtreeUnmounts: [...new Set(state.visibleSubtreeUnmounts)],
|
||||
visibleSubtreeReplacements: state.visibleSubtreeReplacements,
|
||||
visibleSubtreeDropouts: [...new Set(state.visibleSubtreeDropouts)],
|
||||
maxOverlapPx: state.maxOverlap,
|
||||
maxGapPx: state.maxGap,
|
||||
maxPartTopMovementPx: state.maxPartTopMovement,
|
||||
slowestRafGaps: state.slowFrames
|
||||
.sort((a, b) => b.duration - a.duration)
|
||||
.slice(0, 20)
|
||||
.map((frame) => ({
|
||||
durationMs: frame.duration,
|
||||
index: frame.index,
|
||||
phase: frame.phase,
|
||||
tokenSpans: frame.tokenSpans,
|
||||
blocks: frame.blocks,
|
||||
codeBlocks: frame.codeBlocks,
|
||||
heightPx: frame.height,
|
||||
distancePx: frame.distance,
|
||||
})),
|
||||
slowRafGapPhases: Object.fromEntries(
|
||||
["stream", "boundary", "complete", "unknown"].map((phase) => {
|
||||
const frames = state.slowFrames.filter((frame) => frame.phase === phase)
|
||||
return [
|
||||
phase,
|
||||
{
|
||||
count: frames.length,
|
||||
totalMs: frames.reduce((sum, frame) => sum + frame.duration, 0),
|
||||
maxMs: Math.max(0, ...frames.map((frame) => frame.duration)),
|
||||
},
|
||||
]
|
||||
}),
|
||||
),
|
||||
scroll: state.scroll,
|
||||
}
|
||||
: null
|
||||
const geometry = state.minimal
|
||||
? null
|
||||
: {
|
||||
maxDistancePx: Math.max(0, ...state.geometry.map((sample) => sample.distance)),
|
||||
finalDistancePx: state.geometry.at(-1)?.distance ?? 0,
|
||||
final: state.geometry.at(-1),
|
||||
distanceTransitionsPx: state.geometry
|
||||
.map((sample) => Math.round(sample.distance))
|
||||
.filter((value, index, values) => index === 0 || value !== values[index - 1]),
|
||||
bottomDriftTransitions: state.geometry.slice(1).filter((value, index) => {
|
||||
const previous = state.geometry[index]?.distance ?? 0
|
||||
return previous <= 1 && value.distance > 1
|
||||
}).length,
|
||||
blankSamples: state.blanks,
|
||||
}
|
||||
return {
|
||||
capabilities: { visual: state.profileVisual, geometry: !state.minimal },
|
||||
completionObservedMs,
|
||||
deltasPerSecond: Number.isFinite(completionObservedMs) ? finalIndex / (completionObservedMs / 1_000) : null,
|
||||
rafGapSamples: state.frames.length,
|
||||
rafCallbackRate: duration ? (state.frames.length * 1000) / duration : 0,
|
||||
observedProgressWindowRafCallbackRate: busyDuration ? (busyFrames.length * 1000) / busyDuration : null,
|
||||
observedProgressWindowRafGapP95Ms: busySorted[Math.floor(busySorted.length * 0.95)] ?? null,
|
||||
observedProgressWindowRafGaps: busyFrames.length,
|
||||
maxObservedProgressIndex: Math.max(-1, ...state.applied.map((value) => value.index)),
|
||||
observedProgressTransitions: state.applied.length,
|
||||
rafGapP50Ms: sorted[Math.floor(sorted.length * 0.5)] ?? 0,
|
||||
rafGapP95Ms: sorted[Math.floor(sorted.length * 0.95)] ?? 0,
|
||||
rafGapP99Ms: sorted[Math.floor(sorted.length * 0.99)] ?? 0,
|
||||
maxRafGapMs: sorted.at(-1) ?? 0,
|
||||
rafGapsOver33Ms: state.frames.filter((value) => value > 33.34).length,
|
||||
rafGapsOver50Ms: state.frames.filter((value) => value > 50).length,
|
||||
missedFrameBudgetEquivalents: state.frames.reduce(
|
||||
(sum, value) => sum + Math.max(0, Math.round(value / 16.67) - 1),
|
||||
0,
|
||||
),
|
||||
longestRafGapOver33MsStreak: longestSlowStreak,
|
||||
longTaskCount: state.longTasks.length,
|
||||
longTaskTimeMs: state.longTasks.reduce((sum, value) => sum + value, 0),
|
||||
visual,
|
||||
geometry,
|
||||
rowReplaced: row !== state.row,
|
||||
markdownReplaced: markdown !== state.markdown,
|
||||
domTextCharacters: part?.textContent?.length ?? 0,
|
||||
}
|
||||
}, options)
|
||||
}
|
||||
@@ -0,0 +1,335 @@
|
||||
const words = [
|
||||
"alpha",
|
||||
"bravo",
|
||||
"charlie",
|
||||
"delta",
|
||||
"echo",
|
||||
"foxtrot",
|
||||
"golf",
|
||||
"hotel",
|
||||
"india",
|
||||
"juliet",
|
||||
"kilo",
|
||||
"lima",
|
||||
"metro",
|
||||
"nova",
|
||||
"orbit",
|
||||
"pixel",
|
||||
"quartz",
|
||||
"river",
|
||||
"signal",
|
||||
"vector",
|
||||
]
|
||||
|
||||
const sourceID = "ses_smoke_source"
|
||||
const targetID = "ses_smoke_target"
|
||||
const directory = "C:/OpenCode/SmokeProject"
|
||||
const projectID = "proj_smoke_timeline"
|
||||
const model = { providerID: "opencode", modelID: "claude-opus-4-6", variant: "max" }
|
||||
|
||||
type MessageInfo = Record<string, unknown> & { id: string; role: "user" | "assistant" }
|
||||
type MessagePart = Record<string, unknown> & { id: string; type: string; text?: string; tool?: string }
|
||||
type Message = { info: MessageInfo; parts: MessagePart[] }
|
||||
|
||||
function lorem(seed: number, length: number) {
|
||||
let out = ""
|
||||
let i = seed
|
||||
while (out.length < length) {
|
||||
const word = words[i % words.length]
|
||||
out += (out ? " " : "") + word
|
||||
if (i % 17 === 0) out += ".\n\n"
|
||||
i += 7
|
||||
}
|
||||
return out.slice(0, length)
|
||||
}
|
||||
|
||||
function id(prefix: string, value: number) {
|
||||
return `${prefix}_smoke_${String(value).padStart(4, "0")}`
|
||||
}
|
||||
|
||||
function userMessage(sessionID: string, index: number, textLength: number, diffs: unknown[] = []): Message {
|
||||
const messageID = id("msg_user", index)
|
||||
return {
|
||||
info: {
|
||||
id: messageID,
|
||||
sessionID,
|
||||
role: "user",
|
||||
time: { created: 1700000000000 + index * 10_000 },
|
||||
summary: { diffs },
|
||||
agent: "build",
|
||||
model,
|
||||
},
|
||||
parts: [
|
||||
{
|
||||
id: id("prt_user_text", index),
|
||||
sessionID,
|
||||
messageID,
|
||||
type: "text",
|
||||
text: lorem(index, textLength),
|
||||
},
|
||||
],
|
||||
}
|
||||
}
|
||||
|
||||
function assistantMessage(sessionID: string, index: number, parentID: string, parts: MessagePart[]): Message {
|
||||
const messageID = id("msg_assistant", index)
|
||||
return {
|
||||
info: {
|
||||
id: messageID,
|
||||
sessionID,
|
||||
role: "assistant",
|
||||
time: { created: 1700000000000 + index * 10_000 + 1_000, completed: 1700000000000 + index * 10_000 + 8_000 },
|
||||
parentID,
|
||||
modelID: model.modelID,
|
||||
providerID: model.providerID,
|
||||
mode: "build",
|
||||
agent: "build",
|
||||
path: { cwd: directory, root: directory },
|
||||
cost: 0.01,
|
||||
tokens: { input: 100, output: 200, reasoning: 0, cache: { read: 0, write: 0 } },
|
||||
variant: "max",
|
||||
finish: "stop",
|
||||
},
|
||||
parts: parts.map((part) => ({
|
||||
...part,
|
||||
sessionID,
|
||||
messageID,
|
||||
})),
|
||||
}
|
||||
}
|
||||
|
||||
function textPart(index: number, partIndex: number, length: number): MessagePart {
|
||||
const prose = lorem(index * 13 + partIndex, length)
|
||||
const text =
|
||||
index % 12 === 0
|
||||
? `${prose}\n\n\`\`\`ts\n${code(index, 80)}\n\`\`\``
|
||||
: index % 5 === 0
|
||||
? `${prose}\n\n\`\`\`ts\nexport const value = "${lorem(index, 220)}"\n\`\`\``
|
||||
: index % 7 === 0
|
||||
? `${prose}\n\nThe wrapped inline value is \`${lorem(index, 180)}\`.`
|
||||
: prose
|
||||
return { id: id(`prt_text_${partIndex}`, index), type: "text", text }
|
||||
}
|
||||
|
||||
function reasoningPart(index: number, partIndex: number, length: number): MessagePart {
|
||||
return {
|
||||
id: id(`prt_reasoning_${partIndex}`, index),
|
||||
type: "reasoning",
|
||||
text: lorem(index * 19 + partIndex, length),
|
||||
time: { start: 1700000000000 + index * 10_000, end: 1700000000000 + index * 10_000 + 500 },
|
||||
}
|
||||
}
|
||||
|
||||
function toolPart(
|
||||
index: number,
|
||||
partIndex: number,
|
||||
tool: string,
|
||||
input: Record<string, unknown>,
|
||||
outputLength = 160,
|
||||
): MessagePart {
|
||||
const metadata =
|
||||
tool === "apply_patch"
|
||||
? { files: [patchFile(index, "update"), patchFile(index + 1, index % 2 === 0 ? "add" : "delete")] }
|
||||
: tool === "edit" || tool === "write"
|
||||
? {
|
||||
filediff: fileDiff(String(input.filePath ?? `src/generated/file-${index}.ts`), index),
|
||||
diff: patch(index, outputLength),
|
||||
preview: patch(index + 1, 420),
|
||||
}
|
||||
: tool === "question"
|
||||
? { answers: [["Proceed"], ["Keep sample output"]] }
|
||||
: {}
|
||||
return {
|
||||
id: id(`prt_tool_${tool}_${partIndex}`, index),
|
||||
type: "tool",
|
||||
callID: id("call", index * 10 + partIndex),
|
||||
tool,
|
||||
state: {
|
||||
status: "completed",
|
||||
input,
|
||||
output: lorem(index * 23 + partIndex, outputLength),
|
||||
title: tool === "bash" ? "Verify generated output" : input.filePath || input.path || input.pattern || "completed",
|
||||
metadata,
|
||||
time: { start: 1700000000000 + index * 10_000, end: 1700000000000 + index * 10_000 + 400 },
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
function patchFile(seed: number, type: "add" | "update" | "delete") {
|
||||
return {
|
||||
filePath: `src/generated/patch-${seed}.ts`,
|
||||
relativePath: `src/generated/patch-${seed}.ts`,
|
||||
type,
|
||||
additions: (seed % 7) + 1,
|
||||
deletions: type === "add" ? 0 : seed % 4,
|
||||
patch: patch(seed, 520),
|
||||
before: type === "add" ? undefined : code(seed, 18),
|
||||
after: type === "delete" ? undefined : code(seed + 1, 24),
|
||||
}
|
||||
}
|
||||
|
||||
function fileDiff(file: string, seed: number) {
|
||||
const lines = seed % 12 === 0 ? 300 : seed % 8 === 0 ? 2 : 38
|
||||
const before = code(seed, lines, seed % 10 === 0 ? 280 : 32)
|
||||
const after =
|
||||
lines === 2
|
||||
? before.replace("value1", "updatedValue1")
|
||||
: lines === 300
|
||||
? code(seed + 1, lines, seed % 10 === 0 ? 280 : 32)
|
||||
: before.replace("value4", "updatedValue4").replace("value20", "updatedValue20")
|
||||
return {
|
||||
file,
|
||||
additions: lines === 300 ? 300 : lines === 2 ? 1 : 2,
|
||||
deletions: lines === 300 ? 300 : lines === 2 ? 1 : 2,
|
||||
before,
|
||||
after,
|
||||
}
|
||||
}
|
||||
|
||||
function patch(seed: number, length: number) {
|
||||
return `diff --git a/src/generated/file-${seed}.ts b/src/generated/file-${seed}.ts\n+${lorem(seed, length).replace(/\n/g, "\n+")}`
|
||||
}
|
||||
|
||||
function code(seed: number, lines: number, width = 32) {
|
||||
return Array.from(
|
||||
{ length: lines },
|
||||
(_, index) => `export const value${index} = "${lorem(seed + index, width)}"`,
|
||||
).join("\n")
|
||||
}
|
||||
|
||||
function turn(index: number): Message[] {
|
||||
const diff = index % 9 === 0 ? [fileDiff(`src/generated/summary-${index}.ts`, index)] : []
|
||||
const user = userMessage(targetID, index, 100 + (index % 4) * 80, diff)
|
||||
const parts = [
|
||||
...(index % 5 === 0 ? [reasoningPart(index, 0, 420)] : []),
|
||||
...(index % 3 === 0
|
||||
? [
|
||||
toolPart(index, 0, "read", { filePath: `src/generated/file-${index}.ts`, offset: 0, limit: 80 }, 220),
|
||||
toolPart(index, 5, "glob", { path: directory, pattern: `**/*sample-${index}*.ts` }, 140),
|
||||
toolPart(index, 1, "grep", { path: directory, pattern: `sample-${index}`, include: "*.ts" }, 180),
|
||||
toolPart(index, 6, "list", { path: `src/generated/${index}` }, 120),
|
||||
]
|
||||
: []),
|
||||
textPart(index, 2, 160 + (index % 6) * 90),
|
||||
...(index % 4 === 0 ? [toolPart(index, 3, "edit", { filePath: `src/generated/file-${index}.ts` }, 700)] : []),
|
||||
...(index % 6 === 0
|
||||
? [toolPart(index, 7, "write", { filePath: `src/generated/write-${index}.ts`, content: code(index, 28) }, 560)]
|
||||
: []),
|
||||
...(index % 8 === 0
|
||||
? [toolPart(index, 8, "apply_patch", { files: [`src/generated/patch-${index}.ts`] }, 620)]
|
||||
: []),
|
||||
...(index % 7 === 0
|
||||
? [toolPart(index, 4, "bash", { command: "bun typecheck", description: "Verify generated output" }, 620)]
|
||||
: []),
|
||||
...(index % 10 === 0 ? [toolPart(index, 9, "webfetch", { url: "https://example.com/docs/sample" }, 120)] : []),
|
||||
...(index % 11 === 0 ? [toolPart(index, 10, "websearch", { query: "sample movement notes" }, 240)] : []),
|
||||
...(index % 13 === 0
|
||||
? [
|
||||
toolPart(
|
||||
index,
|
||||
11,
|
||||
"question",
|
||||
{ questions: [{ question: "Use generated fixture?" }, { question: "Keep same row shape?" }] },
|
||||
120,
|
||||
),
|
||||
]
|
||||
: []),
|
||||
...(index % 17 === 0
|
||||
? [toolPart(index, 12, "task", { description: "Inspect generated fixture", subagent_type: "explore" }, 160)]
|
||||
: []),
|
||||
]
|
||||
return [user, assistantMessage(targetID, index, user.info.id, parts)]
|
||||
}
|
||||
|
||||
const targetMessages = Array.from({ length: 72 }, (_, index) => turn(index)).flat()
|
||||
const sourceMessages = Array.from({ length: 12 }, (_, index) => [
|
||||
userMessage(sourceID, index + 1000, 120),
|
||||
assistantMessage(sourceID, index + 1000, id("msg_user", index + 1000), [textPart(index + 1000, 0, 240)]),
|
||||
]).flat()
|
||||
|
||||
function renderable(part: MessagePart) {
|
||||
if (part.type === "tool" && part.tool === "todowrite") return false
|
||||
if (part.type === "text") return !!part.text.trim()
|
||||
if (part.type === "reasoning") return !!part.text.trim()
|
||||
return part.type !== "step-start" && part.type !== "step-finish" && part.type !== "patch"
|
||||
}
|
||||
|
||||
function orderedParts(message: Message) {
|
||||
return message.parts.slice().sort((a, b) => a.id.localeCompare(b.id))
|
||||
}
|
||||
|
||||
export const fixture = {
|
||||
directory,
|
||||
project: {
|
||||
id: projectID,
|
||||
worktree: directory,
|
||||
vcs: "git",
|
||||
name: "smoke-project",
|
||||
time: { created: 1700000000000, updated: 1700000000000 },
|
||||
sandboxes: [],
|
||||
},
|
||||
provider: {
|
||||
all: [
|
||||
{
|
||||
id: "opencode",
|
||||
name: "OpenCode",
|
||||
models: { "claude-opus-4-6": { id: "claude-opus-4-6", name: "Claude Opus 4.6", limit: { context: 200_000 } } },
|
||||
},
|
||||
],
|
||||
connected: ["opencode"],
|
||||
default: { providerID: "opencode", modelID: "claude-opus-4-6" },
|
||||
},
|
||||
sessions: [
|
||||
{
|
||||
id: sourceID,
|
||||
slug: "source",
|
||||
projectID,
|
||||
directory,
|
||||
title: "Uncommitted changes inquiry",
|
||||
version: "dev",
|
||||
time: { created: 1700000000000, updated: 1700000000000 },
|
||||
},
|
||||
{
|
||||
id: targetID,
|
||||
slug: "target",
|
||||
projectID,
|
||||
directory,
|
||||
title: "Example Game: sample jump movement & sample physics analysis",
|
||||
version: "dev",
|
||||
time: { created: 1700000001000, updated: 1700000001000 },
|
||||
},
|
||||
],
|
||||
sourceID,
|
||||
targetID,
|
||||
messages: { [sourceID]: sourceMessages, [targetID]: targetMessages },
|
||||
expected: {
|
||||
sourceTitle: "Uncommitted changes inquiry",
|
||||
targetTitle: "Example Game: sample jump movement & sample physics analysis",
|
||||
sourceMessageIDs: sourceMessages
|
||||
.filter((message) => message.info.role === "user")
|
||||
.map((message) => message.info.id),
|
||||
targetMessageIDs: targetMessages
|
||||
.filter((message) => message.info.role === "user")
|
||||
.map((message) => message.info.id),
|
||||
targetPartIDs: targetMessages.flatMap((message) =>
|
||||
orderedParts(message)
|
||||
.filter(renderable)
|
||||
.map((part) => part.id),
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
export function pageMessages(sessionID: string, limit: number, before?: string) {
|
||||
const messages = fixture.messages[sessionID as keyof typeof fixture.messages] ?? []
|
||||
const end = before
|
||||
? Math.max(
|
||||
0,
|
||||
messages.findIndex((message) => message.info.id === before),
|
||||
)
|
||||
: messages.length
|
||||
const start = Math.max(0, end - limit)
|
||||
return {
|
||||
items: messages.slice(start, end),
|
||||
cursor: start > 0 ? messages[start]!.info.id : undefined,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
import type { Page } from "@playwright/test"
|
||||
import { base64Encode } from "@opencode-ai/core/util/encode"
|
||||
import { mockOpenCodeServer } from "../../utils/mock-server"
|
||||
import { fixture } from "./session-timeline-stress.fixture"
|
||||
|
||||
export async function installTimelineSettings(page: Page) {
|
||||
await page.addInitScript(() => {
|
||||
localStorage.setItem(
|
||||
"settings.v3",
|
||||
JSON.stringify({
|
||||
general: {
|
||||
editToolPartsExpanded: true,
|
||||
shellToolPartsExpanded: true,
|
||||
showReasoningSummaries: true,
|
||||
showSessionProgressBar: true,
|
||||
},
|
||||
}),
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
export function mockStressTimeline(page: Page) {
|
||||
return mockOpenCodeServer(page, {
|
||||
sessions: fixture.sessions,
|
||||
provider: fixture.provider,
|
||||
directory: fixture.directory,
|
||||
project: fixture.project,
|
||||
pageMessages: (sessionID) => ({ items: fixture.messages[sessionID as keyof typeof fixture.messages] ?? [] }),
|
||||
})
|
||||
}
|
||||
|
||||
export async function installStressSessionTabs(page: Page) {
|
||||
const server = `http://${process.env.PLAYWRIGHT_SERVER_HOST ?? "127.0.0.1"}:${process.env.PLAYWRIGHT_SERVER_PORT ?? "4096"}`
|
||||
await page.addInitScript(
|
||||
({ directory, sourceID, targetID, dirBase64, server }) => {
|
||||
localStorage.setItem(
|
||||
"opencode.global.dat:server",
|
||||
JSON.stringify({
|
||||
projects: { local: [{ worktree: directory, expanded: true }] },
|
||||
lastProject: { local: directory },
|
||||
}),
|
||||
)
|
||||
localStorage.setItem(
|
||||
"opencode.global.dat:tabs",
|
||||
JSON.stringify(
|
||||
[sourceID, targetID].map((sessionId) => ({
|
||||
type: "session",
|
||||
server,
|
||||
dirBase64,
|
||||
sessionId,
|
||||
})),
|
||||
),
|
||||
)
|
||||
},
|
||||
{
|
||||
directory: fixture.directory,
|
||||
sourceID: fixture.sourceID,
|
||||
targetID: fixture.targetID,
|
||||
dirBase64: base64Encode(fixture.directory),
|
||||
server,
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
export function stressSessionHref(sessionID: string) {
|
||||
return `/${base64Encode(fixture.directory)}/session/${sessionID}`
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
import { expect, test } from "bun:test"
|
||||
import { mkdtemp, rm } from "node:fs/promises"
|
||||
import path from "node:path"
|
||||
import os from "node:os"
|
||||
import { prepareChromeTrace } from "../chrome-trace"
|
||||
|
||||
test("creates the configured trace directory", async () => {
|
||||
const root = await mkdtemp(path.join(os.tmpdir(), "opencode-trace-"))
|
||||
try {
|
||||
const file = await prepareChromeTrace(path.join(root, "nested", "traces"), "session/tab", false, "test")
|
||||
expect(file).toEndWith("-session-tab-458ed9e3-test.json")
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
@@ -0,0 +1,42 @@
|
||||
import { expect, test } from "bun:test"
|
||||
import { compressCachedRepaintTrace, layoutShiftSample } from "../timeline/session-tab-repaint-probe"
|
||||
|
||||
test("compresses repeated repaint states without losing frame samples", () => {
|
||||
const state = {
|
||||
root: 1,
|
||||
scrollTop: 10,
|
||||
scrollHeight: 20,
|
||||
bottomErrorPx: 0,
|
||||
last: true,
|
||||
rows: [{ key: "row", node: 2, top: 0, bottom: 10 }],
|
||||
mounted: 1,
|
||||
center: "content",
|
||||
}
|
||||
const trace = {
|
||||
timeOriginEpochMs: 1_000,
|
||||
startedAtPerformanceMs: 100,
|
||||
samples: [
|
||||
{ observedAtMs: 16, ...state, destination: ["target"], source: [] },
|
||||
{ observedAtMs: 32, ...state, destination: ["target"], source: [] },
|
||||
{ observedAtMs: 48, ...state, scrollTop: 11, destination: ["target"], source: [] },
|
||||
],
|
||||
mutations: [{ observedAtMs: 20, changed: [{ type: "add", node: 2 }] }],
|
||||
shifts: [{ occurredAtMs: 24, value: 0.1 }],
|
||||
windowMs: 1_000,
|
||||
running: false,
|
||||
stop() {},
|
||||
}
|
||||
const compressed = compressCachedRepaintTrace(trace)
|
||||
const samples = compressed.samples.flatMap((group) =>
|
||||
group.observedAtMs.map((observedAtMs) => ({ observedAtMs, ...group.state })),
|
||||
)
|
||||
|
||||
expect(samples).toEqual(trace.samples)
|
||||
expect(compressed.mutations).toEqual(trace.mutations)
|
||||
expect(compressed.shifts).toEqual(trace.shifts)
|
||||
})
|
||||
|
||||
test("records layout shifts at occurrence time within the probe window", () => {
|
||||
expect(layoutShiftSample({ startTime: 99, value: 0.1 }, 100)).toBeUndefined()
|
||||
expect(layoutShiftSample({ startTime: 124, value: 0.2 }, 100)).toEqual({ occurredAtMs: 24, value: 0.2 })
|
||||
})
|
||||
@@ -0,0 +1,54 @@
|
||||
import { expect, test } from "bun:test"
|
||||
import { classifySessionSwitch } from "../timeline/session-tab-switch-metrics"
|
||||
|
||||
test("counts source and blank samples before the destination is observed", () => {
|
||||
const result = classifySessionSwitch([
|
||||
{ observedAtMs: 16, destination: [], source: ["source"], hasVisibleRows: true, last: false },
|
||||
{ observedAtMs: 32, destination: [], source: [], hasVisibleRows: false, last: false },
|
||||
{ observedAtMs: 48, destination: ["destination"], source: [], hasVisibleRows: true, last: true, bottomErrorPx: 0 },
|
||||
{ observedAtMs: 64, destination: ["destination"], source: [], hasVisibleRows: true, last: true, bottomErrorPx: 0 },
|
||||
{ observedAtMs: 80, destination: ["destination"], source: [], hasVisibleRows: true, last: true, bottomErrorPx: 0 },
|
||||
])
|
||||
|
||||
expect(result.blankSamples).toBe(1)
|
||||
expect(result.sourceSamples).toBe(1)
|
||||
expect(result.unknownSamples).toBe(0)
|
||||
expect(result.firstDestinationObservedMs).toBe(48)
|
||||
expect(result.stableObservedMs).toBe(80)
|
||||
})
|
||||
|
||||
test("does not classify mixed source and destination content as correct", () => {
|
||||
const result = classifySessionSwitch([
|
||||
{
|
||||
observedAtMs: 16,
|
||||
destination: ["destination"],
|
||||
source: ["source"],
|
||||
hasVisibleRows: true,
|
||||
last: true,
|
||||
bottomErrorPx: 0,
|
||||
},
|
||||
{ observedAtMs: 32, destination: ["destination"], source: [], hasVisibleRows: true, last: true, bottomErrorPx: 0 },
|
||||
{ observedAtMs: 48, destination: ["destination"], source: [], hasVisibleRows: true, last: true, bottomErrorPx: 0 },
|
||||
{ observedAtMs: 64, destination: ["destination"], source: [], hasVisibleRows: true, last: true, bottomErrorPx: 0 },
|
||||
])
|
||||
|
||||
expect(result.firstCorrectObservedMs).toBe(32)
|
||||
expect(result.stableObservedMs).toBe(64)
|
||||
})
|
||||
|
||||
test("reports missing correctness without throwing", () => {
|
||||
const result = classifySessionSwitch([
|
||||
{
|
||||
observedAtMs: 16,
|
||||
destination: ["destination"],
|
||||
source: ["source"],
|
||||
hasVisibleRows: true,
|
||||
last: true,
|
||||
bottomErrorPx: 0,
|
||||
},
|
||||
])
|
||||
|
||||
expect(result.firstDestinationObservedMs).toBe(16)
|
||||
expect(result.firstCorrectObservedMs).toBeNull()
|
||||
expect(result.stableObservedMs).toBeNull()
|
||||
})
|
||||
@@ -0,0 +1,14 @@
|
||||
import { expect, test } from "bun:test"
|
||||
import { streamChunk } from "../timeline/session-timeline-benchmark.fixture"
|
||||
import { streamProgress } from "../timeline/session-timeline-stream-probe"
|
||||
|
||||
test("classifies emitted stream markers using the fixture cycle", () => {
|
||||
expect(streamProgress("before stream-17 after stream-18")).toEqual({ index: 18, phase: "boundary" })
|
||||
expect(streamProgress("before stream-18 after stream-19")).toEqual({ index: 19, phase: "stream" })
|
||||
expect(streamProgress("benchmark-complete stream-36")).toEqual({ index: 36, phase: "complete" })
|
||||
expect(streamProgress("no marker")).toEqual({ index: -1, phase: "unknown" })
|
||||
})
|
||||
|
||||
test("emits progress markers at fixture boundaries", () => {
|
||||
expect(streamProgress(streamChunk(18, 160))).toEqual({ index: 18, phase: "boundary" })
|
||||
})
|
||||
@@ -0,0 +1,16 @@
|
||||
import { expect, test } from "bun:test"
|
||||
import { layoutShiftValue, removeVisibleRow } from "../timeline/session-timeline-stream-probe"
|
||||
|
||||
test("excludes layout shifts before the probe window and recent input", () => {
|
||||
expect(layoutShiftValue({ startTime: 9, value: 0.1 }, 10)).toBeUndefined()
|
||||
expect(layoutShiftValue({ startTime: 10, value: 0.2, hadRecentInput: true }, 10)).toBeUndefined()
|
||||
expect(layoutShiftValue({ startTime: 11, value: 0.3 }, 10)).toBe(0.3)
|
||||
})
|
||||
|
||||
test("classifies removed rows from their last painted visibility", () => {
|
||||
const row = {}
|
||||
const visible = new Set([row])
|
||||
|
||||
expect(removeVisibleRow(visible, row)).toBe(true)
|
||||
expect(removeVisibleRow(visible, row)).toBe(false)
|
||||
})
|
||||
@@ -44,7 +44,10 @@ export async function mockOpenCodeServer(page: Page, config: MockServerConfig) {
|
||||
await page.route("**/*", async (route) => {
|
||||
const url = new URL(route.request().url())
|
||||
const targetPort = process.env.PLAYWRIGHT_SERVER_PORT ?? "4096"
|
||||
if (url.port !== targetPort) return route.fallback()
|
||||
const appPort = new URL(
|
||||
process.env.PLAYWRIGHT_BASE_URL ?? `http://127.0.0.1:${process.env.PLAYWRIGHT_PORT ?? "3000"}`,
|
||||
).port
|
||||
if (url.port !== targetPort && url.port !== appPort) return route.fallback()
|
||||
|
||||
const path = url.pathname
|
||||
if (path === "/global/event" || path === "/event") return sse(route, config.events?.(), config.eventRetry)
|
||||
@@ -72,7 +75,8 @@ export async function mockOpenCodeServer(page: Page, config: MockServerConfig) {
|
||||
return json(route, pageData.items, pageData.cursor ? { "x-next-cursor": pageData.cursor } : undefined)
|
||||
}
|
||||
|
||||
return json(route, {})
|
||||
if (url.port === targetPort && targetPort !== appPort) return json(route, {})
|
||||
return route.fallback()
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -24,7 +24,8 @@
|
||||
"test:e2e": "playwright test",
|
||||
"test:e2e:local": "playwright test",
|
||||
"test:e2e:ui": "playwright test --ui",
|
||||
"test:e2e:report": "playwright show-report e2e/playwright-report"
|
||||
"test:e2e:report": "playwright show-report e2e/playwright-report",
|
||||
"test:bench": "bun test ./e2e/performance/unit && playwright test --config e2e/performance/playwright.config.ts"
|
||||
},
|
||||
"license": "MIT",
|
||||
"devDependencies": {
|
||||
|
||||
@@ -9,6 +9,7 @@ const reuse = !process.env.CI
|
||||
const workers = Number(process.env.PLAYWRIGHT_WORKERS ?? (process.env.CI ? 5 : 0)) || undefined
|
||||
export default defineConfig({
|
||||
testDir: "./e2e",
|
||||
testIgnore: process.env.OPENCODE_PERFORMANCE === "1" ? "performance/**/*.test.ts" : "performance/**",
|
||||
outputDir: "./e2e/test-results",
|
||||
timeout: 60_000,
|
||||
expect: {
|
||||
|
||||
@@ -23,7 +23,7 @@ export const Plugin = PluginV2.define({
|
||||
skill: new SkillV2.Info({
|
||||
name: "customize-opencode",
|
||||
description:
|
||||
"Use ONLY when the user is editing or creating opencode's own configuration: opencode.json, opencode.jsonc, files under .opencode/, or files under ~/.config/opencode/. Also use when creating or fixing opencode agents, subagents, skills, plugins, MCP servers, or permission rules. Do not use for the user's own application code, or for any project that is not configuring opencode itself.",
|
||||
"Use ONLY when the user is editing or creating opencode's own configuration: opencode.json, opencode.jsonc, files under .opencode/, or files under ~/.config/opencode/. Also use when creating or fixing opencode agents, subagents, commands, skills, plugins, MCP servers, or permission rules. Do not use for the user's own application code, or for any project that is not configuring opencode itself.",
|
||||
location: AbsolutePath.make("/builtin/customize-opencode.md"),
|
||||
content: CustomizeOpencodeContent,
|
||||
}),
|
||||
|
||||
@@ -43,6 +43,8 @@ already-loaded config until then.
|
||||
| Global config | `~/.config/opencode/opencode.json` (NOT `~/.opencode/`) |
|
||||
| Project agents | `.opencode/agent/<name>.md` or `.opencode/agents/<name>.md` |
|
||||
| Global agents | `~/.config/opencode/agent(s)/<name>.md` |
|
||||
| Project commands | `.opencode/command/<name>.md` or `.opencode/commands/<name>.md` |
|
||||
| Global commands | `~/.config/opencode/command(s)/<name>.md` |
|
||||
| Project skills | `.opencode/skill(s)/<name>/SKILL.md` |
|
||||
| Global skills | `~/.config/opencode/skill(s)/<name>/SKILL.md` |
|
||||
| External skills (auto-loaded) | `~/.claude/skills/<name>/SKILL.md`, `~/.agents/skills/<name>/SKILL.md` |
|
||||
@@ -96,7 +98,7 @@ Every field is optional.
|
||||
},
|
||||
|
||||
"command": {
|
||||
"deploy": { "description": "...", "prompt": "..." }
|
||||
"deploy": { "description": "...", "template": "..." }
|
||||
},
|
||||
|
||||
"provider": {
|
||||
@@ -151,6 +153,7 @@ Shape notes worth being explicit about:
|
||||
- `skills` is an object with `paths` and/or `urls`, not an array.
|
||||
- `references` is an object keyed by alias. Each value is a local path, Git repository, or string shorthand.
|
||||
- `agent` is an object keyed by agent name, not an array.
|
||||
- `command` is an object keyed by command name, not an array.
|
||||
- `plugin` is an array of strings or `[name, options]` tuples, not an object.
|
||||
- `mcp[name].command` is an array of strings, never a single string. `type` is required.
|
||||
- `permission` is either a string action or an object keyed by tool name.
|
||||
@@ -277,6 +280,31 @@ opencode ships with `build`, `plan`, `general`, `explore`. Hidden internal agent
|
||||
`compaction`, `title`, `summary`. To override a built-in's fields, define the
|
||||
same key in `agent: { <name>: { ... } }`.
|
||||
|
||||
## Commands
|
||||
|
||||
opencode's command loader scans for `**/*.md` inside command directories. The
|
||||
file is named after the command, and lives directly inside the `command` folder:
|
||||
|
||||
```
|
||||
.opencode/command/deploy.md
|
||||
```
|
||||
|
||||
Frontmatter:
|
||||
|
||||
```markdown
|
||||
---
|
||||
description: One sentence describing what the command does.
|
||||
agent: build
|
||||
model: anthropic/claude-sonnet-4-6
|
||||
---
|
||||
|
||||
(command body in markdown: the prompt opencode runs, with $ARGUMENTS for the user's input)
|
||||
```
|
||||
|
||||
- `template` is the command body — everything below the frontmatter — and is required: it is the prompt opencode runs when the command is invoked. Do not also put a `template:` key in the frontmatter.
|
||||
- `$ARGUMENTS` is replaced with everything the user typed after the command; `$1`, `$2`, … pull individual positional arguments.
|
||||
- Optional: `description`, `agent`, `model`, `variant`, `subtask`.
|
||||
|
||||
## Plugins
|
||||
|
||||
`plugin:` is an array. Each entry is one of:
|
||||
@@ -415,8 +443,8 @@ When a user's config is broken and opencode won't start, these env vars help:
|
||||
exact shape, or the field is not covered in this skill, fetch
|
||||
`https://opencode.ai/config.json` and read the schema rather than guessing.
|
||||
- Preserve `$schema` and any existing fields the user did not ask to change.
|
||||
- For agent, skill, and plugin definitions, prefer creating new files in the
|
||||
correct location over inlining everything in `opencode.json`.
|
||||
- For agent, command, skill, and plugin definitions, prefer creating new files
|
||||
in the correct location over inlining everything in `opencode.json`.
|
||||
- If the user's existing config is malformed, point them at the env-var escape
|
||||
hatches above so they can edit from inside opencode without breaking their
|
||||
session.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
export * as SessionRunner from "./index"
|
||||
|
||||
import type { LLMError } from "@opencode-ai/llm"
|
||||
import { Context, Effect, Schema } from "effect"
|
||||
import { Context, Effect } from "effect"
|
||||
import { SessionSchema } from "../schema"
|
||||
import type { ContextSnapshotDecodeError, MessageDecodeError } from "../error"
|
||||
import { SessionRunnerModel } from "./model"
|
||||
@@ -9,20 +9,11 @@ import type { SystemContext } from "../../system-context/index"
|
||||
import type { SessionContextEpoch } from "../context-epoch"
|
||||
import type { ToolOutputStore } from "../../tool-output-store"
|
||||
|
||||
export class StepLimitExceededError extends Schema.TaggedErrorClass<StepLimitExceededError>()(
|
||||
"SessionRunner.StepLimitExceededError",
|
||||
{
|
||||
sessionID: SessionSchema.ID,
|
||||
limit: Schema.Int,
|
||||
},
|
||||
) {}
|
||||
|
||||
export type RunError =
|
||||
| LLMError
|
||||
| SessionRunnerModel.Error
|
||||
| MessageDecodeError
|
||||
| ContextSnapshotDecodeError
|
||||
| StepLimitExceededError
|
||||
| SystemContext.InitializationBlocked
|
||||
| SessionContextEpoch.AgentReplacementBlocked
|
||||
| ToolOutputStore.Error
|
||||
|
||||
@@ -3,6 +3,7 @@ import {
|
||||
LLMClient,
|
||||
LLMError,
|
||||
LLMEvent,
|
||||
Message,
|
||||
SystemPart,
|
||||
isContextOverflowFailure,
|
||||
type ProviderErrorEvent,
|
||||
@@ -29,10 +30,11 @@ import { SessionHistory } from "../history"
|
||||
import { SessionInput } from "../input"
|
||||
import { SessionSchema } from "../schema"
|
||||
import { SessionStore } from "../store"
|
||||
import { type RunError, Service, StepLimitExceededError } from "./index"
|
||||
import { type RunError, Service } from "./index"
|
||||
import { SessionRunnerModel } from "./model"
|
||||
import { createLLMEventPublisher } from "./publish-llm-event"
|
||||
import { toLLMMessages } from "./to-llm-message"
|
||||
import { MAX_STEPS_PROMPT } from "./max-steps"
|
||||
|
||||
/**
|
||||
* Runs one durable coding-agent Session until it settles.
|
||||
@@ -45,7 +47,7 @@ import { toLLMMessages } from "./to-llm-message"
|
||||
* - [ ] Replace local ownership with durable multi-node ownership when clustered.
|
||||
* - [ ] Mark busy, retrying, idle, interrupted, or terminal-failure status durably.
|
||||
* - [ ] Honor interruption and reject stale work after runtime attachment replacement.
|
||||
* - [x] Bound model steps.
|
||||
* - [x] Honor optional agent step limits.
|
||||
* - [ ] Bound provider retries and repeated identical tool calls.
|
||||
*
|
||||
* - Runtime context assembly
|
||||
@@ -80,13 +82,10 @@ import { toLLMMessages } from "./to-llm-message"
|
||||
* Durable activity recovery remains a separate future slice with an explicit retry policy.
|
||||
*
|
||||
* The current slice loads V2 history, translates it, resolves a model through a core service, and persists one
|
||||
* provider turn. Registry definitions are advertised, local tool calls are settled durably, and a
|
||||
* bounded explicit loop starts the next provider turn after local settlement.
|
||||
* provider turn. Registry definitions are advertised, local tool calls are settled durably, and an
|
||||
* explicit loop starts the next provider turn after local settlement. Configured agent step limits bound the loop.
|
||||
*/
|
||||
|
||||
// QUESTION: Did this exist previously, or did we add this limit? Does it make sense?
|
||||
const MAX_STEPS = 25
|
||||
|
||||
export const layer = Layer.effect(
|
||||
Service,
|
||||
Effect.gen(function* () {
|
||||
@@ -175,6 +174,7 @@ export const layer = Layer.effect(
|
||||
const runTurnAttempt = Effect.fn("SessionRunner.runTurn")(function* (
|
||||
sessionID: SessionSchema.ID,
|
||||
promotion: SessionInput.Delivery | undefined,
|
||||
step: number,
|
||||
recoverOverflow?: typeof compaction.compactAfterOverflow,
|
||||
) {
|
||||
const session = yield* getSession(sessionID)
|
||||
@@ -214,7 +214,8 @@ export const layer = Layer.effect(
|
||||
const model = yield* models.resolve(session)
|
||||
const entries = yield* SessionHistory.entriesForRunner(db, session.id, system.baselineSeq)
|
||||
const context = entries.map((entry) => entry.message)
|
||||
const toolMaterialization = yield* tools.materialize(agent.info?.permissions)
|
||||
const isLastStep = agent.info?.steps !== undefined && step >= agent.info.steps
|
||||
const toolMaterialization = isLastStep ? undefined : yield* tools.materialize(agent.info?.permissions)
|
||||
const promptCacheKey = /^ses_[0-9a-f]{64}$/.test(session.id) ? session.id.slice(4) : session.id
|
||||
const request = LLM.request({
|
||||
model,
|
||||
@@ -222,8 +223,9 @@ export const layer = Layer.effect(
|
||||
system: [agent.info?.system, system.baseline]
|
||||
.filter((part): part is string => part !== undefined && part.length > 0)
|
||||
.map(SystemPart.make),
|
||||
messages: toLLMMessages(context, model),
|
||||
tools: toolMaterialization.definitions,
|
||||
messages: [...toLLMMessages(context, model), ...(isLastStep ? [Message.assistant(MAX_STEPS_PROMPT)] : [])],
|
||||
tools: toolMaterialization?.definitions ?? [],
|
||||
toolChoice: isLastStep ? "none" : undefined,
|
||||
})
|
||||
if (yield* compaction.compactIfNeeded({ sessionID: session.id, entries, model, request }))
|
||||
return yield* Effect.die(rebuildPreparedTurn())
|
||||
@@ -254,6 +256,10 @@ export const layer = Layer.effect(
|
||||
}
|
||||
yield* publish(event)
|
||||
if (event.type !== "tool-call" || event.providerExecuted) return
|
||||
if (!toolMaterialization) {
|
||||
yield* withPublication(publisher.failUnsettledTools("Tools are disabled after the maximum agent steps"))
|
||||
return
|
||||
}
|
||||
needsContinuation = true
|
||||
const assistantMessageID = yield* publisher.assistantMessageID(event.id)
|
||||
yield* Effect.uninterruptibleMask((restore) =>
|
||||
@@ -340,31 +346,32 @@ export const layer = Layer.effect(
|
||||
type RunTurn = (
|
||||
sessionID: SessionSchema.ID,
|
||||
promotion: SessionInput.Delivery | undefined,
|
||||
step: number,
|
||||
) => Effect.Effect<boolean, RunError>
|
||||
|
||||
const runAfterOverflowCompaction: RunTurn = Effect.fnUntraced(function* (sessionID, promotion) {
|
||||
return yield* runTurnAttempt(sessionID, promotion).pipe(
|
||||
const runAfterOverflowCompaction: RunTurn = Effect.fnUntraced(function* (sessionID, promotion, step) {
|
||||
return yield* runTurnAttempt(sessionID, promotion, step).pipe(
|
||||
Effect.catchDefect(
|
||||
Effect.fnUntraced(function* (defect) {
|
||||
if (!(defect instanceof TurnTransitionError)) return yield* Effect.die(defect)
|
||||
if (defect.transition._tag === "ContinueAfterOverflowCompaction")
|
||||
return yield* Effect.die("Post-compaction provider attempt cannot recover another overflow")
|
||||
yield* Effect.yieldNow
|
||||
return yield* runAfterOverflowCompaction(sessionID, defect.transition.promotion)
|
||||
return yield* runAfterOverflowCompaction(sessionID, defect.transition.promotion, step)
|
||||
}),
|
||||
),
|
||||
)
|
||||
})
|
||||
|
||||
const runTurn: RunTurn = Effect.fnUntraced(function* (sessionID, promotion) {
|
||||
return yield* runTurnAttempt(sessionID, promotion, compaction.compactAfterOverflow).pipe(
|
||||
const runTurn: RunTurn = Effect.fnUntraced(function* (sessionID, promotion, step) {
|
||||
return yield* runTurnAttempt(sessionID, promotion, step, compaction.compactAfterOverflow).pipe(
|
||||
Effect.catchDefect(
|
||||
Effect.fnUntraced(function* (defect) {
|
||||
if (!(defect instanceof TurnTransitionError)) return yield* Effect.die(defect)
|
||||
yield* Effect.yieldNow
|
||||
if (defect.transition._tag === "ContinueAfterOverflowCompaction")
|
||||
return yield* runAfterOverflowCompaction(sessionID, undefined)
|
||||
return yield* runTurn(sessionID, defect.transition.promotion)
|
||||
return yield* runAfterOverflowCompaction(sessionID, undefined, step)
|
||||
return yield* runTurn(sessionID, defect.transition.promotion, step)
|
||||
}),
|
||||
),
|
||||
)
|
||||
@@ -382,14 +389,11 @@ export const layer = Layer.effect(
|
||||
let openActivity = input.force === true || hasSteer || hasQueue
|
||||
while (openActivity) {
|
||||
let needsContinuation = true
|
||||
for (let step = 0; step < MAX_STEPS; step++) {
|
||||
needsContinuation = yield* runTurn(input.sessionID, promotion)
|
||||
for (let step = 1; needsContinuation; step++) {
|
||||
needsContinuation = yield* runTurn(input.sessionID, promotion, step)
|
||||
promotion = "steer"
|
||||
if (!needsContinuation) needsContinuation = yield* SessionInput.hasPending(db, input.sessionID, "steer")
|
||||
if (!needsContinuation) break
|
||||
}
|
||||
if (needsContinuation)
|
||||
return yield* new StepLimitExceededError({ sessionID: input.sessionID, limit: MAX_STEPS })
|
||||
openActivity = yield* SessionInput.hasPending(db, input.sessionID, "queue")
|
||||
promotion = openActivity ? "queue" : undefined
|
||||
}
|
||||
|
||||
+2
-2
@@ -1,4 +1,4 @@
|
||||
CRITICAL - MAXIMUM STEPS REACHED
|
||||
export const MAX_STEPS_PROMPT = `CRITICAL - MAXIMUM STEPS REACHED
|
||||
|
||||
The maximum number of steps allowed for this task has been reached. Tools are disabled until next user input. Respond with text only.
|
||||
|
||||
@@ -13,4 +13,4 @@ Response must include:
|
||||
- List of any remaining tasks that were not completed
|
||||
- Recommendations for what should be done next
|
||||
|
||||
Any attempt to use tools is a critical violation. Respond with text ONLY.
|
||||
Any attempt to use tools is a critical violation. Respond with text ONLY.`
|
||||
@@ -1461,6 +1461,7 @@ describe("SessionRunnerLLM", () => {
|
||||
})
|
||||
|
||||
requests.length = 0
|
||||
executions.length = 0
|
||||
responses = [
|
||||
fragmentFixture("text", "text-summary-2", ["## Goal\n- Preserve the updated task"]).completeEvents,
|
||||
fragmentFixture("text", "text-final-2", ["Continued again"]).completeEvents,
|
||||
@@ -3177,7 +3178,7 @@ describe("SessionRunnerLLM", () => {
|
||||
}),
|
||||
)
|
||||
|
||||
it.effect("fails after the bounded number of local tool continuation steps", () =>
|
||||
it.effect("continues past 25 local tool steps when the agent has no step limit", () =>
|
||||
Effect.gen(function* () {
|
||||
yield* setup
|
||||
const session = yield* SessionV2.Service
|
||||
@@ -3188,62 +3189,10 @@ describe("SessionRunnerLLM", () => {
|
||||
executions.length = 0
|
||||
streamGate = undefined
|
||||
streamStarted = undefined
|
||||
responses = Array.from({ length: 25 }, (_, index) => [
|
||||
LLMEvent.stepStart({ index: 0 }),
|
||||
LLMEvent.toolCall({ id: `call-echo-${index}`, name: "echo", input: { text: `${index}` } }),
|
||||
LLMEvent.stepFinish({ index: 0, reason: "tool-calls" }),
|
||||
LLMEvent.finish({ reason: "tool-calls" }),
|
||||
])
|
||||
|
||||
const failure = yield* session.resume(sessionID).pipe(Effect.flip)
|
||||
|
||||
expect(failure).toMatchObject({ _tag: "SessionRunner.StepLimitExceededError", sessionID, limit: 25 })
|
||||
expect(requests).toHaveLength(25)
|
||||
expect(executions).toHaveLength(25)
|
||||
}),
|
||||
)
|
||||
|
||||
it.effect("does not restart a capped tool loop for a coalesced stale wake", () =>
|
||||
Effect.gen(function* () {
|
||||
yield* setup
|
||||
const session = yield* SessionV2.Service
|
||||
const coordinator = yield* SessionRunCoordinator.Service
|
||||
yield* session.prompt({ sessionID, prompt: new Prompt({ text: "Loop forever" }), resume: false })
|
||||
|
||||
requests.length = 0
|
||||
responses = Array.from({ length: 25 }, (_, index) => [
|
||||
LLMEvent.stepStart({ index: 0 }),
|
||||
LLMEvent.toolCall({ id: `call-capped-${index}`, name: "echo", input: { text: `${index}` } }),
|
||||
LLMEvent.stepFinish({ index: 0, reason: "tool-calls" }),
|
||||
LLMEvent.finish({ reason: "tool-calls" }),
|
||||
])
|
||||
streamGate = yield* Deferred.make<void>()
|
||||
streamStarted = yield* Deferred.make<void>()
|
||||
|
||||
const run = yield* session.resume(sessionID).pipe(Effect.forkChild)
|
||||
yield* Deferred.await(streamStarted)
|
||||
yield* coordinator.wake(sessionID)
|
||||
yield* Deferred.succeed(streamGate, undefined)
|
||||
expect(yield* Fiber.join(run).pipe(Effect.flip)).toMatchObject({ _tag: "SessionRunner.StepLimitExceededError" })
|
||||
streamGate = undefined
|
||||
streamStarted = undefined
|
||||
yield* Effect.yieldNow
|
||||
|
||||
expect(requests).toHaveLength(25)
|
||||
}),
|
||||
)
|
||||
|
||||
it.effect("accepts a terminal response on the final bounded provider turn", () =>
|
||||
Effect.gen(function* () {
|
||||
yield* setup
|
||||
const session = yield* SessionV2.Service
|
||||
yield* session.prompt({ sessionID, prompt: new Prompt({ text: "Finish at the limit" }), resume: false })
|
||||
|
||||
requests.length = 0
|
||||
responses = [
|
||||
...Array.from({ length: 24 }, (_, index) => [
|
||||
...Array.from({ length: 25 }, (_, index) => [
|
||||
LLMEvent.stepStart({ index: 0 }),
|
||||
LLMEvent.toolCall({ id: `call-terminal-${index}`, name: "echo", input: { text: `${index}` } }),
|
||||
LLMEvent.toolCall({ id: `call-echo-${index}`, name: "echo", input: { text: `${index}` } }),
|
||||
LLMEvent.stepFinish({ index: 0, reason: "tool-calls" }),
|
||||
LLMEvent.finish({ reason: "tool-calls" }),
|
||||
]),
|
||||
@@ -3256,7 +3205,56 @@ describe("SessionRunnerLLM", () => {
|
||||
|
||||
yield* session.resume(sessionID)
|
||||
|
||||
expect(requests).toHaveLength(25)
|
||||
expect(requests).toHaveLength(26)
|
||||
expect(executions).toHaveLength(25)
|
||||
}),
|
||||
)
|
||||
|
||||
it.effect("forces a text response on an agent's configured final step", () =>
|
||||
Effect.gen(function* () {
|
||||
yield* setup
|
||||
const agents = yield* AgentV2.Service
|
||||
yield* agents.update((editor) =>
|
||||
editor.update(AgentV2.ID.make("build"), (agent) => {
|
||||
agent.steps = 2
|
||||
}),
|
||||
)
|
||||
const session = yield* SessionV2.Service
|
||||
yield* session.prompt({ sessionID, prompt: new Prompt({ text: "Finish at the limit" }), resume: false })
|
||||
|
||||
requests.length = 0
|
||||
executions.length = 0
|
||||
responses = [
|
||||
[
|
||||
LLMEvent.stepStart({ index: 0 }),
|
||||
LLMEvent.toolCall({ id: "call-terminal", name: "echo", input: { text: "done" } }),
|
||||
LLMEvent.stepFinish({ index: 0, reason: "tool-calls" }),
|
||||
LLMEvent.finish({ reason: "tool-calls" }),
|
||||
],
|
||||
[
|
||||
LLMEvent.stepStart({ index: 0 }),
|
||||
LLMEvent.toolCall({ id: "call-forbidden", name: "echo", input: { text: "forbidden" } }),
|
||||
LLMEvent.stepFinish({ index: 0, reason: "tool-calls" }),
|
||||
LLMEvent.finish({ reason: "tool-calls" }),
|
||||
],
|
||||
]
|
||||
|
||||
yield* session.resume(sessionID)
|
||||
|
||||
expect(requests).toHaveLength(2)
|
||||
expect(requests[0]?.toolChoice).toBeUndefined()
|
||||
expect(requests[1]?.toolChoice).toMatchObject({ type: "none" })
|
||||
expect(requests[1]?.tools).toEqual([])
|
||||
expect(requests[1]?.messages.at(-1)).toMatchObject({
|
||||
role: "assistant",
|
||||
content: [{ type: "text", text: expect.stringContaining("MAXIMUM STEPS REACHED") }],
|
||||
})
|
||||
expect(executions).toEqual(["done"])
|
||||
expect(yield* session.context(sessionID)).toMatchObject([
|
||||
{ type: "user", text: "Finish at the limit" },
|
||||
{ type: "assistant", content: [{ type: "tool", id: "call-terminal", state: { status: "completed" } }] },
|
||||
{ type: "assistant", content: [{ type: "tool", id: "call-forbidden", state: { status: "error" } }] },
|
||||
])
|
||||
}),
|
||||
)
|
||||
|
||||
|
||||
@@ -412,6 +412,9 @@ const fromRequest = Effect.fn("BedrockConverse.fromRequest")(function* (request:
|
||||
stopSequences: generation?.stop,
|
||||
},
|
||||
toolConfig,
|
||||
// Converse's base inferenceConfig has no topK; Anthropic/Nova accept it
|
||||
// as a model-specific field, so it goes through additionalModelRequestFields.
|
||||
additionalModelRequestFields: generation?.topK === undefined ? undefined : { top_k: generation.topK },
|
||||
}
|
||||
})
|
||||
|
||||
|
||||
@@ -83,6 +83,26 @@ describe("Bedrock Converse route", () => {
|
||||
}),
|
||||
)
|
||||
|
||||
it.effect("passes topK through additionalModelRequestFields as top_k", () =>
|
||||
Effect.gen(function* () {
|
||||
const prepared = yield* LLMClient.prepare<BedrockConverse.BedrockConverseBody>(
|
||||
LLM.updateRequest(baseRequest, { generation: { maxTokens: 64, temperature: 0, topK: 40 } }),
|
||||
)
|
||||
|
||||
// Converse's inferenceConfig has no topK; Anthropic/Nova read it from
|
||||
// additionalModelRequestFields as top_k.
|
||||
expect(prepared.body.inferenceConfig).toEqual({ maxTokens: 64, temperature: 0 })
|
||||
expect(prepared.body.additionalModelRequestFields).toEqual({ top_k: 40 })
|
||||
}),
|
||||
)
|
||||
|
||||
it.effect("omits additionalModelRequestFields when topK is unset", () =>
|
||||
Effect.gen(function* () {
|
||||
const prepared = yield* LLMClient.prepare<BedrockConverse.BedrockConverseBody>(baseRequest)
|
||||
expect(prepared.body.additionalModelRequestFields).toBeUndefined()
|
||||
}),
|
||||
)
|
||||
|
||||
it.effect("lowers chronological system updates to wrapped user text in order", () =>
|
||||
Effect.gen(function* () {
|
||||
const prepared = yield* LLMClient.prepare<BedrockConverse.BedrockConverseBody>(
|
||||
|
||||
@@ -218,7 +218,7 @@ function normalizeMessages(
|
||||
if (
|
||||
model.providerID === "mistral" ||
|
||||
model.api.id.toLowerCase().includes("mistral") ||
|
||||
model.api.id.toLocaleLowerCase().includes("devstral")
|
||||
model.api.id.toLowerCase().includes("devstral")
|
||||
) {
|
||||
const scrub = (id: string) => {
|
||||
return id
|
||||
@@ -666,6 +666,9 @@ export function variants(model: Provider.Model): Record<string, Record<string, a
|
||||
if (!model.capabilities.reasoning) return {}
|
||||
|
||||
const id = model.id.toLowerCase()
|
||||
const glm52 = ["glm-5.2", "glm-5-2", "glm-5p2"].some(
|
||||
(name) => id.includes(name) || model.api.id.toLowerCase().includes(name),
|
||||
)
|
||||
if (
|
||||
model.api.id.toLowerCase().includes("minimax-m3") &&
|
||||
["@ai-sdk/anthropic", "@ai-sdk/openai-compatible"].includes(model.api.npm)
|
||||
@@ -677,13 +680,32 @@ export function variants(model: Provider.Model): Record<string, Record<string, a
|
||||
}
|
||||
const adaptiveThinkingOmitted = anthropicOmitsThinking(model.api.id)
|
||||
const adaptiveEfforts = anthropicAdaptiveEfforts(model.api.id)
|
||||
if (glm52 && model.api.npm === "@openrouter/ai-sdk-provider") {
|
||||
// OpenRouter maps xhigh to GLM-5.2's native max effort.
|
||||
return {
|
||||
high: { reasoning: { effort: "high" } },
|
||||
xhigh: { reasoning: { effort: "xhigh" } },
|
||||
}
|
||||
}
|
||||
if (glm52 && model.api.npm === "@ai-sdk/openai-compatible") {
|
||||
return {
|
||||
high: { reasoningEffort: "high" },
|
||||
max: { reasoningEffort: "max" },
|
||||
}
|
||||
}
|
||||
if (glm52 && model.api.npm === "@ai-sdk/anthropic") {
|
||||
return {
|
||||
high: { effort: "high" },
|
||||
max: { effort: "max" },
|
||||
}
|
||||
}
|
||||
if (
|
||||
id.includes("deepseek-chat") ||
|
||||
id.includes("deepseek-reasoner") ||
|
||||
id.includes("deepseek-r1") ||
|
||||
id.includes("deepseek-v3") ||
|
||||
id.includes("minimax") ||
|
||||
id.includes("glm") ||
|
||||
(id.includes("glm") && !glm52) ||
|
||||
id.includes("kimi") ||
|
||||
id.includes("k2p") ||
|
||||
id.includes("qwen") ||
|
||||
|
||||
@@ -16,7 +16,7 @@ import { SessionCompaction } from "./compaction"
|
||||
import { SystemPrompt } from "./system"
|
||||
import { Instruction } from "./instruction"
|
||||
import { Plugin } from "../plugin"
|
||||
import MAX_STEPS from "../session/prompt/max-steps.txt"
|
||||
import { MAX_STEPS_PROMPT } from "@opencode-ai/core/session/runner/max-steps"
|
||||
import { ToolRegistry } from "@/tool/registry"
|
||||
import { MCP } from "../mcp"
|
||||
import { LSP } from "@/lsp/lsp"
|
||||
@@ -1304,24 +1304,6 @@ export const layer = Layer.effect(
|
||||
if (step === 1)
|
||||
yield* summary.summarize({ sessionID, messageID: lastUser.id }).pipe(Effect.ignore, Effect.forkIn(scope))
|
||||
|
||||
if (step > 1 && lastFinished) {
|
||||
for (const m of msgs) {
|
||||
if (m.info.role !== "user" || m.info.id <= lastFinished.id) continue
|
||||
for (const p of m.parts) {
|
||||
if (p.type !== "text" || p.ignored || p.synthetic) continue
|
||||
if (!p.text.trim()) continue
|
||||
p.text = [
|
||||
"<system-reminder>",
|
||||
"The user sent the following message:",
|
||||
p.text,
|
||||
"",
|
||||
"Please address this message and continue with your tasks.",
|
||||
"</system-reminder>",
|
||||
].join("\n")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
yield* plugin.trigger("experimental.chat.messages.transform", {}, { messages: msgs })
|
||||
|
||||
const [skills, env, instructions, modelMsgs] = yield* Effect.all([
|
||||
@@ -1340,7 +1322,10 @@ export const layer = Layer.effect(
|
||||
sessionID,
|
||||
parentSessionID: session.parentID,
|
||||
system,
|
||||
messages: [...modelMsgs, ...(isLastStep ? [{ role: "assistant" as const, content: MAX_STEPS }] : [])],
|
||||
messages: [
|
||||
...modelMsgs,
|
||||
...(isLastStep ? [{ role: "assistant" as const, content: MAX_STEPS_PROMPT }] : []),
|
||||
],
|
||||
tools,
|
||||
model,
|
||||
toolChoice: format.type === "json_schema" ? "required" : undefined,
|
||||
|
||||
@@ -2895,6 +2895,102 @@ describe("ProviderTransform.variants", () => {
|
||||
expect(result).toEqual({})
|
||||
})
|
||||
|
||||
test("glm-5.2 returns native effort variants for openai-compatible providers", () => {
|
||||
const model = createMockModel({
|
||||
id: "zhipuai/glm-5.2",
|
||||
providerID: "zhipuai",
|
||||
api: {
|
||||
id: "glm-5.2",
|
||||
url: "https://open.bigmodel.cn/api/paas/v4",
|
||||
npm: "@ai-sdk/openai-compatible",
|
||||
},
|
||||
})
|
||||
expect(ProviderTransform.variants(model)).toEqual({
|
||||
high: { reasoningEffort: "high" },
|
||||
max: { reasoningEffort: "max" },
|
||||
})
|
||||
})
|
||||
|
||||
test("recognizes GLM-5.2 provider model IDs", () => {
|
||||
for (const id of ["accounts/fireworks/models/glm-5p2", "zai-org-glm-5-2", "umans-glm-5.2"]) {
|
||||
const model = createMockModel({
|
||||
id: `test/${id}`,
|
||||
api: {
|
||||
id,
|
||||
url: "https://api.test.com",
|
||||
npm: "@ai-sdk/openai-compatible",
|
||||
},
|
||||
})
|
||||
expect(ProviderTransform.variants(model)).toEqual({
|
||||
high: { reasoningEffort: "high" },
|
||||
max: { reasoningEffort: "max" },
|
||||
})
|
||||
}
|
||||
})
|
||||
|
||||
test("recognizes GLM-5.2 from the API ID when the configured model ID is an alias", () => {
|
||||
const model = createMockModel({
|
||||
id: "custom/my-glm",
|
||||
api: {
|
||||
id: "accounts/fireworks/models/glm-5p2",
|
||||
url: "https://api.fireworks.ai/inference/v1",
|
||||
npm: "@ai-sdk/openai-compatible",
|
||||
},
|
||||
})
|
||||
expect(ProviderTransform.variants(model)).toEqual({
|
||||
high: { reasoningEffort: "high" },
|
||||
max: { reasoningEffort: "max" },
|
||||
})
|
||||
})
|
||||
|
||||
test("glm-5.2 returns openrouter effort variants for openrouter", () => {
|
||||
const model = createMockModel({
|
||||
id: "openrouter/z-ai/glm-5.2",
|
||||
providerID: "openrouter",
|
||||
api: {
|
||||
id: "z-ai/glm-5.2",
|
||||
url: "https://openrouter.ai/api/v1",
|
||||
npm: "@openrouter/ai-sdk-provider",
|
||||
},
|
||||
})
|
||||
expect(ProviderTransform.variants(model)).toEqual({
|
||||
high: { reasoning: { effort: "high" } },
|
||||
xhigh: { reasoning: { effort: "xhigh" } },
|
||||
})
|
||||
})
|
||||
|
||||
test("glm-5.2 returns effort variants for anthropic-compatible providers", () => {
|
||||
const model = createMockModel({
|
||||
id: "zai-coding-plan/glm-5.2",
|
||||
providerID: "zai-coding-plan",
|
||||
api: {
|
||||
id: "glm-5.2",
|
||||
url: "https://api.z.ai/api/anthropic",
|
||||
npm: "@ai-sdk/anthropic",
|
||||
},
|
||||
})
|
||||
expect(ProviderTransform.variants(model)).toEqual({
|
||||
high: { effort: "high" },
|
||||
max: { effort: "max" },
|
||||
})
|
||||
})
|
||||
|
||||
test("glm-5.2 falls back to provider defaults for other packages", () => {
|
||||
const model = createMockModel({
|
||||
id: "test/glm-5.2",
|
||||
api: {
|
||||
id: "glm-5.2",
|
||||
url: "https://api.test.com",
|
||||
npm: "@ai-sdk/amazon-bedrock",
|
||||
},
|
||||
})
|
||||
expect(ProviderTransform.variants(model)).toEqual({
|
||||
low: { reasoningConfig: { type: "enabled", maxReasoningEffort: "low" } },
|
||||
medium: { reasoningConfig: { type: "enabled", maxReasoningEffort: "medium" } },
|
||||
high: { reasoningConfig: { type: "enabled", maxReasoningEffort: "high" } },
|
||||
})
|
||||
})
|
||||
|
||||
test("mistral models with reasoning support return variants", () => {
|
||||
const model = createMockModel({
|
||||
id: "mistral/mistral-small-latest",
|
||||
|
||||
@@ -1346,7 +1346,9 @@ it.instance(
|
||||
|
||||
const inputs = yield* llm.inputs
|
||||
expect(inputs).toHaveLength(2)
|
||||
expect(JSON.stringify(inputs.at(-1)?.messages)).toContain("second")
|
||||
const messages = inputs.at(-1)?.messages
|
||||
if (!Array.isArray(messages)) throw new Error("expected LLM messages")
|
||||
expect(messages.at(-1)).toEqual({ role: "user", content: "second" })
|
||||
}),
|
||||
3_000,
|
||||
)
|
||||
|
||||
@@ -2,6 +2,7 @@ import { PermissionV1 } from "@opencode-ai/core/v1/permission"
|
||||
import { describe, expect } from "bun:test"
|
||||
import path from "path"
|
||||
import { Effect } from "effect"
|
||||
import { LayerNode } from "@opencode-ai/core/effect/layer-node"
|
||||
import { CrossSpawnSpawner } from "@opencode-ai/core/cross-spawn-spawner"
|
||||
import type { Tool } from "@/tool/tool"
|
||||
import { assertExternalDirectoryEffect } from "../../src/tool/external-directory"
|
||||
@@ -11,7 +12,7 @@ import type { Permission } from "../../src/permission"
|
||||
import { SessionID, MessageID } from "../../src/session/schema"
|
||||
import { testEffect } from "../lib/effect"
|
||||
|
||||
const it = testEffect(CrossSpawnSpawner.defaultLayer)
|
||||
const it = testEffect(LayerNode.buildLayer(CrossSpawnSpawner.node))
|
||||
|
||||
const baseCtx: Omit<Tool.Context, "ask"> = {
|
||||
sessionID: SessionID.make("ses_test"),
|
||||
|
||||
@@ -7,8 +7,11 @@ import "./app.css"
|
||||
function AppMeta() {
|
||||
return (
|
||||
<>
|
||||
<Title>OpenCode Data</Title>
|
||||
<Meta name="description" content="OpenCode usage data, market share, token cost, and session cost." />
|
||||
<Title>AI Model Usage Rankings | OpenCode Data</Title>
|
||||
<Meta
|
||||
name="description"
|
||||
content="Explore OpenCode Go usage across AI models, including token volume, rankings, market share, token pricing, session cost, cache ratio, and geo breakdowns."
|
||||
/>
|
||||
</>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -45,6 +45,7 @@ const statsUnfurlUrl = new URL(statsUnfurlPath, statsCanonicalBaseUrl).toString(
|
||||
const modelHeaderLinks: readonly HeaderLink[] = [
|
||||
{ href: "#overview", label: "Overview" },
|
||||
{ href: "#usage", label: "Usage" },
|
||||
{ href: "#users", label: "Users" },
|
||||
{ href: "#efficiency", label: "Efficiency" },
|
||||
{ href: "#geo-breakdown", label: "Geo Breakdown" },
|
||||
{ href: "#peers", label: "Peers" },
|
||||
@@ -120,11 +121,10 @@ export default function StatsModel() {
|
||||
const [themePreference, setThemePreference] = createSignal<ThemePreference>("system")
|
||||
const modelName = createMemo(() => catalogEntry()?.name ?? stats()?.model ?? modelParam() ?? "Model")
|
||||
const labName = createMemo(() => formatCatalogLabName(catalogEntry()?.lab ?? stats()?.provider ?? labParam()))
|
||||
const modelTitle = createMemo(() => `${modelName()} Data`)
|
||||
const modelDescription = createMemo(() =>
|
||||
stats()
|
||||
? `${modelName()} usage, rank, token mix, cost, geo breakdown, and peer data across OpenCode Go.`
|
||||
: `${modelName()} model facts, limits, and OpenCode Go usage availability.`,
|
||||
const modelTitle = createMemo(() => `${modelName()} Usage, Cost & Rank | OpenCode Data`)
|
||||
const modelDescription = createMemo(
|
||||
() =>
|
||||
`View ${modelName()} OpenCode Go usage data, including token volume, weekly rank, token mix, costs, cache ratio, sessions, geo breakdowns, and peer models.`,
|
||||
)
|
||||
const modelUrl = createMemo(() =>
|
||||
new URL(
|
||||
@@ -176,6 +176,7 @@ export default function StatsModel() {
|
||||
<ModelHero data={stats() ?? null} catalog={catalogEntry() ?? null} labName={labName()} />
|
||||
<ModelOverview data={stats() ?? null} />
|
||||
<ModelUsageSection data={stats()?.usage ?? []} />
|
||||
<ModelUsersSection data={stats()?.usage ?? []} />
|
||||
<ModelEfficiencySection data={stats() ?? null} catalog={catalogEntry() ?? null} />
|
||||
<ModelGeoBreakdownSection data={stats()?.country ?? emptyCountryRecord()} />
|
||||
<ModelPeersSection data={stats() ?? null} />
|
||||
@@ -359,14 +360,6 @@ function ModelOverview(props: { data: StatsModelData | null }) {
|
||||
}
|
||||
|
||||
function ModelUsageSection(props: { data: ModelUsagePoint[] }) {
|
||||
const [activeIndex, setActiveIndex] = createSignal<number>()
|
||||
const max = createMemo(() => Math.max(0, ...props.data.map((item) => item.tokens)) || 1)
|
||||
const activePoint = createMemo(() => {
|
||||
const index = activeIndex()
|
||||
if (index === undefined) return undefined
|
||||
return props.data[index]
|
||||
})
|
||||
|
||||
return (
|
||||
<section id="usage" data-section="model-panel">
|
||||
<SectionTitle title="Usage" description="Daily OpenCode Go token volume over the recent two-month window." />
|
||||
@@ -374,92 +367,147 @@ function ModelUsageSection(props: { data: ModelUsagePoint[] }) {
|
||||
when={props.data.some((item) => item.tokens > 0)}
|
||||
fallback={<ModelEmptyState title="No usage" description="No usage landed in the current window." />}
|
||||
>
|
||||
<div
|
||||
data-component="model-usage-chart"
|
||||
data-dense-labels={isModelUsageDense(props.data.length) ? "true" : undefined}
|
||||
role="img"
|
||||
aria-label="Daily token usage chart"
|
||||
style={{ "--model-usage-count": props.data.length } as JSX.CSSProperties}
|
||||
onPointerLeave={(event) => {
|
||||
if (event.pointerType === "touch") return
|
||||
setActiveIndex(undefined)
|
||||
}}
|
||||
>
|
||||
<div data-slot="model-usage-axis" aria-hidden="true">
|
||||
<For each={props.data}>
|
||||
{(point, index) => (
|
||||
<div
|
||||
data-active={activeIndex() === index() ? "true" : undefined}
|
||||
data-label-hidden={isModelUsageLabelHidden(index(), props.data.length) ? "true" : undefined}
|
||||
>
|
||||
<span data-slot="model-usage-label">
|
||||
<span data-slot="model-usage-total">{formatTokens(point.tokens)}</span>
|
||||
<span data-slot="model-usage-date">{point.date}</span>
|
||||
</span>
|
||||
</div>
|
||||
)}
|
||||
</For>
|
||||
</div>
|
||||
<div data-slot="model-usage-bars">
|
||||
<For each={props.data}>
|
||||
{(point, index) => (
|
||||
<div
|
||||
data-slot="model-usage-column"
|
||||
role="button"
|
||||
tabIndex={0}
|
||||
aria-label={`${point.date} ${formatTokens(point.tokens)} tokens`}
|
||||
data-active={activeIndex() === index() ? "true" : undefined}
|
||||
data-muted={activeIndex() !== undefined && activeIndex() !== index() ? "true" : undefined}
|
||||
onPointerDown={(event) => {
|
||||
if (event.pointerType !== "touch") return
|
||||
setActiveIndex(index())
|
||||
}}
|
||||
onPointerEnter={() => setActiveIndex(index())}
|
||||
onPointerMove={(event) => {
|
||||
if (event.pointerType === "touch") return
|
||||
setActiveIndex(index())
|
||||
}}
|
||||
onClick={() => setActiveIndex(index())}
|
||||
onFocus={() => setActiveIndex(index())}
|
||||
onBlur={() => setActiveIndex(undefined)}
|
||||
onKeyDown={(event) => {
|
||||
if (event.key !== "Enter" && event.key !== " ") return
|
||||
event.preventDefault()
|
||||
setActiveIndex(index())
|
||||
}}
|
||||
>
|
||||
<div
|
||||
data-slot="model-usage-bar"
|
||||
style={{ "--model-usage-fill": `${modelUsageHeight(point.tokens, max())}%` } as JSX.CSSProperties}
|
||||
/>
|
||||
<Show when={activeIndex() === index() && activePoint()}>
|
||||
{(active) => (
|
||||
<div
|
||||
data-component="chart-tooltip"
|
||||
data-placement={index() > props.data.length * 0.62 ? "left" : "right"}
|
||||
>
|
||||
<strong>{active().date}</strong>
|
||||
<span>{formatTokens(active().tokens)} tokens</span>
|
||||
<div data-slot="tooltip-divider" />
|
||||
<p>
|
||||
<span data-slot="tooltip-label">
|
||||
<i /> Daily tokens
|
||||
</span>
|
||||
<b>{formatTokens(active().tokens)}</b>
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
</Show>
|
||||
</div>
|
||||
)}
|
||||
</For>
|
||||
</div>
|
||||
</div>
|
||||
<ModelColumnChart data={props.data} metric="tokens" ariaLabel="Daily token usage chart" />
|
||||
</Show>
|
||||
</section>
|
||||
)
|
||||
}
|
||||
|
||||
function ModelUsersSection(props: { data: ModelUsagePoint[] }) {
|
||||
return (
|
||||
<section id="users" data-section="model-panel">
|
||||
<SectionTitle
|
||||
title="Unique Users"
|
||||
description="Daily unique OpenCode Go users over the recent two-month window."
|
||||
/>
|
||||
<Show
|
||||
when={props.data.some((item) => item.users > 0)}
|
||||
fallback={
|
||||
<ModelEmptyState title="No user data" description="No user-bearing rows landed in the current window." />
|
||||
}
|
||||
>
|
||||
<ModelColumnChart data={props.data} metric="users" ariaLabel="Daily unique user chart" />
|
||||
</Show>
|
||||
</section>
|
||||
)
|
||||
}
|
||||
|
||||
function ModelColumnChart(props: { data: ModelUsagePoint[]; metric: "tokens" | "users"; ariaLabel: string }) {
|
||||
const [activeIndex, setActiveIndex] = createSignal<number>()
|
||||
const max = createMemo(() => Math.max(0, ...props.data.map((item) => modelUsageMetricValue(item, props.metric))) || 1)
|
||||
const activePoint = createMemo(() => {
|
||||
const index = activeIndex()
|
||||
if (index === undefined) return undefined
|
||||
return props.data[index]
|
||||
})
|
||||
|
||||
return (
|
||||
<div
|
||||
data-component="model-usage-chart"
|
||||
data-metric={props.metric}
|
||||
data-dense-labels={isModelUsageDense(props.data.length) ? "true" : undefined}
|
||||
role="img"
|
||||
aria-label={props.ariaLabel}
|
||||
style={{ "--model-usage-count": props.data.length } as JSX.CSSProperties}
|
||||
onPointerLeave={(event) => {
|
||||
if (event.pointerType === "touch") return
|
||||
setActiveIndex(undefined)
|
||||
}}
|
||||
>
|
||||
<div data-slot="model-usage-axis" aria-hidden="true">
|
||||
<For each={props.data}>
|
||||
{(point, index) => (
|
||||
<div
|
||||
data-active={activeIndex() === index() ? "true" : undefined}
|
||||
data-label-hidden={isModelUsageLabelHidden(index(), props.data.length) ? "true" : undefined}
|
||||
>
|
||||
<span data-slot="model-usage-label">
|
||||
<span data-slot="model-usage-total">{formatModelUsageValue(point, props.metric)}</span>
|
||||
<span data-slot="model-usage-date">{point.date}</span>
|
||||
</span>
|
||||
</div>
|
||||
)}
|
||||
</For>
|
||||
</div>
|
||||
<div data-slot="model-usage-bars">
|
||||
<For each={props.data}>
|
||||
{(point, index) => (
|
||||
<div
|
||||
data-slot="model-usage-column"
|
||||
role="button"
|
||||
tabIndex={0}
|
||||
aria-label={`${point.date} ${formatModelUsageValue(point, props.metric)} ${modelUsageLabel(props.metric)}`}
|
||||
data-active={activeIndex() === index() ? "true" : undefined}
|
||||
data-muted={activeIndex() !== undefined && activeIndex() !== index() ? "true" : undefined}
|
||||
onPointerDown={(event) => {
|
||||
if (event.pointerType !== "touch") return
|
||||
setActiveIndex(index())
|
||||
}}
|
||||
onPointerEnter={() => setActiveIndex(index())}
|
||||
onPointerMove={(event) => {
|
||||
if (event.pointerType === "touch") return
|
||||
setActiveIndex(index())
|
||||
}}
|
||||
onClick={() => setActiveIndex(index())}
|
||||
onFocus={() => setActiveIndex(index())}
|
||||
onBlur={() => setActiveIndex(undefined)}
|
||||
onKeyDown={(event) => {
|
||||
if (event.key !== "Enter" && event.key !== " ") return
|
||||
event.preventDefault()
|
||||
setActiveIndex(index())
|
||||
}}
|
||||
>
|
||||
<div
|
||||
data-slot="model-usage-bar"
|
||||
style={
|
||||
{
|
||||
"--model-usage-fill": `${modelUsageHeight(modelUsageMetricValue(point, props.metric), max())}%`,
|
||||
} as JSX.CSSProperties
|
||||
}
|
||||
/>
|
||||
<Show when={activeIndex() === index() && activePoint()}>
|
||||
{(active) => (
|
||||
<div
|
||||
data-component="chart-tooltip"
|
||||
data-placement={index() > props.data.length * 0.62 ? "left" : "right"}
|
||||
>
|
||||
<strong>{active().date}</strong>
|
||||
<span>
|
||||
{formatModelUsageValue(active(), props.metric)} {modelUsageLabel(props.metric)}
|
||||
</span>
|
||||
<div data-slot="tooltip-divider" />
|
||||
<p>
|
||||
<span data-slot="tooltip-label">
|
||||
<i /> Daily {modelUsageLabel(props.metric)}
|
||||
</span>
|
||||
<b>{formatModelUsageValue(active(), props.metric)}</b>
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
</Show>
|
||||
</div>
|
||||
)}
|
||||
</For>
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
function modelUsageMetricValue(point: ModelUsagePoint, metric: "tokens" | "users") {
|
||||
if (metric === "users") return point.users
|
||||
return point.tokens
|
||||
}
|
||||
|
||||
function formatModelUsageValue(point: ModelUsagePoint, metric: "tokens" | "users") {
|
||||
if (metric === "users") return formatUsers(point.users)
|
||||
return formatTokens(point.tokens)
|
||||
}
|
||||
|
||||
function modelUsageLabel(metric: "tokens" | "users") {
|
||||
if (metric === "users") return "users"
|
||||
return "tokens"
|
||||
}
|
||||
|
||||
function ModelEfficiencySection(props: { data: StatsModelData | null; catalog: ModelCatalogEntry | null }) {
|
||||
return (
|
||||
<section id="efficiency" data-section="model-panel">
|
||||
@@ -835,6 +883,12 @@ function formatInteger(value: number) {
|
||||
return new Intl.NumberFormat("en").format(value)
|
||||
}
|
||||
|
||||
function formatUsers(value: number) {
|
||||
if (value >= 1_000_000) return `${trimNumber(value / 1_000_000, value >= 10_000_000 ? 0 : 1)}M`
|
||||
if (value >= 1_000) return `${trimNumber(value / 1_000, value >= 10_000 ? 0 : 1)}K`
|
||||
return formatInteger(Math.round(value))
|
||||
}
|
||||
|
||||
function formatPercent(value: number) {
|
||||
return `${value.toFixed(value > 0 && value < 10 ? 1 : 0)}%`
|
||||
}
|
||||
|
||||
@@ -69,10 +69,10 @@ export default function StatsLab() {
|
||||
const githubStars = createAsync(() => getGitHubStars())
|
||||
const [themePreference, setThemePreference] = createSignal<ThemePreference>("system")
|
||||
const labName = createMemo(() => lab()?.name ?? formatCatalogLabName(labParam()))
|
||||
const labTitle = createMemo(() => `${labName()} Models`)
|
||||
const labTitle = createMemo(() => `${labName()} AI Model Usage & Rankings | OpenCode Data`)
|
||||
const labDescription = createMemo(
|
||||
() =>
|
||||
`Explore ${labName()} models used in OpenCode, with recent token usage, context windows, release dates, and model-specific data.`,
|
||||
`Compare ${labName()} models used in OpenCode Go, including token usage, model rankings, context windows, release dates, costs, and model-specific data.`,
|
||||
)
|
||||
const labUrl = createMemo(() => new URL(lab()?.id ?? labParam(), statsCanonicalBaseUrl).toString())
|
||||
const updateThemePreference = (preference: ThemePreference) => {
|
||||
|
||||
@@ -1546,7 +1546,7 @@
|
||||
color: var(--stats-text);
|
||||
}
|
||||
|
||||
[data-page="stats"] [data-section="top-models"] [data-component="chart-tooltip"] {
|
||||
[data-page="stats"] :is([data-section="top-models"], [data-section="unique-users"]) [data-component="chart-tooltip"] {
|
||||
top: 110px;
|
||||
box-sizing: border-box;
|
||||
display: flex;
|
||||
@@ -1564,40 +1564,59 @@
|
||||
color: var(--stats-text);
|
||||
}
|
||||
|
||||
[data-page="stats"] [data-section="top-models"] [data-component="chart-tooltip"][data-placement="right"] {
|
||||
[data-page="stats"]
|
||||
:is([data-section="top-models"], [data-section="unique-users"])
|
||||
[data-component="chart-tooltip"][data-placement="right"] {
|
||||
right: auto;
|
||||
left: calc(100% + 8px);
|
||||
}
|
||||
|
||||
[data-page="stats"] [data-section="top-models"] [data-component="chart-tooltip"][data-placement="left"] {
|
||||
[data-page="stats"]
|
||||
:is([data-section="top-models"], [data-section="unique-users"])
|
||||
[data-component="chart-tooltip"][data-placement="left"] {
|
||||
right: calc(100% + 8px);
|
||||
left: auto;
|
||||
}
|
||||
|
||||
[data-page="stats"] [data-section="top-models"] [data-component="chart-tooltip"] strong,
|
||||
[data-page="stats"] [data-section="top-models"] [data-component="chart-tooltip"] > span {
|
||||
[data-page="stats"]
|
||||
:is([data-section="top-models"], [data-section="unique-users"])
|
||||
[data-component="chart-tooltip"]
|
||||
strong,
|
||||
[data-page="stats"]
|
||||
:is([data-section="top-models"], [data-section="unique-users"])
|
||||
[data-component="chart-tooltip"]
|
||||
> span {
|
||||
display: block;
|
||||
font-size: 11px;
|
||||
line-height: 12px;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
[data-page="stats"] [data-section="top-models"] [data-component="chart-tooltip"] strong {
|
||||
[data-page="stats"]
|
||||
:is([data-section="top-models"], [data-section="unique-users"])
|
||||
[data-component="chart-tooltip"]
|
||||
strong {
|
||||
padding: 8px 8px 0;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
[data-page="stats"] [data-section="top-models"] [data-component="chart-tooltip"] > span {
|
||||
[data-page="stats"]
|
||||
:is([data-section="top-models"], [data-section="unique-users"])
|
||||
[data-component="chart-tooltip"]
|
||||
> span {
|
||||
padding: 4px 8px 8px;
|
||||
color: var(--stats-muted);
|
||||
}
|
||||
|
||||
[data-page="stats"] [data-section="top-models"] [data-component="chart-tooltip"] [data-slot="tooltip-divider"] {
|
||||
[data-page="stats"]
|
||||
:is([data-section="top-models"], [data-section="unique-users"])
|
||||
[data-component="chart-tooltip"]
|
||||
[data-slot="tooltip-divider"] {
|
||||
height: 0.5px;
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
[data-page="stats"] [data-section="top-models"] [data-component="chart-tooltip"] p {
|
||||
[data-page="stats"] :is([data-section="top-models"], [data-section="unique-users"]) [data-component="chart-tooltip"] p {
|
||||
grid-template-columns: minmax(0, 1fr) auto;
|
||||
gap: 4px;
|
||||
height: 16px;
|
||||
@@ -1608,36 +1627,50 @@
|
||||
line-height: 12px;
|
||||
}
|
||||
|
||||
[data-page="stats"] [data-section="top-models"] [data-component="chart-tooltip"] p[data-muted="true"] {
|
||||
[data-page="stats"]
|
||||
:is([data-section="top-models"], [data-section="unique-users"])
|
||||
[data-component="chart-tooltip"]
|
||||
p[data-muted="true"] {
|
||||
opacity: 0.46;
|
||||
}
|
||||
|
||||
[data-page="stats"] [data-section="top-models"] [data-component="chart-tooltip"] [data-slot="tooltip-divider"] + p {
|
||||
[data-page="stats"]
|
||||
:is([data-section="top-models"], [data-section="unique-users"])
|
||||
[data-component="chart-tooltip"]
|
||||
[data-slot="tooltip-divider"]
|
||||
+ p {
|
||||
margin-top: 8px;
|
||||
}
|
||||
|
||||
[data-page="stats"] [data-section="top-models"] [data-component="chart-tooltip"] p:last-child {
|
||||
[data-page="stats"]
|
||||
:is([data-section="top-models"], [data-section="unique-users"])
|
||||
[data-component="chart-tooltip"]
|
||||
p:last-child {
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
|
||||
[data-page="stats"] [data-section="top-models"] [data-component="chart-tooltip"] [data-slot="tooltip-label"] {
|
||||
[data-page="stats"]
|
||||
:is([data-section="top-models"], [data-section="unique-users"])
|
||||
[data-component="chart-tooltip"]
|
||||
[data-slot="tooltip-label"] {
|
||||
grid-template-columns: 16px minmax(0, 1fr);
|
||||
gap: 4px;
|
||||
}
|
||||
|
||||
[data-page="stats"] [data-section="top-models"] [data-component="chart-tooltip"] i {
|
||||
[data-page="stats"] :is([data-section="top-models"], [data-section="unique-users"]) [data-component="chart-tooltip"] i {
|
||||
width: 6px;
|
||||
height: 6px;
|
||||
justify-self: center;
|
||||
}
|
||||
|
||||
[data-page="stats"] [data-section="top-models"] [data-component="chart-tooltip"] b {
|
||||
[data-page="stats"] :is([data-section="top-models"], [data-section="unique-users"]) [data-component="chart-tooltip"] b {
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
[data-page="stats"]
|
||||
:is(
|
||||
[data-section="leaderboard"],
|
||||
[data-section="unique-users"],
|
||||
[data-section="market-share"],
|
||||
[data-section="geo-breakdown"],
|
||||
[data-section="token-cost"],
|
||||
@@ -3264,10 +3297,12 @@
|
||||
background: #242424f2;
|
||||
}
|
||||
|
||||
[data-page="stats"][data-theme="dark"] [data-section="top-models"] [data-component="chart-tooltip"],
|
||||
[data-page="stats"][data-theme="dark"]
|
||||
:is([data-section="top-models"], [data-section="unique-users"])
|
||||
[data-component="chart-tooltip"],
|
||||
:root[data-stats-theme="dark"]
|
||||
[data-page="stats"]:not([data-theme="light"])
|
||||
[data-section="top-models"]
|
||||
:is([data-section="top-models"], [data-section="unique-users"])
|
||||
[data-component="chart-tooltip"] {
|
||||
background: #242424f2;
|
||||
box-shadow:
|
||||
@@ -3276,10 +3311,13 @@
|
||||
0 4px 8px #00000052;
|
||||
}
|
||||
|
||||
[data-page="stats"][data-theme="dark"] [data-section="top-models"] [data-component="chart-tooltip"] > span,
|
||||
[data-page="stats"][data-theme="dark"]
|
||||
:is([data-section="top-models"], [data-section="unique-users"])
|
||||
[data-component="chart-tooltip"]
|
||||
> span,
|
||||
:root[data-stats-theme="dark"]
|
||||
[data-page="stats"]:not([data-theme="light"])
|
||||
[data-section="top-models"]
|
||||
:is([data-section="top-models"], [data-section="unique-users"])
|
||||
[data-component="chart-tooltip"]
|
||||
> span {
|
||||
color: var(--stats-faint);
|
||||
@@ -3544,6 +3582,7 @@
|
||||
@media (max-width: 74rem) {
|
||||
[data-page="stats"] [data-section="top-models"],
|
||||
[data-page="stats"] [data-section="leaderboard"],
|
||||
[data-page="stats"] [data-section="unique-users"],
|
||||
[data-page="stats"] [data-section="market-share"],
|
||||
[data-page="stats"] [data-section="geo-breakdown"],
|
||||
[data-page="stats"] [data-section="token-cost"],
|
||||
@@ -3703,6 +3742,7 @@
|
||||
@media (max-width: 47.999rem) {
|
||||
[data-page="stats"] [data-section="top-models"],
|
||||
[data-page="stats"] [data-section="leaderboard"],
|
||||
[data-page="stats"] [data-section="unique-users"],
|
||||
[data-page="stats"] [data-section="market-share"],
|
||||
[data-page="stats"] [data-section="geo-breakdown"],
|
||||
[data-page="stats"] [data-section="token-cost"],
|
||||
@@ -4011,7 +4051,7 @@
|
||||
display: block;
|
||||
}
|
||||
|
||||
[data-page="stats"] [data-section="top-models"] [data-component="chart-tooltip"] {
|
||||
[data-page="stats"] :is([data-section="top-models"], [data-section="unique-users"]) [data-component="chart-tooltip"] {
|
||||
position: fixed;
|
||||
top: auto;
|
||||
right: 12px;
|
||||
@@ -4025,7 +4065,9 @@
|
||||
transform: none;
|
||||
}
|
||||
|
||||
[data-page="stats"] [data-section="top-models"] [data-component="chart-tooltip"][data-placement] {
|
||||
[data-page="stats"]
|
||||
:is([data-section="top-models"], [data-section="unique-users"])
|
||||
[data-component="chart-tooltip"][data-placement] {
|
||||
right: 12px;
|
||||
left: 12px;
|
||||
}
|
||||
|
||||
@@ -49,8 +49,9 @@ const rangeLabels: Record<UsageRange, string> = {
|
||||
"1M": "1 Month",
|
||||
"2M": "2 Months",
|
||||
}
|
||||
const statsHomeTitle = "OpenCode Data"
|
||||
const statsHomeDescription = "OpenCode usage data, market share, token cost, and session cost."
|
||||
const statsHomeTitle = "AI Model Usage Rankings | OpenCode Data"
|
||||
const statsHomeDescription =
|
||||
"Explore OpenCode Go usage across AI models, including token volume, rankings, market share, token pricing, session cost, cache ratio, and geo breakdowns."
|
||||
const statsHomeFallbackUrl = "https://opencode.ai/data/"
|
||||
const statsUnfurlPath = "banner.jpg"
|
||||
const statsUnfurlAlt = "OpenCode Data wordmark on a dark patterned background"
|
||||
@@ -170,6 +171,7 @@ export default function StatsHome() {
|
||||
<>
|
||||
<Hero updatedAt={stats().updatedAt} />
|
||||
<TopModelsSection data={stats().usage} leaderboard={stats().leaderboard} />
|
||||
<UniqueUsersSection data={stats().users} />
|
||||
<SessionCostSection data={stats().sessionCost} />
|
||||
<TokenCostSection data={stats().tokenCost} catalog={catalog() ?? null} />
|
||||
<CacheRatioSection data={stats().cacheRatio} />
|
||||
@@ -597,6 +599,8 @@ function FilterPills<T extends string>(props: {
|
||||
function TopModelsChart(props: {
|
||||
data: UsagePoint[]
|
||||
range: UsageRange
|
||||
metric?: "tokens" | "users"
|
||||
ariaLabel?: string
|
||||
activeModel: string | undefined
|
||||
onActiveModelChange: (model: string | undefined) => void
|
||||
}) {
|
||||
@@ -605,6 +609,7 @@ function TopModelsChart(props: {
|
||||
const maxTotal = createMemo(() => getTopModelsMaxTotal(props.data))
|
||||
const segmentOrder = createMemo(() => getTopModelsSegmentOrder(props.data))
|
||||
const activePoint = createMemo(() => props.data[activeIndex() ?? -1])
|
||||
const metric = createMemo(() => props.metric ?? "tokens")
|
||||
|
||||
createEffect(() => scrollDenseChartToEnd(chartRef, props.range, props.data.length))
|
||||
|
||||
@@ -613,9 +618,10 @@ function TopModelsChart(props: {
|
||||
ref={chartRef}
|
||||
data-component="top-models-chart"
|
||||
data-range={props.range}
|
||||
data-metric={metric()}
|
||||
data-dense-labels={isDenseColumnRange(props.range) ? "true" : undefined}
|
||||
role="img"
|
||||
aria-label="Stacked top model usage chart"
|
||||
aria-label={props.ariaLabel ?? "Stacked top model usage chart"}
|
||||
style={{ "--top-models-count": props.data.length } as JSX.CSSProperties}
|
||||
onPointerLeave={(event) => {
|
||||
if (event.pointerType === "touch") return
|
||||
@@ -632,7 +638,7 @@ function TopModelsChart(props: {
|
||||
data-mobile-hidden={isTopModelsMobileAxisHidden(index(), props.data.length) ? "true" : undefined}
|
||||
>
|
||||
<span data-slot="axis-label">
|
||||
<span data-slot="axis-total">{formatTokens(usageTotal(day))}</span>
|
||||
<span data-slot="axis-total">{formatUsageChartValue(usageTotal(day), metric())}</span>
|
||||
<span data-slot="axis-date">
|
||||
<span data-slot="axis-date-full">{day.date}</span>
|
||||
<span data-slot="axis-date-mobile">{formatTopModelsMobileDate(day.date, props.range)}</span>
|
||||
@@ -656,7 +662,7 @@ function TopModelsChart(props: {
|
||||
data-slot="top-models-bar"
|
||||
role="button"
|
||||
tabIndex={0}
|
||||
aria-label={`${day.date} ${formatTokens(usageTotal(day))}`}
|
||||
aria-label={`${day.date} ${formatUsageChartValue(usageTotal(day), metric())} ${usageChartTotalLabel(metric())}`}
|
||||
data-active={activeIndex() === dayIndex() ? "true" : undefined}
|
||||
data-muted={activeIndex() !== undefined && activeIndex() !== dayIndex() ? "true" : undefined}
|
||||
style={{ "--top-models-bar-height": `${getTopModelsBarHeight(usageTotal(day), maxTotal())}%` }}
|
||||
@@ -738,7 +744,9 @@ function TopModelsChart(props: {
|
||||
data-placement={dayIndex() > props.data.length * 0.62 ? "left" : "right"}
|
||||
>
|
||||
<strong>{point().date}</strong>
|
||||
<span>{formatTokens(usageTotal(point()))} total</span>
|
||||
<span>
|
||||
{formatUsageChartValue(usageTotal(point()), metric())} {usageChartTotalLabel(metric())}
|
||||
</span>
|
||||
<div data-slot="tooltip-divider" />
|
||||
<For each={visibleTopModelsSegments(point())}>
|
||||
{(item) => (
|
||||
@@ -758,7 +766,7 @@ function TopModelsChart(props: {
|
||||
/>{" "}
|
||||
{item.segment.model}
|
||||
</span>
|
||||
<b>{formatTokens(item.segment.value)}</b>
|
||||
<b>{formatUsageChartValue(item.segment.value, metric())}</b>
|
||||
</p>
|
||||
)}
|
||||
</For>
|
||||
@@ -773,6 +781,33 @@ function TopModelsChart(props: {
|
||||
)
|
||||
}
|
||||
|
||||
function UniqueUsersSection(props: { data: StatsHomeData["users"] }) {
|
||||
const [activeModel, setActiveModel] = createSignal<string>()
|
||||
const data = createMemo(() => props.data.Go["2M"])
|
||||
|
||||
return (
|
||||
<section id="unique-users" data-section="unique-users">
|
||||
<SectionBridge label="TOP MODELS" href="#top-models" />
|
||||
<SectionTitle title="Unique Users" description="Daily unique OpenCode Go users by model." />
|
||||
<Show
|
||||
when={data().some((item) => usageTotal(item) > 0)}
|
||||
fallback={
|
||||
<EmptyState title="No user data" description="No user-bearing model_stat rows matched this window." />
|
||||
}
|
||||
>
|
||||
<TopModelsChart
|
||||
data={data()}
|
||||
range="2M"
|
||||
metric="users"
|
||||
ariaLabel="Stacked unique user chart by model"
|
||||
activeModel={activeModel()}
|
||||
onActiveModelChange={setActiveModel}
|
||||
/>
|
||||
</Show>
|
||||
</section>
|
||||
)
|
||||
}
|
||||
|
||||
function isTopModelsBlankHover(bar: HTMLElement, clientY: number) {
|
||||
const stack = bar.querySelector<HTMLElement>('[data-slot="top-models-stack"]')
|
||||
if (!stack) return true
|
||||
@@ -863,6 +898,22 @@ function formatTokens(value: number) {
|
||||
return `${Math.round(value * 1000)}B`
|
||||
}
|
||||
|
||||
function formatUsageChartValue(value: number, metric: "tokens" | "users") {
|
||||
if (metric === "users") return formatUsers(value)
|
||||
return formatTokens(value)
|
||||
}
|
||||
|
||||
function usageChartTotalLabel(metric: "tokens" | "users") {
|
||||
if (metric === "users") return "model users"
|
||||
return "total"
|
||||
}
|
||||
|
||||
function formatUsers(value: number) {
|
||||
if (value >= 1_000_000) return `${(value / 1_000_000).toFixed(value >= 10_000_000 ? 0 : 1)}M`
|
||||
if (value >= 1_000) return `${(value / 1_000).toFixed(value >= 10_000 ? 0 : 1)}K`
|
||||
return new Intl.NumberFormat("en").format(Math.round(value))
|
||||
}
|
||||
|
||||
function Leaderboard(props: {
|
||||
data: LeaderboardEntry[]
|
||||
activeModel: string | undefined
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
ALTER TABLE `geo_stat` ADD `unique_users` bigint NOT NULL DEFAULT 0;--> statement-breakpoint
|
||||
ALTER TABLE `model_stat` ADD `unique_users` bigint NOT NULL DEFAULT 0;--> statement-breakpoint
|
||||
ALTER TABLE `provider_stat` ADD `unique_users` bigint NOT NULL DEFAULT 0;
|
||||
@@ -123,6 +123,7 @@ function metricColumns() {
|
||||
return {
|
||||
sessions: bigint({ mode: "number" }).notNull().default(0),
|
||||
requests: bigint({ mode: "number" }).notNull().default(0),
|
||||
unique_users: bigint({ mode: "number" }).notNull().default(0),
|
||||
input_tokens: bigint({ mode: "number" }).notNull().default(0),
|
||||
output_tokens: bigint({ mode: "number" }).notNull().default(0),
|
||||
reasoning_tokens: bigint({ mode: "number" }).notNull().default(0),
|
||||
|
||||
@@ -8,6 +8,8 @@ import {
|
||||
chunks,
|
||||
collapseRows,
|
||||
inserted,
|
||||
isMissingUniqueUsersColumn,
|
||||
omitUniqueUsers,
|
||||
rankRowsWithMarketShare,
|
||||
statPeriodKey,
|
||||
statRowScope,
|
||||
@@ -136,48 +138,59 @@ export class GeoStatRepo extends Context.Service<GeoStatRepo, GeoStatRepo.Servic
|
||||
chunks(rows, UPSERT_CHUNK_SIZE),
|
||||
(chunk) =>
|
||||
Effect.tryPromise({
|
||||
try: () =>
|
||||
db
|
||||
.insert(geoStat)
|
||||
.values(chunk)
|
||||
.onDuplicateKeyUpdate({
|
||||
set: {
|
||||
continent: inserted("continent"),
|
||||
sessions: inserted("sessions"),
|
||||
requests: inserted("requests"),
|
||||
input_tokens: inserted("input_tokens"),
|
||||
output_tokens: inserted("output_tokens"),
|
||||
reasoning_tokens: inserted("reasoning_tokens"),
|
||||
cache_read_tokens: inserted("cache_read_tokens"),
|
||||
total_tokens: inserted("total_tokens"),
|
||||
input_cost_microcents: inserted("input_cost_microcents"),
|
||||
output_cost_microcents: inserted("output_cost_microcents"),
|
||||
total_cost_microcents: inserted("total_cost_microcents"),
|
||||
avg_duration_ms: inserted("avg_duration_ms"),
|
||||
p50_duration_ms: inserted("p50_duration_ms"),
|
||||
p95_duration_ms: inserted("p95_duration_ms"),
|
||||
avg_ttfb_ms: inserted("avg_ttfb_ms"),
|
||||
p50_ttfb_ms: inserted("p50_ttfb_ms"),
|
||||
p95_ttfb_ms: inserted("p95_ttfb_ms"),
|
||||
avg_output_tps: inserted("avg_output_tps"),
|
||||
success_count: inserted("success_count"),
|
||||
error_count: inserted("error_count"),
|
||||
sample_count: inserted("sample_count"),
|
||||
market_share_tokens: inserted("market_share_tokens"),
|
||||
market_share_requests: inserted("market_share_requests"),
|
||||
market_share_sessions: inserted("market_share_sessions"),
|
||||
rank_by_tokens: inserted("rank_by_tokens"),
|
||||
rank_by_requests: inserted("rank_by_requests"),
|
||||
rank_by_sessions: inserted("rank_by_sessions"),
|
||||
rank_by_cost: inserted("rank_by_cost"),
|
||||
},
|
||||
}),
|
||||
try: async () => {
|
||||
try {
|
||||
return await upsertGeoChunk(chunk, true)
|
||||
} catch (cause) {
|
||||
if (!isMissingUniqueUsersColumn(cause)) throw cause
|
||||
return upsertGeoChunk(chunk, false)
|
||||
}
|
||||
},
|
||||
catch: (cause) => DatabaseError.make({ cause }),
|
||||
}),
|
||||
{ discard: true },
|
||||
)
|
||||
})
|
||||
|
||||
function upsertGeoChunk(chunk: GeoStatRow[], includeUniqueUsers: boolean) {
|
||||
return db
|
||||
.insert(geoStat)
|
||||
.values(includeUniqueUsers ? chunk : omitUniqueUsers(chunk))
|
||||
.onDuplicateKeyUpdate({
|
||||
set: {
|
||||
continent: inserted("continent"),
|
||||
sessions: inserted("sessions"),
|
||||
requests: inserted("requests"),
|
||||
...(includeUniqueUsers ? { unique_users: inserted("unique_users") } : {}),
|
||||
input_tokens: inserted("input_tokens"),
|
||||
output_tokens: inserted("output_tokens"),
|
||||
reasoning_tokens: inserted("reasoning_tokens"),
|
||||
cache_read_tokens: inserted("cache_read_tokens"),
|
||||
total_tokens: inserted("total_tokens"),
|
||||
input_cost_microcents: inserted("input_cost_microcents"),
|
||||
output_cost_microcents: inserted("output_cost_microcents"),
|
||||
total_cost_microcents: inserted("total_cost_microcents"),
|
||||
avg_duration_ms: inserted("avg_duration_ms"),
|
||||
p50_duration_ms: inserted("p50_duration_ms"),
|
||||
p95_duration_ms: inserted("p95_duration_ms"),
|
||||
avg_ttfb_ms: inserted("avg_ttfb_ms"),
|
||||
p50_ttfb_ms: inserted("p50_ttfb_ms"),
|
||||
p95_ttfb_ms: inserted("p95_ttfb_ms"),
|
||||
avg_output_tps: inserted("avg_output_tps"),
|
||||
success_count: inserted("success_count"),
|
||||
error_count: inserted("error_count"),
|
||||
sample_count: inserted("sample_count"),
|
||||
market_share_tokens: inserted("market_share_tokens"),
|
||||
market_share_requests: inserted("market_share_requests"),
|
||||
market_share_sessions: inserted("market_share_sessions"),
|
||||
rank_by_tokens: inserted("rank_by_tokens"),
|
||||
rank_by_requests: inserted("rank_by_requests"),
|
||||
rank_by_sessions: inserted("rank_by_sessions"),
|
||||
rank_by_cost: inserted("rank_by_cost"),
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
const deleteRetiredDimensions = Effect.fn("GeoStatRepo.deleteRetiredDimensions")(function* (rows: GeoStatRow[]) {
|
||||
const scope = statRowScope(rows)
|
||||
if (!scope) return
|
||||
|
||||
@@ -21,7 +21,7 @@ export type TokenCostEntry = { model: string; total: number; input: number; outp
|
||||
export type CacheRatioEntry = { model: string; ratio: number; cached: number; uncached: number; total: number }
|
||||
export type SessionCostEntry = { model: string; cost: number; tokens: number }
|
||||
export type CountryEntry = { country: string; continent: string; tokens: number; share: number; rank: number }
|
||||
export type ModelUsagePoint = { date: string; tokens: number; sessions: number; cost: number }
|
||||
export type ModelUsagePoint = { date: string; tokens: number; users: number; sessions: number; cost: number }
|
||||
export type ModelMixEntry = { label: string; tokens: number; share: number }
|
||||
export type ModelPeerEntry = {
|
||||
model: string
|
||||
@@ -82,6 +82,7 @@ export type StatsLabData = {
|
||||
export type StatsHomeData = {
|
||||
updatedAt: string | null
|
||||
usage: Record<UsageProduct, Record<UsageRange, UsagePoint[]>>
|
||||
users: Record<UsageProduct, Record<UsageRange, UsagePoint[]>>
|
||||
leaderboard: Record<UsageProduct, Record<UsageRange, LeaderboardEntry[]>>
|
||||
market: Record<UsageRange, MarketDay[]>
|
||||
tokenCost: Record<TokenProduct, TokenCostEntry[]>
|
||||
@@ -118,6 +119,7 @@ type ModelAggregate = {
|
||||
model: string
|
||||
provider: string
|
||||
sessions: number
|
||||
uniqueUsers: number
|
||||
inputTokens: number
|
||||
outputTokens: number
|
||||
reasoningTokens: number
|
||||
@@ -200,6 +202,18 @@ function buildStatsHomeData(
|
||||
),
|
||||
),
|
||||
),
|
||||
users: createUsageProductRecord((product) =>
|
||||
createRangeRecord((range) =>
|
||||
buildUsagePoints(
|
||||
normalized,
|
||||
product,
|
||||
range,
|
||||
getWindow(range, earliest, latest),
|
||||
getWindow("1W", earliest, latest),
|
||||
"users",
|
||||
),
|
||||
),
|
||||
),
|
||||
leaderboard: createUsageProductRecord((product) =>
|
||||
createRangeRecord((range) => buildLeaderboard(normalized, product, getWindow("1W", earliest, latest))),
|
||||
),
|
||||
@@ -248,15 +262,15 @@ function buildStatsModelData(
|
||||
)
|
||||
.filter((item) => item.totalTokens > 0)
|
||||
.toSorted((a, b) => b.totalTokens - a.totalTokens || a.model.localeCompare(b.model))
|
||||
const peers = aggregateByModelName(rowsForProduct(normalized, SITE_PRODUCT, window.start, window.end))
|
||||
const windowPeers = aggregateByModelName(rowsForProduct(normalized, SITE_PRODUCT, window.start, window.end))
|
||||
.filter((item) => item.totalTokens > 0)
|
||||
.toSorted((a, b) => b.totalTokens - a.totalTokens || a.model.localeCompare(b.model))
|
||||
const rankIndex = rankPeers.findIndex((item) => item.model === model)
|
||||
const rank = rankIndex >= 0 ? rankIndex + 1 : null
|
||||
const previousRankIndex = previousRankPeers.findIndex((item) => item.model === model)
|
||||
const peerRankIndex = peers.findIndex((item) => item.model === model)
|
||||
const peerRank = peerRankIndex >= 0 ? peerRankIndex + 1 : 1
|
||||
const totalTokens = peers.reduce((sum, item) => sum + item.totalTokens, 0)
|
||||
const peerRank = rankIndex >= 0 ? rankIndex + 1 : 1
|
||||
const totalTokens = windowPeers.reduce((sum, item) => sum + item.totalTokens, 0)
|
||||
const peerTokens = rankPeers.reduce((sum, item) => sum + item.totalTokens, 0)
|
||||
|
||||
return {
|
||||
updatedAt: Number.isFinite(latestUpdate) ? new Date(latestUpdate).toISOString() : null,
|
||||
@@ -266,7 +280,7 @@ function buildStatsModelData(
|
||||
author: formatProvider(current.provider),
|
||||
rank,
|
||||
previousRank: previousRankIndex >= 0 ? previousRankIndex + 1 : null,
|
||||
totalModels: peers.length,
|
||||
totalModels: windowPeers.length,
|
||||
tokenShare: totalTokens > 0 ? round((current.totalTokens / totalTokens) * 100, 2) : 0,
|
||||
tokenChange: percentChange(current.totalTokens, previous.totalTokens),
|
||||
totals: {
|
||||
@@ -285,7 +299,7 @@ function buildStatsModelData(
|
||||
usage: buildModelUsage(currentRows, window, "2M"),
|
||||
tokenMix: buildModelTokenMix(current),
|
||||
country: createRangeRecord((range) => buildCountryStats(geo, getWindow(range, earliest, latest))),
|
||||
peers: buildModelPeers(peers, peerRank, totalTokens),
|
||||
peers: buildModelPeers(rankPeers, peerRank, peerTokens),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -340,6 +354,7 @@ function emptyStatsHomeData(): StatsHomeData {
|
||||
return {
|
||||
updatedAt: null,
|
||||
usage: createUsageProductRecord(() => createRangeRecord(() => [])),
|
||||
users: createUsageProductRecord(() => createRangeRecord(() => [])),
|
||||
leaderboard: createUsageProductRecord(() => createRangeRecord(() => [])),
|
||||
market: createRangeRecord(() => []),
|
||||
tokenCost: createTokenProductRecord(() => []),
|
||||
@@ -355,28 +370,39 @@ function buildUsagePoints(
|
||||
range: UsageRange,
|
||||
window: DateWindow,
|
||||
rankWindow: DateWindow,
|
||||
metric: "tokens" | "users" = "tokens",
|
||||
) {
|
||||
const modelOrder = aggregateByModelName(rowsForProduct(rows, product, rankWindow.start, rankWindow.end))
|
||||
.toSorted((a, b) => b.totalTokens - a.totalTokens)
|
||||
.toSorted((a, b) => modelUsageValue(b, metric) - modelUsageValue(a, metric))
|
||||
.slice(0, TOP_MODEL_SEGMENT_LIMIT)
|
||||
.map((item) => item.model)
|
||||
|
||||
return createBuckets(window, range).map((bucket) => {
|
||||
const bucketRows = aggregateByModelName(rowsForProduct(rows, product, bucket.start, bucket.end))
|
||||
const byModel = new Map(bucketRows.map((item) => [item.model, item.totalTokens]))
|
||||
const segmentTokens = modelOrder.map((model) => ({ model, tokens: byModel.get(model) ?? 0 }))
|
||||
const knownTokens = segmentTokens.reduce((sum, item) => sum + item.tokens, 0)
|
||||
const totalTokens = bucketRows.reduce((sum, item) => sum + item.totalTokens, 0)
|
||||
const byModel = new Map(bucketRows.map((item) => [item.model, modelUsageValue(item, metric)]))
|
||||
const segments = modelOrder.map((model) => ({ model, value: byModel.get(model) ?? 0 }))
|
||||
const knownValue = segments.reduce((sum, item) => sum + item.value, 0)
|
||||
const totalValue = bucketRows.reduce((sum, item) => sum + modelUsageValue(item, metric), 0)
|
||||
return {
|
||||
date: bucket.label,
|
||||
segments: [
|
||||
...segmentTokens.map((item) => ({ model: item.model, value: round(item.tokens / 1_000_000_000_000, 4) })),
|
||||
{ model: "Other", value: round(Math.max(totalTokens - knownTokens, 0) / 1_000_000_000_000, 4) },
|
||||
...segments.map((item) => ({ model: item.model, value: usagePointValue(item.value, metric) })),
|
||||
{ model: "Other", value: usagePointValue(Math.max(totalValue - knownValue, 0), metric) },
|
||||
],
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
function modelUsageValue(item: ModelAggregate, metric: "tokens" | "users") {
|
||||
if (metric === "users") return item.uniqueUsers
|
||||
return item.totalTokens
|
||||
}
|
||||
|
||||
function usagePointValue(value: number, metric: "tokens" | "users") {
|
||||
if (metric === "users") return value
|
||||
return round(value / 1_000_000_000_000, 4)
|
||||
}
|
||||
|
||||
function buildLeaderboard(rows: StatMetricRow[], product: UsageProduct, rankWindow: DateWindow) {
|
||||
const previous = new Map(
|
||||
aggregateByModelName(rowsForProduct(rows, product, rankWindow.previousStart, rankWindow.previousEnd)).map(
|
||||
@@ -502,6 +528,7 @@ function buildModelUsage(rows: StatMetricRow[], window: DateWindow, range: Usage
|
||||
return {
|
||||
date: bucket.label,
|
||||
tokens: aggregate.totalTokens,
|
||||
users: aggregate.uniqueUsers,
|
||||
sessions: aggregate.sessions,
|
||||
cost: round(microcentsToDollars(aggregate.totalCostMicrocents), 2),
|
||||
}
|
||||
@@ -601,6 +628,7 @@ function combineRowsForModel(model: string, rows: StatMetricRow[]): ModelAggrega
|
||||
model,
|
||||
provider: "unknown",
|
||||
sessions: 0,
|
||||
uniqueUsers: 0,
|
||||
inputTokens: 0,
|
||||
outputTokens: 0,
|
||||
reasoningTokens: 0,
|
||||
@@ -617,6 +645,7 @@ function combineModelAggregate(current: ModelAggregate | undefined, row: StatMet
|
||||
model: row.model,
|
||||
provider: row.provider,
|
||||
sessions: (current?.sessions ?? 0) + row.sessions,
|
||||
uniqueUsers: (current?.uniqueUsers ?? 0) + row.uniqueUsers,
|
||||
inputTokens: (current?.inputTokens ?? 0) + row.inputTokens,
|
||||
outputTokens: (current?.outputTokens ?? 0) + row.outputTokens,
|
||||
reasoningTokens: (current?.reasoningTokens ?? 0) + row.reasoningTokens,
|
||||
|
||||
@@ -40,6 +40,7 @@ export function buildStatsQuery(periodStart: Date, periodEnd: Date, dimension: S
|
||||
const aggregateColumns = `
|
||||
COUNT(DISTINCT session) AS sessions,
|
||||
COUNT(*) AS requests,
|
||||
COUNT(DISTINCT user_key) AS unique_users,
|
||||
COALESCE(SUM(tokens_input), 0) AS input_tokens,
|
||||
COALESCE(SUM(tokens_output), 0) AS output_tokens,
|
||||
COALESCE(SUM(tokens_reasoning), 0) AS reasoning_tokens,
|
||||
@@ -70,6 +71,8 @@ WITH normalized AS (
|
||||
UPPER(COALESCE(NULLIF(cf_country, ''), 'ZZ')) AS country,
|
||||
COALESCE(NULLIF(cf_continent, ''), '') AS continent,
|
||||
session,
|
||||
COALESCE(NULLIF(workspace, ''), '') AS workspace,
|
||||
COALESCE(NULLIF(api_key, ''), '') AS api_key,
|
||||
status,
|
||||
duration AS duration_ms,
|
||||
time_to_first_byte AS ttfb_ms,
|
||||
@@ -108,6 +111,7 @@ WITH normalized AS (
|
||||
country,
|
||||
continent,
|
||||
session,
|
||||
COALESCE(NULLIF(workspace, ''), NULLIF(api_key, '')) AS user_key,
|
||||
status,
|
||||
duration_ms,
|
||||
ttfb_ms,
|
||||
@@ -197,6 +201,7 @@ function toStatBaseAggregate(data: AthenaData): StatBaseAggregate[] {
|
||||
tier: normalizeTier(data.tier || "unknown"),
|
||||
sessions: integer(data, "sessions"),
|
||||
requests: integer(data, "requests"),
|
||||
unique_users: integer(data, "unique_users"),
|
||||
input_tokens: integer(data, "input_tokens"),
|
||||
output_tokens: integer(data, "output_tokens"),
|
||||
reasoning_tokens: integer(data, "reasoning_tokens"),
|
||||
|
||||
@@ -8,6 +8,8 @@ import {
|
||||
chunks,
|
||||
collapseRows,
|
||||
inserted,
|
||||
isMissingUniqueUsersColumn,
|
||||
omitUniqueUsers,
|
||||
rankBy,
|
||||
statPeriodKey,
|
||||
statRowScope,
|
||||
@@ -27,6 +29,7 @@ export type ModelStatMetric = {
|
||||
provider: string
|
||||
model: string
|
||||
sessions: number
|
||||
uniqueUsers: number
|
||||
inputTokens: number
|
||||
outputTokens: number
|
||||
reasoningTokens: number
|
||||
@@ -55,34 +58,55 @@ export class ModelStatRepo extends Context.Service<ModelStatRepo, ModelStatRepo.
|
||||
|
||||
const listDaily = Effect.fn("ModelStatRepo.listDaily")(function* () {
|
||||
return yield* Effect.tryPromise({
|
||||
try: () =>
|
||||
db
|
||||
.select({
|
||||
periodKey: modelStat.period_key,
|
||||
updatedAt: modelStat.updated_at,
|
||||
tier: modelStat.tier,
|
||||
provider: modelStat.provider,
|
||||
model: modelStat.model,
|
||||
sessions: modelStat.sessions,
|
||||
inputTokens: modelStat.input_tokens,
|
||||
outputTokens: modelStat.output_tokens,
|
||||
reasoningTokens: modelStat.reasoning_tokens,
|
||||
cacheReadTokens: modelStat.cache_read_tokens,
|
||||
totalTokens: modelStat.total_tokens,
|
||||
inputCostMicrocents: modelStat.input_cost_microcents,
|
||||
outputCostMicrocents: modelStat.output_cost_microcents,
|
||||
totalCostMicrocents: modelStat.total_cost_microcents,
|
||||
})
|
||||
.from(modelStat)
|
||||
.where(
|
||||
and(
|
||||
eq(modelStat.grain, "day"),
|
||||
eq(modelStat.client, "all"),
|
||||
eq(modelStat.source, "all"),
|
||||
inArray(modelStat.tier, ["Go", "go"]),
|
||||
),
|
||||
)
|
||||
.orderBy(asc(modelStat.period_key)),
|
||||
try: async () => {
|
||||
try {
|
||||
return await db
|
||||
.select({
|
||||
periodKey: modelStat.period_key,
|
||||
updatedAt: modelStat.updated_at,
|
||||
tier: modelStat.tier,
|
||||
provider: modelStat.provider,
|
||||
model: modelStat.model,
|
||||
sessions: modelStat.sessions,
|
||||
uniqueUsers: modelStat.unique_users,
|
||||
inputTokens: modelStat.input_tokens,
|
||||
outputTokens: modelStat.output_tokens,
|
||||
reasoningTokens: modelStat.reasoning_tokens,
|
||||
cacheReadTokens: modelStat.cache_read_tokens,
|
||||
totalTokens: modelStat.total_tokens,
|
||||
inputCostMicrocents: modelStat.input_cost_microcents,
|
||||
outputCostMicrocents: modelStat.output_cost_microcents,
|
||||
totalCostMicrocents: modelStat.total_cost_microcents,
|
||||
})
|
||||
.from(modelStat)
|
||||
.where(modelDailyScope())
|
||||
.orderBy(asc(modelStat.period_key))
|
||||
} catch (cause) {
|
||||
if (!isMissingUniqueUsersColumn(cause)) throw cause
|
||||
return (
|
||||
await db
|
||||
.select({
|
||||
periodKey: modelStat.period_key,
|
||||
updatedAt: modelStat.updated_at,
|
||||
tier: modelStat.tier,
|
||||
provider: modelStat.provider,
|
||||
model: modelStat.model,
|
||||
sessions: modelStat.sessions,
|
||||
inputTokens: modelStat.input_tokens,
|
||||
outputTokens: modelStat.output_tokens,
|
||||
reasoningTokens: modelStat.reasoning_tokens,
|
||||
cacheReadTokens: modelStat.cache_read_tokens,
|
||||
totalTokens: modelStat.total_tokens,
|
||||
inputCostMicrocents: modelStat.input_cost_microcents,
|
||||
outputCostMicrocents: modelStat.output_cost_microcents,
|
||||
totalCostMicrocents: modelStat.total_cost_microcents,
|
||||
})
|
||||
.from(modelStat)
|
||||
.where(modelDailyScope())
|
||||
.orderBy(asc(modelStat.period_key))
|
||||
).map((row) => ({ ...row, uniqueUsers: 0 }))
|
||||
}
|
||||
},
|
||||
catch: (cause) => DatabaseError.make({ cause }),
|
||||
})
|
||||
})
|
||||
@@ -92,44 +116,55 @@ export class ModelStatRepo extends Context.Service<ModelStatRepo, ModelStatRepo.
|
||||
chunks(rows, UPSERT_CHUNK_SIZE),
|
||||
(chunk) =>
|
||||
Effect.tryPromise({
|
||||
try: () =>
|
||||
db
|
||||
.insert(modelStat)
|
||||
.values(chunk)
|
||||
.onDuplicateKeyUpdate({
|
||||
set: {
|
||||
provider_model: inserted("provider_model"),
|
||||
sessions: inserted("sessions"),
|
||||
requests: inserted("requests"),
|
||||
input_tokens: inserted("input_tokens"),
|
||||
output_tokens: inserted("output_tokens"),
|
||||
reasoning_tokens: inserted("reasoning_tokens"),
|
||||
cache_read_tokens: inserted("cache_read_tokens"),
|
||||
total_tokens: inserted("total_tokens"),
|
||||
input_cost_microcents: inserted("input_cost_microcents"),
|
||||
output_cost_microcents: inserted("output_cost_microcents"),
|
||||
total_cost_microcents: inserted("total_cost_microcents"),
|
||||
avg_duration_ms: inserted("avg_duration_ms"),
|
||||
p50_duration_ms: inserted("p50_duration_ms"),
|
||||
p95_duration_ms: inserted("p95_duration_ms"),
|
||||
avg_ttfb_ms: inserted("avg_ttfb_ms"),
|
||||
p50_ttfb_ms: inserted("p50_ttfb_ms"),
|
||||
p95_ttfb_ms: inserted("p95_ttfb_ms"),
|
||||
avg_output_tps: inserted("avg_output_tps"),
|
||||
success_count: inserted("success_count"),
|
||||
error_count: inserted("error_count"),
|
||||
sample_count: inserted("sample_count"),
|
||||
rank_by_tokens: inserted("rank_by_tokens"),
|
||||
rank_by_requests: inserted("rank_by_requests"),
|
||||
rank_by_cost: inserted("rank_by_cost"),
|
||||
},
|
||||
}),
|
||||
try: async () => {
|
||||
try {
|
||||
return await upsertModelChunk(chunk, true)
|
||||
} catch (cause) {
|
||||
if (!isMissingUniqueUsersColumn(cause)) throw cause
|
||||
return upsertModelChunk(chunk, false)
|
||||
}
|
||||
},
|
||||
catch: (cause) => DatabaseError.make({ cause }),
|
||||
}),
|
||||
{ discard: true },
|
||||
)
|
||||
})
|
||||
|
||||
function upsertModelChunk(chunk: ModelStatRow[], includeUniqueUsers: boolean) {
|
||||
return db
|
||||
.insert(modelStat)
|
||||
.values(includeUniqueUsers ? chunk : omitUniqueUsers(chunk))
|
||||
.onDuplicateKeyUpdate({
|
||||
set: {
|
||||
provider_model: inserted("provider_model"),
|
||||
sessions: inserted("sessions"),
|
||||
requests: inserted("requests"),
|
||||
...(includeUniqueUsers ? { unique_users: inserted("unique_users") } : {}),
|
||||
input_tokens: inserted("input_tokens"),
|
||||
output_tokens: inserted("output_tokens"),
|
||||
reasoning_tokens: inserted("reasoning_tokens"),
|
||||
cache_read_tokens: inserted("cache_read_tokens"),
|
||||
total_tokens: inserted("total_tokens"),
|
||||
input_cost_microcents: inserted("input_cost_microcents"),
|
||||
output_cost_microcents: inserted("output_cost_microcents"),
|
||||
total_cost_microcents: inserted("total_cost_microcents"),
|
||||
avg_duration_ms: inserted("avg_duration_ms"),
|
||||
p50_duration_ms: inserted("p50_duration_ms"),
|
||||
p95_duration_ms: inserted("p95_duration_ms"),
|
||||
avg_ttfb_ms: inserted("avg_ttfb_ms"),
|
||||
p50_ttfb_ms: inserted("p50_ttfb_ms"),
|
||||
p95_ttfb_ms: inserted("p95_ttfb_ms"),
|
||||
avg_output_tps: inserted("avg_output_tps"),
|
||||
success_count: inserted("success_count"),
|
||||
error_count: inserted("error_count"),
|
||||
sample_count: inserted("sample_count"),
|
||||
rank_by_tokens: inserted("rank_by_tokens"),
|
||||
rank_by_requests: inserted("rank_by_requests"),
|
||||
rank_by_cost: inserted("rank_by_cost"),
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
const deleteRetiredDimensions = Effect.fn("ModelStatRepo.deleteRetiredDimensions")(function* (
|
||||
rows: ModelStatRow[],
|
||||
) {
|
||||
@@ -162,6 +197,15 @@ export class ModelStatRepo extends Context.Service<ModelStatRepo, ModelStatRepo.
|
||||
)
|
||||
}
|
||||
|
||||
function modelDailyScope() {
|
||||
return and(
|
||||
eq(modelStat.grain, "day"),
|
||||
eq(modelStat.client, "all"),
|
||||
eq(modelStat.source, "all"),
|
||||
inArray(modelStat.tier, ["Go", "go"]),
|
||||
)
|
||||
}
|
||||
|
||||
export function rowsFromAggregates(aggregates: ModelStatAggregate[]) {
|
||||
return rankRows([
|
||||
...synthesizeAllTierRows(
|
||||
|
||||
@@ -8,6 +8,8 @@ import {
|
||||
chunks,
|
||||
collapseRows,
|
||||
inserted,
|
||||
isMissingUniqueUsersColumn,
|
||||
omitUniqueUsers,
|
||||
rankRowsWithMarketShare,
|
||||
statRowScope,
|
||||
synthesizeAllTierRows,
|
||||
@@ -107,47 +109,58 @@ export class ProviderStatRepo extends Context.Service<ProviderStatRepo, Provider
|
||||
chunks(rows, UPSERT_CHUNK_SIZE),
|
||||
(chunk) =>
|
||||
Effect.tryPromise({
|
||||
try: () =>
|
||||
db
|
||||
.insert(providerStat)
|
||||
.values(chunk)
|
||||
.onDuplicateKeyUpdate({
|
||||
set: {
|
||||
sessions: inserted("sessions"),
|
||||
requests: inserted("requests"),
|
||||
input_tokens: inserted("input_tokens"),
|
||||
output_tokens: inserted("output_tokens"),
|
||||
reasoning_tokens: inserted("reasoning_tokens"),
|
||||
cache_read_tokens: inserted("cache_read_tokens"),
|
||||
total_tokens: inserted("total_tokens"),
|
||||
input_cost_microcents: inserted("input_cost_microcents"),
|
||||
output_cost_microcents: inserted("output_cost_microcents"),
|
||||
total_cost_microcents: inserted("total_cost_microcents"),
|
||||
avg_duration_ms: inserted("avg_duration_ms"),
|
||||
p50_duration_ms: inserted("p50_duration_ms"),
|
||||
p95_duration_ms: inserted("p95_duration_ms"),
|
||||
avg_ttfb_ms: inserted("avg_ttfb_ms"),
|
||||
p50_ttfb_ms: inserted("p50_ttfb_ms"),
|
||||
p95_ttfb_ms: inserted("p95_ttfb_ms"),
|
||||
avg_output_tps: inserted("avg_output_tps"),
|
||||
success_count: inserted("success_count"),
|
||||
error_count: inserted("error_count"),
|
||||
sample_count: inserted("sample_count"),
|
||||
market_share_tokens: inserted("market_share_tokens"),
|
||||
market_share_requests: inserted("market_share_requests"),
|
||||
market_share_sessions: inserted("market_share_sessions"),
|
||||
rank_by_tokens: inserted("rank_by_tokens"),
|
||||
rank_by_requests: inserted("rank_by_requests"),
|
||||
rank_by_sessions: inserted("rank_by_sessions"),
|
||||
rank_by_cost: inserted("rank_by_cost"),
|
||||
},
|
||||
}),
|
||||
try: async () => {
|
||||
try {
|
||||
return await upsertProviderChunk(chunk, true)
|
||||
} catch (cause) {
|
||||
if (!isMissingUniqueUsersColumn(cause)) throw cause
|
||||
return upsertProviderChunk(chunk, false)
|
||||
}
|
||||
},
|
||||
catch: (cause) => DatabaseError.make({ cause }),
|
||||
}),
|
||||
{ discard: true },
|
||||
)
|
||||
})
|
||||
|
||||
function upsertProviderChunk(chunk: ProviderStatRow[], includeUniqueUsers: boolean) {
|
||||
return db
|
||||
.insert(providerStat)
|
||||
.values(includeUniqueUsers ? chunk : omitUniqueUsers(chunk))
|
||||
.onDuplicateKeyUpdate({
|
||||
set: {
|
||||
sessions: inserted("sessions"),
|
||||
requests: inserted("requests"),
|
||||
...(includeUniqueUsers ? { unique_users: inserted("unique_users") } : {}),
|
||||
input_tokens: inserted("input_tokens"),
|
||||
output_tokens: inserted("output_tokens"),
|
||||
reasoning_tokens: inserted("reasoning_tokens"),
|
||||
cache_read_tokens: inserted("cache_read_tokens"),
|
||||
total_tokens: inserted("total_tokens"),
|
||||
input_cost_microcents: inserted("input_cost_microcents"),
|
||||
output_cost_microcents: inserted("output_cost_microcents"),
|
||||
total_cost_microcents: inserted("total_cost_microcents"),
|
||||
avg_duration_ms: inserted("avg_duration_ms"),
|
||||
p50_duration_ms: inserted("p50_duration_ms"),
|
||||
p95_duration_ms: inserted("p95_duration_ms"),
|
||||
avg_ttfb_ms: inserted("avg_ttfb_ms"),
|
||||
p50_ttfb_ms: inserted("p50_ttfb_ms"),
|
||||
p95_ttfb_ms: inserted("p95_ttfb_ms"),
|
||||
avg_output_tps: inserted("avg_output_tps"),
|
||||
success_count: inserted("success_count"),
|
||||
error_count: inserted("error_count"),
|
||||
sample_count: inserted("sample_count"),
|
||||
market_share_tokens: inserted("market_share_tokens"),
|
||||
market_share_requests: inserted("market_share_requests"),
|
||||
market_share_sessions: inserted("market_share_sessions"),
|
||||
rank_by_tokens: inserted("rank_by_tokens"),
|
||||
rank_by_requests: inserted("rank_by_requests"),
|
||||
rank_by_sessions: inserted("rank_by_sessions"),
|
||||
rank_by_cost: inserted("rank_by_cost"),
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
const deleteRetiredDimensions = Effect.fn("ProviderStatRepo.deleteRetiredDimensions")(function* (
|
||||
rows: ProviderStatRow[],
|
||||
) {
|
||||
|
||||
@@ -12,6 +12,7 @@ export type StatBaseAggregate = {
|
||||
tier: string
|
||||
sessions: number
|
||||
requests: number
|
||||
unique_users: number
|
||||
input_tokens: number
|
||||
output_tokens: number
|
||||
reasoning_tokens: number
|
||||
@@ -41,6 +42,7 @@ export type StatBaseRow = {
|
||||
source?: string
|
||||
sessions?: number
|
||||
requests?: number
|
||||
unique_users?: number
|
||||
input_tokens?: number
|
||||
output_tokens?: number
|
||||
reasoning_tokens?: number
|
||||
@@ -71,6 +73,7 @@ export function toStatBaseRow(data: StatBaseAggregate) {
|
||||
source: "all",
|
||||
sessions: data.sessions,
|
||||
requests: data.requests,
|
||||
unique_users: data.unique_users,
|
||||
input_tokens: data.input_tokens,
|
||||
output_tokens: data.output_tokens,
|
||||
reasoning_tokens: data.reasoning_tokens,
|
||||
@@ -122,6 +125,7 @@ export function combineRows<T extends StatBaseRow>(left: T, right: T): T {
|
||||
...left,
|
||||
sessions: (left.sessions ?? 0) + (right.sessions ?? 0),
|
||||
requests: (left.requests ?? 0) + (right.requests ?? 0),
|
||||
unique_users: (left.unique_users ?? 0) + (right.unique_users ?? 0),
|
||||
input_tokens: (left.input_tokens ?? 0) + (right.input_tokens ?? 0),
|
||||
output_tokens: (left.output_tokens ?? 0) + (right.output_tokens ?? 0),
|
||||
reasoning_tokens: (left.reasoning_tokens ?? 0) + (right.reasoning_tokens ?? 0),
|
||||
@@ -143,6 +147,18 @@ export function combineRows<T extends StatBaseRow>(left: T, right: T): T {
|
||||
}
|
||||
}
|
||||
|
||||
export function isMissingUniqueUsersColumn(cause: unknown): boolean {
|
||||
return errorText(cause).includes("Unknown column 'unique_users'")
|
||||
}
|
||||
|
||||
export function omitUniqueUsers<T extends { unique_users?: number }>(rows: T[]) {
|
||||
return rows.map((row) => {
|
||||
const result = { ...row }
|
||||
delete result.unique_users
|
||||
return result
|
||||
})
|
||||
}
|
||||
|
||||
export function statPeriodKey(row: StatBaseRow) {
|
||||
return [row.grain, row.period_key, row.dataset, row.tier, row.client, row.source].join("\u0000")
|
||||
}
|
||||
@@ -238,6 +254,15 @@ export function inserted(column: string) {
|
||||
return sql.raw(`values(\`${column}\`)`)
|
||||
}
|
||||
|
||||
function errorText(cause: unknown): string {
|
||||
if (cause instanceof Error) return `${cause.message} ${errorText((cause as { cause?: unknown }).cause)}`
|
||||
if (typeof cause === "object" && cause)
|
||||
return Object.values(cause as Record<string, unknown>)
|
||||
.map(errorText)
|
||||
.join(" ")
|
||||
return String(cause)
|
||||
}
|
||||
|
||||
export function weightedAverage(
|
||||
left: number | null | undefined,
|
||||
leftWeight = 0,
|
||||
|
||||
@@ -242,6 +242,7 @@ function metricQuery(breakdowns: string[], limit: number, filters: ReturnType<ty
|
||||
calculations: [
|
||||
{ op: "COUNT_DISTINCT", column: "session" },
|
||||
{ op: "COUNT" },
|
||||
{ op: "COUNT_DISTINCT", column: "workspace" },
|
||||
{ op: "SUM", column: "tokens.input" },
|
||||
{ op: "SUM", column: "tokens.output" },
|
||||
{ op: "SUM", column: "tokens.reasoning" },
|
||||
@@ -374,9 +375,15 @@ function classifyRows(file: string, rows: RawRow[]): ImportKey {
|
||||
}
|
||||
|
||||
function hasMetricHeaders(headers: Set<string>) {
|
||||
return ["sumtokens", "sumtokensinput", "inputtokens", "totaltokens", "avgduration", "countdistinctsession"].some(
|
||||
(header) => headers.has(header),
|
||||
)
|
||||
return [
|
||||
"sumtokens",
|
||||
"sumtokensinput",
|
||||
"inputtokens",
|
||||
"totaltokens",
|
||||
"avgduration",
|
||||
"countdistinctsession",
|
||||
"countdistinctworkspace",
|
||||
].some((header) => headers.has(header))
|
||||
}
|
||||
|
||||
function hasHeader(headers: Set<string>, names: string[]) {
|
||||
@@ -447,6 +454,7 @@ function baseAggregate(row: RawRow, grain: Grain, opts: ImportOptions): StatBase
|
||||
tier: tier(row),
|
||||
sessions: integer(row, "sessions", ["COUNT_DISTINCT(session)"]),
|
||||
requests: integer(row, "requests", ["COUNT", "COUNT()"]),
|
||||
unique_users: integer(row, "unique_users", ["COUNT_DISTINCT(workspace)", "COUNT_DISTINCT(api_key)"]),
|
||||
input_tokens: integer(row, "input_tokens", ["SUM(tokens.input)", "SUM(tokens_input)"]),
|
||||
output_tokens: integer(row, "output_tokens", ["SUM(tokens.output)", "SUM(tokens_output)"]),
|
||||
reasoning_tokens: integer(row, "reasoning_tokens", ["SUM(tokens.reasoning)", "SUM(tokens_reasoning)"]),
|
||||
@@ -808,6 +816,7 @@ async function upsertModelRows(db: ReturnType<typeof drizzle>, rows: ModelStatRo
|
||||
provider_model: inserted("provider_model"),
|
||||
sessions: inserted("sessions"),
|
||||
requests: inserted("requests"),
|
||||
unique_users: inserted("unique_users"),
|
||||
input_tokens: inserted("input_tokens"),
|
||||
output_tokens: inserted("output_tokens"),
|
||||
reasoning_tokens: inserted("reasoning_tokens"),
|
||||
@@ -845,6 +854,7 @@ async function upsertProviderRows(db: ReturnType<typeof drizzle>, rows: Provider
|
||||
set: {
|
||||
sessions: inserted("sessions"),
|
||||
requests: inserted("requests"),
|
||||
unique_users: inserted("unique_users"),
|
||||
input_tokens: inserted("input_tokens"),
|
||||
output_tokens: inserted("output_tokens"),
|
||||
reasoning_tokens: inserted("reasoning_tokens"),
|
||||
@@ -887,6 +897,7 @@ async function upsertGeoRows(db: ReturnType<typeof drizzle>, rows: GeoStatRow[],
|
||||
continent: inserted("continent"),
|
||||
sessions: inserted("sessions"),
|
||||
requests: inserted("requests"),
|
||||
unique_users: inserted("unique_users"),
|
||||
input_tokens: inserted("input_tokens"),
|
||||
output_tokens: inserted("output_tokens"),
|
||||
reasoning_tokens: inserted("reasoning_tokens"),
|
||||
|
||||
@@ -11,6 +11,7 @@ import { startOfIsoWeek } from "./domain/stat"
|
||||
const DATALAKE_INGESTION_LAG_MS = 5 * 60_000
|
||||
const STATS_DATA_START_MS = new Date("2026-05-28T00:00:00.000Z").getTime()
|
||||
const WEEK_MS = 7 * 86_400_000
|
||||
const DISPLAY_WINDOW_MS = 56 * 86_400_000
|
||||
|
||||
export type SyncStatsResult = { ok: true; rows: number; startedAt: string; periodStart: string; periodEnd: string }
|
||||
export type SyncStatsError = AthenaQueryError | AthenaQueryTimeoutError | DatabaseError
|
||||
@@ -23,7 +24,12 @@ export const syncStats: () => Effect.Effect<
|
||||
const startedAt = yield* DateTime.nowAsDate
|
||||
const periodEnd = new Date(Math.floor((startedAt.getTime() - DATALAKE_INGESTION_LAG_MS) / 60_000) * 60_000)
|
||||
// May 27 was partial, so keep Athena stats anchored at the first complete day.
|
||||
const periodStart = new Date(Math.max(startOfIsoWeek(periodEnd).getTime() - WEEK_MS, STATS_DATA_START_MS))
|
||||
const periodStart = new Date(
|
||||
Math.max(
|
||||
Math.min(startOfIsoWeek(periodEnd).getTime() - WEEK_MS, periodEnd.getTime() - DISPLAY_WINDOW_MS),
|
||||
STATS_DATA_START_MS,
|
||||
),
|
||||
)
|
||||
const athena = yield* Athena
|
||||
const modelStats = yield* ModelStatRepo
|
||||
const providerStats = yield* ProviderStatRepo
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
import { createResource, createMemo } from "solid-js"
|
||||
import { createResource, createMemo, createSignal } from "solid-js"
|
||||
import { TextAttributes } from "@opentui/core"
|
||||
import { DialogSelect } from "../ui/dialog-select"
|
||||
import { useSDK } from "../context/sdk"
|
||||
import { useDialog } from "../ui/dialog"
|
||||
import { useToast } from "../ui/toast"
|
||||
import { useTheme } from "../context/theme"
|
||||
import { errorMessage } from "../util/error"
|
||||
import type { ExperimentalConsoleListOrgsResponse } from "@opencode-ai/sdk/v2"
|
||||
|
||||
type OrgOption = ExperimentalConsoleListOrgsResponse["orgs"][number]
|
||||
@@ -25,14 +27,26 @@ export function DialogConsoleOrg() {
|
||||
const toast = useToast()
|
||||
const { theme } = useTheme()
|
||||
|
||||
const [orgs] = createResource(async () => {
|
||||
const result = await sdk.client.experimental.console.listOrgs({}, { throwOnError: true })
|
||||
return result.data?.orgs ?? []
|
||||
})
|
||||
const [loadError, setLoadError] = createSignal<unknown>()
|
||||
|
||||
const [orgs] = createResource(() =>
|
||||
sdk.client.experimental.console
|
||||
.listOrgs({}, { throwOnError: true })
|
||||
.then((result) => result.data?.orgs ?? [])
|
||||
// Catch so the rejected resource never reaches the memos below: reading
|
||||
// orgs() in an errored state re-throws and tears down the dialog.
|
||||
.catch((error) => {
|
||||
setLoadError(error)
|
||||
return undefined
|
||||
}),
|
||||
)
|
||||
|
||||
const showError = createMemo(() => Boolean(loadError()))
|
||||
|
||||
const current = createMemo(() => orgs()?.find((item) => item.active))
|
||||
|
||||
const options = createMemo(() => {
|
||||
if (showError()) return []
|
||||
const listed = orgs()
|
||||
if (listed === undefined) {
|
||||
return [
|
||||
@@ -99,5 +113,23 @@ export function DialogConsoleOrg() {
|
||||
}))
|
||||
})
|
||||
|
||||
return <DialogSelect<string | OrgOption> title="Switch org" options={options()} current={current()} />
|
||||
return (
|
||||
<DialogSelect<string | OrgOption>
|
||||
title="Switch org"
|
||||
options={options()}
|
||||
current={current()}
|
||||
renderFilter={!showError()}
|
||||
locked={showError()}
|
||||
emptyView={
|
||||
showError() ? (
|
||||
<box paddingLeft={4} paddingRight={4}>
|
||||
<text fg={theme.error} attributes={TextAttributes.BOLD}>
|
||||
Could not load orgs
|
||||
</text>
|
||||
<text fg={theme.textMuted}>{errorMessage(loadError())}</text>
|
||||
</box>
|
||||
) : undefined
|
||||
}
|
||||
/>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -91,6 +91,8 @@ const GO_UPSELL_ACCOUNT_RATE_LIMIT_DONT_SHOW = "go_upsell_account_rate_limit_don
|
||||
const GO_UPSELL_WINDOW = 86_400_000 // 24 hrs
|
||||
const GO_UPSELL_PROVIDERS = new Set(["opencode", "opencode-go"])
|
||||
|
||||
export const alwaysSeparate = new WeakSet<BoxRenderable>()
|
||||
|
||||
type RetryAction = Extract<SessionStatus, { type: "retry" }>["action"]
|
||||
|
||||
function goUpsellKeys(action: RetryAction) {
|
||||
@@ -160,7 +162,6 @@ const context = createContext<{
|
||||
showTimestamps: () => boolean
|
||||
showDetails: () => boolean
|
||||
showGenericToolOutput: () => boolean
|
||||
userMessageIDs: () => ReadonlySet<string>
|
||||
diffWrapMode: () => "word" | "none"
|
||||
providers: () => ReadonlyMap<string, Provider>
|
||||
sync: ReturnType<typeof useSync>
|
||||
@@ -218,14 +219,6 @@ export function Session() {
|
||||
)
|
||||
: [],
|
||||
)
|
||||
const userMessageIDs = createMemo(
|
||||
() =>
|
||||
new Set(
|
||||
messages()
|
||||
.filter((message) => message.role === "user")
|
||||
.map((message) => message.id),
|
||||
),
|
||||
)
|
||||
const permissions = createMemo(() => {
|
||||
if (session()?.parentID) return []
|
||||
return children().flatMap((x) => sync.data.permission[x.id] ?? [])
|
||||
@@ -1158,7 +1151,6 @@ export function Session() {
|
||||
showTimestamps,
|
||||
showDetails,
|
||||
showGenericToolOutput,
|
||||
userMessageIDs,
|
||||
diffWrapMode,
|
||||
providers,
|
||||
sync,
|
||||
@@ -1395,6 +1387,7 @@ function UserMessage(props: {
|
||||
<Show when={text()}>
|
||||
<box
|
||||
id={props.message.id}
|
||||
ref={(el: BoxRenderable) => alwaysSeparate.add(el)}
|
||||
border={["left"]}
|
||||
borderColor={color()}
|
||||
customBorderChars={SplitBorder.customBorderChars}
|
||||
@@ -1532,7 +1525,7 @@ function AssistantMessage(props: { message: AssistantMessage; parts: Part[]; las
|
||||
</Show>
|
||||
<Show when={props.message.error && props.message.error.name !== "MessageAbortedError"}>
|
||||
<box
|
||||
id={`assistant-error-${props.message.id}`}
|
||||
ref={(el: BoxRenderable) => alwaysSeparate.add(el)}
|
||||
border={["left"]}
|
||||
paddingTop={1}
|
||||
paddingBottom={1}
|
||||
@@ -1547,7 +1540,7 @@ function AssistantMessage(props: { message: AssistantMessage; parts: Part[]; las
|
||||
</Show>
|
||||
<Switch>
|
||||
<Match when={props.last || final() || props.message.error?.name === "MessageAbortedError"}>
|
||||
<box id={`assistant-summary-${props.message.id}`} paddingLeft={3}>
|
||||
<box ref={(el: BoxRenderable) => alwaysSeparate.add(el)} paddingLeft={3}>
|
||||
<text marginTop={1}>
|
||||
<span
|
||||
style={{
|
||||
@@ -1613,7 +1606,7 @@ function ReasoningPart(props: { last: boolean; part: ReasoningPart; message: Ass
|
||||
return (
|
||||
<Show when={content()}>
|
||||
<box
|
||||
id={`text-${props.part.messageID}-${props.part.id}`}
|
||||
ref={(el: BoxRenderable) => alwaysSeparate.add(el)}
|
||||
paddingLeft={3}
|
||||
marginTop={1}
|
||||
flexDirection="column"
|
||||
@@ -1695,7 +1688,7 @@ function TextPart(props: { last: boolean; part: TextPart; message: AssistantMess
|
||||
const { theme, syntax } = useTheme()
|
||||
return (
|
||||
<Show when={props.part.text.trim()}>
|
||||
<box id={`text-${props.part.messageID}-${props.part.id}`} paddingLeft={3} marginTop={1} flexShrink={0}>
|
||||
<box ref={(el: BoxRenderable) => alwaysSeparate.add(el)} paddingLeft={3} marginTop={1} flexShrink={0}>
|
||||
<markdown
|
||||
syntaxStyle={syntax()}
|
||||
streaming={true}
|
||||
@@ -1845,7 +1838,6 @@ function InlineTool(props: {
|
||||
pending: string
|
||||
failure?: string
|
||||
spinner?: boolean
|
||||
subagent?: boolean
|
||||
children: JSX.Element
|
||||
part: ToolPart
|
||||
onClick?: () => void
|
||||
@@ -1886,7 +1878,6 @@ function InlineTool(props: {
|
||||
|
||||
return (
|
||||
<InlineToolRow
|
||||
id={`tool-inline-${props.subagent ? "subagent-" : ""}${props.part.messageID}-${props.part.id}`}
|
||||
icon={props.icon}
|
||||
iconColor={props.iconColor}
|
||||
color={fg()}
|
||||
@@ -1899,8 +1890,6 @@ function InlineTool(props: {
|
||||
pending={props.pending}
|
||||
failure={props.failure}
|
||||
spinner={props.spinner}
|
||||
subagent={props.subagent}
|
||||
separateAfter={(id) => id !== undefined && ctx.userMessageIDs().has(id)}
|
||||
onMouseOver={() => clickable() && setHover(true)}
|
||||
onMouseOut={() => setHover(false)}
|
||||
onMouseUp={() => {
|
||||
@@ -1918,7 +1907,6 @@ function InlineTool(props: {
|
||||
}
|
||||
|
||||
export function InlineToolRow(props: {
|
||||
id?: string
|
||||
icon: string
|
||||
iconColor?: RGBA
|
||||
color?: RGBA
|
||||
@@ -1931,32 +1919,20 @@ export function InlineToolRow(props: {
|
||||
pending: string
|
||||
failure?: string
|
||||
spinner?: boolean
|
||||
subagent?: boolean
|
||||
children: JSX.Element
|
||||
separateAfter?: (id: string | undefined) => boolean
|
||||
onMouseOver?: () => void
|
||||
onMouseOut?: () => void
|
||||
onMouseUp?: () => void
|
||||
}) {
|
||||
return (
|
||||
<box
|
||||
id={props.id}
|
||||
paddingLeft={3}
|
||||
onMouseOver={props.onMouseOver}
|
||||
onMouseOut={props.onMouseOut}
|
||||
onMouseUp={props.onMouseUp}
|
||||
ref={(el: BoxRenderable) => {
|
||||
setPreLayoutSiblingMargin(el, (previous) => {
|
||||
const previousInline = previous?.id.startsWith("tool-inline-") ?? false
|
||||
const previousSubagent = previous?.id.startsWith("tool-inline-subagent-") ?? false
|
||||
return previous?.id.startsWith("text-") ||
|
||||
previous?.id.startsWith("tool-block-") ||
|
||||
previous?.id.startsWith("assistant-error-") ||
|
||||
previous?.id.startsWith("assistant-summary-") ||
|
||||
(previousInline && previousSubagent !== Boolean(props.subagent)) ||
|
||||
props.separateAfter?.(previous?.id)
|
||||
? 1
|
||||
: 0
|
||||
return previous instanceof BoxRenderable && (previous.height > 1 || alwaysSeparate.has(previous)) ? 1 : 0
|
||||
})
|
||||
}}
|
||||
>
|
||||
@@ -2018,7 +1994,7 @@ function BlockTool(props: {
|
||||
const error = createMemo(() => (props.part?.state.status === "error" ? props.part.state.error : undefined))
|
||||
return (
|
||||
<box
|
||||
id={props.part ? `tool-block-${props.part.messageID}-${props.part.id}` : undefined}
|
||||
ref={(el: BoxRenderable) => alwaysSeparate.add(el)}
|
||||
border={["left"]}
|
||||
paddingTop={1}
|
||||
paddingBottom={1}
|
||||
@@ -2184,8 +2160,8 @@ function Read(props: ToolProps) {
|
||||
Read {pathFormatter.format(stringValue(props.input.filePath))} {input(props.input, ["filePath"])}
|
||||
</InlineTool>
|
||||
<For each={loaded()}>
|
||||
{(filepath, index) => (
|
||||
<box id={`tool-inline-loaded-${props.part.messageID}-${props.part.id}-${index()}`} paddingLeft={3}>
|
||||
{(filepath) => (
|
||||
<box paddingLeft={3}>
|
||||
<text paddingLeft={3} fg={theme.textMuted}>
|
||||
↳ Loaded {pathFormatter.format(filepath)}
|
||||
</text>
|
||||
@@ -2305,7 +2281,6 @@ function Task(props: ToolProps) {
|
||||
return (
|
||||
<InlineTool
|
||||
icon={props.part.state.status === "completed" ? "✓" : "│"}
|
||||
subagent={true}
|
||||
color={retry() ? theme.error : undefined}
|
||||
spinner={isRunning()}
|
||||
complete={stringValue(props.input.description)}
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
exports[`TUI inline tool wrapping snapshots consecutive grep, glob, and read rows at a narrow width 1`] = `
|
||||
" ✱ Grep "OPENCODE.*DB|database|sqlite|drizzle|dev.*db|data.
|
||||
*dir|xdg|APPDATA" in packages/opencode/src (151 matches)
|
||||
|
||||
✱ Glob "**/*db*" in packages/opencode (6 matches)
|
||||
→ Read packages/opencode/src/storage/db.ts [offset=1, limit=130]
|
||||
→ Read packages/opencode/src/index.ts [offset=1, limit=100]
|
||||
@@ -13,10 +14,12 @@ exports[`TUI inline tool wrapping snapshots consecutive grep, glob, and read row
|
||||
exports[`TUI inline tool wrapping snapshots expanded tool errors under the tool text 1`] = `
|
||||
" ✱ Grep "OPENCODE.*DB|database|sqlite|drizzle|dev.*db|data.
|
||||
*dir|xdg|APPDATA" in packages/opencode/src (151 matches)
|
||||
|
||||
✱ Glob "**/*db*" in packages/opencode (6 matches)
|
||||
→ Read packages/opencode/src/storage/db.ts [offset=1, limit=130]
|
||||
→ Read packages/opencode/src/index.ts [offset=1, limit=100]
|
||||
No LSP server available for this file type.
|
||||
|
||||
✱ Grep "export const OPENCODE_DB|OPENCODE_DB|OPENCODE_DEV|Global\\.
|
||||
Path\\.data|data =" in packages/opencode/src (115 matches)"
|
||||
`;
|
||||
@@ -32,6 +35,7 @@ exports[`TUI inline tool wrapping keeps separation after a shell output block 1`
|
||||
|
||||
✱ Grep "OPENCODE.*DB|database|sqlite|drizzle|dev.*db|data.
|
||||
*dir|xdg|APPDATA" in packages/opencode/src (151 matches)
|
||||
|
||||
✱ Glob "**/*db*" in packages/opencode (6 matches)
|
||||
→ Read packages/opencode/src/storage/db.ts [offset=1, limit=130]
|
||||
→ Read packages/opencode/src/index.ts [offset=1, limit=100]
|
||||
@@ -46,6 +50,7 @@ exports[`TUI inline tool wrapping keeps separation after a padded user message 1
|
||||
|
||||
✱ Grep "OPENCODE.*DB|database|sqlite|drizzle|dev.*db|data.
|
||||
*dir|xdg|APPDATA" in packages/opencode/src (151 matches)
|
||||
|
||||
✱ Glob "**/*db*" in packages/opencode (6 matches)
|
||||
→ Read packages/opencode/src/storage/db.ts [offset=1, limit=130]
|
||||
→ Read packages/opencode/src/index.ts [offset=1, limit=100]
|
||||
@@ -53,9 +58,8 @@ exports[`TUI inline tool wrapping keeps separation after a padded user message 1
|
||||
Path\\.data|data =" in packages/opencode/src (115 matches)"
|
||||
`;
|
||||
|
||||
exports[`TUI inline tool wrapping separates a contiguous subagent group from inline tools 1`] = `
|
||||
exports[`TUI inline tool wrapping separates after a multi-line task row 1`] = `
|
||||
" ✱ Grep "Task" (2 matches)
|
||||
|
||||
⠙ Explore Task — Inspect active task spacing
|
||||
✓ General Task — Confirm completed task spacing
|
||||
↳ 1 toolcall · 501ms
|
||||
@@ -63,22 +67,21 @@ exports[`TUI inline tool wrapping separates a contiguous subagent group from inl
|
||||
→ Read src/cli/cmd/tui/routes/session/index.tsx"
|
||||
`;
|
||||
|
||||
exports[`TUI inline tool wrapping separates a subagent group after an expanded read 1`] = `
|
||||
exports[`TUI inline tool wrapping does not treat task rows differently from other inline rows 1`] = `
|
||||
" → Read src/cli/cmd/tui/routes/session/index.tsx
|
||||
↳ Loaded src/cli/cmd/tui/routes/session/tools.tsx
|
||||
|
||||
✓ Explore Task — Inspect active task spacing
|
||||
↳ 1 toolcall · 501ms"
|
||||
`;
|
||||
|
||||
exports[`TUI inline tool wrapping separates a subagent from the previous assistant summary 1`] = `
|
||||
exports[`TUI inline tool wrapping separates an inline row from the previous assistant summary 1`] = `
|
||||
" ▣ Build · Little Frank · 53.1s
|
||||
|
||||
✓ Build Task — Review changes
|
||||
↳ 48 toolcalls · 1m 40s"
|
||||
`;
|
||||
|
||||
exports[`TUI inline tool wrapping separates a subagent from the previous assistant error 1`] = `
|
||||
exports[`TUI inline tool wrapping separates an inline row from the previous assistant error 1`] = `
|
||||
"│
|
||||
│ Managed inference requires an active Member plan
|
||||
│
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { afterEach, describe, expect, test } from "bun:test"
|
||||
import { createSignal, For, Show } from "solid-js"
|
||||
import type { ScrollBoxRenderable } from "@opentui/core"
|
||||
import type { BoxRenderable, ScrollBoxRenderable } from "@opentui/core"
|
||||
import { testRender, type JSX } from "@opentui/solid"
|
||||
import {
|
||||
formatCompletedSubagentDetail,
|
||||
@@ -13,6 +13,7 @@ import {
|
||||
parseQuestionAnswers,
|
||||
parseQuestions,
|
||||
parseTodos,
|
||||
alwaysSeparate,
|
||||
toolDisplay,
|
||||
} from "../../../src/routes/session"
|
||||
|
||||
@@ -53,7 +54,14 @@ const tools: readonly ToolFixture[] = [
|
||||
|
||||
function ShellOutput() {
|
||||
return (
|
||||
<box id="tool-block-shell" marginTop={1} paddingTop={1} paddingBottom={1} paddingLeft={2} gap={1}>
|
||||
<box
|
||||
ref={(el: BoxRenderable) => alwaysSeparate.add(el)}
|
||||
marginTop={1}
|
||||
paddingTop={1}
|
||||
paddingBottom={1}
|
||||
paddingLeft={2}
|
||||
gap={1}
|
||||
>
|
||||
<text paddingLeft={3}># List files</text>
|
||||
<box gap={1}>
|
||||
<text>$ ls</text>
|
||||
@@ -65,7 +73,7 @@ function ShellOutput() {
|
||||
|
||||
function UserMessage() {
|
||||
return (
|
||||
<box id="message-user">
|
||||
<box ref={(el: BoxRenderable) => alwaysSeparate.add(el)}>
|
||||
<box paddingTop={1} paddingBottom={1} paddingLeft={2}>
|
||||
<text>Check whether the next tool remains separated.</text>
|
||||
</box>
|
||||
@@ -88,7 +96,6 @@ function Fixture(props: { errorExpanded?: boolean; before?: "shell" | "user" })
|
||||
failed={Boolean(item.error)}
|
||||
error={item.error}
|
||||
errorExpanded={props.errorExpanded}
|
||||
separateAfter={(id) => id === "message-user"}
|
||||
>
|
||||
{item.label}
|
||||
</InlineToolRow>
|
||||
@@ -99,61 +106,67 @@ function Fixture(props: { errorExpanded?: boolean; before?: "shell" | "user" })
|
||||
)
|
||||
}
|
||||
|
||||
function SubagentGroupFixture() {
|
||||
function TaskRowsFixture() {
|
||||
return (
|
||||
<box flexDirection="column" width={72}>
|
||||
<InlineToolRow id="tool-inline-before" icon="✱" complete={true} pending="">
|
||||
<InlineToolRow icon="✱" complete={true} pending="">
|
||||
Grep "Task" (2 matches)
|
||||
</InlineToolRow>
|
||||
<InlineToolRow id="tool-inline-subagent-one" icon="⠙" complete={true} pending="" subagent={true}>
|
||||
<InlineToolRow icon="⠙" complete={true} pending="">
|
||||
Explore Task — Inspect active task spacing
|
||||
</InlineToolRow>
|
||||
<InlineToolRow id="tool-inline-subagent-two" icon="✓" complete={true} pending="" subagent={true}>
|
||||
<InlineToolRow icon="✓" complete={true} pending="">
|
||||
{"General Task — Confirm completed task spacing\n↳ 1 toolcall · 501ms"}
|
||||
</InlineToolRow>
|
||||
<InlineToolRow id="tool-inline-after" icon="→" complete={true} pending="">
|
||||
<InlineToolRow icon="→" complete={true} pending="">
|
||||
Read src/cli/cmd/tui/routes/session/index.tsx
|
||||
</InlineToolRow>
|
||||
</box>
|
||||
)
|
||||
}
|
||||
|
||||
function LoadedReadBeforeSubagentFixture() {
|
||||
function LoadedReadBeforeTaskFixture() {
|
||||
return (
|
||||
<box flexDirection="column" width={72}>
|
||||
<InlineToolRow id="tool-inline-read" icon="→" complete={true} pending="">
|
||||
<InlineToolRow icon="→" complete={true} pending="">
|
||||
Read src/cli/cmd/tui/routes/session/index.tsx
|
||||
</InlineToolRow>
|
||||
<box id="tool-inline-loaded-read-child" paddingLeft={3}>
|
||||
<box paddingLeft={3}>
|
||||
<text paddingLeft={3}>↳ Loaded src/cli/cmd/tui/routes/session/tools.tsx</text>
|
||||
</box>
|
||||
<InlineToolRow id="tool-inline-subagent-after-read" icon="✓" complete={true} pending="" subagent={true}>
|
||||
<InlineToolRow icon="✓" complete={true} pending="">
|
||||
{"Explore Task — Inspect active task spacing\n↳ 1 toolcall · 501ms"}
|
||||
</InlineToolRow>
|
||||
</box>
|
||||
)
|
||||
}
|
||||
|
||||
function AssistantSummaryBeforeSubagentFixture() {
|
||||
function AssistantSummaryBeforeInlineFixture() {
|
||||
return (
|
||||
<box flexDirection="column" width={72}>
|
||||
<box id="assistant-summary-message-one" paddingLeft={3}>
|
||||
<box ref={(el: BoxRenderable) => alwaysSeparate.add(el)} paddingLeft={3}>
|
||||
<text>▣ Build · Little Frank · 53.1s</text>
|
||||
</box>
|
||||
<InlineToolRow id="tool-inline-subagent-one" icon="✓" complete={true} pending="" subagent={true}>
|
||||
<InlineToolRow icon="✓" complete={true} pending="">
|
||||
{"Build Task — Review changes\n↳ 48 toolcalls · 1m 40s"}
|
||||
</InlineToolRow>
|
||||
</box>
|
||||
)
|
||||
}
|
||||
|
||||
function AssistantErrorBeforeSubagentFixture() {
|
||||
function AssistantErrorBeforeInlineFixture() {
|
||||
return (
|
||||
<box flexDirection="column" width={72}>
|
||||
<box id="assistant-error-message-one" border={["left"]} paddingTop={1} paddingBottom={1} paddingLeft={2}>
|
||||
<box
|
||||
ref={(el: BoxRenderable) => alwaysSeparate.add(el)}
|
||||
border={["left"]}
|
||||
paddingTop={1}
|
||||
paddingBottom={1}
|
||||
paddingLeft={2}
|
||||
>
|
||||
<text>Managed inference requires an active Member plan</text>
|
||||
</box>
|
||||
<InlineToolRow id="tool-inline-subagent-one" icon="✓" complete={true} pending="" subagent={true}>
|
||||
<InlineToolRow icon="✓" complete={true} pending="">
|
||||
{"Build Task — Review changes\n↳ 48 toolcalls · 1m 40s"}
|
||||
</InlineToolRow>
|
||||
</box>
|
||||
@@ -170,7 +183,7 @@ function StickyScrollFixture(props: { separated: boolean; scroll: (scroll: Scrol
|
||||
<text>Second row</text>
|
||||
</box>
|
||||
<Show when={props.separated}>
|
||||
<box id="text-before-tool">
|
||||
<box ref={(el: BoxRenderable) => alwaysSeparate.add(el)}>
|
||||
<text>Assistant text</text>
|
||||
</box>
|
||||
</Show>
|
||||
@@ -200,6 +213,7 @@ function FailedCompleteToolFixture() {
|
||||
async function renderFrame(component: () => JSX.Element, options: { width: number; height: number }) {
|
||||
testSetup = await testRender(component, options)
|
||||
await testSetup.renderOnce()
|
||||
await testSetup.renderOnce()
|
||||
|
||||
return testSetup
|
||||
.captureCharFrame()
|
||||
@@ -294,22 +308,20 @@ describe("TUI inline tool wrapping", () => {
|
||||
expect(await renderFrame(() => <Fixture before="user" />, { width: 72, height: 14 })).toMatchSnapshot()
|
||||
})
|
||||
|
||||
test("separates a contiguous subagent group from inline tools", async () => {
|
||||
expect(await renderFrame(() => <SubagentGroupFixture />, { width: 72, height: 10 })).toMatchSnapshot()
|
||||
test("separates after a multi-line task row", async () => {
|
||||
expect(await renderFrame(() => <TaskRowsFixture />, { width: 72, height: 10 })).toMatchSnapshot()
|
||||
})
|
||||
|
||||
test("separates a subagent group after an expanded read", async () => {
|
||||
expect(await renderFrame(() => <LoadedReadBeforeSubagentFixture />, { width: 72, height: 8 })).toMatchSnapshot()
|
||||
test("does not treat task rows differently from other inline rows", async () => {
|
||||
expect(await renderFrame(() => <LoadedReadBeforeTaskFixture />, { width: 72, height: 8 })).toMatchSnapshot()
|
||||
})
|
||||
|
||||
test("separates a subagent from the previous assistant summary", async () => {
|
||||
expect(
|
||||
await renderFrame(() => <AssistantSummaryBeforeSubagentFixture />, { width: 72, height: 5 }),
|
||||
).toMatchSnapshot()
|
||||
test("separates an inline row from the previous assistant summary", async () => {
|
||||
expect(await renderFrame(() => <AssistantSummaryBeforeInlineFixture />, { width: 72, height: 5 })).toMatchSnapshot()
|
||||
})
|
||||
|
||||
test("separates a subagent from the previous assistant error", async () => {
|
||||
expect(await renderFrame(() => <AssistantErrorBeforeSubagentFixture />, { width: 72, height: 7 })).toMatchSnapshot()
|
||||
test("separates an inline row from the previous assistant error", async () => {
|
||||
expect(await renderFrame(() => <AssistantErrorBeforeInlineFixture />, { width: 72, height: 7 })).toMatchSnapshot()
|
||||
})
|
||||
|
||||
test("updates sticky-bottom geometry when a text separator mounts and unmounts", async () => {
|
||||
|
||||
Reference in New Issue
Block a user