Files
Kilo-Org_kilocode/packages/opencode/script/build.ts
T
Igor Šćekić 8aaa62c794 Session export capture (#10611)
* feat(session-export): scaffold module with config constants

* feat(session-export): add zstd compression wrapper

* feat(session-export): event and envelope type definitions

* feat(session-export): eligibility check with kill-switch

* feat(session-export): org signal collector with auth resolver

* feat(session-export): worker SQLite schema and storage helpers

* feat(session-export): content-addressed chunker with zstd dedup

* feat(session-export): client-side light scrubber

* feat(session-export): IPC contract and inbox with back-pressure

* feat(session-export): persist scrubbed events with chunked payloads

* feat(session-export): worker entry point with inbox drain loop

* feat(session-export): main-thread capture module

* feat(session-export): workspace baseline and delta fibers

* feat(session-export): sync subscriber and tool io chunking

* feat(session-export): bootstrap wiring and compaction hook

* feat(session-export): wire capture hooks into sessions

* feat(session-export): uploader and buffer cap

* chore(session-export): annotate shared session hooks

* changeset(session-export): add release note

* fix(session-export): keep bootstrap non-blocking without instance context

* feat(session-export): respawn worker after failures

* test(session-export): add performance budget assertions

* test(session-export): add worker end-to-end smoke test

* fix(session-export): strip identity and high-risk baseline paths

* test(session-export): gate perf assertions for stable sweeps

* fix(session-export): bundle worker in single-file builds

* fix(session-export): make capture envelopes cloneable

* fix(session-export): drain worker on CLI shutdown

* feat(session-export): capture workspace baseline and deltas

* fix(session-export): preserve request metadata

* test(session-export): cover request metadata capture

* feat(cli): send indexed session export batches

* feat(cli): authorize session export uploads

* fix(cli): flush session export shutdown uploads

* feat(cli): optimize session export replay payloads

* fix(cli): include session export surface metadata

* fix(session-export): decrement chunk refs after upload

decRefChunks was never called, so chunk ref_count stayed at 1 (or
higher with dedup) and DELETE FROM chunk WHERE ref_count <= 0 never
matched. Chunks accumulated in the local SQLite buffer until the 50 GB
cap. Call decRefChunks alongside markUploaded so deleteUploaded can
reclaim the rows.

* fix(session-export): add periodic uploader flush timer

flushIntervalMs and retryBackoffMaxMs were both defined in config and
neither was referenced anywhere; scheduleFlush only ran on inbound
events or reconnect. After a 5xx or network failure the row was backed
off for 1 s, but if no further event arrived the retry never fired and
the events stranded until the CLI restarted. Drive a periodic flush
from a setInterval, unref the handle so it doesn't pin the process, and
clear it from a new dispose() hook the worker calls on shutdown.

* fix(session-export): stop emitting absolute workspace root in baseline

CaptureMetadata.root was the literal absolute filesystem path
(/Users/<name>/Projects/<repo>), shipped in every
workspace_baseline_completed event and not stripped by handlers'
identity filter. The field was set but never read anywhere downstream
— file paths in the baseline are already relative, so the root added
no replay signal. Remove the field outright.

* fix(session-export): cap pendingEvents result set

The SELECT had no LIMIT clause, so under a backlog (network outage,
crashed receiver) it would materialize the full pending table into a
JavaScript array before the byte-limit truncation applied. With a
50 GB buffer cap that is hundreds of MB of heap inside the worker per
drain. Add LIMIT 500 — the drain loop already re-queries until empty,
so no events are missed.

* fix(session-export): exponential retry backoff up to retryBackoffMaxMs

Both the 5xx and network-error branches always retried after the floor
delay regardless of how many attempts had already failed, and
retryBackoffMaxMs was unreferenced. During a sustained outage every
session re-tried at roughly 1 Hz against the dead receiver. Surface
upload_attempts on EventRow and compute the next delay as
min * 2^attempts capped at max.

* fix(session-export): evict superseded workspace snapshots on remember

createWorkspaceProvider retained every captured snapshot — in-memory
and inside the persisted state file — even after the session moved on
to a newer one. For a 1k-file repo over a 100-turn session that is
~2 GB of unreachable heap plus a state file that grows monotonically.
Drop the previous snapshot for the session on remember() when no other
session still references it.

* fix(session-export): anchor aws_secret_key scrubber to key name

The bare 40-char base64 pattern matched every 40-character hex string,
including all git commit SHAs. Tool outputs, diffs, and conversation
messages were silently rewritten as <<REDACTED:aws_secret_key>>,
destroying lineage information in training data. Require the key name
context — naked secrets in unstructured text are rare and the .env /
.aws/credentials high-risk path strip already covers the common case.

* fix(session-export): preserve root linkage in SyncSubscriber events

SyncSubscriber hardcoded rootSessionId = sessionId on every tool,
permission, and feedback event, so sub-agent sessions lost their root
linkage and a future training pipeline could not reconstruct the
agent topology from these side-channel events. Expose the rootSessionId
mapping from Capture and plumb it through the same pattern as
getTurnId.

* fix(session-export): atomic chunk GC after upload

markUploaded + decRefChunks + deleteUploaded were three separate SQL
statements; a crash between markUploaded and decRefChunks would leave
events flagged uploaded (never retried) and chunks with stale
ref_count (never reclaimed by deleteUploaded). Bundle the three calls
into a single transactional commitUploaded helper so either all three
land or none of them do.

* fix(session-export): wait on transient sqlite locks

* fix(session-export): snapshot current workspace directory

* fix(cli): preserve Kilo model export metadata

* fix(cli): send anon id for session export

* fix(cli): fallback to telemetry anon id

* chore(cli): annotate session export config test

* chore: remove session export docs

* fix(cli): harden session export uploads

* fix(cli): preserve stream lifecycle for session export

* fix(kilo-docs): exclude session export ingest link

* fix(cli): restrict session export workspace sync to git repos

* chore: remove session export changeset

* fix(cli): tighten session export payload types

* fix(cli): type session export model payloads

* fix(cli): simplify session export cleanup

* fix(cli): avoid session export shutdown race

* refactor(cli): use drizzle for session export storage

* fix(cli): finalize session export sqlite statements

* fix(cli): link compaction exports to root sessions

* fix(cli): stop exporting raw stream parts

* fix(cli): prune stale workspace snapshots

* refactor(cli): clarify chunk ref counting

* test(cli): clarify dropped upload assertions

* ci: avoid visual path filter action failure

* fix(cli): preserve in-flight workspace snapshots

* fix(cli): stop uploading baseline start events

* fix(cli): trim redundant export metadata

* fix(cli): trim workspace export bookkeeping

* fix(cli): fold terminal outcome into tool exports

* fix(cli): dedupe request context in export batches

* fix(cli): normalize compaction export payloads

* fix(cli): avoid duplicate tool result exports

* test(cli): align session export expectations

* fix(cli): run secretlint during session export scrubbing

* fix(cli): keep retried export batches contiguous

* fix(cli): include agent info in session exports

* fix(cli): flush pending session exports on serve startup

* fix(cli): drop exports when scrubbing fails

* fix(cli): narrow secretlint value extraction

* fix(cli): limit exported agent info

* test(cli): remove brittle agent export source assertion

* fix(cli): ignore corrupt session export workspace state

* fix(cli): keep session export close best effort

* fix(cli): avoid following session export symlinks

* fix(cli): preserve session export chunk ref counts

* fix(cli): retry transient session export uploads

* fix(cli): validate session export ingest endpoint

* fix(cli): tolerate missing export token details

* fix(cli): fail closed on session export org lookup

* fix(cli): decode session export permission replies

* fix(cli): finalize session export on stream close

* fix(cli): bound session export baseline timeout

* fix(cli): avoid persisting workspace file contents

* fix(cli): bound workspace snapshot capture

* fix(cli): validate session export worker messages

* fix(cli): revoke stale session export eligibility

* fix(cli): scope session export workspaces

* fix(cli): extend session export shutdown flush

* fix(cli): infer free Kilo models for export

* fix(cli): avoid duplicate chunk refs

* fix(cli): throttle session export uploads

* fix(cli): harden session export capture

* feat: disclose free model data collection (#10767)

* feat: disclose free model data collection

* chore(cli): document free model footer sorting

* fix(vscode): add free model data translations

* fix: simplify free model data label

* fix: simplify data collection badges

* fix: remove duplicate model info disclosure

* fix: restore composer data tooltip

* fix: align jetbrains data collection indicator

* fix: align free model data indicators

* fix(vscode): show data collection in model preview

* fix: limit data indicators to kilo gateway

* test(cli): relax prompt cancel timeout

* test(cli): annotate prompt cancel timeout

* test(cli): classify prompt queue runtime test

* fix(cli): defer session export startup
2026-06-02 13:58:42 +00:00

366 lines
12 KiB
TypeScript
Executable File

#!/usr/bin/env bun
import { $ } from "bun"
import fs from "fs"
import path from "path"
import { fileURLToPath } from "url"
import { createSolidTransformPlugin } from "@opentui/solid/bun-plugin"
import { createRequire } from "module" // kilocode_change
const __filename = fileURLToPath(import.meta.url)
const __dirname = path.dirname(__filename)
const dir = path.resolve(__dirname, "..")
const require = createRequire(import.meta.url) // kilocode_change
process.chdir(dir)
await import("./generate.ts")
import { Script } from "@opencode-ai/script"
import pkg from "../package.json"
import { LanceDBRuntime } from "../src/kilocode/lancedb" // kilocode_change
// Load migrations from migration directories
const migrationDirs = (
await fs.promises.readdir(path.join(dir, "migration"), {
withFileTypes: true,
})
)
.filter((entry) => entry.isDirectory() && /^\d{4}\d{2}\d{2}\d{2}\d{2}\d{2}/.test(entry.name))
.map((entry) => entry.name)
.sort()
const migrations = await Promise.all(
migrationDirs.map(async (name) => {
const file = path.join(dir, "migration", name, "migration.sql")
const sql = await Bun.file(file).text()
const match = /^(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})/.exec(name)
const timestamp = match
? Date.UTC(
Number(match[1]),
Number(match[2]) - 1,
Number(match[3]),
Number(match[4]),
Number(match[5]),
Number(match[6]),
)
: 0
return { sql, timestamp, name }
}),
)
console.log(`Loaded ${migrations.length} migrations`)
const singleFlag = process.argv.includes("--single")
const baselineFlag = process.argv.includes("--baseline")
const skipInstall = process.argv.includes("--skip-install")
const sourcemapsFlag = process.argv.includes("--sourcemaps")
const plugin = createSolidTransformPlugin()
// kilocode_change - packages/app was removed; the web UI embed step is no longer applicable
// kilocode_change start - codebase indexing
async function copyTreeSitterWasms(outputDir: string) {
const runtimeWasmPath = require.resolve("web-tree-sitter/tree-sitter.wasm")
const languagePackagePath = require.resolve("tree-sitter-wasms/package.json")
const languageWasmDir = path.join(path.dirname(languagePackagePath), "out")
const targetDir = path.join(outputDir, "tree-sitter")
await fs.promises.mkdir(targetDir, { recursive: true })
await fs.promises.copyFile(runtimeWasmPath, path.join(targetDir, "tree-sitter.wasm"))
const languageWasmFiles = (await fs.promises.readdir(languageWasmDir)).filter((file) => file.endsWith(".wasm"))
await Promise.all(
languageWasmFiles.map((file) => fs.promises.copyFile(path.join(languageWasmDir, file), path.join(targetDir, file))),
)
console.log(`copied ${languageWasmFiles.length + 1} tree-sitter wasm files to ${targetDir}`)
}
// kilocode_change end
// kilocode_change start - embed Kilo Console static assets
async function buildKiloConsole() {
const app = path.resolve(dir, "../kilo-console")
const out = path.join(app, "dist")
console.log("building Kilo Console")
const proc = Bun.spawn([process.execPath, "run", "build"], {
cwd: app,
env: { ...process.env, KILO_CONSOLE_BASE: "/console/" },
stdout: "inherit",
stderr: "inherit",
windowsHide: true,
})
const code = await proc.exited
if (code !== 0) throw new Error(`Kilo Console build failed with exit code ${code}`)
return out
}
async function copyKiloConsole(input: string, outputDir: string) {
const target = path.join(outputDir, "console")
await fs.promises.rm(target, { recursive: true, force: true })
await fs.promises.cp(input, target, { recursive: true })
console.log(`copied Kilo Console assets to ${target}`)
}
// kilocode_change end
// kilocode_change start - upstream's createEmbeddedWebUIBundle is intentionally removed because
// Kilo dropped the packages/app web UI. Kept here as a commented reference so future upstream merges
// can see the deliberate divergence rather than treating a re-add as a clean re-introduction.
// const createEmbeddedWebUIBundle = async () => {
// console.log(`Building Web UI to embed in the binary`)
// const appDir = path.join(import.meta.dirname, "../../app")
// const dist = path.join(appDir, "dist")
// await $`bun run --cwd ${appDir} build`
// const files = (await Array.fromAsync(new Bun.Glob("**/*").scan({ cwd: dist })))
// .map((file) => file.replaceAll("\\", "/"))
// .filter((file) => !file.endsWith(".map"))
// .sort()
// const imports = files.map((file, i) => {
// const spec = path.relative(dir, path.join(dist, file)).replaceAll("\\", "/")
// return `import file_${i} from ${JSON.stringify(spec.startsWith(".") ? spec : `./${spec}`)} with { type: "file" };`
// })
// const entries = files.map((file, i) => ` ${JSON.stringify(file)}: file_${i},`)
// return [
// `// Import all files as file_$i with type: "file"`,
// ...imports,
// `// Export with original mappings`,
// `export default {`,
// ...entries,
// `}`,
// ].join("\n")
// }
// kilocode_change end
const allTargets: {
os: string
arch: "arm64" | "x64"
abi?: "musl"
avx2?: false
}[] = [
{
os: "linux",
arch: "arm64",
},
{
os: "linux",
arch: "x64",
},
{
os: "linux",
arch: "x64",
avx2: false,
},
{
os: "linux",
arch: "arm64",
abi: "musl",
},
{
os: "linux",
arch: "x64",
abi: "musl",
},
{
os: "linux",
arch: "x64",
abi: "musl",
avx2: false,
},
{
os: "darwin",
arch: "arm64",
},
{
os: "darwin",
arch: "x64",
},
{
os: "darwin",
arch: "x64",
avx2: false,
},
{
os: "win32",
arch: "arm64",
},
{
os: "win32",
arch: "x64",
},
{
os: "win32",
arch: "x64",
avx2: false,
},
]
const targets = singleFlag
? allTargets.filter((item) => {
if (item.os !== process.platform || item.arch !== process.arch) {
return false
}
// When building for the current platform, prefer a single native binary by default.
// Baseline binaries require additional Bun artifacts and can be flaky to download.
if (item.avx2 === false) {
return baselineFlag
}
// also skip abi-specific builds for the same reason
if (item.abi !== undefined) {
return false
}
return true
})
: allTargets
await $`rm -rf dist` // kilocode_change
const kiloConsoleDist = await buildKiloConsole() // kilocode_change
const binaries: Record<string, string> = {}
if (!skipInstall) {
await $`bun install --os="*" --cpu="*" @opentui/core@${pkg.dependencies["@opentui/core"]}`
await $`bun install --os="*" --cpu="*" @parcel/watcher@${pkg.dependencies["@parcel/watcher"]}`
}
for (const item of targets) {
const name = [
pkg.name,
// changing to win32 flags npm for some reason
item.os === "win32" ? "windows" : item.os,
item.arch,
item.avx2 === false ? "baseline" : undefined,
item.abi === undefined ? undefined : item.abi,
]
.filter(Boolean)
.join("-")
console.log(`building ${name}`)
await $`mkdir -p dist/${name}/bin`
const localPath = path.resolve(dir, "node_modules/@opentui/core/parser.worker.js")
const rootPath = path.resolve(dir, "../../node_modules/@opentui/core/parser.worker.js")
const parserWorker = fs.realpathSync(fs.existsSync(localPath) ? localPath : rootPath)
const workerPath = "./src/cli/cmd/tui/worker.ts"
const sessionExportWorkerPath = "./src/kilocode/session-export/worker.ts" // kilocode_change
const indexingWorkerPath = "./src/kilocode/indexing-worker.ts" // kilocode_change
// Use platform-specific bunfs root path based on target OS // kilocode_change
const bunfsRoot = item.os === "win32" ? "B:/~BUN/root/" : "/$bunfs/root/"
const workerRelativePath = path.relative(dir, parserWorker).replaceAll("\\", "/")
await Bun.build({
conditions: ["browser"],
tsconfig: "./tsconfig.json",
plugins: [plugin], // kilocode_change
// kilocode_change start - skip sourcemaps for release builds (each .js.map adds ~50 MB per target → ~600 MB total)
sourcemap: Script.release ? "none" : "external",
// kilocode_change end
external: ["node-gyp", ...LanceDBRuntime.external], // kilocode_change
format: "esm",
minify: true,
splitting: true,
compile: {
autoloadBunfig: false,
autoloadDotenv: false,
autoloadTsconfig: true,
autoloadPackageJson: true,
target: name.replace(pkg.name, "bun") as any,
outfile: `dist/${name}/bin/kilo`, // kilocode_change
execArgv: [`--user-agent=kilo/${Script.version}`, "--use-system-ca", "--"], // kilocode_change
windows: {},
},
// kilocode_change start - packages/app was removed; no embedded web UI
files: {},
entrypoints: ["./src/index.ts", parserWorker, workerPath, sessionExportWorkerPath, indexingWorkerPath],
// kilocode_change end
define: {
KILO_VERSION: `'${Script.version}'`,
KILO_MIGRATIONS: JSON.stringify(migrations),
OTUI_TREE_SITTER_WORKER_PATH: bunfsRoot + workerRelativePath,
KILO_WORKER_PATH: workerPath,
KILO_SESSION_EXPORT_WORKER_PATH: sessionExportWorkerPath, // kilocode_change
KILO_INDEXING_WORKER_PATH: indexingWorkerPath, // kilocode_change
KILO_CHANNEL: `'${Script.channel}'`,
KILO_LIBC: item.os === "linux" ? `'${item.abi ?? "glibc"}'` : "",
KILO_BUILD_KIND: Script.release ? `'release'` : `'source'`, // kilocode_change
},
})
await copyTreeSitterWasms(path.resolve(dir, `dist/${name}/bin`)) // kilocode_change
await copyKiloConsole(kiloConsoleDist, path.resolve(dir, `dist/${name}/bin`)) // kilocode_change
// kilocode_change start - fix Nix-specific ELF interpreter paths for Linux binaries
if (item.os === "linux") {
const interpreters: Record<string, string> = {
x64: "/lib64/ld-linux-x86-64.so.2",
arm64: "/lib/ld-linux-aarch64.so.1",
"x64-musl": "/lib/ld-musl-x86_64.so.1",
"arm64-musl": "/lib/ld-musl-aarch64.so.1",
}
const key = item.abi === "musl" ? `${item.arch}-musl` : item.arch
const interpreter = interpreters[key]
if (interpreter) {
try {
await $`patchelf --set-interpreter ${interpreter} dist/${name}/bin/kilo`
console.log(`patched interpreter for ${name} -> ${interpreter}`)
} catch {
console.warn(`patchelf not available, skipping interpreter fix for ${name}`)
}
}
}
// kilocode_change end
// Smoke test: only run if binary is for current platform
if (item.os === process.platform && item.arch === process.arch && !item.abi) {
const binaryPath = `dist/${name}/bin/kilo` // kilocode_change
console.log(`Running smoke test: ${binaryPath} --version`)
try {
const versionOutput = await $`${binaryPath} --version`.text()
console.log(`Smoke test passed: ${versionOutput.trim()}`)
} catch (e) {
console.error(`Smoke test failed for ${name}:`, e)
process.exit(1)
}
}
await $`rm -rf ./dist/${name}/bin/tui`
await Bun.file(`dist/${name}/package.json`).write(
JSON.stringify(
{
name,
version: Script.version,
os: [item.os],
cpu: [item.arch],
// kilocode_change start
repository: {
type: "git",
url: "https://github.com/Kilo-Org/kilocode",
},
// kilocode_change end
},
null,
2,
),
)
binaries[name] = Script.version
}
if (Script.release) {
const archives: string[] = [] // kilocode_change
for (const key of Object.keys(binaries)) {
const archive = key.replace(pkg.name, "kilo") // kilocode_change
if (key.includes("linux")) {
const out = path.resolve("dist", `${archive}.tar.gz`) // kilocode_change
await $`tar -czf ${out} *`.cwd(`dist/${key}/bin`) // kilocode_change
archives.push(out) // kilocode_change
} else {
const out = path.resolve("dist", `${archive}.zip`) // kilocode_change
await $`zip -r ${out} *`.cwd(`dist/${key}/bin`) // kilocode_change
archives.push(out) // kilocode_change
}
}
await $`gh release upload v${Script.version} ${archives} --clobber` // kilocode_change
}
export { binaries }