From ee5ba49268edf21bbb36e53d6913849601ce8129 Mon Sep 17 00:00:00 2001 From: Mark IJbema Date: Tue, 17 Feb 2026 14:41:36 +0100 Subject: [PATCH 1/7] docs: add telemetry plan for vscode extension --- .../docs/non-agent-features/telemetry.md | 306 ++++++++++++++++++ .../docs/opencode-migration-plan.md | 5 +- 2 files changed, 309 insertions(+), 2 deletions(-) create mode 100644 packages/kilo-vscode/docs/non-agent-features/telemetry.md diff --git a/packages/kilo-vscode/docs/non-agent-features/telemetry.md b/packages/kilo-vscode/docs/non-agent-features/telemetry.md new file mode 100644 index 000000000..abbcc0a4f --- /dev/null +++ b/packages/kilo-vscode/docs/non-agent-features/telemetry.md @@ -0,0 +1,306 @@ +# Telemetry Implementation Plan + +## Architecture Overview + +The extension uses a **dual-layer telemetry architecture**: server-side (PostHog Node SDK in the extension host) and client-side (PostHog JS in the webview). Both layers communicate with PostHog US (`https://us.i.posthog.com`). + +```mermaid +graph TD + A[Extension Host - Node.js] -->|PostHog Node SDK| P[PostHog US] + B[Webview - React/Browser] -->|PostHog JS SDK| P + A -->|setProvider| C[ClineProvider - TelemetryPropertiesProvider] + C -->|getTelemetryProperties| A + D[TelemetryService - Singleton] --> E[PostHogTelemetryClient] + D --> F[DebugTelemetryClient - dev only] + B --> G[TelemetryClient - webview singleton] +``` + +--- + +## 1. Core Components + +### 1.1 `TelemetryService` — Singleton Facade + +**File:** `packages/telemetry/src/TelemetryService.ts` + +- Created once at extension activation via `TelemetryService.createInstance()` +- Holds an array of `TelemetryClient` implementations +- Exposes typed convenience methods for each event: `captureTaskCreated()`, `captureLlmCompletion()`, etc. +- Also handles: `captureException()`, `updateIdentity()`, `updateTelemetryState()`, `shutdown()` + +### 1.2 `BaseTelemetryClient` — Abstract Base + +**File:** `packages/telemetry/src/BaseTelemetryClient.ts` + +- Holds a `WeakRef` to a `TelemetryPropertiesProvider` (the `ClineProvider`) +- Implements event subscription/filtering (include/exclude lists via `TelemetryEventSubscription`) +- Implements property filtering via `isPropertyCapturable()` — subclasses can override +- Merges provider properties with event-specific properties via `getEventProperties()` + +### 1.3 `PostHogTelemetryClient` — Production Client + +**File:** `packages/telemetry/src/PostHogTelemetryClient.ts` + +- Uses `posthog-node` SDK +- **Distinct ID**: defaults to `vscode.env.machineId`, upgrades to user email when authenticated via `updateIdentity()` +- **Privacy filters**: + - Git properties (`repositoryUrl`, `repositoryName`, `defaultBranch`) always filtered + - Error details (`errorMessage`, `cliPath`, `stderrPreview`) filtered for organization users +- **Opt-in logic**: Requires BOTH VSCode global telemetry level = `"all"` AND user opt-in +- **Event exclusions**: `TASK_MESSAGE` is excluded from PostHog (too verbose) +- **Exception capture**: `captureException()` sends structured errors to PostHog + +### 1.4 `DebugTelemetryClient` — Development Client + +**File:** `packages/telemetry/src/DebugTelemetryClient.ts` + +- Always enabled, logs to console +- Registered only in `NODE_ENV === "development"` + +### 1.5 `TelemetryClient` (Webview) — Browser-side PostHog + +**File:** `webview-ui/src/utils/TelemetryClient.ts` + +- Uses `posthog-js` (browser SDK) +- Initialized with API key from extension state, distinct ID = `machineId` +- Used for frontend-specific events (tab views, button clicks, marketplace interactions) + +--- + +## 2. Initialization Flow + +```mermaid +sequenceDiagram + participant Ext as extension.ts + participant TS as TelemetryService + participant PH as PostHogTelemetryClient + participant CP as ClineProvider + participant WV as Webview TelemetryClient + + Ext->>TS: createInstance + Ext->>PH: new PostHogTelemetryClient + Ext->>TS: register PostHogTelemetryClient + Note over PH: API key from env KILOCODE_POSTHOG_API_KEY + CP->>TS: setProvider - ClineProvider as provider + Note over TS,CP: ClineProvider implements TelemetryPropertiesProvider + WV->>WV: updateTelemetryState with apiKey + machineId + Note over WV: posthog.init + posthog.identify +``` + +**Key code paths:** + +1. `extension.ts:119` — `TelemetryService.createInstance()` +2. `extension.ts:133` — `new PostHogTelemetryClient()` registered +3. `ClineProvider` constructor — `TelemetryService.instance.setProvider(this)` +4. Extension shutdown — `TelemetryService.instance.shutdown()` + +--- + +## 3. Telemetry Events + +All events are defined in `TelemetryEventName` enum (`packages/types/src/telemetry.ts`). Here are the categories: + +### 3.1 Task Lifecycle + +| Event | Properties | Capture Method | +|-------|-----------|-------------| +| `Task Created` | `taskId` | `captureTaskCreated()` | +| `Task Reopened` | `taskId` | `captureTaskRestarted()` | +| `Task Completed` | `taskId` | `captureTaskCompleted()` | +| `Conversation Message` | `taskId`, `source` (user/assistant) | `captureConversationMessage()` | + +### 3.2 LLM & AI + +| Event | Properties | +|-------|-----------| +| `LLM Completion` | `taskId`, `inputTokens`, `outputTokens`, `cacheWriteTokens`, `cacheReadTokens`, `cost`, `completionTime`, `inferenceProvider` | +| `Context Condensed` | `taskId`, `isAutomaticTrigger`, `usedCustomPrompt`, `usedCustomApiHandler` | +| `Sliding Window Truncation` | `taskId` | + +### 3.3 Tools & Modes + +| Event | Properties | +|-------|-----------| +| `Tool Used` | `taskId`, `tool`, `toolProtocol` | +| `Mode Switched` | `taskId`, `newMode` | +| `Mode Setting Changed` | `settingName` | +| `Custom Mode Created` | `modeSlug`, `modeName` | +| `Code Action Used` | `actionType` | + +### 3.4 Checkpoints + +| Event | Properties | +|-------|-----------| +| `Checkpoint Created` | `taskId` | +| `Checkpoint Restored` | `taskId` | +| `Checkpoint Diffed` | `taskId` | + +### 3.5 UI Interactions + +| Event | Properties | +|-------|-----------| +| `Tab Shown` | `tab` | +| `Title Button Clicked` | `button` | +| `Prompt Enhanced` | `taskId` (optional) | + +### 3.6 Marketplace + +| Event | Properties | +|-------|-----------| +| `Marketplace Item Installed` | `itemId`, `itemType`, `itemName`, `target`, additional props | +| `Marketplace Item Removed` | `itemId`, `itemType`, `itemName`, `target` | + +### 3.7 Account & Auth + +| Event | Properties | +|-------|-----------| +| `Account Connect Clicked` | — | +| `Account Connect Success` | — | +| `Account Logout Clicked` | — | +| `Account Logout Success` | — | + +### 3.8 Error Tracking + +| Event | Properties | +|-------|-----------| +| `Schema Validation Error` | `schemaName`, `error` (Zod formatted) | +| `Diff Application Error` | `taskId`, `consecutiveMistakeCount` | +| `Shell Integration Error` | `taskId` | +| `Consecutive Mistake Error` | `taskId` | +| Exceptions via `captureException()` | Structured via `ApiProviderError` and `ConsecutiveMistakeError` | + +### 3.9 Autocomplete (Kilo-specific) + +| Event | Properties | +|-------|-----------| +| `Autocomplete Suggestion Requested` | `languageId`, `modelId`, `provider`, `autocompleteType` | +| `Autocomplete LLM Request Completed` | `latencyMs`, `cost`, `inputTokens`, `outputTokens`, context | +| `Autocomplete LLM Request Failed` | `latencyMs`, `error`, context | +| `Autocomplete LLM Suggestion Returned` | context, `suggestionLength` | +| `Autocomplete Suggestion Cache Hit` | `matchType`, `suggestionLength`, context | +| `Autocomplete Accept Suggestion` | `suggestionLength` | +| `Autocomplete Suggestion Filtered` | `reason`, context | +| `Autocomplete Unique Suggestion Shown` | context (only after 300ms visibility) | + +### 3.10 Other (Kilo-specific) + +- `Commit Message Generated` +- `Agent Manager Opened/Session Started/Session Completed/Session Stopped/Session Error/Login Issue` +- `Auto Purge Started/Completed/Failed`, `Manual Purge Triggered` +- `Webview Memory Usage`, `Memory Warning Shown` +- `Ask Approval` +- `Notification Clicked` +- `Suggestion Button Clicked` +- `Free Models Link Clicked`, `Create Organization Link Clicked` +- `Ghost Service Disabled` + +--- + +## 4. Properties Attached to Every Event + +Every event gets enriched with properties from `ClineProvider.getTelemetryProperties()`: + +### Static App Properties (computed once) + +- `appName`, `appVersion`, `vscodeVersion`, `platform`, `editorName` +- `wrapped`, `wrapper`, `wrapperCode`, `wrapperVersion`, `wrapperTitle` — wrapper/IDE detection +- `machineId`, `vscodeIsTelemetryEnabled` + +### Dynamic Properties (per-event) + +- `language`, `mode`, `taskId`, `parentTaskId`, `apiProvider`, `modelId`, `diffStrategy`, `isSubtask` +- `currentTaskSize`, `taskHistorySize`, `toolStyle` (XML vs native) +- `todos` object: `{ total, completed, inProgress, pending }` +- `memory` (process memory usage) +- `fastApply` settings, `openRouter` routing config, `autoApprove` settings +- `kilocodeOrganizationId` (when present) + +### Git Properties (computed once) + +- `repositoryUrl`, `repositoryName`, `defaultBranch` (filtered out before sending by PostHog client) + +--- + +## 5. Privacy & Consent + +### User Opt-in Model + +- Three states: `"unset"`, `"enabled"`, `"disabled"` (see `TelemetrySetting` type) +- Telemetry enabled only when: **VSCode telemetry level = "all"** AND **user setting ≠ "disabled"** +- Wrapper apps can force telemetry enabled via environment variable + +### Identity Management + +- Default: `vscode.env.machineId` (anonymous) +- Authenticated: user email fetched from `api.kilo.ai/api/profile` via `updateIdentity()` +- Identity updates are race-safe (counter-based) + +### Data Filtering + +- Git repository info is **always** stripped before sending +- Error details and file paths are stripped for **organization** users +- `TASK_MESSAGE` events are excluded from PostHog (contain full conversation) +- Expected API errors (429, 402) are not reported via `shouldReportApiErrorToTelemetry()` + +--- + +## 6. Structured Error Classes + +The extension defines reusable error classes for structured exception tracking: + +### `ApiProviderError` + +```typescript +class ApiProviderError extends Error { + provider: string + modelId: string + operation: string + errorCode?: number +} +``` + +### `ConsecutiveMistakeError` + +```typescript +class ConsecutiveMistakeError extends Error { + taskId: string + consecutiveMistakeCount: number + consecutiveMistakeLimit: number + reason: "no_tools_used" | "tool_repetition" | "unknown" + provider?: string + modelId?: string +} +``` + +Both have type guards (`isApiProviderError()`, `isConsecutiveMistakeError()`) and property extractors for telemetry. + +--- + +## 7. Implementation Recommendations for New Extension + +1. **Use PostHog** as the analytics backend — the extension uses `posthog-node` server-side and `posthog-js` client-side +2. **Singleton service pattern** — single `TelemetryService` instance, multiple pluggable clients +3. **Properties provider pattern** — the main provider class implements `TelemetryPropertiesProvider` to inject context +4. **Typed events** — all event names in an enum, with typed capture methods on the service +5. **Event subscription/filtering** — clients can include/exclude specific events +6. **Property filtering** — per-client property filtering (privacy controls) +7. **Dual opt-in** — respect both IDE-level and extension-level telemetry settings +8. **Identity upgrade** — anonymous by default, upgrade to user identity on auth +9. **Graceful degradation** — never crash on telemetry failures; all capture calls are fire-and-forget +10. **Debug client** — separate console-logging client for development + +--- + +## 8. Package Dependencies + +### Server-side (Extension Host) + +- `posthog-node` — PostHog Node.js SDK + +### Client-side (Webview) + +- `posthog-js` — PostHog browser SDK + +### Shared Types + +- `zod` — for schema validation of telemetry properties diff --git a/packages/kilo-vscode/docs/opencode-migration-plan.md b/packages/kilo-vscode/docs/opencode-migration-plan.md index e59a1baf3..c286de235 100644 --- a/packages/kilo-vscode/docs/opencode-migration-plan.md +++ b/packages/kilo-vscode/docs/opencode-migration-plan.md @@ -95,8 +95,9 @@ The rebuild has a working foundation: | [Settings UI](non-agent-features/settings-ui.md) | 🔨 Partial | 15-tab settings shell exists. BrowserTab has real settings controls (enable/disable, system Chrome, headless toggles). LanguageTab has working locale selector. Remaining 13 tabs are stubs. [#170](https://github.com/Kilo-Org/kilo/issues/170) | CLI exposes config; extension provides settings forms | P1 | | [Skills System](non-agent-features/skills-system.md) | ❌ Not started | No skill discovery, management, or hot-reload in extension. | CLI has skills runtime; extension provides packaging/UI | P2 | | [Speech-to-Text](non-agent-features/speech-to-text.md) | ❌ Not started | No voice input or streaming STT. | Webview (mic capture); CLI-compatible STT optional | P3 | -| [Task History](non-agent-features/task-history.md) | 🔨 Partial | Session list exists but lacks search, metadata, and full persistence. [#167](https://github.com/Kilo-Org/kilo/issues/167) | CLI session storage; extension provides history UI | P1 | -| [Terminal / Shell Integration](non-agent-features/terminal-shell-integration.md) | ❌ Not started | No VS Code terminal integration for command execution display, exit code tracking, or working directory changes. | CLI executes commands; extension provides terminal UX | P1 | +| [Task History](non-agent-features/task-history.md) | 🔨 Partial | Session list exists but lacks search, metadata, and full persistence. [#167](https://github.com/Kilo-Org/kilo/issues/167) | CLI session storage; extension provides history UI | P1 | +| [Telemetry](non-agent-features/telemetry.md) | ❌ Not started | Dual-layer telemetry (PostHog Node server-side + PostHog JS client-side) for extension usage, errors, LLM completions, and AI interactions. Includes privacy controls, typed events, and structured error tracking. See detailed plan. | Extension-side (PostHog + kilo-telemetry) | P1 | +| [Terminal / Shell Integration](non-agent-features/terminal-shell-integration.md) | ❌ Not started | No VS Code terminal integration for command execution display, exit code tracking, or working directory changes. | CLI executes commands; extension provides terminal UX | P1 | --- From 52d6feae57415b40449a2763a120e60676dd6d85 Mon Sep 17 00:00:00 2001 From: Mark IJbema Date: Tue, 17 Feb 2026 16:34:05 +0100 Subject: [PATCH 2/7] docs: update telemetry plan for new extension architecture --- .../docs/non-agent-features/telemetry.md | 113 +++++++++++------- 1 file changed, 73 insertions(+), 40 deletions(-) diff --git a/packages/kilo-vscode/docs/non-agent-features/telemetry.md b/packages/kilo-vscode/docs/non-agent-features/telemetry.md index abbcc0a4f..cbbbfba25 100644 --- a/packages/kilo-vscode/docs/non-agent-features/telemetry.md +++ b/packages/kilo-vscode/docs/non-agent-features/telemetry.md @@ -2,21 +2,39 @@ ## Architecture Overview -The extension uses a **dual-layer telemetry architecture**: server-side (PostHog Node SDK in the extension host) and client-side (PostHog JS in the webview). Both layers communicate with PostHog US (`https://us.i.posthog.com`). +The extension uses a **proxy-based telemetry architecture**: all telemetry flows through the CLI's `kilo-telemetry` package, which uses `posthog-node` internally. The extension host and webview never talk to PostHog directly. ```mermaid graph TD - A[Extension Host - Node.js] -->|PostHog Node SDK| P[PostHog US] - B[Webview - React/Browser] -->|PostHog JS SDK| P - A -->|setProvider| C[ClineProvider - TelemetryPropertiesProvider] - C -->|getTelemetryProperties| A - D[TelemetryService - Singleton] --> E[PostHogTelemetryClient] + B[Webview - SolidJS/Browser] -->|postMessage| A[Extension Host - Node.js] + A -->|POST /telemetry/capture| CLI[CLI Server - kilo-telemetry] + CLI -->|posthog-node| P[PostHog US] + A -->|enriches with VS Code properties| A + KP[KiloProvider - TelemetryPropertiesProvider] -->|getTelemetryProperties| A + D[TelemetryService - Singleton] --> E[CLITelemetryClient] D --> F[DebugTelemetryClient - dev only] - B --> G[TelemetryClient - webview singleton] ``` --- +## Architecture — New Extension + +Key architectural decisions for telemetry in the new extension: + +1. **CLI/core owns all telemetry sending** — The `kilo-telemetry` package (which uses `posthog-node` internally) is the single point of contact with PostHog. Neither the extension host nor the webview include any PostHog SDK. + +2. **Extension proxies UI events through CLI** — The extension host forwards telemetry events to the CLI via `POST /telemetry/capture`. The CLI enriches them with server-side context and sends to PostHog. + +3. **Webview sends events via `postMessage`** — The webview has no PostHog SDK. Instead, it fires `postMessage({ type: "telemetry", event, properties })` to the extension host, which enriches the event with VS Code properties and forwards to the CLI endpoint. + +4. **VS Code telemetry consent is the master kill switch** — `vscode.env.telemetryLevel` is passed as a startup parameter to the CLI server. If VS Code telemetry is disabled, the CLI suppresses all PostHog sending. + +5. **Same PostHog project as old extension** — Events go to the same PostHog project. Old vs new extension events are distinguished via properties (e.g., `extensionVersion`, `architecture: "new"`). + +6. **All events from old plan are kept** — Every event category is preserved, including autocomplete, agent manager, marketplace, and memory tracking events. + +--- + ## 1. Core Components ### 1.1 `TelemetryService` — Singleton Facade @@ -32,23 +50,24 @@ graph TD **File:** `packages/telemetry/src/BaseTelemetryClient.ts` -- Holds a `WeakRef` to a `TelemetryPropertiesProvider` (the `ClineProvider`) +- Holds a `WeakRef` to a `TelemetryPropertiesProvider` (the `KiloProvider`) - Implements event subscription/filtering (include/exclude lists via `TelemetryEventSubscription`) - Implements property filtering via `isPropertyCapturable()` — subclasses can override - Merges provider properties with event-specific properties via `getEventProperties()` -### 1.3 `PostHogTelemetryClient` — Production Client +### 1.3 `CLITelemetryClient` — Production Client -**File:** `packages/telemetry/src/PostHogTelemetryClient.ts` +**File:** `packages/telemetry/src/CLITelemetryClient.ts` -- Uses `posthog-node` SDK +- Forwards events to the CLI server via `POST /telemetry/capture` +- The CLI uses `kilo-telemetry` (which wraps `posthog-node`) to send events to PostHog - **Distinct ID**: defaults to `vscode.env.machineId`, upgrades to user email when authenticated via `updateIdentity()` - **Privacy filters**: - Git properties (`repositoryUrl`, `repositoryName`, `defaultBranch`) always filtered - Error details (`errorMessage`, `cliPath`, `stderrPreview`) filtered for organization users - **Opt-in logic**: Requires BOTH VSCode global telemetry level = `"all"` AND user opt-in -- **Event exclusions**: `TASK_MESSAGE` is excluded from PostHog (too verbose) -- **Exception capture**: `captureException()` sends structured errors to PostHog +- **Event exclusions**: `TASK_MESSAGE` is excluded (too verbose) +- **Exception capture**: `captureException()` sends structured errors via the CLI endpoint ### 1.4 `DebugTelemetryClient` — Development Client @@ -57,13 +76,17 @@ graph TD - Always enabled, logs to console - Registered only in `NODE_ENV === "development"` -### 1.5 `TelemetryClient` (Webview) — Browser-side PostHog +### 1.5 Webview Telemetry — `postMessage` Proxy -**File:** `webview-ui/src/utils/TelemetryClient.ts` +**File:** `webview-ui/src/utils/telemetry.ts` -- Uses `posthog-js` (browser SDK) -- Initialized with API key from extension state, distinct ID = `machineId` -- Used for frontend-specific events (tab views, button clicks, marketplace interactions) +The webview does **not** use any PostHog SDK. Instead: + +1. The webview calls a thin helper that fires `postMessage({ type: "telemetry", event, properties })` to the extension host. +2. The extension host receives the message, enriches it with VS Code-specific properties (from `KiloProvider.getTelemetryProperties()`), and forwards the event to the CLI via `POST /telemetry/capture`. +3. The CLI's `kilo-telemetry` package sends the event to PostHog. + +This keeps the webview dependency-free for analytics and ensures all telemetry flows through a single, auditable path. --- @@ -73,26 +96,28 @@ graph TD sequenceDiagram participant Ext as extension.ts participant TS as TelemetryService - participant PH as PostHogTelemetryClient - participant CP as ClineProvider - participant WV as Webview TelemetryClient + participant CLI as CLI Server + participant KP as KiloProvider + participant WV as Webview - Ext->>TS: createInstance - Ext->>PH: new PostHogTelemetryClient - Ext->>TS: register PostHogTelemetryClient - Note over PH: API key from env KILOCODE_POSTHOG_API_KEY - CP->>TS: setProvider - ClineProvider as provider - Note over TS,CP: ClineProvider implements TelemetryPropertiesProvider - WV->>WV: updateTelemetryState with apiKey + machineId - Note over WV: posthog.init + posthog.identify + Ext->>CLI: Start CLI server (pass telemetryLevel from vscode.env) + Ext->>TS: createInstance() + Ext->>TS: register CLITelemetryClient (points to CLI endpoint) + KP->>TS: setProvider(this) + Note over TS,KP: KiloProvider implements TelemetryPropertiesProvider + WV->>Ext: postMessage({ type: "telemetry", event, properties }) + Ext->>Ext: Enrich with VS Code properties + Ext->>CLI: POST /telemetry/capture + CLI->>CLI: kilo-telemetry → posthog-node → PostHog ``` **Key code paths:** -1. `extension.ts:119` — `TelemetryService.createInstance()` -2. `extension.ts:133` — `new PostHogTelemetryClient()` registered -3. `ClineProvider` constructor — `TelemetryService.instance.setProvider(this)` -4. Extension shutdown — `TelemetryService.instance.shutdown()` +1. `extension.ts` — Start CLI server with `telemetryLevel` from `vscode.env.telemetryLevel` +2. `extension.ts` — `TelemetryService.createInstance()`, register `CLITelemetryClient` +3. `KiloProvider` constructor — `TelemetryService.instance.setProvider(this)` +4. Webview posts telemetry messages via `postMessage` → extension host → CLI +5. Extension shutdown — `TelemetryService.instance.shutdown()` --- @@ -198,7 +223,7 @@ All events are defined in `TelemetryEventName` enum (`packages/types/src/telemet ## 4. Properties Attached to Every Event -Every event gets enriched with properties from `ClineProvider.getTelemetryProperties()`: +Every event gets enriched with properties from `KiloProvider.getTelemetryProperties()`: ### Static App Properties (computed once) @@ -217,7 +242,7 @@ Every event gets enriched with properties from `ClineProvider.getTelemetryProper ### Git Properties (computed once) -- `repositoryUrl`, `repositoryName`, `defaultBranch` (filtered out before sending by PostHog client) +- `repositoryUrl`, `repositoryName`, `defaultBranch` (filtered out before sending by telemetry client) --- @@ -229,6 +254,10 @@ Every event gets enriched with properties from `ClineProvider.getTelemetryProper - Telemetry enabled only when: **VSCode telemetry level = "all"** AND **user setting ≠ "disabled"** - Wrapper apps can force telemetry enabled via environment variable +### VS Code Telemetry Level as Master Control + +`vscode.env.telemetryLevel` is passed as a startup parameter to the CLI server. If VS Code telemetry is disabled (level is `"off"` or `"crash"`), the CLI suppresses all PostHog sending entirely. This ensures the user's VS Code telemetry preference is respected across the entire stack — webview, extension host, and CLI. + ### Identity Management - Default: `vscode.env.machineId` (anonymous) @@ -278,9 +307,9 @@ Both have type guards (`isApiProviderError()`, `isConsecutiveMistakeError()`) an ## 7. Implementation Recommendations for New Extension -1. **Use PostHog** as the analytics backend — the extension uses `posthog-node` server-side and `posthog-js` client-side +1. **Use `kilo-telemetry` via CLI proxy** — all PostHog communication goes through the CLI's `POST /telemetry/capture` endpoint. The extension does not include `posthog-node` or `posthog-js` directly. 2. **Singleton service pattern** — single `TelemetryService` instance, multiple pluggable clients -3. **Properties provider pattern** — the main provider class implements `TelemetryPropertiesProvider` to inject context +3. **Properties provider pattern** — `KiloProvider` implements `TelemetryPropertiesProvider` to inject VS Code context into every event 4. **Typed events** — all event names in an enum, with typed capture methods on the service 5. **Event subscription/filtering** — clients can include/exclude specific events 6. **Property filtering** — per-client property filtering (privacy controls) @@ -293,13 +322,17 @@ Both have type guards (`isApiProviderError()`, `isConsecutiveMistakeError()`) an ## 8. Package Dependencies -### Server-side (Extension Host) +### Server-side (CLI) -- `posthog-node` — PostHog Node.js SDK +- `kilo-telemetry` package (already in monorepo, uses `posthog-node` internally) ### Client-side (Webview) -- `posthog-js` — PostHog browser SDK +- No PostHog SDK needed — telemetry events are sent via `postMessage` to the extension host + +### Extension Host + +- No PostHog SDK — events are proxied through the CLI's `POST /telemetry/capture` endpoint ### Shared Types From 310b05aecfa2693e9685633bd52f021ceca019fb Mon Sep 17 00:00:00 2001 From: Mark IJbema Date: Tue, 17 Feb 2026 16:38:38 +0100 Subject: [PATCH 3/7] docs: fix core components section for CLI-owned telemetry --- .../docs/non-agent-features/telemetry.md | 67 +++++++++++-------- 1 file changed, 40 insertions(+), 27 deletions(-) diff --git a/packages/kilo-vscode/docs/non-agent-features/telemetry.md b/packages/kilo-vscode/docs/non-agent-features/telemetry.md index cbbbfba25..606648cec 100644 --- a/packages/kilo-vscode/docs/non-agent-features/telemetry.md +++ b/packages/kilo-vscode/docs/non-agent-features/telemetry.md @@ -37,44 +37,57 @@ Key architectural decisions for telemetry in the new extension: ## 1. Core Components -### 1.1 `TelemetryService` — Singleton Facade +Telemetry is split across two layers: the **CLI** (which owns all agent/LLM/tool events) and the **extension** (which captures UI-only events and proxies them to the CLI). -**File:** `packages/telemetry/src/TelemetryService.ts` +### 1.1 Telemetry Ownership — CLI vs Extension -- Created once at extension activation via `TelemetryService.createInstance()` -- Holds an array of `TelemetryClient` implementations -- Exposes typed convenience methods for each event: `captureTaskCreated()`, `captureLlmCompletion()`, etc. -- Also handles: `captureException()`, `updateIdentity()`, `updateTelemetryState()`, `shutdown()` +#### CLI side: `kilo-telemetry` package -### 1.2 `BaseTelemetryClient` — Abstract Base +**Package:** `packages/kilo-telemetry/` -**File:** `packages/telemetry/src/BaseTelemetryClient.ts` +The CLI owns all agent, LLM, and tool telemetry. The `kilo-telemetry` package provides: -- Holds a `WeakRef` to a `TelemetryPropertiesProvider` (the `KiloProvider`) -- Implements event subscription/filtering (include/exclude lists via `TelemetryEventSubscription`) -- Implements property filtering via `isPropertyCapturable()` — subclasses can override -- Merges provider properties with event-specific properties via `getEventProperties()` +- `Telemetry.capture()` — sends events to PostHog via `posthog-node` +- `Telemetry.getTracer()` — OpenTelemetry tracing for spans/traces +- Identity management (distinct ID, user properties) +- Consent enforcement (respects VS Code telemetry level passed at CLI startup) -### 1.3 `CLITelemetryClient` — Production Client +All events related to task lifecycle, LLM completions, tool usage, context condensation, checkpoints, etc. are captured internally by the CLI — the extension never sees or sends these. -**File:** `packages/telemetry/src/CLITelemetryClient.ts` +#### Extension side: `TelemetryProxy` service -- Forwards events to the CLI server via `POST /telemetry/capture` -- The CLI uses `kilo-telemetry` (which wraps `posthog-node`) to send events to PostHog -- **Distinct ID**: defaults to `vscode.env.machineId`, upgrades to user email when authenticated via `updateIdentity()` -- **Privacy filters**: - - Git properties (`repositoryUrl`, `repositoryName`, `defaultBranch`) always filtered - - Error details (`errorMessage`, `cliPath`, `stderrPreview`) filtered for organization users -- **Opt-in logic**: Requires BOTH VSCode global telemetry level = `"all"` AND user opt-in -- **Event exclusions**: `TASK_MESSAGE` is excluded (too verbose) -- **Exception capture**: `captureException()` sends structured errors via the CLI endpoint +**File:** `packages/kilo-vscode/src/services/telemetry/TelemetryProxy.ts` *(planned)* -### 1.4 `DebugTelemetryClient` — Development Client +A lightweight service in the extension that: -**File:** `packages/telemetry/src/DebugTelemetryClient.ts` +- Captures **UI-only events**: tab views, button clicks, auth UI events, marketplace interactions +- Exposes typed convenience methods for extension events: `captureTabShown()`, `captureTitleButtonClicked()`, `captureAuthEvent()`, `captureMarketplaceAction()`, etc. +- Forwards all events to the CLI via `POST /telemetry/capture` +- Enriches events with VS Code-specific properties (from `KiloProvider.getTelemetryProperties()`) +- Does **NOT** have methods like `captureLlmCompletion()` or `captureToolUsed()` — those are CLI-internal -- Always enabled, logs to console -- Registered only in `NODE_ENV === "development"` +### 1.2 `BaseTelemetryClient` — Old Extension Only + +> **Note:** This abstraction existed in the old extension's in-process architecture, where the extension itself needed pluggable telemetry backends. In the new architecture, the extension simply proxies events to the CLI — no pluggable client abstraction is needed on the extension side. The CLI's `kilo-telemetry` package handles all backend concerns internally. + +### 1.3 `PostHogTelemetryClient` — CLI-side (`kilo-telemetry`) + +**Package:** `packages/kilo-telemetry/` + +- The CLI uses `posthog-node` internally via the `kilo-telemetry` package +- The extension **never** touches PostHog directly — all events flow through `POST /telemetry/capture` +- **Distinct ID**: defaults to `vscode.env.machineId` (passed at CLI startup), upgrades to user email on auth +- **Privacy filters**: Git properties always filtered; error details filtered for organization users +- **Opt-in logic**: Requires BOTH VS Code telemetry level = `"all"` AND user opt-in +- **Event exclusions**: `TASK_MESSAGE` excluded (too verbose) + +### 1.4 `DebugTelemetryClient` — CLI-side Only + +**Package:** `packages/kilo-telemetry/` + +- Logs telemetry events to console in development mode +- CLI-side only — registered when `NODE_ENV === "development"` +- The extension can independently log telemetry events to its own VS Code output channel for debugging, but this is separate from the CLI's debug client ### 1.5 Webview Telemetry — `postMessage` Proxy From bc3a50b40b9d1118625484a91fe269d5a01a5e0b Mon Sep 17 00:00:00 2001 From: Mark IJbema Date: Tue, 17 Feb 2026 16:41:32 +0100 Subject: [PATCH 4/7] =?UTF-8?q?docs:=20simplify=20telemetry=20core=20compo?= =?UTF-8?q?nents=20=E2=80=94=20single=20singleton,=20no=20inheritance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../docs/non-agent-features/telemetry.md | 60 +++++-------------- 1 file changed, 15 insertions(+), 45 deletions(-) diff --git a/packages/kilo-vscode/docs/non-agent-features/telemetry.md b/packages/kilo-vscode/docs/non-agent-features/telemetry.md index 606648cec..6746f32e2 100644 --- a/packages/kilo-vscode/docs/non-agent-features/telemetry.md +++ b/packages/kilo-vscode/docs/non-agent-features/telemetry.md @@ -39,55 +39,25 @@ Key architectural decisions for telemetry in the new extension: Telemetry is split across two layers: the **CLI** (which owns all agent/LLM/tool events) and the **extension** (which captures UI-only events and proxies them to the CLI). -### 1.1 Telemetry Ownership — CLI vs Extension +### 1.1 Extension-side: `TelemetryProxy` — Simple Singleton -#### CLI side: `kilo-telemetry` package +A single class that captures UI events from the extension host and webview, then forwards them to the CLI. -**Package:** `packages/kilo-telemetry/` +- One singleton instance, created at extension activation +- `capture(event, properties)` — sends to CLI via `POST /telemetry/capture` and logs to `console.log` +- Receives webview events via `postMessage` handler +- Enriches all events with VS Code properties (`vscodeVersion`, `editorName`, `machineId`) +- No inheritance, no pluggable backends, no abstract classes +- Respects `vscode.env.telemetryLevel` — if disabled, `capture()` is a no-op -The CLI owns all agent, LLM, and tool telemetry. The `kilo-telemetry` package provides: +### 1.2 CLI-side: `kilo-telemetry` Package -- `Telemetry.capture()` — sends events to PostHog via `posthog-node` -- `Telemetry.getTracer()` — OpenTelemetry tracing for spans/traces -- Identity management (distinct ID, user properties) -- Consent enforcement (respects VS Code telemetry level passed at CLI startup) - -All events related to task lifecycle, LLM completions, tool usage, context condensation, checkpoints, etc. are captured internally by the CLI — the extension never sees or sends these. - -#### Extension side: `TelemetryProxy` service - -**File:** `packages/kilo-vscode/src/services/telemetry/TelemetryProxy.ts` *(planned)* - -A lightweight service in the extension that: - -- Captures **UI-only events**: tab views, button clicks, auth UI events, marketplace interactions -- Exposes typed convenience methods for extension events: `captureTabShown()`, `captureTitleButtonClicked()`, `captureAuthEvent()`, `captureMarketplaceAction()`, etc. -- Forwards all events to the CLI via `POST /telemetry/capture` -- Enriches events with VS Code-specific properties (from `KiloProvider.getTelemetryProperties()`) -- Does **NOT** have methods like `captureLlmCompletion()` or `captureToolUsed()` — those are CLI-internal - -### 1.2 `BaseTelemetryClient` — Old Extension Only - -> **Note:** This abstraction existed in the old extension's in-process architecture, where the extension itself needed pluggable telemetry backends. In the new architecture, the extension simply proxies events to the CLI — no pluggable client abstraction is needed on the extension side. The CLI's `kilo-telemetry` package handles all backend concerns internally. - -### 1.3 `PostHogTelemetryClient` — CLI-side (`kilo-telemetry`) - -**Package:** `packages/kilo-telemetry/` - -- The CLI uses `posthog-node` internally via the `kilo-telemetry` package -- The extension **never** touches PostHog directly — all events flow through `POST /telemetry/capture` -- **Distinct ID**: defaults to `vscode.env.machineId` (passed at CLI startup), upgrades to user email on auth -- **Privacy filters**: Git properties always filtered; error details filtered for organization users -- **Opt-in logic**: Requires BOTH VS Code telemetry level = `"all"` AND user opt-in -- **Event exclusions**: `TASK_MESSAGE` excluded (too verbose) - -### 1.4 `DebugTelemetryClient` — CLI-side Only - -**Package:** `packages/kilo-telemetry/` - -- Logs telemetry events to console in development mode -- CLI-side only — registered when `NODE_ENV === "development"` -- The extension can independently log telemetry events to its own VS Code output channel for debugging, but this is separate from the CLI's debug client +Already exists at `packages/kilo-telemetry/`. Handles: +- All PostHog communication (`posthog-node`) +- OpenTelemetry integration for AI SDK spans +- Identity management (machineId → email upgrade on auth) +- Privacy filtering (git info stripped, error details filtered for org users) +- The `POST /telemetry/capture` endpoint (to be added) routes through this package ### 1.5 Webview Telemetry — `postMessage` Proxy From 53ed970334bedd05a36307e6717ee6a598631640 Mon Sep 17 00:00:00 2001 From: Mark IJbema Date: Tue, 17 Feb 2026 16:47:32 +0100 Subject: [PATCH 5/7] docs: document VS Code telemetry levels and mapping --- .../docs/non-agent-features/telemetry.md | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/packages/kilo-vscode/docs/non-agent-features/telemetry.md b/packages/kilo-vscode/docs/non-agent-features/telemetry.md index 6746f32e2..34812ecf6 100644 --- a/packages/kilo-vscode/docs/non-agent-features/telemetry.md +++ b/packages/kilo-vscode/docs/non-agent-features/telemetry.md @@ -234,12 +234,26 @@ Every event gets enriched with properties from `KiloProvider.getTelemetryPropert ### User Opt-in Model - Three states: `"unset"`, `"enabled"`, `"disabled"` (see `TelemetrySetting` type) -- Telemetry enabled only when: **VSCode telemetry level = "all"** AND **user setting ≠ "disabled"** +- Full telemetry enabled only when: **VS Code telemetry level = `"all"`** AND **user setting ≠ `"disabled"`** +- VS Code exposes four telemetry levels via `vscode.env.telemetryLevel`: + - `"off"` — No telemetry at all + - `"crash"` — Only crash reports + - `"error"` — Crash reports + errors + - `"all"` — Full telemetry enabled - Wrapper apps can force telemetry enabled via environment variable ### VS Code Telemetry Level as Master Control -`vscode.env.telemetryLevel` is passed as a startup parameter to the CLI server. If VS Code telemetry is disabled (level is `"off"` or `"crash"`), the CLI suppresses all PostHog sending entirely. This ensures the user's VS Code telemetry preference is respected across the entire stack — webview, extension host, and CLI. +`vscode.env.telemetryLevel` is passed as a startup parameter to the CLI server. The CLI maps each level to a specific telemetry behavior: + +| VS Code Level | What we send | +|---------------|-------------| +| `"off"` | Nothing — CLI telemetry fully disabled | +| `"crash"` | Only `captureException()` for crashes | +| `"error"` | Exceptions + error events | +| `"all"` | All telemetry events | + +This ensures the user's VS Code telemetry preference is respected across the entire stack — webview, extension host, and CLI. The mapping is enforced at the CLI server level, so even if the extension host or webview attempts to send an event, the CLI will suppress it based on the configured level. ### Identity Management From 98fd05c1b83fa6c24b7554645c41856a317a7973 Mon Sep 17 00:00:00 2001 From: Mark IJbema Date: Tue, 17 Feb 2026 16:57:35 +0100 Subject: [PATCH 6/7] docs: simplify telemetry consent to all-or-nothing --- .../docs/non-agent-features/telemetry.md | 16 +++------------- 1 file changed, 3 insertions(+), 13 deletions(-) diff --git a/packages/kilo-vscode/docs/non-agent-features/telemetry.md b/packages/kilo-vscode/docs/non-agent-features/telemetry.md index 34812ecf6..0dc31ba95 100644 --- a/packages/kilo-vscode/docs/non-agent-features/telemetry.md +++ b/packages/kilo-vscode/docs/non-agent-features/telemetry.md @@ -235,25 +235,15 @@ Every event gets enriched with properties from `KiloProvider.getTelemetryPropert - Three states: `"unset"`, `"enabled"`, `"disabled"` (see `TelemetrySetting` type) - Full telemetry enabled only when: **VS Code telemetry level = `"all"`** AND **user setting ≠ `"disabled"`** -- VS Code exposes four telemetry levels via `vscode.env.telemetryLevel`: - - `"off"` — No telemetry at all - - `"crash"` — Only crash reports - - `"error"` — Crash reports + errors - - `"all"` — Full telemetry enabled - Wrapper apps can force telemetry enabled via environment variable ### VS Code Telemetry Level as Master Control -`vscode.env.telemetryLevel` is passed as a startup parameter to the CLI server. The CLI maps each level to a specific telemetry behavior: +`vscode.env.telemetryLevel` is passed as a startup parameter to the CLI server. -| VS Code Level | What we send | -|---------------|-------------| -| `"off"` | Nothing — CLI telemetry fully disabled | -| `"crash"` | Only `captureException()` for crashes | -| `"error"` | Exceptions + error events | -| `"all"` | All telemetry events | +Telemetry is only active when `vscode.env.telemetryLevel` is `"all"`. Any other level (`"off"`, `"crash"`, `"error"`) disables all telemetry — the CLI startup parameter will indicate telemetry is disabled and no events will be sent. -This ensures the user's VS Code telemetry preference is respected across the entire stack — webview, extension host, and CLI. The mapping is enforced at the CLI server level, so even if the extension host or webview attempts to send an event, the CLI will suppress it based on the configured level. +This ensures the user's VS Code telemetry preference is respected across the entire stack — webview, extension host, and CLI. ### Identity Management From a391b1523cdad943cd375ab998d2c597c527ab64 Mon Sep 17 00:00:00 2001 From: Mark IJbema Date: Tue, 17 Feb 2026 17:04:38 +0100 Subject: [PATCH 7/7] docs: add gateway integration, identity continuity, and schema change process from Pedro's feedback --- .../docs/non-agent-features/telemetry.md | 42 +++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/packages/kilo-vscode/docs/non-agent-features/telemetry.md b/packages/kilo-vscode/docs/non-agent-features/telemetry.md index 0dc31ba95..bfb4e5f5c 100644 --- a/packages/kilo-vscode/docs/non-agent-features/telemetry.md +++ b/packages/kilo-vscode/docs/non-agent-features/telemetry.md @@ -258,6 +258,17 @@ This ensures the user's VS Code telemetry preference is respected across the ent - `TASK_MESSAGE` events are excluded from PostHog (contain full conversation) - Expected API errors (429, 402) are not reported via `shouldReportApiErrorToTelemetry()` +### Identity Continuity (Old → New Extension) + +Users switching from the old extension to the new one must not appear as two different users in PostHog. + +- **Old extension**: Uses `vscode.env.machineId` as PostHog `distinct_id` +- **CLI (`kilo-telemetry`)**: Generates a random UUID stored in `~/.config/kilo/telemetry-id` +- **Authenticated users**: Both old and new call `alias()` on auth, which merges identities server-side — no issue +- **Unauthenticated users**: Will have completely different `distinct_id` values with no link + +**Action required:** The extension must pass `vscode.env.machineId` to the CLI at startup so `kilo-telemetry` uses it as the `distinct_id` instead of (or aliased with) its own generated UUID. This ensures unauthenticated users maintain identity continuity. + --- ## 6. Structured Error Classes @@ -324,3 +335,34 @@ Both have type guards (`isApiProviderError()`, `isConsecutiveMistakeError()`) an ### Shared Types - `zod` — for schema validation of telemetry properties + +--- + +## 9. Gateway Integration + +### 9.1 `editor_name` Header + +The Kilo Gateway records `editor_name` on every API request for cost attribution by feature. The CLI currently defaults to `"Kilo CLI"`, so when the extension spawns the CLI server, all gateway usage gets misattributed as CLI usage. + +**Action required:** The extension must pass `editor_name` (e.g., `"Kilo VSCode"`) as a CLI startup parameter so the gateway correctly attributes requests to the VS Code extension. + +### 9.2 `platform` Field in Session Ingest + +The session ingest API records a `platform` field per session, which also defaults to `"cli"`. This must be set to identify the new extension (e.g., `"vscode"`) so sessions are correctly attributed. + +**Action required:** Pass `platform` as a CLI startup parameter alongside `editor_name`. + +### 9.3 `X-KILOCODE-TASKID` Persistence (Nice-to-have) + +The gateway already sends `X-KILOCODE-TASKID` (session ID) on every request, but the backend doesn't persist it in `microdollar_usage_metadata`. If persisted, usage records could be joined to sessions for deterministic cost attribution, eliminating the time-matching heuristic currently used for ~20% of records. This is a backend change, not an extension change. + +--- + +## 10. Schema Change Process + +If any telemetry events or properties are intentionally dropped, renamed, or have their semantics changed, the analytics team (Pedro) must be notified **before** the change ships. This allows analytics dashboards and queries to be updated in parallel. + +Checklist for schema changes: +- [ ] Document the change in this file +- [ ] Create a GitHub issue tagged with `telemetry` +- [ ] Notify @pedroheyerdahl in the PR description