docs(ai): remove stale design documents (#43512)
This commit is contained in:
Kit Langton
@@ -49,7 +49,7 @@ Filter or narrow `LLMEvent` streams with `LLMEvent.is.*` (camelCase guards, e.g.
|
||||
|
||||
### Routes
|
||||
|
||||
A route is the registered, runnable composition of four orthogonal pieces:
|
||||
A route is the runnable composition of four orthogonal pieces:
|
||||
|
||||
- **`Protocol`** (`src/route/protocol.ts`) — semantic API contract. Owns request body construction (`body.from`), the body schema (`body.schema`), the streaming-event schema (`stream.event`), and the event-to-`LLMEvent` state machine (`stream.step`). `Route.make(...)` validates and JSON-encodes the body from `body.schema` and decodes frames with `stream.event`. Examples: `OpenAIChat.protocol`, `OpenResponses.protocol`, `OpenAIResponses.protocol`, `AnthropicMessages.protocol`, `Gemini.protocol`, `BedrockConverse.protocol`.
|
||||
- **`Endpoint`** (`src/route/endpoint.ts`) — URL construction. The host, path, and route query live on the endpoint. `Endpoint.path("/chat/completions", { baseURL })` is the common case; pass a function for paths that embed the model id or a body field (e.g. `Endpoint.path(({ body }) => `/model/${body.modelId}/converse-stream`)`).
|
||||
@@ -66,7 +66,7 @@ export const route = Route.make({
|
||||
endpoint: Endpoint.path("/chat/completions", {
|
||||
baseURL: "https://api.openai.com/v1",
|
||||
}),
|
||||
auth: Auth.bearer(),
|
||||
auth: Auth.bearer(Auth.config("OPENAI_API_KEY")),
|
||||
framing: Framing.sse,
|
||||
})
|
||||
```
|
||||
@@ -79,7 +79,7 @@ When a provider supports multiple physical transports, selection remains executi
|
||||
|
||||
### URL Construction
|
||||
|
||||
`Endpoint` owns `{ baseURL, path, query }`. Each protocol route includes a canonical endpoint when the provider has one (e.g. `https://api.openai.com/v1`); provider helpers override endpoint fields by configuring the route before selecting a model. Routes that have no canonical URL (OpenAI-compatible Chat, GitHub Copilot) require configuration before execution.
|
||||
`Endpoint` owns `{ baseURL, path, query }`. Each protocol route includes a canonical endpoint when the provider has one (e.g. `https://api.openai.com/v1`); provider helpers override endpoint fields by configuring the route before selecting a model. Generic OpenAI-compatible routes have no canonical URL and require configuration before execution.
|
||||
|
||||
For providers where the URL is derived from typed inputs (Azure resource name, Bedrock region), the provider helper configures the route endpoint before calling `.model(...)`. Use `AtLeastOne<T>` from `route/auth-options.ts` for inputs that accept either of two derivation paths (Azure: `resourceName` or `baseURL`).
|
||||
|
||||
@@ -173,19 +173,17 @@ Routes lower these into provider-native assistant tool-call messages and tool-re
|
||||
|
||||
### Tool dispatch
|
||||
|
||||
`LLM.stream(request)` and `LLM.generate(request)` each run exactly one provider turn. Add tool schemas to `request.tools` with `Tool.toDefinitions(tools)`. When a caller wants the package's typed one-call execution behavior, pass each canonical local `tool-call` event to `ToolRuntime.dispatch(tools, call)`.
|
||||
`LLM.stream(request)` and `LLM.generate(request)` each run exactly one model call. Add tool schemas to `request.tools` with `Tool.toDefinitions(tools)`. When a caller wants the package's typed one-call execution behavior, pass each canonical local `tool-call` event to `ToolRuntime.dispatch(tools, call)`.
|
||||
|
||||
```ts
|
||||
const get_weather = tool({
|
||||
const get_weather = Tool.make({
|
||||
description: "Get current weather for a city",
|
||||
parameters: Schema.Struct({ city: Schema.String }),
|
||||
success: Schema.Struct({ temperature: Schema.Number, condition: Schema.String }),
|
||||
execute: ({ city }) =>
|
||||
execute: (input) =>
|
||||
Effect.gen(function* () {
|
||||
// city: string — typed from parameters Schema
|
||||
const data = yield* WeatherApi.fetch(city)
|
||||
const data = yield* WeatherApi.fetch(input.city)
|
||||
return { temperature: data.temp, condition: data.cond }
|
||||
// return type checked against success Schema
|
||||
}),
|
||||
})
|
||||
|
||||
|
||||
@@ -1,1114 +0,0 @@
|
||||
# AI Library Design
|
||||
|
||||
> Discussion draft. This document describes an intended clean-break redesign of
|
||||
> the current private `@opencode-ai/ai` API. Names and exact TypeScript signatures
|
||||
> are illustrative until implementation, but the domain boundaries and defaults
|
||||
> are deliberate.
|
||||
|
||||
## Status
|
||||
|
||||
- Proposed package: `@opencode-ai/ai`
|
||||
- Initial stable domain: `LLM`
|
||||
- Release posture: pre-1.0, with a stable-core intent
|
||||
- Migration posture: clean break; do not preserve compatibility aliases
|
||||
- Primary audience: general-purpose TypeScript developers using Effect
|
||||
- Secondary audience: OpenCode and other durable agent runtimes
|
||||
|
||||
The package name leaves room for future domains such as embeddings, images, and
|
||||
speech. Those domains are not part of this design and should not be forced into
|
||||
the LLM run/turn model.
|
||||
|
||||
## Goals
|
||||
|
||||
1. Make a useful model call require very little code.
|
||||
2. Make the default behavior good enough that most callers do not configure it.
|
||||
3. Let advanced callers inspect, transform, or replace every important stage.
|
||||
4. Keep provider quirks behind provider and protocol boundaries.
|
||||
5. Preserve one provider turn as an explicit primitive for durable runtimes.
|
||||
6. Keep serializable request data separate from process-local execution behavior.
|
||||
7. Make unsupported combinations fail locally with useful typed errors.
|
||||
8. Stay Effect-native without making package-specific service provisioning part
|
||||
of every call site.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- A global provider or model registry
|
||||
- Durable agent orchestration or persistence
|
||||
- Session history ownership
|
||||
- Permission handling
|
||||
- Cost billing or accounting guarantees
|
||||
- Runtime model-catalog network requests
|
||||
- Compatibility with the current private API
|
||||
- Designing embeddings, image generation, speech, or transcription now
|
||||
|
||||
## Design Principles
|
||||
|
||||
### Progressive disclosure
|
||||
|
||||
The API has four layers:
|
||||
|
||||
1. **Run a model** with `LLM.generate` or `LLM.stream`.
|
||||
2. **Control one provider turn** with `LLM.generateTurn` or `LLM.streamTurn`.
|
||||
3. **Customize execution** with model defaults, call options, hooks, and provider
|
||||
configuration.
|
||||
4. **Author providers** with experimental provider definitions and protocols.
|
||||
|
||||
Normal documentation should teach only the first layer initially.
|
||||
|
||||
### Values over registries
|
||||
|
||||
Provider definitions, configured providers, models, protocols, tools, and hooks
|
||||
are immutable values. Importing a provider does not register anything globally.
|
||||
|
||||
### Portable data, local behavior
|
||||
|
||||
Requests, messages, tool definitions, events, usage, and result projections are
|
||||
plain immutable data with schemas. Configured models, executable tools, hooks,
|
||||
and provider definitions may contain functions and Effect requirements and are
|
||||
not serializable.
|
||||
|
||||
### Strong defaults, explicit overrides
|
||||
|
||||
Defaults should make common calls correct without hiding where behavior comes
|
||||
from. Overrides compose in a documented order and never require patching
|
||||
installed dependencies.
|
||||
|
||||
## Domain Model
|
||||
|
||||
### Provider Definition
|
||||
|
||||
An immutable, declarative description of a provider integration. It owns model
|
||||
selection, option schemas, catalog corrections, protocols, and provider-wide
|
||||
hooks. It is an experimental provider-authoring API.
|
||||
|
||||
### Configured Provider
|
||||
|
||||
A provider definition bound to deployment concerns such as credentials,
|
||||
endpoint, transport, and provider headers.
|
||||
|
||||
`configure(...)` is intentionally deployment-only. It does not establish hidden
|
||||
generation defaults.
|
||||
|
||||
### Model
|
||||
|
||||
A process-local executable model value selected from a configured provider. It
|
||||
contains identity, capabilities, pricing metadata, provider-specific option
|
||||
types, reusable request-behavior defaults, and hidden execution behavior.
|
||||
|
||||
Normal users do not need to learn the current `Route` composite. Protocol,
|
||||
endpoint, auth, transport, and hooks are bound behind `LanguageModel`.
|
||||
|
||||
### Request
|
||||
|
||||
Portable, model-independent input for a model call. It may contain system
|
||||
instructions, messages, tool definitions, generation controls, output intent,
|
||||
cache policy, and metadata. It does not contain a configured model, executable
|
||||
tool handlers, or hooks.
|
||||
|
||||
### Provider Turn
|
||||
|
||||
Exactly one request to a model provider and its normalized response. It does not
|
||||
execute local tools or continue the conversation.
|
||||
|
||||
### Model Run
|
||||
|
||||
A complete interaction consisting of one or more provider turns. A run executes
|
||||
local tools, appends their results, and continues until the model completes or a
|
||||
stopping condition matches.
|
||||
|
||||
### TurnResult
|
||||
|
||||
The result of exactly one provider turn.
|
||||
|
||||
### GenerateResult
|
||||
|
||||
The result of a complete model run. It preserves every turn, tool activity,
|
||||
aggregate usage, and estimated cost while exposing shortcuts to the final output.
|
||||
|
||||
### Protocol
|
||||
|
||||
The provider-wire contract that lowers portable requests into provider-native
|
||||
bodies and raises provider-native stream events into normalized turn events.
|
||||
Protocols are public, reusable, fully inspectable, and immutably patchable, but
|
||||
the entire protocol-authoring API is experimental.
|
||||
|
||||
## Happy Path
|
||||
|
||||
### Effect
|
||||
|
||||
```ts
|
||||
import { Effect } from "effect"
|
||||
import { LLM } from "@opencode-ai/ai"
|
||||
import { OpenAI } from "@opencode-ai/ai/providers/openai"
|
||||
|
||||
// Environment-based credentials are a provider default. No LLMClient layer is
|
||||
// required: the Effect exposes standard runtime dependencies directly.
|
||||
const model = OpenAI.model("gpt-4.1-mini")
|
||||
|
||||
const program = Effect.gen(function* () {
|
||||
const result = yield* LLM.generate({
|
||||
model,
|
||||
system: "You are concise.",
|
||||
prompt: "Explain Effect in one sentence.",
|
||||
})
|
||||
|
||||
// `generate` always returns GenerateResult, even when the run has one turn.
|
||||
console.log(result.text)
|
||||
console.log(result.turns.length) // 1
|
||||
console.log(result.usage)
|
||||
console.log(result.cost) // Estimated cost, or undefined if any turn is unpriced.
|
||||
})
|
||||
```
|
||||
|
||||
The required Effect environment should contain standard services plus services
|
||||
required by tools and hooks. It should not contain an `LLMClient` wrapper service.
|
||||
|
||||
### Current API
|
||||
|
||||
The current README appears similarly small but omits the package-specific service
|
||||
and layer required at runtime:
|
||||
|
||||
```ts
|
||||
// Current API: this request contains an executable model/route value.
|
||||
const request = LLM.request({
|
||||
model: OpenAI.configure({ apiKey }).responses("gpt-4o-mini"),
|
||||
prompt: "Say hello.",
|
||||
})
|
||||
|
||||
// Current API: this performs one provider turn, despite the broad name.
|
||||
const response = yield * LLM.generate(request)
|
||||
|
||||
// Current API: execution also needs LLMClient.layer and RequestExecutor services.
|
||||
```
|
||||
|
||||
The proposal removes mandatory request construction, removes package-specific
|
||||
runtime provisioning, and makes `generate` mean a complete run.
|
||||
|
||||
## Provider And Model Selection
|
||||
|
||||
### Environment defaults
|
||||
|
||||
```ts
|
||||
import { OpenAI } from "@opencode-ai/ai/providers/openai"
|
||||
|
||||
// Open strings receive autocomplete for IDs from the generated models.dev
|
||||
// snapshot but continue to accept newly released and fine-tuned model IDs.
|
||||
const model = OpenAI.model("gpt-4.1-mini")
|
||||
```
|
||||
|
||||
### Deployment configuration
|
||||
|
||||
```ts
|
||||
const openai = OpenAI.configure({
|
||||
apiKey,
|
||||
baseURL: "https://gateway.example.com/openai/v1",
|
||||
headers: {
|
||||
"x-tenant": "acme",
|
||||
},
|
||||
})
|
||||
|
||||
const model = openai.model("gpt-4.1-mini")
|
||||
```
|
||||
|
||||
`configure(...)` owns deployment concerns only:
|
||||
|
||||
- Credentials and authentication
|
||||
- Base URL and deployment location
|
||||
- Transport selection
|
||||
- Provider/deployment headers
|
||||
- Other provider-specific connection setup
|
||||
|
||||
It does not own temperature, maximum output tokens, cache policy, retry policy,
|
||||
tools, output schema, or system instructions.
|
||||
|
||||
### Reusable model defaults
|
||||
|
||||
```ts
|
||||
const model = OpenAI.model("gpt-4.1-mini", {
|
||||
generation: {
|
||||
temperature: 0.2,
|
||||
maxTokens: 2_000,
|
||||
},
|
||||
cache: "auto",
|
||||
provider: {
|
||||
store: false,
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
The second argument may default request behavior but not prompt/history or
|
||||
executable tools. Call-level values override model defaults.
|
||||
|
||||
Provider-specific options are inferred from the concrete model:
|
||||
|
||||
```ts
|
||||
yield *
|
||||
LLM.generate({
|
||||
model: OpenAI.model("gpt-4.1-mini"),
|
||||
prompt: "Hello",
|
||||
provider: {
|
||||
store: false,
|
||||
// OpenAI-specific autocomplete here; no `{ openai: ... }` nesting.
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
Code choosing between providers dynamically must narrow the model before using
|
||||
provider-specific options. Portable generation controls remain available without
|
||||
narrowing.
|
||||
|
||||
### Current API
|
||||
|
||||
```ts
|
||||
// Current API mixes deployment configuration and reusable request behavior.
|
||||
const model = OpenAI.configure({
|
||||
apiKey,
|
||||
generation: { maxTokens: 160 },
|
||||
providerOptions: {
|
||||
openai: { store: false },
|
||||
},
|
||||
}).model("gpt-4o-mini")
|
||||
```
|
||||
|
||||
The proposal separates deployment configuration from selected-model behavior and
|
||||
removes provider-keyed option bags when a concrete model already identifies the
|
||||
provider.
|
||||
|
||||
## Requests
|
||||
|
||||
### Inline input
|
||||
|
||||
```ts
|
||||
const result =
|
||||
yield *
|
||||
LLM.generate({
|
||||
model,
|
||||
system: "You are concise.",
|
||||
prompt: "Summarize this pull request.",
|
||||
generation: { maxTokens: 500 },
|
||||
})
|
||||
```
|
||||
|
||||
### Reusable portable request
|
||||
|
||||
```ts
|
||||
const request = LLM.request({
|
||||
system: "You are concise.",
|
||||
prompt: "Summarize this pull request.",
|
||||
generation: { maxTokens: 500 },
|
||||
})
|
||||
|
||||
// Bind process-local execution behavior only when running.
|
||||
const result = yield * LLM.generate({ model, request })
|
||||
```
|
||||
|
||||
`LLM.request(...)` returns a plain immutable object. Use ordinary object spread
|
||||
to derive another request:
|
||||
|
||||
```ts
|
||||
const longer = {
|
||||
...request,
|
||||
generation: {
|
||||
...request.generation,
|
||||
maxTokens: 1_000,
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
There is no `LLM.updateRequest(...)` helper. The current Schema-backed implementation
|
||||
uses `LLMRequest.update(...)` when canonical request data must be derived.
|
||||
|
||||
### Conversation history
|
||||
|
||||
```ts
|
||||
import { Message } from "@opencode-ai/ai"
|
||||
|
||||
const request = LLM.request({
|
||||
system: "You are concise.",
|
||||
messages: [
|
||||
Message.user("What is Effect?"),
|
||||
Message.assistant("A TypeScript library for typed functional effects."),
|
||||
Message.user("Why would I use it?"),
|
||||
],
|
||||
})
|
||||
```
|
||||
|
||||
Message helpers return plain immutable data. Object literals remain valid when
|
||||
they satisfy the same input type.
|
||||
|
||||
`system` stays separate from chronological messages because it is the initial
|
||||
privileged instruction. A chronological system message represents an instruction
|
||||
change at a specific point in history.
|
||||
|
||||
## Complete Runs
|
||||
|
||||
### Automatic local tool loop
|
||||
|
||||
```ts
|
||||
import { Effect, Schema } from "effect"
|
||||
import { LLM, Tool } from "@opencode-ai/ai"
|
||||
|
||||
const tools = {
|
||||
getWeather: Tool.make({
|
||||
description: "Get current weather for a city.",
|
||||
parameters: Schema.Struct({ city: Schema.String }),
|
||||
success: Schema.Struct({ forecast: Schema.String }),
|
||||
|
||||
// Tool service requirements and typed errors flow into LLM.generate's
|
||||
// Effect environment/error model instead of being erased.
|
||||
execute: ({ city }) => Weather.get(city),
|
||||
|
||||
// Expected domain failures need an explicit model-visible representation.
|
||||
formatError: (error) => ({
|
||||
type: "text",
|
||||
text: `Weather lookup failed: ${error.message}`,
|
||||
}),
|
||||
}),
|
||||
}
|
||||
|
||||
const result =
|
||||
yield *
|
||||
LLM.generate({
|
||||
model,
|
||||
prompt: "What is the weather in London?",
|
||||
tools,
|
||||
})
|
||||
|
||||
// The runtime advertises definitions, dispatches calls, records results, and
|
||||
// continues provider turns automatically.
|
||||
console.log(result.text)
|
||||
console.log(result.turns)
|
||||
console.log(result.toolExecutions)
|
||||
```
|
||||
|
||||
The default stopping condition is equivalent to:
|
||||
|
||||
```ts
|
||||
stopWhen: StopWhen.turnCount(20)
|
||||
```
|
||||
|
||||
This matches the Vercel AI SDK `ToolLoopAgent` default. Reaching the limit is a
|
||||
successful result with `stopReason: "max-turns"`, not an Effect failure.
|
||||
|
||||
### Custom stopping
|
||||
|
||||
```ts
|
||||
const result =
|
||||
yield *
|
||||
LLM.generate({
|
||||
model,
|
||||
prompt,
|
||||
tools,
|
||||
stopWhen: StopWhen.any(StopWhen.turnCount(8), StopWhen.hasToolCall("finalize")),
|
||||
})
|
||||
```
|
||||
|
||||
`stopWhen` accepts one predicate. Composition is explicit through combinators
|
||||
such as `StopWhen.any`, `StopWhen.all`, and `StopWhen.not`.
|
||||
|
||||
Successful run stop reasons are closed:
|
||||
|
||||
```ts
|
||||
type RunStopReason = "completed" | "max-turns" | "stop-condition"
|
||||
```
|
||||
|
||||
### Tool concurrency
|
||||
|
||||
Independent tool calls emitted in one turn run concurrently with a bounded,
|
||||
configurable concurrency limit. Results are appended in deterministic emitted
|
||||
order. The runtime does not infer dependencies between tool calls; the model must
|
||||
request dependent calls in separate turns.
|
||||
|
||||
Tools may declare an optional timeout. The overall run timeout still applies.
|
||||
|
||||
### Current API
|
||||
|
||||
Today callers must manually bridge every layer:
|
||||
|
||||
```ts
|
||||
const request = LLM.request({
|
||||
model,
|
||||
prompt,
|
||||
tools: Tool.toDefinitions(tools),
|
||||
})
|
||||
|
||||
const events = yield * LLM.stream(request).pipe(Stream.runCollect)
|
||||
const call = Array.from(events).find(LLMEvent.is.toolCall)
|
||||
|
||||
if (call && !call.providerExecuted) {
|
||||
const dispatched = yield * ToolRuntime.dispatch(tools, call)
|
||||
const followUp = LLMRequest.update(request, {
|
||||
messages: [...request.messages, Message.assistant([call]), Message.tool({ ...call, result: dispatched.result })],
|
||||
})
|
||||
// Caller must invoke the provider again and repeat the loop.
|
||||
}
|
||||
```
|
||||
|
||||
That explicit flow remains possible through turn APIs, but it is no longer the
|
||||
only tool experience.
|
||||
|
||||
## One Provider Turn
|
||||
|
||||
OpenCode and other durable runtimes need to own persistence, tool settlement,
|
||||
and continuation. They use the explicit turn API:
|
||||
|
||||
```ts
|
||||
const result =
|
||||
yield *
|
||||
LLM.generateTurn({
|
||||
model,
|
||||
request,
|
||||
// Definitions only. generateTurn never dispatches local handlers.
|
||||
tools: {
|
||||
getWeather: Tool.definition({
|
||||
description: "Get current weather for a city.",
|
||||
parameters: WeatherInput,
|
||||
}),
|
||||
},
|
||||
})
|
||||
|
||||
// Persist the TurnResult and settle calls durably before the next turn.
|
||||
for (const call of result.toolCalls) {
|
||||
// Application-owned dispatch and persistence.
|
||||
}
|
||||
```
|
||||
|
||||
`generateTurn` and `streamTurn` make exactly one provider request. They never
|
||||
execute a local tool and never continue automatically.
|
||||
|
||||
This separation is load-bearing:
|
||||
|
||||
- `generate` / `stream`: complete Model Run
|
||||
- `generateTurn` / `streamTurn`: one Provider Turn
|
||||
|
||||
## Portable Tool Definitions
|
||||
|
||||
A portable request may declare serializable definitions, but executable handlers
|
||||
are bound at run time:
|
||||
|
||||
```ts
|
||||
const request = LLM.request({
|
||||
prompt: "What is the weather in London?",
|
||||
tools: {
|
||||
getWeather: Tool.definition({
|
||||
description: "Get current weather for a city.",
|
||||
parameters: WeatherInput,
|
||||
}),
|
||||
},
|
||||
})
|
||||
|
||||
const result =
|
||||
yield *
|
||||
LLM.generate({
|
||||
model,
|
||||
request,
|
||||
tools: {
|
||||
getWeather: Tool.make({
|
||||
description: "Get current weather for a city.",
|
||||
parameters: WeatherInput,
|
||||
success: WeatherOutput,
|
||||
execute: getWeather,
|
||||
formatError,
|
||||
}),
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
Definitions and handlers match by record key. Before the first provider call,
|
||||
the runtime validates that every local definition has a compatible executable
|
||||
binding. Missing or incompatible bindings fail with a typed tool-binding error.
|
||||
|
||||
Provider-hosted tools are distinct typed values:
|
||||
|
||||
```ts
|
||||
const result =
|
||||
yield *
|
||||
LLM.generate({
|
||||
model: OpenAI.model("gpt-4.1"),
|
||||
prompt: "Find today's relevant announcements.",
|
||||
tools: {
|
||||
search: OpenAI.tool.webSearch({ searchContextSize: "medium" }),
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
Hosted tools do not pretend to have local handlers, and callers do not inspect a
|
||||
`providerExecuted` boolean to decide whether dispatch is safe.
|
||||
|
||||
## Streaming
|
||||
|
||||
### Run stream
|
||||
|
||||
`LLM.stream` returns an Effect `Stream<RunEvent, AIError, Requirements>`.
|
||||
Run events explicitly expose orchestration boundaries:
|
||||
|
||||
```ts
|
||||
const program = LLM.stream({ model, prompt, tools }).pipe(
|
||||
Stream.tap((event) =>
|
||||
Effect.sync(() => {
|
||||
switch (event.type) {
|
||||
case "run-start":
|
||||
break
|
||||
case "turn-start":
|
||||
break
|
||||
case "turn-event":
|
||||
// Normalized text, reasoning, tool-call, usage, and finish events.
|
||||
if (event.event.type === "text-delta") {
|
||||
process.stdout.write(event.event.text)
|
||||
}
|
||||
break
|
||||
case "tool-start":
|
||||
break
|
||||
case "tool-finish":
|
||||
break
|
||||
case "turn-finish":
|
||||
break
|
||||
case "run-finish":
|
||||
// Contains the same full GenerateResult returned by LLM.generate.
|
||||
console.log(event.result.usage)
|
||||
break
|
||||
}
|
||||
}),
|
||||
),
|
||||
Stream.runDrain,
|
||||
)
|
||||
```
|
||||
|
||||
Exact event tag spelling remains an implementation detail to finalize, but the
|
||||
algebra is settled:
|
||||
|
||||
- A separate `RunEvent` union for run, turn, and tool lifecycle
|
||||
- A focused `TurnEvent` union for normalized provider output
|
||||
- `streamTurn` emits only `TurnEvent`
|
||||
- The terminal run event contains the full `GenerateResult`
|
||||
|
||||
External cancellation remains Effect interruption. It does not fabricate a
|
||||
successful result with an `interrupted` stop reason.
|
||||
|
||||
## Structured Output
|
||||
|
||||
Structured output is an option on `generate`, not a separate operation:
|
||||
|
||||
```ts
|
||||
const Weather = Schema.Struct({
|
||||
city: Schema.String,
|
||||
forecast: Schema.String,
|
||||
highCelsius: Schema.Number,
|
||||
})
|
||||
|
||||
const result =
|
||||
yield *
|
||||
LLM.generate({
|
||||
model,
|
||||
prompt: "Give me today's weather for London.",
|
||||
output: Weather,
|
||||
})
|
||||
|
||||
// Inferred from Weather.
|
||||
result.output.city
|
||||
```
|
||||
|
||||
The model declaration and protocol select the best reliable strategy:
|
||||
|
||||
1. Provider-native structured output when supported and reliable
|
||||
2. Forced tool output when required as a compatibility fallback
|
||||
3. Typed unsupported-capability failure before network execution when neither is
|
||||
available
|
||||
|
||||
Advanced callers may override the strategy when exact provider semantics matter.
|
||||
|
||||
### Current API
|
||||
|
||||
```ts
|
||||
// Current API is a separate operation and always forces a synthetic tool.
|
||||
const result =
|
||||
yield *
|
||||
LLM.generateObject({
|
||||
model,
|
||||
prompt,
|
||||
schema: Weather,
|
||||
})
|
||||
```
|
||||
|
||||
The proposal unifies generation and lets capabilities choose the strategy rather
|
||||
than permanently encoding one cross-provider workaround.
|
||||
|
||||
## Model Catalog
|
||||
|
||||
`models.dev` is the release-time source for:
|
||||
|
||||
- Model ID suggestions
|
||||
- Capabilities and modalities
|
||||
- Context and output limits
|
||||
- Pricing
|
||||
- Other available model metadata
|
||||
|
||||
The package ships a generated, versioned snapshot. Normal execution performs no
|
||||
catalog network requests.
|
||||
|
||||
Provider definitions may correct generated metadata where protocol-specific
|
||||
knowledge is more accurate. Precedence is:
|
||||
|
||||
```text
|
||||
models.dev snapshot
|
||||
< provider-definition correction
|
||||
< provider configuration override
|
||||
< model-selection override
|
||||
< call override
|
||||
```
|
||||
|
||||
Unknown model IDs inherit only capabilities guaranteed by the selected protocol.
|
||||
Unsupported request capabilities fail before network execution unless the caller
|
||||
explicitly overrides the model declaration.
|
||||
|
||||
## Usage And Cost
|
||||
|
||||
`GenerateResult` aggregates normalized usage across every turn, including cache
|
||||
read/write usage where providers report it.
|
||||
|
||||
It also exposes estimated cost using the generated models.dev pricing snapshot:
|
||||
|
||||
```ts
|
||||
result.usage.inputTokens
|
||||
result.usage.outputTokens
|
||||
result.usage.cacheReadInputTokens
|
||||
result.usage.cacheWriteInputTokens
|
||||
|
||||
result.cost?.total
|
||||
result.cost?.currency // e.g. "USD"
|
||||
```
|
||||
|
||||
Cost is an estimate, not a billing guarantee. If reliable pricing is unavailable
|
||||
for any turn, aggregate run cost is unavailable rather than partial or silently
|
||||
zero. Per-turn metadata should retain the catalog/pricing identity used so an
|
||||
estimate can be explained.
|
||||
|
||||
## Caching
|
||||
|
||||
Prompt caching remains `"auto"` by default. The library places protocol-aware
|
||||
cache boundaries where explicit caching is supported and does nothing on the wire
|
||||
where providers cache implicitly.
|
||||
|
||||
```ts
|
||||
yield *
|
||||
LLM.generate({
|
||||
model,
|
||||
prompt,
|
||||
cache: "none", // Explicit opt-out.
|
||||
})
|
||||
```
|
||||
|
||||
Granular cache policy remains available as an advanced request option.
|
||||
|
||||
## Retries, Timeouts, And Cancellation
|
||||
|
||||
### Retries
|
||||
|
||||
The default retry policy is deliberately conservative:
|
||||
|
||||
- Retry bounded transient transport and rate-limit failures
|
||||
- Retry only before observable output
|
||||
- Never silently retry after ambiguous tool execution or other side effects
|
||||
- Allow each call to override or disable retry behavior
|
||||
|
||||
Retry configuration is call-scoped only. Provider and model configuration do not
|
||||
silently inherit custom retry policies.
|
||||
|
||||
### Timeouts
|
||||
|
||||
```ts
|
||||
yield *
|
||||
LLM.generate({
|
||||
model,
|
||||
prompt,
|
||||
timeout: "2 minutes", // Entire run, including tools.
|
||||
turnTimeout: "30 seconds", // Each provider turn.
|
||||
tools,
|
||||
})
|
||||
```
|
||||
|
||||
Exact Duration input spelling follows Effect conventions. Individual tools may
|
||||
also declare optional timeouts.
|
||||
|
||||
### Cancellation
|
||||
|
||||
- Effect API: fiber interruption
|
||||
- Promise API: `AbortSignal`, rejecting with a recognizable abort error
|
||||
- Cancellation is not a successful run stop reason
|
||||
|
||||
## Hooks
|
||||
|
||||
Stable high-level hooks exist at five named stages:
|
||||
|
||||
1. Canonical request
|
||||
2. Provider-native body
|
||||
3. Prepared transport request
|
||||
4. Normalized event
|
||||
5. Error
|
||||
|
||||
Hooks are Effectful. They may transform the stage value or fail with a typed
|
||||
error. They may not secretly short-circuit execution, synthesize a response,
|
||||
retry, or redirect control flow.
|
||||
|
||||
```ts
|
||||
const model = OpenAI.model("gpt-4.1", {
|
||||
hooks: {
|
||||
request: (request) =>
|
||||
Effect.succeed({
|
||||
...request,
|
||||
metadata: { ...request.metadata, tenant: "acme" },
|
||||
}),
|
||||
body: (body, context) => auditBody(body, context),
|
||||
transport: (request) => signInternalGatewayRequest(request),
|
||||
event: (event) => redactProviderMetadata(event),
|
||||
error: (error) => classifyInternalError(error),
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
Hook scopes compose in this order:
|
||||
|
||||
```text
|
||||
provider-definition hooks -> model hooks -> call hooks
|
||||
```
|
||||
|
||||
Each hook sees the prior hook's output. Replacement requires an explicit
|
||||
definition-level patch, not accidental last-writer-wins semantics.
|
||||
|
||||
Provider-definition hooks are authored by provider integrations. They are not
|
||||
passed through `Provider.configure(...)`, which remains deployment-only.
|
||||
|
||||
## HTTP And Provider Escape Hatches
|
||||
|
||||
The request customization ladder is:
|
||||
|
||||
1. Portable generation controls
|
||||
2. Model-typed `provider` options
|
||||
3. Stable staged hooks
|
||||
4. Serializable HTTP/body overlays
|
||||
5. Experimental provider-definition or protocol patching
|
||||
|
||||
```ts
|
||||
yield *
|
||||
LLM.generate({
|
||||
model,
|
||||
prompt,
|
||||
http: {
|
||||
headers: { "x-experimental": "1" },
|
||||
query: { debug: "true" },
|
||||
body: { newlyReleasedProviderField: true },
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
Raw overlays are intentional last-resort support for provider features that ship
|
||||
before the library has a typed option.
|
||||
|
||||
## Provider-Native Metadata
|
||||
|
||||
Normalized message/content/event unions remain closed and exhaustive. Unknown or
|
||||
provider-required round-trip data lives in caller-writable `providerMetadata`.
|
||||
|
||||
```ts
|
||||
const assistant = Message.assistant([
|
||||
{
|
||||
type: "reasoning",
|
||||
text: "...",
|
||||
providerMetadata: {
|
||||
openai: {
|
||||
// Opaque provider data needed for replay or continuation.
|
||||
},
|
||||
},
|
||||
},
|
||||
])
|
||||
```
|
||||
|
||||
Protocols validate metadata they consume. The field is an escape hatch, not a
|
||||
portable semantic guarantee.
|
||||
|
||||
## Error Model
|
||||
|
||||
The Effect error channel is a tagged domain union rather than one `AIError`
|
||||
wrapper with nested reasons. Illustrative categories:
|
||||
|
||||
```ts
|
||||
type AIError =
|
||||
| AuthenticationError
|
||||
| InvalidRequestError
|
||||
| UnsupportedCapabilityError
|
||||
| ToolBindingError
|
||||
| TransportError
|
||||
| ProviderResponseError
|
||||
| InvalidProviderOutputError
|
||||
| HookError
|
||||
```
|
||||
|
||||
Each error retains relevant provider/model/turn/stage context and its underlying
|
||||
cause where available.
|
||||
|
||||
Expected tool errors keep their own typed error channel. `Tool.make` requires an
|
||||
explicit mapping before such errors become model-visible tool results. Expected
|
||||
mapped failures let the model recover; defects and interruption fail the run.
|
||||
|
||||
## Observability
|
||||
|
||||
The core library emits Effect-native spans and metrics for:
|
||||
|
||||
- Model runs
|
||||
- Provider turns
|
||||
- Provider requests
|
||||
- Retries
|
||||
- Tool executions
|
||||
|
||||
Default telemetry records metadata only:
|
||||
|
||||
- Provider and model identity
|
||||
- Timing
|
||||
- Token/cache usage
|
||||
- Estimated cost availability
|
||||
- Finish and stop reasons
|
||||
- Retry counts
|
||||
- Tool names
|
||||
|
||||
Prompts, model output, tool arguments, and tool results are never recorded by
|
||||
default. Explicit hooks or telemetry configuration may opt into content capture.
|
||||
|
||||
## Promise API
|
||||
|
||||
Promise wrappers live at a separate subpath so the root remains unambiguously
|
||||
Effect-first:
|
||||
|
||||
```ts
|
||||
import { LLM } from "@opencode-ai/ai/promise"
|
||||
import { OpenAI } from "@opencode-ai/ai/providers/openai"
|
||||
|
||||
const result = await LLM.generate({
|
||||
model: OpenAI.model("gpt-4.1-mini"),
|
||||
prompt: "Explain Effect in one sentence.",
|
||||
signal: abortController.signal,
|
||||
})
|
||||
```
|
||||
|
||||
Streaming returns an `AsyncIterable<RunEvent>`:
|
||||
|
||||
```ts
|
||||
for await (const event of LLM.stream({ model, prompt, signal })) {
|
||||
if (event.type === "turn-event" && event.event.type === "text-delta") {
|
||||
process.stdout.write(event.event.text)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Top-level Promise functions use a default runtime for built-in services. Custom
|
||||
Effect service requirements use a configured client:
|
||||
|
||||
```ts
|
||||
const client = LLM.makeClient({
|
||||
layer: Layer.mergeAll(WeatherLive, AuditLive),
|
||||
})
|
||||
|
||||
const result = await client.generate({ model, prompt, tools })
|
||||
```
|
||||
|
||||
The Promise API mirrors Effect semantics. It does not invent different run,
|
||||
error, stopping, or cancellation behavior.
|
||||
|
||||
## Schemas
|
||||
|
||||
Schemas live in a dedicated namespace/subpath instead of flooding root exports:
|
||||
|
||||
```ts
|
||||
import { LLMSchema } from "@opencode-ai/ai/schema"
|
||||
|
||||
const request = yield * Schema.decodeUnknown(LLMSchema.Request)(input)
|
||||
```
|
||||
|
||||
Schemas cover only serializable domain values:
|
||||
|
||||
- Requests and messages
|
||||
- Portable tool definitions
|
||||
- Turn and run events
|
||||
- Serializable result projections
|
||||
- Usage and cost estimates
|
||||
- Tagged errors where serializable
|
||||
- Provider metadata containers
|
||||
|
||||
Configured models, executable tools, hooks, provider definitions, and protocols
|
||||
are process-local behavior and do not receive fake serialization schemas.
|
||||
|
||||
## Provider Authoring
|
||||
|
||||
Provider authoring is public but experimental.
|
||||
|
||||
### Declarative provider definition
|
||||
|
||||
```ts
|
||||
import { Provider, Protocol } from "@opencode-ai/ai/provider"
|
||||
|
||||
export const ExampleAI = Provider.define({
|
||||
id: "example",
|
||||
options: ExampleProviderOptions,
|
||||
configure: configureExampleDeployment,
|
||||
protocols: {
|
||||
responses: ExampleResponses,
|
||||
},
|
||||
models: ({ deployment, catalog }) => ({
|
||||
model: (id, defaults) =>
|
||||
Provider.model({
|
||||
id,
|
||||
deployment,
|
||||
protocol: ExampleResponses,
|
||||
metadata: catalog.model(id),
|
||||
defaults,
|
||||
}),
|
||||
}),
|
||||
catalog: generatedExampleCatalog,
|
||||
corrections: exampleCatalogCorrections,
|
||||
hooks: exampleProviderHooks,
|
||||
})
|
||||
```
|
||||
|
||||
The exact builder fields need implementation design, but it must remain one
|
||||
declarative immutable object, infer provider option types, and support `.with(...)`
|
||||
patching. It must not register globally.
|
||||
|
||||
Built-ins export their immutable definition for advanced forking:
|
||||
|
||||
```ts
|
||||
import { OpenAI } from "@opencode-ai/ai/providers/openai"
|
||||
|
||||
const PatchedOpenAI = OpenAI.definition.with({
|
||||
protocols: {
|
||||
responses: OpenAI.protocols.responses.with({
|
||||
// Explicit immutable stage patch.
|
||||
body: {
|
||||
fromRequest: patchResponsesBody,
|
||||
},
|
||||
}),
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
### Protocols
|
||||
|
||||
A protocol exposes all native types and stages:
|
||||
|
||||
- Provider-native request body and schema
|
||||
- Transport frame type
|
||||
- Provider-native event and schema
|
||||
- Parser state
|
||||
- Request lowering
|
||||
- Event stepping
|
||||
- Terminal detection and final flushing
|
||||
|
||||
Every stage is immutably patchable. This is deliberately more open than the AI
|
||||
SDK integrations that motivated this package.
|
||||
|
||||
```ts
|
||||
const PatchedResponses = OpenAIResponses.with({
|
||||
body: {
|
||||
fromRequest: (request) =>
|
||||
OpenAIResponses.body.fromRequest(request).pipe(Effect.map((body) => ({ ...body, custom_field: true }))),
|
||||
},
|
||||
stream: {
|
||||
step: patchResponsesStep,
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
Protocol body, frame, native event, and parser-state types are exported. Because
|
||||
provider wire formats change often, these types and patch APIs are explicitly
|
||||
experimental and do not receive the high-level API's compatibility promise.
|
||||
|
||||
## Package Surface
|
||||
|
||||
Illustrative export layout:
|
||||
|
||||
```text
|
||||
@opencode-ai/ai
|
||||
LLM
|
||||
Message
|
||||
Tool
|
||||
StopWhen
|
||||
stable domain types
|
||||
|
||||
@opencode-ai/ai/promise
|
||||
Promise/AsyncIterable LLM facade
|
||||
|
||||
@opencode-ai/ai/schema
|
||||
serializable domain schemas
|
||||
|
||||
@opencode-ai/ai/provider
|
||||
experimental Provider and Protocol authoring APIs
|
||||
|
||||
@opencode-ai/ai/providers/openai
|
||||
@opencode-ai/ai/providers/anthropic
|
||||
@opencode-ai/ai/providers/google
|
||||
...
|
||||
```
|
||||
|
||||
Providers are imported through individual subpaths. The root does not export all
|
||||
providers, and there is no preferred all-providers barrel.
|
||||
|
||||
## Defaults
|
||||
|
||||
| Concern | Default |
|
||||
| ---------------------------- | --------------------------------------------------- |
|
||||
| `LLM.generate` semantics | Complete Model Run |
|
||||
| `LLM.generateTurn` semantics | Exactly one Provider Turn |
|
||||
| Maximum turns | 20 |
|
||||
| Turn-limit outcome | Successful `max-turns` result |
|
||||
| Tool execution | Automatic in runs |
|
||||
| Tool concurrency | Concurrent, bounded, deterministic result order |
|
||||
| Prompt caching | `auto` |
|
||||
| Retries | Conservative, pre-output transient failures only |
|
||||
| Structured output | Capability-selected native or tool strategy |
|
||||
| Capability mismatch | Typed failure before network execution |
|
||||
| Unknown model capability | Conservative protocol baseline |
|
||||
| Telemetry content | Metadata only |
|
||||
| Cost | Estimated aggregate or unavailable |
|
||||
| Cancellation | Interruption/rejection, never successful completion |
|
||||
|
||||
## Clean-break Migration
|
||||
|
||||
The redesign intentionally removes or changes these current concepts:
|
||||
|
||||
| Current | Proposed |
|
||||
| --------------------------------------- | ----------------------------------------------------------- |
|
||||
| Mandatory `LLM.request({ model, ... })` | Inline calls or model-free portable requests |
|
||||
| `LLM.generate` means one turn | `LLM.generate` means complete run |
|
||||
| `LLMClient.generate/stream` | `LLM.generateTurn/streamTurn` for one turn |
|
||||
| `LLMClient.layer` requirement | Standard Effect requirements exposed directly |
|
||||
| Public `Route` mental model | Hidden behind executable `LanguageModel` |
|
||||
| `Provider.make` structural helper | Experimental declarative `Provider.define` |
|
||||
| Schema classes as canonical values | Plain immutable values plus schema subpath |
|
||||
| `LLM.updateRequest` | Object spread |
|
||||
| `Tool.toDefinitions` in normal calls | Named executable tool records |
|
||||
| Manual `ToolRuntime.dispatch` loop | Automatic run dispatch; explicit turn API for orchestration |
|
||||
| `providerOptions: { openai: ... }` | Model-typed `provider: ...` |
|
||||
| `generateObject` | Typed `output` option on `generate` |
|
||||
| One event union for provider output | Separate `TurnEvent` and `RunEvent` unions |
|
||||
| `providerExecuted` dispatch check | Distinct hosted-tool constructors |
|
||||
| One wrapped `AIError` | Tagged domain error union |
|
||||
|
||||
OpenCode should migrate to `generateTurn` / `streamTurn`, preserving its durable
|
||||
prompt admission, persistence, permission, tool settlement, and continuation
|
||||
boundaries. It should not use the automatic run API for Session orchestration.
|
||||
|
||||
## Remaining Implementation-level Questions
|
||||
|
||||
These do not reopen the main design:
|
||||
|
||||
1. Exact `RunEvent` and `TurnEvent` tag names and payloads
|
||||
2. Exact `GenerateResult` shortcut fields for text, reasoning, output, and messages
|
||||
3. Exact Provider definition TypeScript shape needed for strong inference
|
||||
4. Exact protocol `.with(...)` patch syntax and replacement semantics
|
||||
5. Exact Duration input fields and names
|
||||
6. Exact models.dev generation pipeline and correction-file format
|
||||
7. Exact cost representation and decimal arithmetic strategy
|
||||
8. Exact default retry schedule and bounded tool concurrency number
|
||||
9. Whether request-level serializable HTTP overlays belong in the stable schema
|
||||
10. Which tagged errors are serializable versus process-local
|
||||
|
||||
These should be resolved with call-site sketches and implementation spikes rather
|
||||
than by changing the domain boundaries above.
|
||||
@@ -237,11 +237,11 @@ Prompt caching is **on by default**. Every `LLMRequest` resolves to `cache: "aut
|
||||
|
||||
### Auto placement
|
||||
|
||||
`"auto"` places up to four breakpoints — the last tool definition, the first system part, the last system part when distinct, and the final message boundary. These expose successively larger reusable prefixes for tools, the base agent, project instructions, and the active conversation. The rolling final-message boundary is the load-bearing detail in tool loops: it advances on every request so the previous cache entry stays within Anthropic's 20-block lookback.
|
||||
`"auto"` places up to four breakpoints — the last tool definition, the first system part, the last system part when distinct, and the final message boundary. These expose successively larger reusable prefixes for tools, the base agent, project instructions, and the active conversation. The rolling final-message boundary advances on every request so recent conversation prefixes remain reusable during tool loops.
|
||||
|
||||
Tools precede every system and conversation block in the provider prefix, so tool definitions must remain byte-stable and deterministically ordered for downstream breakpoints to remain reusable.
|
||||
|
||||
The math justifies the default: Anthropic's 5-minute cache write is 1.25× base, read is 0.1×, so a single reuse within 5 minutes already wins. One-shot completions below the per-model minimum-cacheable-token threshold silently no-op on the wire, so the worst case is harmless.
|
||||
Requests below a provider's minimum cacheable size simply do not produce a reusable cache entry.
|
||||
|
||||
### Opting out
|
||||
|
||||
@@ -285,6 +285,7 @@ LLM.request({
|
||||
| ----------------------- | ------------------------------------------------------------------------- |
|
||||
| Anthropic Messages | emits up to 4 `cache_control` markers (4-breakpoint cap enforced) |
|
||||
| Bedrock Converse | emits up to 4 `cachePoint` blocks (4-breakpoint cap enforced) |
|
||||
| OpenRouter | emits up to 4 `cache_control` markers |
|
||||
| OpenAI Chat / Responses | no-op (implicit caching above 1024 tokens) |
|
||||
| Gemini | no-op (implicit caching on 2.5+; explicit `CachedContent` is out-of-band) |
|
||||
|
||||
@@ -387,6 +388,5 @@ This package is built on Effect. Public methods return `Effect` or `Stream`; pro
|
||||
## See also
|
||||
|
||||
- `AGENTS.md` — architecture, route construction, contributor guide
|
||||
- `STATUS.md` — native provider parity status and AI SDK migration gaps
|
||||
- `example/tutorial.ts` — runnable end-to-end walkthrough
|
||||
- `test/provider/*.test.ts` — fixture-first protocol tests; `*.recorded.test.ts` files cover live cassettes
|
||||
|
||||
@@ -1,107 +0,0 @@
|
||||
# LLM Provider Parity Status
|
||||
|
||||
Last reviewed: 2026-08-07
|
||||
|
||||
This file tracks the gap between the native `@opencode-ai/ai` package and the AI SDK provider packages that opencode still depends on for many catalog/runtime paths.
|
||||
|
||||
## Existing Status Sources
|
||||
|
||||
| File | What it tracks | Limitation |
|
||||
| ----------------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------- |
|
||||
| `packages/ai/DESIGN.md` | Future clean-break API proposal for `@opencode-ai/ai`. | Not a provider parity tracker. |
|
||||
| `packages/ai/example/call-sites.md` | Route/value/provider-facade migration checklist and call-site sketches. | Architecture migration only; not AI SDK package parity. |
|
||||
|
||||
## Current Implementation Snapshot
|
||||
|
||||
| Native slice | Source | Current state | Main gaps |
|
||||
| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| OpenAI Chat | `src/protocols/openai-chat.ts`, `src/providers/openai.ts` | Usable. Streams text, reasoning deltas, tool calls, usage, images, and common generation controls. | No typed structured-output / `response_format` path. Limited typed OpenAI option surface compared with SDK escape hatches. |
|
||||
| OpenAI Responses | `src/protocols/open-responses.ts`, `src/protocols/openai-responses.ts`, `src/providers/openai.ts` | Usable over HTTP by default, with optional per-call WebSocket channel execution on the same model and route identity. | No incremental `previous_response_id` path or persistent Session channel manager yet. Typed options cover only a subset of Responses fields. Structured output is still mostly synthetic-tool based. |
|
||||
| OpenAI-compatible Chat | `src/protocols/openai-compatible-chat.ts`, `src/providers/openai-compatible.ts` | Usable for generic Chat and several profiles: Baseten, Cerebras, DeepInfra, DeepSeek, Fireworks, Groq, TogetherAI. | Family quirks are mostly endpoint defaults, not full typed behavior. |
|
||||
| Open Responses-compatible | `src/protocols/open-responses.ts`, `src/protocols/openai-compatible-responses.ts`, `src/providers/openai-compatible-responses.ts` | Usable for deployments that implement the provider-neutral Open Responses protocol. The deployment adapter does not inherit OpenAI tools, events, metadata, or defaults. | No named family profiles or recorded deployment coverage yet. |
|
||||
| Anthropic-compatible Messages | `src/protocols/anthropic-messages.ts`, `src/providers/anthropic-compatible.ts` | Usable for deployments that implement the Anthropic Messages wire protocol. Named Anthropic composes this base; MiniMax M3 has recorded text and tool-loop coverage. | No named compatible family profiles yet. |
|
||||
| Anthropic Messages | `src/protocols/anthropic-messages.ts`, `src/providers/anthropic.ts` | Usable. Supports tools, thinking, cache control, images, server-hosted tool events, and usage. | Provider option surface is small. Beta/header handling, metadata, and newer Messages fields need a typed parity pass. |
|
||||
| Gemini Developer API | `src/protocols/gemini.ts`, `src/providers/google.ts` | Usable for Google API key flow. Supports text, images, tools, thinking signatures, and cache usage. | This is not Vertex. Typed provider options are narrow; many Gemini request fields currently require raw `http.body` overlays. |
|
||||
| Vertex Gemini | `src/protocols/gemini.ts`, `src/providers/google-vertex.ts` | Usable through API-key express mode, explicit OAuth tokens, or ADC with project/location endpoint derivation, including tuned `endpoints/...` deployments. | Core runner/catalog mapping and recorded provider coverage are missing. |
|
||||
| Vertex Chat | `src/protocols/openai-chat.ts`, `src/providers/google-vertex-chat.ts` | Usable for MaaS models through OpenAI-compatible Chat Completions with explicit OAuth tokens or ADC and project/location endpoint derivation. | Core runner/catalog mapping and recorded provider coverage are missing; MaaS family-specific request parity needs review. |
|
||||
| Vertex Responses | `src/protocols/open-responses.ts`, `src/providers/google-vertex-responses.ts` | Usable for Grok models through Open Responses with explicit OAuth tokens or ADC, project/location endpoint derivation, and an explicit `store: false` Vertex default. | Core runner/catalog mapping and recorded provider coverage are missing; stateful continuation is not supported by Vertex. |
|
||||
| Vertex Messages | `src/protocols/anthropic-messages.ts`, `src/providers/google-vertex-messages.ts` | Usable through explicit OAuth tokens or ADC, including global, regional, and `eu`/`us` multi-region endpoints. | Core runner/catalog mapping and recorded provider coverage are missing; Vertex-specific hosted-tool parity needs review. |
|
||||
| Bedrock Converse | `src/protocols/bedrock-converse.ts`, `src/providers/amazon-bedrock.ts` | Partial but real. Supports AWS event-stream framing, SigV4 with supplied credentials, bearer auth, tools, reasoning signatures, media, cache points, and recorded tests. | Native facade does not mirror the AI SDK plugin's default AWS credential chain/profile behavior. Runner/catalog mapping is missing. Guardrails, inference profiles, region-specific model ID fixes, and model-specific request fields need a parity pass. |
|
||||
| Azure OpenAI | `src/providers/azure.ts` using OpenAI Chat/Responses protocols | Partial. Supports resource/base URL setup, API key auth, API version query, Chat, and Responses selectors. | Core runner does not map `@ai-sdk/azure` to this native facade. AAD/token auth and Azure-specific endpoint variants need review. |
|
||||
| Cloudflare AI Gateway / Workers AI | `src/providers/cloudflare.ts` | Present via OpenAI-compatible Chat routes. | Useful but not part of the critical AI SDK replacement set yet. Needs per-product recorded coverage before relying on it broadly. |
|
||||
| OpenRouter | `src/providers/openrouter.ts` | Present with OpenRouter-specific usage/reasoning/prompt-cache options over Chat. | Responses-style OpenRouter support is absent. |
|
||||
| xAI | `src/providers/xai.ts` | Present with Responses and Chat selectors. | Needs package-parity review against the AI SDK xAI provider. |
|
||||
| GitHub Copilot | `src/providers/github-copilot.ts` | Present as explicit-base-URL OpenAI Chat/Responses facade. | Runtime/catalog integration remains specialized and should stay separate from public OpenAI-compatible defaults. |
|
||||
|
||||
## V2 Runner Status
|
||||
|
||||
`packages/core/src/session/runner/model.ts` currently resolves only this native subset from catalog `aisdk` metadata:
|
||||
|
||||
| Catalog API | Native route used today |
|
||||
| --------------------------------------------------- | ---------------------------- |
|
||||
| `aisdk:@ai-sdk/openai` | `OpenAIResponses.route` |
|
||||
| `aisdk:@ai-sdk/anthropic` | `AnthropicMessages.route` |
|
||||
| `aisdk:@ai-sdk/openai-compatible` with explicit URL | `OpenAICompatibleChat.route` |
|
||||
|
||||
Other `aisdk:` packages, including Google Vertex, Azure, and Bedrock, currently fall back through the AI SDK loader in the production runner. The dependency-free resolver seam rejects them with `SessionRunnerModel.UnsupportedPackageError`; they are not native route mappings yet.
|
||||
|
||||
## AI SDK Package Parity Matrix
|
||||
|
||||
| AI SDK package | Intended native target | Status | Biggest gaps |
|
||||
| --------------------------------- | --------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `@ai-sdk/openai` | `OpenAI.chat`, `OpenAI.responses` | Partial / usable | Add complete typed option coverage, structured output strategy, explicit Responses continuation support, and runner execution policy for optional WebSocket channels. |
|
||||
| `@ai-sdk/openai-compatible` | Generic OpenAI-compatible Chat and Responses | Partial / usable | Decide per-family namespace/profile behavior and runner API selection for providers that support Responses versus Chat only. |
|
||||
| `@ai-sdk/anthropic` | `AnthropicMessages` | Partial / usable | Finish Messages API parity for headers/betas/metadata/newer fields and document hosted-tool continuation expectations. |
|
||||
| `@ai-sdk/google` | Gemini Developer API | Partial / usable | Add typed options for safety, response schema/modalities, cached content, grounding/search/code execution, and non-text output modes where supported. |
|
||||
| `@ai-sdk/google-vertex` | Vertex Gemini namespace/facade | Partial / usable | Add runner/catalog mapping, recorded coverage, and broader provider-option parity. |
|
||||
| `@ai-sdk/google-vertex/anthropic` | Anthropic Messages over Vertex namespace/facade | Partial / usable | Add runner/catalog mapping, recorded coverage, and Vertex-specific hosted-tool parity. |
|
||||
| `@ai-sdk/google-vertex/maas` | Vertex Chat | Partial / usable | Add runner/catalog mapping, recorded coverage, and MaaS family-specific request parity. |
|
||||
| `@ai-sdk/google-vertex/xai` | Vertex Chat / Responses | Partial / usable | Decide Chat/Responses selection for catalog models, add runner mapping and recorded coverage, and review xAI-specific request options. |
|
||||
| `@ai-sdk/azure` | Azure OpenAI Chat/Responses facade | Partial | Map runner/catalog metadata to native Azure, handle resourceName/baseURL/apiVersion variants, add AAD/token auth story, and verify Chat vs Responses deployment selection. |
|
||||
| `@ai-sdk/amazon-bedrock` | Bedrock Converse | Partial | Add default AWS credential chain/profile support, region/inference-profile model ID handling, provider option parity via `additionalModelRequestFields`, guardrails/performance config, and runner/catalog mapping. |
|
||||
| `@ai-sdk/amazon-bedrock/mantle` | Bedrock Mantle OpenAI-compatible Chat/Responses namespace | Partial / usable | Add default AWS credential chain/profile support; native catalog mapping currently requires bearer auth or explicit static credentials. |
|
||||
|
||||
## Highest-Risk Gaps
|
||||
|
||||
1. Runner support is narrower than the LLM package. The package has native provider facades for Google, Azure, and Bedrock, but the V2 Session runner only maps OpenAI, Anthropic, and explicit OpenAI-compatible Chat from `aisdk` catalog metadata.
|
||||
2. The Open Responses adapter is available through a separate package entrypoint, but the V2 runner still maps `@ai-sdk/openai-compatible` to Chat only. Catalog selection must become API-aware before Responses deployments can use it.
|
||||
3. Bedrock native auth is not AI SDK parity. The AI SDK plugin uses the default AWS provider chain, profile, container credentials, and Bedrock bearer token env behavior. Native Bedrock currently expects explicit credentials or bearer auth on the facade.
|
||||
4. Vertex Gemini, Vertex Chat, Vertex Responses, and Vertex Messages now have native package entrypoints, but the core runner does not map catalog metadata to them yet and recorded provider coverage is still missing.
|
||||
5. Azure is only a provider facade, not a full runtime replacement. Native Azure exists, but the catalog runner does not select it, and token auth/resource variants need review.
|
||||
6. Provider option typing is uneven. OpenAI, Anthropic, Gemini, Bedrock, and OpenRouter each expose a small typed subset plus raw HTTP overlays; this is useful but not equivalent to AI SDK provider option coverage.
|
||||
7. Structured output is not provider-native yet. `LLM.generateObject` still uses a synthetic tool strategy, while the future design expects native structured output where reliable and tool fallback where needed.
|
||||
8. Package/namespace boundaries for the current native loading set are explicit in docs and exports. Other exported provider facades are not catalog package entrypoints until they implement the contract. Vertex xAI still needs catalog API selection.
|
||||
9. Recorded coverage is uneven. OpenAI, Anthropic, Gemini, Bedrock Converse, Bedrock Mantle, Cloudflare, OpenRouter, and several OpenAI-compatible Chat providers have cassettes. Azure and Vertex still need first-class recorded scenarios before switching defaults.
|
||||
|
||||
## Native Namespace Shape
|
||||
|
||||
These are implementation/API slices, not separate npm packages.
|
||||
|
||||
| API slice | Package-like entrypoint | Purpose |
|
||||
| ----------------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
|
||||
| OpenAI Chat | `@opencode-ai/ai/providers/openai/chat` | OpenAI `/chat/completions` semantics. |
|
||||
| OpenAI Responses | `@opencode-ai/ai/providers/openai/responses` | OpenAI `/responses` semantics with HTTP default and optional per-call WebSocket execution. |
|
||||
| OpenAI-compatible Chat | `@opencode-ai/ai/providers/openai-compatible` | Generic OpenAI-compatible `/chat/completions`. |
|
||||
| Open Responses-compatible | `@opencode-ai/ai/providers/openai-compatible/responses` | Generic provider-neutral `/responses`. |
|
||||
| Anthropic-compatible Messages | `@opencode-ai/ai/providers/anthropic-compatible` | Generic Anthropic-compatible `/messages`. |
|
||||
| Anthropic Messages | `@opencode-ai/ai/providers/anthropic` | Anthropic Messages API. |
|
||||
| Gemini Developer API | `@opencode-ai/ai/providers/google` | Google AI Studio Gemini API. |
|
||||
| Vertex Gemini | `@opencode-ai/ai/providers/google-vertex/gemini` | Vertex Gemini API; `providers/google-vertex` is the default alias. |
|
||||
| Vertex Chat | `@opencode-ai/ai/providers/google-vertex/chat` | Vertex OpenAI-compatible Chat Completions for MaaS models. |
|
||||
| Vertex Responses | `@opencode-ai/ai/providers/google-vertex/responses` | Vertex Open Responses for Grok models. |
|
||||
| Vertex Messages | `@opencode-ai/ai/providers/google-vertex/messages` | Vertex-hosted Anthropic Messages API. |
|
||||
| Bedrock Converse | `@opencode-ai/ai/providers/amazon-bedrock` | AWS Bedrock Converse API. |
|
||||
| Bedrock Mantle Chat | `@opencode-ai/ai/providers/amazon-bedrock/mantle/chat` | AWS Bedrock Mantle OpenAI-compatible Chat API. |
|
||||
| Bedrock Mantle Responses | `@opencode-ai/ai/providers/amazon-bedrock/mantle/responses` | AWS Bedrock Mantle OpenAI-compatible Responses API. |
|
||||
| Azure OpenAI Chat | `@opencode-ai/ai/providers/azure/chat` | Azure specialization of OpenAI Chat. |
|
||||
| Azure OpenAI Responses | `@opencode-ai/ai/providers/azure/responses` | Azure specialization of OpenAI Responses. |
|
||||
|
||||
## Suggested Next Work Slices
|
||||
|
||||
1. Add native runner/catalog mappings for `@ai-sdk/azure`, `@ai-sdk/google`, and `@ai-sdk/amazon-bedrock` where the existing native facades are already close.
|
||||
2. Add API-aware runner/catalog selection between OpenAI-compatible Chat and Responses.
|
||||
3. Bring Bedrock native auth/config to AI SDK parity: region, profile, default AWS credential chain, bearer token env, endpoint override, and cross-region inference profile handling.
|
||||
4. Add runner/catalog mappings and recorded scenarios for the native Vertex Gemini, Chat, Responses, and Messages entrypoints.
|
||||
5. Decide Chat/Responses selection for `@ai-sdk/google-vertex/xai` catalog models.
|
||||
6. Expand typed provider options from the existing V1 lowerer knowledge in `packages/core/src/v1/config/provider-options.ts` before adding more raw overlay examples.
|
||||
7. Add recorded provider tests for Azure, Vertex Gemini, Vertex Chat, Vertex Responses, Vertex Messages, and Bedrock credential-chain behavior before making native runtime the default for those packages.
|
||||
@@ -1,606 +0,0 @@
|
||||
# LLM Call Site Sketches
|
||||
|
||||
Scratchpad for examples first, abstractions second. Current direction: routes
|
||||
execute, provider facades organize configured route sets, and models carry route
|
||||
values directly.
|
||||
|
||||
## Conversation Summary
|
||||
|
||||
Kit and Aidan want provider-specific LLM behavior to move out of opencode's AI
|
||||
SDK transform path and into `packages/ai` where possible. The goal is not a big
|
||||
generic transform layer; the goal is small composable route definitions backed by
|
||||
recorded golden tests.
|
||||
|
||||
Things to keep testing against:
|
||||
|
||||
- Cache placement: `cache: "auto"`, manual cache breakpoints, provider cache usage.
|
||||
- Images: golden image tests for providers/protocols that claim image support.
|
||||
- Reasoning: canonical reasoning parts/events versus provider-native knobs.
|
||||
- Auth: bearer, custom headers, multiple credentials, query auth, SigV4, OAuth, no auth.
|
||||
- OpenAI-compatible providers: DeepSeek, Together, Groq, Alibaba/DashScope, custom routers.
|
||||
- Provider switching: stale signatures, encrypted reasoning, provider metadata, incompatible parts.
|
||||
- Error quality: typed errors instead of generic SDK/server failures.
|
||||
|
||||
## Final Guide: Routes Execute, Providers Organize
|
||||
|
||||
Do not introduce a first-class `Deployment` abstraction unless it gains real
|
||||
semantics. Provider facades are ergonomic configured route groups, not execution
|
||||
registries. The executable/composable thing is still a route. Do not make route
|
||||
construction publish to a global registry; models should carry their route value
|
||||
directly.
|
||||
|
||||
Keep durable identity separate from runtime capability:
|
||||
|
||||
- Durable identity is small serializable data like `{ providerID, modelID }` for
|
||||
config, sessions, logs, and catalogs.
|
||||
- Runtime capability is a `LanguageModel` with a route value, protocol, transport, auth,
|
||||
and defaults. It is allowed to contain functions and schemas.
|
||||
- If persisted identity needs to become executable, resolve it through an app
|
||||
boundary first. Do not make `LLMRequest` recover behavior from a global route
|
||||
side table.
|
||||
|
||||
Keep unconfigured behavior values as values, not factories. A transport like
|
||||
`HttpTransport.sseJson` should be a reusable immutable value. Use a function only
|
||||
when the caller supplies options or when construction needs fresh state.
|
||||
|
||||
Use constants to remove repetition before inventing abstractions. Provider ids
|
||||
are branded once per provider facade and reused across routes; a plain exported
|
||||
object is enough for the provider-facing API unless a helper earns its keep by
|
||||
removing repeated route projection.
|
||||
|
||||
Expose default configured provider instances, and put provider-specific setup on
|
||||
`.configure(...)`. Model selectors stay pure: `model(id)`, `responses(id)`,
|
||||
`chat(id)`, etc. Endpoint/auth/resource/api-version configuration happens before
|
||||
model selection, not as a second argument to model selection.
|
||||
|
||||
Use provider/product facades consistently:
|
||||
|
||||
- One coherent provider/product config surface gets one top-level facade.
|
||||
- APIs/model kinds that share that config are methods on the facade.
|
||||
- Different products with different required config get separate top-level
|
||||
facades, not a shared namespace with unrelated children.
|
||||
- Default facades are exposed only when concrete defaults or lazy env/credential
|
||||
defaults make the facade valid.
|
||||
|
||||
Examples:
|
||||
|
||||
```ts
|
||||
OpenAI.responses("gpt-4o")
|
||||
OpenAI.chat("gpt-4o")
|
||||
|
||||
Azure.configure({ resourceName, apiKey }).responses("my-deployment")
|
||||
AmazonBedrock.configure({ region, credentials }).model("anthropic.claude-3-5-sonnet-20241022-v2:0")
|
||||
|
||||
CloudflareAIGateway.configure({ accountId, gatewayId, gatewayApiKey, apiKey }).model("openai/gpt-4o")
|
||||
CloudflareWorkersAI.configure({ accountId, apiKey }).model("@cf/meta/llama-3.1-8b-instruct")
|
||||
|
||||
OpenAICompatible.configure({
|
||||
provider: "custom",
|
||||
baseURL: "https://custom.example/v1",
|
||||
auth: Auth.bearer(apiKey),
|
||||
}).model("custom-model")
|
||||
```
|
||||
|
||||
Standardize the provider facade contract before abstracting construction. A
|
||||
plain object is enough at first; add a helper only if repeated route projection
|
||||
starts hiding the real provider-specific config.
|
||||
|
||||
`Route.with(...)` patch semantics should be boring and explicit:
|
||||
|
||||
- Omitted fields inherit from the original route.
|
||||
- `endpoint` patches merge with the existing endpoint, so overriding `baseURL`
|
||||
keeps the existing `path`.
|
||||
- `endpoint.query` merges by default; later values win.
|
||||
- `auth` replaces.
|
||||
- `headers` merge by default; undefined values are omitted.
|
||||
- `id` is optional in patches. Route ids are diagnostic/provider API labels, not
|
||||
global runtime registry keys.
|
||||
|
||||
1. **Route**
|
||||
- route id
|
||||
- provider id
|
||||
- protocol
|
||||
- body schema
|
||||
- body builder
|
||||
- stream event schema
|
||||
- parser/state machine
|
||||
- transport
|
||||
- method / IO shape
|
||||
- framing
|
||||
- request preparation
|
||||
- constants when unconfigured; functions only when configured
|
||||
- endpoint
|
||||
- base URL
|
||||
- static path
|
||||
- body/model-derived path
|
||||
- query params
|
||||
- auth
|
||||
- bearer
|
||||
- custom header
|
||||
- multiple credentials
|
||||
- SigV4
|
||||
- none
|
||||
- defaults
|
||||
- headers
|
||||
- generation defaults
|
||||
- provider options
|
||||
- limits
|
||||
2. **Provider Facade**
|
||||
- default configured provider instance
|
||||
- provider-specific `.configure(...)`
|
||||
- plain object/function facade over one or more routes
|
||||
- top-level export only when it represents one coherent config surface
|
||||
- no passive `Provider.make(...)` wrapper unless it gains runtime behavior
|
||||
3. **Model Selector**
|
||||
- route/provider-owned selector
|
||||
- accepts model id only
|
||||
- returns executable models
|
||||
- does not accept endpoint/auth/deployment overrides
|
||||
4. **Language Model**
|
||||
- model id
|
||||
- route value
|
||||
- provider id
|
||||
- configured route value at selection time
|
||||
5. **LLM Request**
|
||||
- model
|
||||
- messages/tools
|
||||
- generation/cache/reasoning/response-format options
|
||||
- request-level HTTP overlays for per-request headers/query/body additions,
|
||||
not provider endpoint/auth reconfiguration
|
||||
6. **Compile**
|
||||
- read route from model
|
||||
- merge route defaults and request overrides
|
||||
- build final URL from route endpoint
|
||||
- apply auth from the configured route
|
||||
- build body with protocol
|
||||
- execute with transport and parse with protocol
|
||||
|
||||
## Provider Facade Shape
|
||||
|
||||
The provider abstraction is a facade over configured routes, not the runtime
|
||||
execution mechanism:
|
||||
|
||||
```ts
|
||||
type ProviderFacade<APIs, Config> = {
|
||||
readonly id: ProviderID
|
||||
readonly model: (id: string) => LanguageModel
|
||||
readonly configure: (input?: Config) => ProviderFacade<APIs, Config>
|
||||
} & APIs
|
||||
```
|
||||
|
||||
Manual construction is fine and should be the default until duplication earns a
|
||||
helper:
|
||||
|
||||
```ts
|
||||
export const OpenAI = {
|
||||
id: openAIProvider,
|
||||
model: openAIResponses.model,
|
||||
responses: openAIResponses.model,
|
||||
chat: openAIChat.model,
|
||||
configure: configureOpenAI,
|
||||
} satisfies ProviderFacade<
|
||||
{
|
||||
responses: (id: string) => LanguageModel
|
||||
chat: (id: string) => LanguageModel
|
||||
},
|
||||
OpenAIConfig
|
||||
>
|
||||
```
|
||||
|
||||
If several providers repeat the same projection from route values to model
|
||||
methods, the helper can stay deliberately tiny:
|
||||
|
||||
```ts
|
||||
const configureOpenAI = (input: OpenAIConfig = {}) =>
|
||||
Provider.define({
|
||||
id: openAIProvider,
|
||||
routes: {
|
||||
responses: openAIResponses.with(openAIConfig(input)),
|
||||
chat: openAIChat.with(openAIConfig(input)),
|
||||
},
|
||||
default: "responses",
|
||||
configure: configureOpenAI,
|
||||
})
|
||||
|
||||
export const OpenAI = configureOpenAI()
|
||||
```
|
||||
|
||||
`Provider.define(...)` would only project route methods and preserve types:
|
||||
|
||||
```ts
|
||||
OpenAI.model("gpt-4o")
|
||||
OpenAI.responses("gpt-4o")
|
||||
OpenAI.chat("gpt-4o")
|
||||
OpenAI.configure({ apiKey }).responses("gpt-4o")
|
||||
```
|
||||
|
||||
It must not register routes, select routes dynamically, or participate in
|
||||
execution. Execution still reads the route value carried by the model.
|
||||
|
||||
## Ideal Call Sites
|
||||
|
||||
Define concrete routes for a native provider, then project them through a
|
||||
provider facade:
|
||||
|
||||
```ts
|
||||
const openAIProvider = ProviderID.make("openai")
|
||||
|
||||
const openAIResponses = Route.make({
|
||||
id: "openai-responses",
|
||||
provider: openAIProvider,
|
||||
protocol: OpenAIResponses.protocol,
|
||||
transport: HttpTransport.sseJson,
|
||||
endpoint: {
|
||||
baseURL: "https://api.openai.com/v1",
|
||||
path: "/responses",
|
||||
},
|
||||
auth: Auth.envBearer("OPENAI_API_KEY"),
|
||||
})
|
||||
|
||||
const openAIChat = Route.make({
|
||||
id: "openai-chat",
|
||||
provider: openAIProvider,
|
||||
protocol: OpenAIChat.protocol,
|
||||
transport: HttpTransport.sseJson,
|
||||
endpoint: {
|
||||
baseURL: "https://api.openai.com/v1",
|
||||
path: "/chat/completions",
|
||||
},
|
||||
auth: Auth.envBearer("OPENAI_API_KEY"),
|
||||
})
|
||||
|
||||
const openAIConfig = (input: OpenAIConfig) => ({
|
||||
endpoint: input.endpoint,
|
||||
auth: input.auth ?? (input.apiKey ? Auth.bearer(input.apiKey) : undefined),
|
||||
headers: {
|
||||
"OpenAI-Organization": input.organization,
|
||||
"OpenAI-Project": input.project,
|
||||
},
|
||||
})
|
||||
|
||||
const configureOpenAI = (input: OpenAIConfig = {}) => {
|
||||
const responses = openAIResponses.with(openAIConfig(input))
|
||||
const chat = openAIChat.with(openAIConfig(input))
|
||||
|
||||
return {
|
||||
id: openAIProvider,
|
||||
responses: responses.model,
|
||||
chat: chat.model,
|
||||
model: responses.model,
|
||||
configure: configureOpenAI,
|
||||
}
|
||||
}
|
||||
|
||||
export const OpenAI = configureOpenAI()
|
||||
```
|
||||
|
||||
Specialize it functionally for concrete providers:
|
||||
|
||||
```ts
|
||||
const deepSeekProvider = ProviderID.make("deepseek")
|
||||
|
||||
const deepseekChat = openAIChat.with({
|
||||
id: "deepseek-chat",
|
||||
provider: deepSeekProvider,
|
||||
endpoint: {
|
||||
baseURL: "https://api.deepseek.com/v1",
|
||||
},
|
||||
auth: Auth.envBearer("DEEPSEEK_API_KEY"),
|
||||
})
|
||||
|
||||
const configureDeepSeek = (input: OpenAICompatibleConfig = {}) => {
|
||||
const route = deepseekChat.with({
|
||||
endpoint: input.endpoint,
|
||||
auth: input.auth ?? (input.apiKey ? Auth.bearer(input.apiKey) : undefined),
|
||||
})
|
||||
|
||||
return {
|
||||
id: deepSeekProvider,
|
||||
model: route.model,
|
||||
configure: configureDeepSeek,
|
||||
}
|
||||
}
|
||||
|
||||
export const DeepSeek = {
|
||||
id: deepSeekProvider,
|
||||
model: deepseekChat.model,
|
||||
configure: configureDeepSeek,
|
||||
}
|
||||
```
|
||||
|
||||
Provider-specific configuration happens before model selection:
|
||||
|
||||
```ts
|
||||
const deepseek = DeepSeek.configure({
|
||||
endpoint: {
|
||||
baseURL: "https://proxy.example.com/v1",
|
||||
},
|
||||
auth: Auth.bearer(apiKey),
|
||||
})
|
||||
|
||||
const model = deepseek.model("deepseek-chat")
|
||||
```
|
||||
|
||||
Final request call site stays boring:
|
||||
|
||||
```ts
|
||||
const response =
|
||||
yield *
|
||||
LLM.generate(
|
||||
LLM.request({
|
||||
model: DeepSeek.model("deepseek-chat"),
|
||||
prompt: "Hello.",
|
||||
}),
|
||||
)
|
||||
```
|
||||
|
||||
For direct provider-facade calls, Responses has one semantic model and route:
|
||||
|
||||
```ts
|
||||
OpenAI.responses("gpt-4o")
|
||||
```
|
||||
|
||||
The package-like OpenAI Responses entrypoint has the same transport-neutral
|
||||
`model(...)` contract:
|
||||
|
||||
```ts
|
||||
import { model } from "@opencode-ai/ai/providers/openai/responses"
|
||||
|
||||
model("gpt-4o", { apiKey })
|
||||
```
|
||||
|
||||
Vertex keeps Gemini, Chat, Responses, and Messages as separate package-like entrypoints,
|
||||
while sharing project/location resolution and ADC authentication internally:
|
||||
|
||||
```ts
|
||||
import { model } from "@opencode-ai/ai/providers/google-vertex/gemini"
|
||||
|
||||
model("gemini-3.5-flash", { project, location: "global" })
|
||||
```
|
||||
|
||||
```ts
|
||||
import { model } from "@opencode-ai/ai/providers/google-vertex/chat"
|
||||
|
||||
model("deepseek-ai/deepseek-v3.2-maas", { project, location: "global" })
|
||||
```
|
||||
|
||||
```ts
|
||||
import { model } from "@opencode-ai/ai/providers/google-vertex/responses"
|
||||
|
||||
model("xai/grok-4.20-reasoning", { project, location: "global" })
|
||||
```
|
||||
|
||||
```ts
|
||||
import { model } from "@opencode-ai/ai/providers/google-vertex/messages"
|
||||
|
||||
model("claude-sonnet-4-6", { project, location: "global" })
|
||||
```
|
||||
|
||||
The client does not require a different public layer for WebSocket execution.
|
||||
Responses routes use HTTP by default, and callers may pass a channel executor per
|
||||
call. Routes without channel support simply ignore that execution capability.
|
||||
|
||||
Azure is a route specialization with auth/path/default changes plus input
|
||||
mapping. The public API configures the Azure resource once, then selects
|
||||
deployment ids with pure model selectors:
|
||||
|
||||
```ts
|
||||
const azureProvider = ProviderID.make("azure")
|
||||
|
||||
const azureResponses = openAIResponses.with({
|
||||
id: "azure-openai-responses",
|
||||
provider: azureProvider,
|
||||
auth: Auth.envHeader("api-key", "AZURE_OPENAI_API_KEY"),
|
||||
})
|
||||
|
||||
const configureAzure = (input: AzureConfig = {}) => {
|
||||
const route = azureResponses.with({
|
||||
endpoint: {
|
||||
baseURL:
|
||||
input.baseURL ??
|
||||
Endpoint.envBaseURL(
|
||||
"AZURE_RESOURCE_NAME",
|
||||
(resourceName) => `https://${resourceName}.openai.azure.com/openai/v1`,
|
||||
),
|
||||
query: { "api-version": input.apiVersion ?? "v1" },
|
||||
},
|
||||
auth: input.apiKey ? Auth.header("api-key", input.apiKey) : Auth.envHeader("api-key", "AZURE_OPENAI_API_KEY"),
|
||||
})
|
||||
|
||||
return {
|
||||
id: azureProvider,
|
||||
model: route.model,
|
||||
responses: route.model,
|
||||
configure: configureAzure,
|
||||
}
|
||||
}
|
||||
|
||||
export const Azure = configureAzure()
|
||||
|
||||
const azure = Azure.configure({
|
||||
resourceName: "my-resource",
|
||||
apiVersion: "v1",
|
||||
})
|
||||
|
||||
const model = azure.responses("my-deployment")
|
||||
```
|
||||
|
||||
Default provider facades are only valid when required configuration has a lazy
|
||||
default source. `Azure.responses("my-deployment")` can be valid if endpoint
|
||||
resolution reads `AZURE_RESOURCE_NAME` lazily and fails with a typed
|
||||
configuration error when missing. If a provider has no sensible lazy default,
|
||||
do not expose a default model selector; expose only a configured entrypoint.
|
||||
|
||||
Cloudflare AI Gateway and Workers AI are separate product facades because their
|
||||
configuration surfaces differ. Do not make a root `Cloudflare.configure(...)`
|
||||
pretend there is one coherent Cloudflare provider configuration:
|
||||
|
||||
```ts
|
||||
const cloudflareProvider = ProviderID.make("cloudflare-ai-gateway")
|
||||
|
||||
const cloudflareOpenAIChat = openAIChat.with({
|
||||
id: "cloudflare-ai-gateway-openai-chat",
|
||||
provider: cloudflareProvider,
|
||||
auth: Auth.bearerHeader("cf-aig-authorization").andThen(Auth.bearer()),
|
||||
})
|
||||
|
||||
const configureCloudflareAIGateway = (input: CloudflareAIGatewayConfig) => {
|
||||
const route = cloudflareOpenAIChat.with({
|
||||
endpoint: {
|
||||
baseURL: `https://gateway.ai.cloudflare.com/v1/${input.accountId}/${input.gatewayId}/openai`,
|
||||
},
|
||||
auth: Auth.bearerHeader("cf-aig-authorization", input.gatewayApiKey).andThen(Auth.bearer(input.apiKey)),
|
||||
})
|
||||
|
||||
return {
|
||||
id: cloudflareProvider,
|
||||
model: (modelID: string) => route.model({ id: modelID }),
|
||||
configure: configureCloudflareAIGateway,
|
||||
}
|
||||
}
|
||||
|
||||
export const CloudflareAIGateway = {
|
||||
id: cloudflareProvider,
|
||||
configure: configureCloudflareAIGateway,
|
||||
}
|
||||
|
||||
const gateway = CloudflareAIGateway.configure({
|
||||
accountId: "account",
|
||||
gatewayId: "gateway",
|
||||
gatewayApiKey,
|
||||
apiKey,
|
||||
})
|
||||
|
||||
const model = gateway.model("openai/gpt-4o")
|
||||
```
|
||||
|
||||
If a Cloudflare product gains a full lazy env default, it can expose a direct
|
||||
selector too. Until then, omitting `CloudflareAIGateway.model(...)` makes missing
|
||||
account/gateway configuration unrepresentable.
|
||||
|
||||
opencode's dynamic runtime should construct executable models at its app
|
||||
boundary instead of exposing a giant unstructured public model constructor or a
|
||||
generic dynamic resolver:
|
||||
|
||||
```ts
|
||||
const model =
|
||||
providerID === "azure" ? Azure.configure(resolvedAzureConfig).responses(apiModelID) : OpenAI.responses(apiModelID)
|
||||
```
|
||||
|
||||
That boundary can branch on durable config/catalog metadata and call typed
|
||||
provider APIs directly. Transport selection remains execution policy: a Session
|
||||
or other caller may pass a WebSocket channel executor per call without changing
|
||||
the model constructed by this boundary.
|
||||
|
||||
## Competitive Shape
|
||||
|
||||
This follows the strongest parts of adjacent libraries:
|
||||
|
||||
- AI SDK: configured provider instances expose provider-specific model methods.
|
||||
- Effect AI: executable models carry provider requirements and can be resolved by
|
||||
an app boundary.
|
||||
- LiteLLM/opencode config: dynamic `providerID/modelID` branching belongs at the
|
||||
app boundary, not in the typed public provider API or a global runtime
|
||||
resolver.
|
||||
- LangChain/LlamaIndex: constructor-style config plus model id is convenient,
|
||||
but we avoid making model selection also configure endpoint/auth.
|
||||
|
||||
The chosen split is:
|
||||
|
||||
```txt
|
||||
Route = execution mechanics
|
||||
Provider facade = configured route group
|
||||
LanguageModel = selected executable model carrying route value
|
||||
App boundary = explicit durable-config -> typed-provider call
|
||||
```
|
||||
|
||||
## What This Removes
|
||||
|
||||
- No `Provider.make(...)` as a core abstraction.
|
||||
- No `Provider.make(...)` wrapper just to bind an id to model functions. Use a
|
||||
branded provider id constant and a plain exported provider facade.
|
||||
- No `Deployment.define(...)` unless future examples force it.
|
||||
- No global route registry as the normal execution path.
|
||||
- No import side effects required before a model can execute.
|
||||
- No duplicate `provider.id` object when selected models already carry provider
|
||||
id.
|
||||
- No `model(id, overrides)` escape hatch. Model selection takes the model id;
|
||||
endpoint/auth/deployment customization happens by configuring the route first.
|
||||
- No transport setting on a provider or executable model. OpenAI Responses uses
|
||||
HTTP by default and accepts an optional per-call channel executor as execution policy.
|
||||
- No separate public `LLMClient.layerWithWebSocket`. The runtime should expose one
|
||||
client layer with the available transport capabilities.
|
||||
- No executable `ModelRef`. The executable handle is `LanguageModel`; durable model
|
||||
identity stays separate and cannot execute on its own.
|
||||
|
||||
## Implementation Todo
|
||||
|
||||
- [x] Replace the current executable `ModelRef` with `LanguageModel`.
|
||||
- [x] Change `LanguageModel.route` to carry a route value, not a `RouteID` string.
|
||||
- [ ] Keep a separate durable model identity type for persisted/session/catalog
|
||||
data, likely `{ providerID, modelID }`, and make it clear that it cannot
|
||||
execute without resolver context.
|
||||
- [x] Change route model selectors so `route.model(id)` returns an executable
|
||||
model with the route value attached, not a globally registered route id.
|
||||
- [x] Remove the standalone `Route.model(route, defaults, mapInput)` helper;
|
||||
configured route instances own model selection.
|
||||
- [x] Remove endpoint/auth escape hatches from route model selection; callers must
|
||||
configure endpoint/auth through `route.with(...)` or provider facades before
|
||||
calling `.model(...)`.
|
||||
- [x] Remove request-shaping defaults from `LanguageModel`; selected models now carry only
|
||||
id, provider, and configured route while defaults live on routes or requests.
|
||||
- [x] Rework `LLMClient.stream` / `generate` to read
|
||||
`request.model.route` directly instead of calling `registeredRoute(...)`.
|
||||
- [x] Remove `Route.make(...)` global registration from the normal execution
|
||||
path; keep route ids only as diagnostics/provider API labels.
|
||||
- [x] Model endpoint as `{ baseURL, path, query }` on routes, then remove the
|
||||
current split where host/query live on the model and path lives in route
|
||||
transport setup.
|
||||
- [x] Define `Route.with(...)` with explicit patch semantics for endpoint merge,
|
||||
query merge, header merge, auth replacement, and optional diagnostic id.
|
||||
- [x] Make unconfigured transports reusable constants such as
|
||||
`HttpTransport.sseJson`; keep transport functions only for configured/fresh
|
||||
state construction.
|
||||
- [x] Collapse the public WebSocket runtime split so one `LLMClient.layer` accepts
|
||||
optional per-call channel execution without changing route identity.
|
||||
- [x] Convert OpenAI provider APIs to provider-facade shape:
|
||||
`OpenAI.configure(config).responses(id)` and `.chat(id)`.
|
||||
- [x] Convert Azure to a configured facade where resource/base URL/api version
|
||||
setup happens before selecting deployment ids.
|
||||
- [x] Split Cloudflare products into separate facades such as
|
||||
`CloudflareAIGateway` and `CloudflareWorkersAI`; do not expose a shared root
|
||||
config surface unless one product actually exists.
|
||||
- [x] Migrate remaining built-in provider facades one at a time so configuration
|
||||
happens before model selection and selectors accept only ids:
|
||||
xAI, GitHub Copilot, OpenRouter, OpenAI-compatible families, Anthropic,
|
||||
Google/Gemini, and Amazon Bedrock now use configured facades such as
|
||||
`Provider.configure(options).model(id)` with named selectors where needed.
|
||||
- [ ] Decide whether a tiny `Provider.define(...)` helper is warranted after two
|
||||
or three provider conversions; start with plain objects if duplication is not
|
||||
yet painful.
|
||||
- [x] Keep executable model construction transport-neutral at the Session boundary;
|
||||
Session-scoped execution policy supplies channel capability separately.
|
||||
- [ ] Update tests so direct route/provider tests assert route values are carried
|
||||
by executable models, and opencode/native tests assert boundary-based route
|
||||
selection.
|
||||
- [ ] Remove compatibility exports or stale docs only after internal call sites
|
||||
are migrated; do not keep duplicate constructor paths without an external
|
||||
compatibility need.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Default facades with required setup: should providers like Azure and Bedrock
|
||||
expose default model selectors only when all required setup has lazy env or
|
||||
credential-chain defaults? If not, omit the default selector so missing config
|
||||
is impossible at the type/API level.
|
||||
- Lazy endpoint/auth values: should `Endpoint.envBaseURL(...)` and env-backed
|
||||
auth produce typed configuration/authentication errors at compile/prepare time
|
||||
or only when executing the transport?
|
||||
- `Route.with(...)` clearing semantics: endpoint/query/header patches merge by
|
||||
default, but what is the explicit way to remove an inherited value?
|
||||
- Provider facade helper: keep plain objects until duplication hurts, or add a
|
||||
tiny `Provider.define(...)` immediately to enforce shape and method projection?
|
||||
- Auth shape: should auth stay as today's composable `Auth`, or split into an
|
||||
auth placement/strategy and credential sources?
|
||||
- Naming: is `baseURL` still the right endpoint field name, or should it be
|
||||
`origin` / `urlPrefix` to clarify that route `path` is appended?
|
||||
@@ -188,7 +188,7 @@ const FakeProtocol = Protocol.make<FakeBody, string, string, void>({
|
||||
},
|
||||
})
|
||||
|
||||
// An route is the runnable binding for that protocol. It adds the deployment
|
||||
// A route is the runnable binding for that protocol. It adds the deployment
|
||||
// axes that the protocol deliberately does not know: URL, auth, and framing.
|
||||
const FakeAdapter = Route.make({
|
||||
id: "fake-echo",
|
||||
|
||||
@@ -5,8 +5,8 @@
|
||||
// The default `"auto"` shape places breakpoints at the last tool definition,
|
||||
// the first and last distinct system parts, and the conversation tail. This
|
||||
// exposes reusable tool, base-agent, project, and session prefixes while
|
||||
// advancing the tail after each tool result keeps the previous cache entry
|
||||
// within Anthropic's 20-block lookback during long agent turns.
|
||||
// advancing the tail after each tool result keeps recent conversation prefixes
|
||||
// reusable during long agent runs.
|
||||
//
|
||||
// Manual `cache: CacheHint` placements on individual parts are preserved and
|
||||
// count against the four-breakpoint budget; auto only fills remaining slots.
|
||||
@@ -23,9 +23,7 @@ const NONE: CachePolicyObject = {}
|
||||
const BREAKPOINT_CAP = 4
|
||||
|
||||
// Resolution rules:
|
||||
// - undefined → "auto" — caching is on by default. The math favors it:
|
||||
// Anthropic 5m-cache write is 1.25x base, read is 0.1x,
|
||||
// so a single reuse within 5 minutes already wins.
|
||||
// - undefined → "auto" — caching is on by default.
|
||||
// - "auto" → tools + first/last system + final message boundary.
|
||||
// - "none" → no auto placement; manual `CacheHint`s still flow.
|
||||
// - object form → exactly what the caller asked for.
|
||||
|
||||
@@ -1015,8 +1015,8 @@ const step = (state: ParserState, event: AnthropicEvent) => {
|
||||
// =============================================================================
|
||||
/**
|
||||
* The Anthropic Messages protocol — request body construction, body schema,
|
||||
* and the streaming-event state machine. Used by native Anthropic Cloud and
|
||||
* (once registered) Vertex Anthropic / Bedrock-hosted Anthropic passthrough.
|
||||
* and the streaming-event state machine shared by Anthropic-compatible and
|
||||
* Vertex-hosted Messages routes.
|
||||
*/
|
||||
export const protocol = Protocol.make({
|
||||
id: ADAPTER,
|
||||
|
||||
@@ -630,8 +630,7 @@ const step = (state: ParserState, event: GeminiEvent) => {
|
||||
// =============================================================================
|
||||
/**
|
||||
* The Gemini protocol — request body construction, body schema, and the
|
||||
* streaming-event state machine. Used by Google AI Studio Gemini and (once
|
||||
* registered) Vertex Gemini.
|
||||
* streaming-event state machine shared by Google AI Studio and Vertex Gemini.
|
||||
*/
|
||||
export const protocol = Protocol.make({
|
||||
id: ADAPTER,
|
||||
|
||||
@@ -340,7 +340,7 @@ export const lowerTool = Effect.fn("OpenResponses.lowerTool")(function* (
|
||||
name: tool.name,
|
||||
description: tool.description,
|
||||
parameters: ToolSchemaProjection.responses(inputSchema),
|
||||
// TODO: Read this from Responses tool options so direct LLM callers can opt into strict schemas.
|
||||
// The common tool definition does not currently express Responses strict-schema policy.
|
||||
strict: false,
|
||||
}
|
||||
})
|
||||
|
||||
@@ -41,12 +41,10 @@ export interface ToolAccumulator {
|
||||
* when at least one is defined. Returns `undefined` when neither input nor
|
||||
* output is known so routes don't publish a misleading `0`.
|
||||
*
|
||||
* Under the additive `AI.Usage` contract, `inputTokens` and `outputTokens`
|
||||
* are the non-cached input and visible output only. The provider-supplied
|
||||
* `total` is the source of truth when present; the computed fallback
|
||||
* under-counts cache and reasoning by design and exists mainly so
|
||||
* Anthropic-style providers (which don't surface a total) still get a
|
||||
* sensible aggregate on the input + output axes.
|
||||
* Under the inclusive `AI.Usage` contract, `inputTokens` includes cached input
|
||||
* and `outputTokens` includes reasoning. Protocol mappers normalize those
|
||||
* inclusive values before calling this helper. The provider-supplied total is
|
||||
* the source of truth when present; otherwise their sum is the canonical total.
|
||||
*/
|
||||
export const totalTokens = (
|
||||
inputTokens: number | undefined,
|
||||
@@ -67,7 +65,7 @@ export const totalTokens = (
|
||||
*
|
||||
* If `total` is `undefined`, returns `undefined` (we don't fabricate
|
||||
* counts). If `subtrahend` is `undefined`, returns `total` unchanged. The
|
||||
* provider-native breakdown stays available on `Usage.native` for debugging.
|
||||
* provider-native breakdown stays available on `Usage.providerMetadata` for debugging.
|
||||
*/
|
||||
export const subtractTokens = (total: number | undefined, subtrahend: number | undefined): number | undefined => {
|
||||
if (total === undefined) return undefined
|
||||
@@ -199,8 +197,8 @@ export const errorText = (error: unknown) => {
|
||||
|
||||
/**
|
||||
* `framing` step for Server-Sent Events. Decodes UTF-8, runs the SSE channel
|
||||
* decoder, and drops empty / `[DONE]` keep-alive events so the downstream
|
||||
* `decodeChunk` sees one JSON string per element. The SSE channel emits a
|
||||
* decoder, and drops empty / `[DONE]` keep-alive events so the protocol event
|
||||
* schema sees one JSON string per element. The SSE channel emits a
|
||||
* `Retry` control event on its error channel; we drop it here (we don't
|
||||
* implement client-driven retries). Decoder failures become provider output
|
||||
* errors so the public error channel stays `AIError`.
|
||||
@@ -216,11 +214,7 @@ export const sseFraming = (bytes: Stream.Stream<Uint8Array, AIError>): Stream.St
|
||||
)
|
||||
|
||||
/**
|
||||
* Canonical invalid-request constructor. Lift one-line `const invalid =
|
||||
* (message) => invalidRequest(message)` aliases out of every
|
||||
* route so the error constructor lives in one place. If we ever extend
|
||||
* `InvalidRequestReason` with route context or trace metadata, the change
|
||||
* lands here.
|
||||
* Canonical invalid-request constructor shared by protocol lowering.
|
||||
*/
|
||||
export const invalidRequest = (message: string) =>
|
||||
new AIError({
|
||||
|
||||
@@ -4,7 +4,7 @@ import { newBreakpoints, ttlBucket, type Breakpoints } from "./cache.js"
|
||||
|
||||
// Bedrock cache markers are positional: emit a `cachePoint` block immediately
|
||||
// after the content the caller wants treated as a cacheable prefix. Bedrock
|
||||
// accepts optional `ttl: "5m" | "1h"` on cachePoint, mirroring Anthropic.
|
||||
// accepts optional `ttl: "5m" | "1h"` on cachePoint.
|
||||
export const CachePointBlock = Schema.Struct({
|
||||
cachePoint: Schema.Struct({
|
||||
type: Schema.tag("default"),
|
||||
@@ -13,9 +13,8 @@ export const CachePointBlock = Schema.Struct({
|
||||
})
|
||||
export type CachePointBlock = Schema.Schema.Type<typeof CachePointBlock>
|
||||
|
||||
// Bedrock-Claude enforces the same 4-breakpoint cap as the Anthropic Messages
|
||||
// API. Callers pass a shared counter through every `block()` call site so the
|
||||
// budget is respected across `system`, `messages`, and `tools`.
|
||||
// Callers pass a shared counter through every `block()` call site so the
|
||||
// four-breakpoint budget is respected across `system`, `messages`, and `tools`.
|
||||
export const BEDROCK_BREAKPOINT_CAP = 4
|
||||
|
||||
export type { Breakpoints } from "./cache.js"
|
||||
|
||||
@@ -1,6 +1,4 @@
|
||||
// Shared helpers for provider cache-marker lowering. Anthropic and Bedrock
|
||||
// both enforce a 4-breakpoint cap per request and accept the same `5m`/`1h`
|
||||
// TTL buckets, so the counter and TTL mapping live here.
|
||||
// Shared counter and TTL mapping for provider cache-marker lowering.
|
||||
|
||||
export interface Breakpoints {
|
||||
remaining: number
|
||||
@@ -9,8 +7,7 @@ export interface Breakpoints {
|
||||
|
||||
export const newBreakpoints = (cap: number): Breakpoints => ({ remaining: cap, dropped: 0 })
|
||||
|
||||
// Returns `"1h"` for any `ttlSeconds >= 3600`, otherwise `undefined` (the
|
||||
// provider default 5m). Anthropic & Bedrock both treat anything shorter than
|
||||
// an hour as 5m.
|
||||
// Requests of at least one hour use the explicit `"1h"` bucket; shorter
|
||||
// requests omit the wire TTL and use the provider default.
|
||||
export const ttlBucket = (ttlSeconds: number | undefined): "1h" | undefined =>
|
||||
ttlSeconds !== undefined && ttlSeconds >= 3600 ? "1h" : undefined
|
||||
|
||||
@@ -13,8 +13,8 @@ import type { AIError } from "../schema/index.js"
|
||||
* - AWS event stream — length-prefixed binary frames with CRC checksums.
|
||||
* Each emitted frame is one parsed binary event record.
|
||||
*
|
||||
* The frame type is opaque to this layer; the protocol's `decode` step turns
|
||||
* a frame into a typed chunk.
|
||||
* The frame type is opaque to this layer; the protocol's event schema decodes
|
||||
* each frame before its state machine handles it.
|
||||
*/
|
||||
export interface Definition<Frame> {
|
||||
readonly id: string
|
||||
|
||||
@@ -73,8 +73,7 @@ export interface ProtocolStream<Frame, Event, State> {
|
||||
*
|
||||
* Provider implementations should usually call `Protocol.make({ ... })`
|
||||
* without explicit type arguments; the schemas and parser functions are the
|
||||
* source of truth. The constructor remains as the public seam for future
|
||||
* cross-cutting concerns such as tracing or instrumentation.
|
||||
* source of truth.
|
||||
*/
|
||||
export const make = <Body, Frame, Event, State>(
|
||||
input: Protocol<Body, Frame, Event, State>,
|
||||
|
||||
@@ -272,10 +272,9 @@ export class CacheHint extends Schema.Class<CacheHint>("LLM.CacheHint")({
|
||||
// Auto-placement policy for prompt caching. The protocol-neutral lowering step
|
||||
// reads this and injects `CacheHint`s at the configured boundaries; the
|
||||
// per-protocol body builders then translate those hints into wire markers as
|
||||
// usual. `"auto"` is the recommended default for agent loops — it places
|
||||
// usual. `"auto"` is the default for agent loops — it places
|
||||
// breakpoints at the last tool definition, the first and last distinct system
|
||||
// parts, and the conversation tail. The rolling message breakpoint keeps a
|
||||
// prior cache entry within Anthropic/Bedrock's 20-block lookback during long
|
||||
// parts, and the conversation tail so recent prefixes remain reusable during
|
||||
// tool loops.
|
||||
//
|
||||
// Pass `"none"` to opt out entirely (the legacy behavior). Pass the granular
|
||||
|
||||
@@ -55,7 +55,7 @@ const schema_only_weather = Tool.make({
|
||||
})
|
||||
|
||||
describe("LLMClient tools", () => {
|
||||
it.effect("uses the registered model route when adding runtime tools", () =>
|
||||
it.effect("uses the selected model route when adding runtime tools", () =>
|
||||
Effect.gen(function* () {
|
||||
const layer = scriptedResponses([
|
||||
sseEvents(deltaChunk({ role: "assistant", content: "Done." }), finishChunk("stop")),
|
||||
|
||||
Reference in New Issue
Block a user