Mirror_Repos/anomalyco_opencode
2
0
Fork 0
Code Issues Pull Requests Actions 61 Packages Projects Releases Wiki Activity

refactor(llm): replace patches with transforms

Browse Source
This commit is contained in:
Kit Langton
2026-05-05 17:58:58 -04:00
parent 172c382f00
commit b3568ab0f0
60 changed files with 666 additions and 2595 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/llm/AGENTS.md
+28 -25
View File
@@ -31,7 +31,7 @@ const request = LLM.request({
const response = yield* LLMClient.make({ adapters: [OpenAIChat.adapter] }).generate(request)
```
`LLM.request(...)` builds an `LLMRequest`. `LLMClient.make(...)` selects an adapter by `request.model.protocol`, applies patches, prepares a typed provider payload, asks the adapter for a real `HttpClientRequest.HttpClientRequest`, sends it through `RequestExecutor.Service`, parses the provider stream into common `LLMEvent`s, and finally returns an `LLMResponse`.
`LLM.request(...)` builds an `LLMRequest`. `LLMClient.make(...)` selects an adapter by `request.model.adapter`, applies runtime transforms, prepares a typed provider payload, applies adapter-local payload transforms, asks the adapter for a real `HttpClientRequest.HttpClientRequest`, sends it through `RequestExecutor.Service`, parses the provider stream into common `LLMEvent`s, and finally returns an `LLMResponse`.
Use `LLMClient.make(...).stream(request)` when callers want incremental `LLMEvent`s. Use `LLMClient.make(...).generate(request)` when callers want those same events collected into an `LLMResponse`. Use `LLMClient.make(...).prepare<Payload>(request)` to compile a request through the adapter pipeline without sending it — the optional `Payload` type argument narrows `.payload` to the adapter's native shape (e.g. `prepare<OpenAIChatPayload>(...)` returns a `PreparedRequestOf<OpenAIChatPayload>`). The runtime payload is identical; the generic is a type-level assertion.
@@ -71,30 +71,33 @@ packages/llm/src/
llm.ts // request constructors and convenience helpers
adapter.ts // Adapter.make + LLMClient.make
executor.ts // RequestExecutor service + transport error mapping
patch.ts // Patch system (request/prompt/tool-schema/payload/stream)
transform.ts // Transform system (request/prompt/tool-schema/payload/stream)
protocol.ts // Protocol type + Protocol.define
endpoint.ts // Endpoint type + Endpoint.baseURL
auth.ts // Auth type + Auth.bearer / Auth.apiKeyHeader / Auth.passthrough
framing.ts // Framing type + Framing.sse
provider-transform.ts // ProviderTransform helpers (defaults, capability gates)
provider/
protocols/
shared.ts // ProviderShared toolkit used inside protocol impls
patch.ts // ProviderPatch helpers (defaults, capability gates)
openai-chat.ts // protocol + adapter (compose OpenAIChat.protocol)
openai-responses.ts
anthropic-messages.ts
gemini.ts
bedrock-converse.ts
openai-compatible-chat.ts // adapter that reuses OpenAIChat.protocol
openai-compatible-family.ts // family lookups (deepseek, togetherai, ...)
providers/
openai-compatible.ts // generic compatible helper + family model helpers
openai-compatible-profile.ts // family defaults (deepseek, togetherai, ...)
azure.ts / amazon-bedrock.ts / github-copilot.ts / google.ts / xai.ts / ... // provider model helpers
tool.ts // typed tool() helper
tool-runtime.ts // ToolRuntime.run with full tool-loop type safety
```
The dependency arrow points down: `provider/*.ts` files import `protocol`, `endpoint`, `auth`, `framing` and never the other direction. Lower-level modules know nothing about specific providers.
The dependency arrow points down: `providers/*.ts` files import `protocols`, `endpoint`, `auth`, and `framing`; protocols do not import provider metadata. Lower-level modules know nothing about specific providers.
### Shared adapter helpers
@@ -110,25 +113,25 @@ The dependency arrow points down: `provider/*.ts` files import `protocol`, `endp
If you find yourself copying a 3-to-5-line snippet between two protocols, lift it into `ProviderShared` next to these helpers rather than duplicating.
### Patches
### Transforms
Patches are the forcing function for provider/model quirks, similar to OpenCode's `ProviderTransform`: payload cleanup, provider option shaping, schema sanitization, and payload-level body tweaks. If a behavior is not universal enough for common IR, keep it as a named patch with a trace entry. Good examples:
Transforms are the forcing function for provider/model quirks, similar to OpenCode's `ProviderTransform`: prompt cleanup, provider option shaping, schema sanitization, and payload-level body tweaks. If a behavior is not universal enough for common IR, keep it as a named transform at the right pipeline boundary. Good examples:
- OpenAI Chat streaming usage: `payload.openai-chat.include-usage` adds `stream_options.include_usage`.
- Anthropic prompt caching: map common cache hints onto selected content/message blocks.
- Mistral/OpenAI-compatible prompt cleanup: normalize empty text content or tool-call IDs only for affected models.
- Reasoning models: map common reasoning intent to provider-specific effort, summary, or encrypted-content fields.
Do not grow common request schemas just to fit one provider. Prefer adapter-local payload schemas plus patches selected by provider/model predicates. Patches must not reroute a request: `model.provider`, `model.id`, and `model.protocol` are fixed before patches run, and request patches that change them are rejected.
Do not grow common request schemas just to fit one provider. Prefer runtime transforms for common IR and adapter-local payload transforms for provider-native payload fields. Runtime transforms cannot touch provider-native payloads, and transforms must not reroute a request: `model.provider`, `model.id`, `model.adapter`, and `model.protocol` are fixed before transforms run.
Current OpenCode parity map:
| Native location | OpenCode source | Status |
| --- | --- | --- |
| `ProviderPatch.removeEmptyAnthropicContent` | `ProviderTransform.normalizeMessages(...)` empty-content filtering for Anthropic/Bedrock. | Ported default patch. |
| `ProviderPatch.scrubClaudeToolIds` | `ProviderTransform.normalizeMessages(...)` Claude tool id scrub. | Ported default patch. |
| `ProviderPatch.scrubMistralToolIds` | `ProviderTransform.normalizeMessages(...)` Mistral/Devstral tool id scrub. | Partially ported; sequence repair still TODO. |
| `ProviderPatch.cachePromptHints` | `ProviderTransform.applyCaching(...)`. | Ported default patch. |
| `ProviderTransform.removeEmptyAnthropicContent` | `ProviderTransform.normalizeMessages(...)` empty-content filtering for Anthropic/Bedrock. | Ported default transform. |
| `ProviderTransform.scrubClaudeToolIds` | `ProviderTransform.normalizeMessages(...)` Claude tool id scrub. | Ported default transform. |
| `ProviderTransform.scrubMistralToolIds` | `ProviderTransform.normalizeMessages(...)` Mistral/Devstral tool id scrub. | Partially ported; sequence repair still TODO. |
| `ProviderTransform.cachePromptHints` | `ProviderTransform.applyCaching(...)`. | Ported default transform. |
| `Gemini` schema sanitizer/projector | `ProviderTransform.schema(...)` Gemini branch. | Ported inside the adapter protocol. |
| Provider option namespacing and model-specific reasoning defaults | `ProviderTransform.providerOptions(...)`, `options(...)`, `variants(...)`. | TODO/native bridge fallback today. |
@@ -255,11 +258,11 @@ Do not blanket re-record an entire test file when adding one cassette. `RECORD=t
- [x] Expand OpenAI Chat support for assistant tool-call messages followed by tool-result messages.
- [x] Add OpenAI Chat recorded tests for tool-result follow-up and usage chunks.
- [x] Add deterministic fixture tests for unsupported content paths, including media in user messages and unsupported assistant content.
- [x] Add provider patch examples from real opencode quirks, starting with prompt normalization and payload-level provider options.
- [x] Add provider transform examples from real opencode quirks, starting with prompt normalization and adapter-local payload options.
- [x] Add an OpenAI Responses adapter once the Chat adapter shape feels stable.
- [x] Add Anthropic Messages adapter coverage after Responses, especially content block mapping, tool use/result mapping, and cache hints.
- [x] Add Gemini adapter coverage for text, media input, tool calls, reasoning deltas, finish reasons, usage, and recorded cassettes.
- [x] Extract or port OpenCode's `ProviderTransform.schema` Gemini sanitizer into a tested `packages/llm` tool-schema patch; do not keep a divergent adapter-local copy long term.
- [x] Extract or port OpenCode's `ProviderTransform.schema` Gemini sanitizer into a tested `packages/llm` tool-schema transform; do not keep a divergent adapter-local copy long term.
### Provider Coverage
@@ -268,19 +271,19 @@ Do not blanket re-record an entire test file when adding one cassette. `RECORD=t
- [x] Cover OpenAI-compatible provider families that can share the generic adapter first: DeepSeek, TogetherAI, Cerebras, Baseten, Fireworks, DeepInfra, and similar providers.
- [ ] Decide which providers need thin dedicated wrappers over OpenAI-compatible Chat because they have custom parsing/options: Mistral, Groq, Perplexity, and Cohere. xAI already has a thin model helper that routes to OpenAI Responses.
- [x] Add Bedrock Converse support: wire format (messages / system / inferenceConfig / toolConfig), AWS event stream binary framing via `@smithy/eventstream-codec`, SigV4 signing via `aws4fetch` (or Bearer API key path), text/reasoning/tool/usage/finish decoding, cache hints, image/document content, deterministic tests, and recorded basic text/tool cassettes. Additional model-specific fields are still TODO.
- [ ] Decide Vertex shape after Bedrock/OpenAI-compatible are stable: Vertex Gemini as Gemini payload/http patch vs adapter, and Vertex Anthropic as Anthropic payload/http patch vs adapter.
- [ ] Add Gateway/OpenRouter-style routing support only after the generic OpenAI-compatible adapter and provider option patch model are stable.
- [ ] Decide Vertex shape after Bedrock/OpenAI-compatible are stable: Vertex Gemini as Gemini payload/http transform vs adapter, and Vertex Anthropic as Anthropic payload/http transform vs adapter.
- [ ] Add Gateway/OpenRouter-style routing support only after the generic OpenAI-compatible adapter and provider option transform model are stable.
### OpenCode Parity Patches
- [ ] Port Anthropic tool-use ordering into a prompt patch.
- [ ] Finish Mistral/OpenAI-compatible cleanup patches, including message sequence repair after tool messages.
- [ ] Port Anthropic tool-use ordering into a prompt transform.
- [ ] Finish Mistral/OpenAI-compatible cleanup transforms, including message sequence repair after tool messages.
- [ ] Port DeepSeek reasoning handling and interleaved reasoning field mapping.
- [ ] Add unsupported attachment fallback patches keyed by model capabilities.
- [ ] Add cache hint patches for Anthropic, OpenRouter, Bedrock, OpenAI-compatible, Copilot, and Alibaba-style providers.
- [ ] Add provider option namespacing patches for Gateway, OpenRouter, OpenAI-compatible wrappers, and other provider-specific option bags. Azure already has model-helper support for base URL, `api-version`, and Chat-vs-Responses routing; future Azure work should cover any remaining provider-specific option mapping.
- [ ] Add model-specific reasoning option patches for providers that need effort, summary, or native reasoning fields.
- [ ] Add provider-specific metadata extraction patches only where OpenCode needs returned reasoning, citations, usage details, or provider-native fields.
- [ ] Add unsupported attachment fallback transforms keyed by model capabilities.
- [ ] Add cache hint transforms for Anthropic, OpenRouter, Bedrock, OpenAI-compatible, Copilot, and Alibaba-style providers.
- [ ] Add provider option namespacing transforms for Gateway, OpenRouter, OpenAI-compatible wrappers, and other provider-specific option bags. Azure already has model-helper support for base URL, `api-version`, and Chat-vs-Responses routing; future Azure work should cover any remaining provider-specific option mapping.
- [ ] Add model-specific reasoning option transforms for providers that need effort, summary, or native reasoning fields.
- [ ] Add provider-specific metadata extraction transforms only where OpenCode needs returned reasoning, citations, usage details, or provider-native fields.
### OpenCode Bridge
@@ -330,5 +333,5 @@ Do not blanket re-record an entire test file when adding one cassette. `RECORD=t
- [ ] Mistral, Groq, Perplexity, and Cohere basic/tool cassettes after deciding whether each stays generic OpenAI-compatible or gets a thin wrapper.
- [ ] xAI basic/tool cassettes for its OpenAI Responses model helper path.
- [x] Bedrock Converse basic text and tool-call cassettes (recorded against `us.amazon.nova-micro-v1:0` in us-east-1). Cache-hint cassettes still TODO.
- [ ] Vertex Gemini and Vertex Anthropic basic/tool cassettes after the Vertex adapter/patch shape is decided.
- [ ] Vertex Gemini and Vertex Anthropic basic/tool cassettes after the Vertex adapter/transform shape is decided.
- [ ] Gateway/OpenRouter routing-header cassettes after routing support lands.
packages/llm/ARCHITECTURE.layered.md
-334
View File
@@ -1,334 +0,0 @@
# LLM Architecture
This package has one public shape:
```ts
const model = OpenAI.model("gpt-4o-mini", { apiKey })
const response = yield* LLM.generate({ model, prompt: "Say hello." })
```
Everything below explains how that stays simple while still supporting OpenAI, Anthropic, Gemini, Bedrock, OpenRouter, Azure, local OpenAI-compatible gateways, provider quirks, hosted tools, cache hints, and request replay.
Read this as layers. Stop when the next layer is not relevant to your task.
| Layer | Use it when... |
| --- | --- |
| 1. Public API | You are writing application code or examples. |
| 2. Model Routing | You need to understand why provider, model, and protocol are separate. |
| 3. Request Lifecycle | You are debugging what happens after `LLM.generate`. |
| 4. Provider Composition | You are wiring a new deployment or protocol. |
| 5. Provider Patches | You are preserving provider-specific behavior without polluting common schemas. |
| 6. Design Tradeoffs | You are relating this to AI SDK or OpenCode's current provider stack. |
## 1. Public API
Most code should live here.
```ts
import { Effect, Layer } from "effect"
import { LLM, RequestExecutor } from "@opencode-ai/llm"
import { OpenAI } from "@opencode-ai/llm/providers"
const model = OpenAI.model("gpt-4o-mini", {
apiKey: Bun.env.OPENAI_API_KEY,
})
const program = Effect.gen(function* () {
const response = yield* LLM.generate({
model,
prompt: "Say hello.",
})
console.log(response.text)
}).pipe(
Effect.provide(Layer.mergeAll(
LLM.layer({ providers: [OpenAI] }),
RequestExecutor.defaultLayer,
)),
)
```
The public rule is:
```txt
provider helper -> model reference -> LLM.generate / LLM.stream
```
Provider helpers should feel boring at use sites.
```ts
OpenAI.model("gpt-4o-mini", { apiKey })
Anthropic.model("claude-3-5-sonnet-latest", { apiKey })
Google.model("gemini-2.0-flash", { apiKey })
OpenRouter.model("openai/gpt-4o-mini", { apiKey })
OpenAICompatible.model("gpt-4o-mini", {
name: "local-gateway",
baseURL: "http://localhost:11434/v1",
})
```
For OpenAI, `OpenAI.model(...)` means Responses. Use `OpenAI.chat(...)` only when you specifically need Chat Completions.
<details>
<summary>Hidden implementation details</summary>
The call site does not name adapters, protocols, endpoints, auth, framing, patches, provider payloads, or stream parsers.
Those are runtime concerns. They should be inspectable and composable, but not required for normal use.
</details>
## 2. Model Routing
A model reference is a route card. It says which model to call, which provider owns the deployment, and which wire protocol can talk to it.
```txt
OpenAI.model("gpt-4o-mini", { apiKey })
-> provider: openai
-> protocol: openai-responses
-> id: gpt-4o-mini
OpenRouter.model("openai/gpt-4o-mini", { apiKey })
-> provider: openrouter
-> protocol: openai-compatible-chat
-> id: openai/gpt-4o-mini
OpenAICompatible.model("gpt-4o-mini", { name: "local-gateway", baseURL })
-> provider: local-gateway
-> protocol: openai-compatible-chat
-> id: gpt-4o-mini
```
This split is the core design choice.
| Concept | Question it answers |
| --- | --- |
| `provider` | Who is the deployment or product surface? |
| `protocol` | Which request/response shape should the runtime use? |
| `id` | Which model/deployment id should be sent? |
| `baseURL` | Where should HTTP go? |
| `apiKey`, `headers`, `queryParams`, `native` | What deployment-specific transport data is needed? |
| `capabilities`, `limits` | What normalized features and constraints should callers see? |
Provider identity and wire protocol often differ. OpenRouter is not OpenAI, but many OpenRouter models speak enough OpenAI Chat shape to reuse the OpenAI Chat protocol.
<details>
<summary>Conceptual ModelRef shape</summary>
```ts
type ModelRef = {
id: ModelID
provider: ProviderID
protocol: ProtocolID
baseURL?: string
apiKey?: string
headers?: Record<string, string>
queryParams?: Record<string, string>
capabilities: ModelCapabilities
limits: ModelLimits
native?: Record<string, unknown>
}
```
`ModelRef` is not a provider client. It does not send requests. It is the stable, serializable description of what should be called.
</details>
## 3. Request Lifecycle
At runtime, every request follows the same path.
```txt
LLM.generate({ model, prompt })
-> LLM.request(...)
-> LLMClient
-> adapter selected by model.protocol
-> provider-native payload
-> HttpClientRequest
-> RequestExecutor
-> provider response stream
-> LLMEvent stream
-> LLMResponse
```
The high-level API hides that pipeline.
```ts
const response = yield* LLM.generate({
model: OpenAI.model("gpt-4o-mini", { apiKey }),
prompt: "Say hello.",
})
```
The lower-level runtime sees this shape.
```ts
const request = LLM.request({
model,
prompt: "Say hello.",
})
const client = LLMClient.make({
adapters: [OpenAIResponses.adapter, OpenAIChat.adapter],
patches: ProviderPatch.defaults,
})
const response = yield* client.generate(request)
```
<details>
<summary>Adapter pipeline</summary>
The adapter is selected by `request.model.protocol`.
```ts
const adapter = adapters.get(request.model.protocol)
const draft = adapter.prepare(request)
const patched = applyTargetPatches(draft)
const target = adapter.validate(patched)
const http = adapter.toHttp(target)
const response = yield* RequestExecutor.execute(http)
const events = adapter.parse(response)
```
`generate` collects the same `LLMEvent` stream that `stream` exposes incrementally.
</details>
## 4. Provider Composition
Provider behavior is split across reusable layers instead of one large provider class.
```txt
Provider helper
creates ModelRef values
Provider module
exports adapters and helper constructors
Adapter
composes Protocol + Endpoint + Auth + Framing
Protocol
owns provider-native request and stream semantics
```
The composition rule is:
```txt
Adapter = Protocol + Endpoint + Auth + Framing
```
OpenAI Chat is a normal adapter composition.
```ts
export const adapter = Adapter.make({
id: "openai-chat",
protocol: OpenAIChat.protocol,
endpoint: Endpoint.baseURL({
default: "https://api.openai.com/v1",
path: "/chat/completions",
}),
auth: Auth.openAI,
framing: Framing.sse,
})
```
OpenAI-compatible Chat is the same protocol with different deployment axes.
```txt
OpenAI-compatible Chat adapter
= OpenAIChat.protocol
+ required baseURL endpoint
+ bearer auth
+ SSE framing
```
That is why these can share implementation without pretending they are the same provider.
```ts
OpenAI.chat("gpt-4o-mini", { apiKey })
OpenRouter.model("openai/gpt-4o-mini", { apiKey })
OpenAICompatible.model("gpt-4o-mini", { name: "local-gateway", baseURL })
```
<details>
<summary>Layer responsibilities</summary>
| Layer | Owns |
| --- | --- |
| Provider helper | Public constructor, defaults, provider identity, model capabilities, limits. |
| Provider module | Exported adapters and helpers passed to `LLM.layer({ providers })`. |
| Adapter | Runtime registration and composition. |
| Protocol | Request lowering, payload schema, chunk schema, stream state machine. |
| Endpoint | URL construction, base URL, path, query params, deployment routing. |
| Auth | Bearer tokens, API-key headers, SigV4, future IAM/AAD signing. |
| Framing | Bytes to frames before protocol parsing, usually SSE. |
</details>
<details>
<summary>When to add what</summary>
| Need | Add |
| --- | --- |
| A new hosted product speaks an existing protocol | Provider helper plus adapter composition. |
| A provider has a unique request/response shape | New protocol plus adapter composition. |
| A provider has the same protocol but different auth | Reuse protocol, add auth axis. |
| A provider has the same protocol but different URL rules | Reuse protocol, add endpoint axis. |
| A provider streams non-SSE frames | Reuse or add protocol, add framing axis. |
| A model needs a one-off body tweak | Patch, not a common schema field. |
</details>
## 5. Provider Patches
Patches are named, traceable provider/model transformations inspired by OpenCode's existing `ProviderTransform` layer.
Use a patch when behavior is real but not universal enough to belong in the common request schema.
```txt
cache.prompt-hints
anthropic.scrub-tool-call-ids
target.openai-chat.include-usage
```
Each patch has an id, phase, predicate, and reason. Applied patches appear in `patchTrace`.
Patches are not a routing mechanism. Adapter selection happens from the original `request.model`; request patches may change payload details, but changing `model.provider`, `model.id`, or `model.protocol` is rejected. If a call needs a different provider, model, or protocol, construct a different model handle before building the request.
The rule is:
```txt
Common request shape stays small.
Provider quirks stay named and auditable.
Model routing stays explicit at the call site.
```
Good patch candidates include cache hint lowering, model-specific reasoning fields, OpenAI-compatible message cleanup, hosted-tool shape differences, metadata extraction, and provider option namespacing.
Bad patch candidates are behaviors that every provider supports the same way. Those belong in the common request model.
## 6. Design Tradeoffs
AI SDK has an excellent use-site shape.
```ts
openai("gpt-4o-mini")
openai.chat("gpt-4o-mini")
createOpenAICompatible({ baseURL })("gpt-4o-mini")
```
This package keeps the use-site shape familiar.
```ts
OpenAI.model("gpt-4o-mini", { apiKey })
OpenAI.chat("gpt-4o-mini", { apiKey })
OpenAICompatible.model("gpt-4o-mini", { name, baseURL, apiKey })
```
The difference is below the public API.
| Concern | AI SDK | This package |
| --- | --- | --- |
| Use site | Provider creates runnable model object. | Provider creates `ModelRef`; `LLM` runtime runs it. |
| Provider implementation | Usually provider-package-specific language model classes. | Protocol, endpoint, auth, framing, and patches are separate axes. |
| OpenAI-compatible reuse | Dedicated OpenAI-compatible implementation. | Reuses `OpenAIChat.protocol` with different deployment axes. |
| Debug/replay/parity | Mostly hidden behind provider implementation. | Exposed through request lowering, patches, adapters, and events. |
The tradeoff is intentional. The public API should feel small. The internals should be inspectable enough for OpenCode to preserve provider parity, replay HTTP, diff native payloads, and migrate provider-by-provider without cloning whole adapter classes.
packages/llm/ARCHITECTURE.md
-606
View File
@@ -1,606 +0,0 @@
# LLM Architecture
This package has one public shape:
```ts
const model = OpenAI.model("gpt-4o-mini", { apiKey })
const response = yield* LLM.generate({ model, prompt: "Say hello." })
```
Everything below explains how that stays simple while still supporting OpenAI, Anthropic, Gemini, Bedrock, OpenRouter, Azure, local OpenAI-compatible gateways, provider quirks, hosted tools, cache hints, and request replay.
Read this document as terraces. Stop when the next layer is not useful for your task.
| Terrace | You need this when... |
| --- | --- |
| 1. Use the API | You are writing application code or examples. |
| 2. Choose a route | You need to understand why provider, model, and protocol are separate. |
| 3. Follow a request | You are debugging what happens after `LLM.generate`. |
| 4. Add a provider | You are wiring a new deployment or protocol. |
| 5. Patch a quirk | You are preserving provider-specific behavior without polluting common schemas. |
| 6. Compare designs | You are relating this to AI SDK or OpenCode's current provider stack. |
## Terrace 1: Use The API
Most code should live here.
```ts
import { Effect, Layer } from "effect"
import { LLM, RequestExecutor } from "@opencode-ai/llm"
import { OpenAI } from "@opencode-ai/llm/providers"
const model = OpenAI.model("gpt-4o-mini", {
apiKey: Bun.env.OPENAI_API_KEY,
})
const program = Effect.gen(function* () {
const response = yield* LLM.generate({
model,
prompt: "Say hello.",
})
console.log(response.text)
}).pipe(
Effect.provide(Layer.mergeAll(
LLM.layer(),
RequestExecutor.defaultLayer,
)),
)
```
The public rule is:
```txt
provider helper -> model handle -> LLM.generate / LLM.stream
```
Provider helpers should feel boring at use sites.
```ts
OpenAI.model("gpt-4o-mini", { apiKey })
Anthropic.model("claude-3-5-sonnet-latest", { apiKey })
Google.model("gemini-2.0-flash", { apiKey })
OpenRouter.model("openai/gpt-4o-mini", { apiKey })
OpenAICompatible.model("gpt-4o-mini", {
provider: "local-gateway",
baseURL: "http://localhost:11434/v1",
})
```
For OpenAI, `OpenAI.model(...)` means Responses. Use `OpenAI.chat(...)` only when you specifically need Chat Completions.
<details>
<summary>What this terrace intentionally hides</summary>
The call site does not name adapters, protocols, endpoints, auth, framing, patches, provider payloads, or stream parsers.
Those things are runtime concerns. They should be inspectable and composable, but not required for normal use.
</details>
## Terrace 2: Choose A Route
A model reference is a route card. It says which model to call, which provider owns the deployment, and which wire protocol can talk to it.
```txt
OpenAI.model("gpt-4o-mini", { apiKey })
-> provider: openai
-> protocol: openai-responses
-> id: gpt-4o-mini
OpenRouter.model("openai/gpt-4o-mini", { apiKey })
-> provider: openrouter
-> protocol: openai-compatible-chat
-> id: openai/gpt-4o-mini
OpenAICompatible.model("gpt-4o-mini", { provider: "local-gateway", baseURL })
-> provider: local-gateway
-> protocol: openai-compatible-chat
-> id: gpt-4o-mini
```
This split is the core design choice.
| Concept | Question it answers |
| --- | --- |
| `provider` | Who is the deployment or product surface? |
| `protocol` | Which request/response shape should the runtime use? This is an open string so custom providers can add new protocol ids. |
| `id` | Which model/deployment id should be sent? |
| `baseURL` | Where should HTTP go? |
| `apiKey`, `headers`, `queryParams`, `native` | What deployment-specific transport data is needed? |
| `capabilities`, `limits` | What normalized features and constraints should callers see? |
Provider identity and wire protocol often differ. OpenRouter is not OpenAI, but many OpenRouter models speak enough OpenAI Chat shape to reuse the OpenAI Chat protocol.
<details>
<summary>Conceptual ModelRef shape</summary>
```ts
type ModelRef = {
id: ModelID
provider: ProviderID
protocol: ProtocolID
baseURL?: string
apiKey?: string
headers?: Record<string, string>
queryParams?: Record<string, string>
capabilities: ModelCapabilities
limits: ModelLimits
native?: Record<string, unknown>
}
```
`ModelRef` is the stable, serializable description of what should be called. Provider helpers also bind an in-memory adapter to the returned model handle so direct call sites do not need to manually register adapters; serialized copies fall back to `model.protocol` registry lookup.
</details>
## Terrace 3: Follow A Request
At runtime, the flow is a staircase.
```txt
LLM.generate({ model, prompt })
-> LLM.request(...)
-> LLMClient
-> adapter from the model handle, or explicit registry fallback
-> provider-native payload
-> HttpClientRequest
-> RequestExecutor
-> provider response stream
-> LLMEvent stream
-> LLMResponse
```
The high-level API hides that pipeline.
```ts
const response = yield* LLM.generate({
model: OpenAI.model("gpt-4o-mini", { apiKey }),
prompt: "Say hello.",
})
```
The lower-level runtime sees this shape.
```ts
const request = LLM.request({
model,
prompt: "Say hello.",
})
const client = LLMClient.make({
adapters: [],
patches: ProviderPatch.defaults,
})
const response = yield* client.generate(request)
```
<details>
<summary>Adapter pipeline</summary>
Explicit adapters passed to `LLMClient.make(...)` win first. If no explicit adapter matches, the adapter bound to the in-memory model handle is used. If the model was serialized and revived, `LLMClient` falls back to the explicit registry keyed by `request.model.protocol`.
```ts
const adapter = adapters.get(request.model.protocol) ?? modelAdapters.get(request.model)
const candidate = adapter.prepare(request)
const patched = applyPayloadPatches(candidate)
const payload = adapter.validate(patched)
const http = adapter.toHttp(payload)
const response = yield* RequestExecutor.execute(http)
const events = adapter.parse(response)
```
`generate` collects the same `LLMEvent` stream that `stream` exposes incrementally.
</details>
### How Adapter Is Used Today
Keeping the current names, an `Adapter` is the runnable implementation for one registered request route.
It is selected from the model handle when the provider helper created the model in the same process. Explicit adapter registration overrides that default and remains the fallback for revived models, OpenCode config bridges, and low-level tests.
```ts
const adapters = new Map(
options.adapters.map((adapter) => [adapter.protocol, adapter] as const),
)
const adapter = adapters.get(request.model.protocol) ?? modelAdapters.get(request.model)
```
That means `protocol` has two jobs only in fallback paths:
| Job | Example |
| --- | --- |
| Describes the wire API shape | `openai-responses`, `anthropic-messages`, `gemini`. |
| Selects the adapter after serialization | `LLMClient` looks up `adapters.get(request.model.protocol)`. |
The adapter then owns the full compile/run boundary for that selected route.
| Adapter field | Used for |
| --- | --- |
| `id` | Human/debug name, prepared request metadata, patch namespace. |
| `protocol` | Registry key used by `LLMClient` lookup. |
| `patches` | Adapter-local payload patches. |
| `prepare(request)` | Lowers common `LLMRequest` into a provider-native payload candidate. |
| `validate(candidate)` | Validates and normalizes the payload candidate with the protocol payload schema. |
| `toHttp(payload, context)` | Builds the real `HttpClientRequest`. |
| `parse(response)` | Converts the provider response stream into common `LLMEvent`s. |
`Adapter.make(...)` is the normal constructor. It builds those methods by composing four pieces.
```txt
Adapter.make(...)
= Protocol.prepare / payload Schema / chunk Schema / process
+ Endpoint URL construction
+ Auth header/signing behavior
+ Framing bytes-to-frames behavior
```
`Protocol` no longer has a separate `encode` function in the normal path. The adapter validates payload patches and JSON-encodes the final payload from `protocol.payload`.
So the current relationship is:
```txt
ModelRef.protocol
-> selects Adapter after serialization / registry lookup
-> Adapter composes Protocol + Endpoint + Auth + Framing
-> Adapter compiles the request and parses the response
```
`model.provider` is still useful, but it is not the adapter lookup key. It identifies the deployment/product surface for defaults, capabilities, provider-specific options, patch predicates, debugging, telemetry, and OpenCode provider parity.
The odd-looking case is OpenAI-compatible Chat. It reuses the OpenAI Chat protocol implementation, but registers under a different protocol id.
```txt
OpenAICompatible.model(...)
-> provider: local-gateway
-> protocol: openai-compatible-chat
OpenAI-compatible adapter
-> registry key: openai-compatible-chat
-> reused Protocol implementation: OpenAIChat.protocol
-> custom Endpoint/Auth/Framing deployment axes
```
That keeps provider identity separate from the reusable wire behavior, even though the current `protocol` name is carrying both “wire shape” and “adapter lookup key” meaning.
## Terrace 4: Add A Provider
Provider behavior is split across reusable layers instead of one large provider class.
```txt
Provider helper
creates model handles backed by ModelRef values
Provider module
exports adapters and helper constructors
Adapter
composes Protocol + Endpoint + Auth + Framing
Protocol
owns provider-native request and stream semantics
```
The composition rule is:
```txt
Adapter = Protocol + Endpoint + Auth + Framing
```
OpenAI Chat is a normal adapter composition.
```ts
export const adapter = Adapter.make({
id: "openai-chat",
protocol: OpenAIChat.protocol,
endpoint: Endpoint.baseURL({
default: "https://api.openai.com/v1",
path: "/chat/completions",
}),
auth: Auth.openAI,
framing: Framing.sse,
})
```
OpenAI-compatible Chat is the same protocol with different deployment axes.
```txt
OpenAI-compatible Chat adapter
= OpenAIChat.protocol
+ required baseURL endpoint
+ bearer auth
+ SSE framing
```
That is why these can share implementation without pretending they are the same provider.
```ts
OpenAI.chat("gpt-4o-mini", { apiKey })
OpenRouter.model("openai/gpt-4o-mini", { apiKey })
OpenAICompatible.model("gpt-4o-mini", { provider: "local-gateway", baseURL })
```
<details>
<summary>Layer responsibilities</summary>
| Layer | Owns |
| --- | --- |
| Provider helper | Public constructor, defaults, provider identity, model capabilities, limits, in-process adapter binding. |
| Provider module | Exported adapters and helpers for explicit registry fallback. |
| Adapter | Runtime registration and composition. |
| Protocol | Request lowering, payload schema, chunk schema, stream state machine. |
| Endpoint | URL construction, base URL, path, query params, deployment routing. |
| Auth | Bearer tokens, API-key headers, SigV4, future IAM/AAD signing. |
| Framing | Bytes to frames before protocol parsing, usually SSE. |
</details>
<details>
<summary>When to add what</summary>
| Need | Add |
| --- | --- |
| A new hosted product speaks an existing protocol | Provider helper plus adapter composition. |
| A provider has a unique request/response shape | New protocol plus adapter composition. |
| A provider has the same protocol but different auth | Reuse protocol, add auth axis. |
| A provider has the same protocol but different URL rules | Reuse protocol, add endpoint axis. |
| A provider streams non-SSE frames | Reuse or add protocol, add framing axis. |
| A model needs a one-off body tweak | Patch, not a common schema field. |
</details>
## Terrace 5: Patch A Quirk
Patches are named, traceable provider/model transformations inspired by OpenCode's existing `ProviderTransform` layer.
Use a patch when behavior is real but not universal enough to belong in the common request schema.
```txt
cache.prompt-hints
anthropic.scrub-tool-call-ids
payload.openai-chat.include-usage
```
Each patch has an id, phase, predicate, and reason. Applied patches appear in `patchTrace`.
Patches are not a routing mechanism. Adapter selection happens from the original `request.model`; request patches may change payload details, but changing `model.provider`, `model.id`, or `model.protocol` is rejected. If a call needs a different provider, model, or protocol, construct a different model handle before building the request.
The rule is:
```txt
Common request shape stays small.
Provider quirks stay named and auditable.
Model routing stays explicit at the call site.
```
Good patch candidates include cache hint lowering, model-specific reasoning fields, OpenAI-compatible message cleanup, hosted-tool shape differences, metadata extraction, and provider option namespacing.
Bad patch candidates are behaviors that every provider supports the same way. Those belong in the common request model.
### OpenCode Transform Map
The native patch layer exists to preserve the behavior OpenCode previously centralized in `packages/opencode/src/provider/transform.ts`, but with named phases and `patchTrace` entries.
1. Empty Anthropic / Bedrock content
Old OpenCode shape:
```ts
// ProviderTransform.normalizeMessages(...)
if (model.api.npm === "@ai-sdk/anthropic" || model.api.npm === "@ai-sdk/amazon-bedrock") {
msgs = msgs
.map((msg) => removeEmptyTextAndReasoningParts(msg))
.filter((msg) => msg.content !== "" && msg.content.length > 0)
}
```
Native shape:
```ts
ProviderPatch.removeEmptyAnthropicContent
// prompt.anthropic.remove-empty-content
```
Status: ported default prompt patch. Anthropic and Bedrock reject empty text/reasoning blocks, so this stays as a provider/model quirk instead of forbidding empty content in the common request model.
2. Claude tool-call id scrub
Old OpenCode shape:
```ts
// ProviderTransform.normalizeMessages(...)
if (model.api.id.includes("claude")) {
toolCallId = toolCallId.replace(/[^a-zA-Z0-9_-]/g, "_")
}
```
Native shape:
```ts
ProviderPatch.scrubClaudeToolIds
// prompt.anthropic.scrub-tool-call-ids
```
Status: ported default prompt patch. The common request model can preserve original tool ids; Claude-specific transport constraints are applied late and traced.
3. Mistral / Devstral tool-call id scrub
Old OpenCode shape:
```ts
// ProviderTransform.normalizeMessages(...)
if (model.providerID === "mistral" || model.api.id.includes("devstral")) {
toolCallId = toolCallId.replace(/[^a-zA-Z0-9]/g, "").substring(0, 9).padEnd(9, "0")
}
```
Native shape:
```ts
ProviderPatch.scrubMistralToolIds
// prompt.mistral.scrub-tool-call-ids
```
Status: partially ported default prompt patch. The id scrub is ported. The old OpenCode message-sequence repair for `tool -> user` is still an OpenCode parity TODO.
4. Prompt caching markers
Old OpenCode shape:
```ts
// ProviderTransform.applyCaching(...)
const system = msgs.filter((msg) => msg.role === "system").slice(0, 2)
const final = msgs.filter((msg) => msg.role !== "system").slice(-2)
for (const msg of unique([...system, ...final])) {
msg.providerOptions = mergeDeep(msg.providerOptions ?? {}, providerCacheOptions)
}
```
Native shape:
```ts
ProviderPatch.cachePromptHints
// prompt.cache.prompt-hints
```
Status: ported default prompt patch. The patch marks the first two system parts and last two messages with a common `CacheHint`. Adapters lower that hint to provider-native shapes like Anthropic `cache_control` or Bedrock `cachePoint`.
5. Gemini tool-schema sanitization
Old OpenCode shape:
```ts
// ProviderTransform.schema(...)
if (model.providerID === "google" || model.api.id.includes("gemini")) {
schema = sanitizeGemini(schema)
}
```
Native shape:
```ts
// packages/llm/src/provider/gemini.ts
lowerToolSchema(tool.inputSchema)
```
Status: ported inside `Gemini.protocol`, not as a registered patch. Gemini has a distinct schema dialect, so the adapter owns both the historical sanitizer and the lossy projection into Gemini's accepted keys.
6. OpenAI Chat / OpenAI-compatible streaming usage
Old OpenCode shape:
```ts
// ProviderTransform.options(...), provider-specific option shaping
result["usage"] = { include: true }
```
Native shape:
```ts
OpenAIChat.adapter.patch("include-usage", ...)
OpenAICompatibleChat.adapter.patch("include-usage", ...)
// payload.openai-chat.include-usage
```
Status: ported as adapter-local payload patches. This is payload shape, not common request shape.
7. DeepSeek reasoning replay and interleaved reasoning fields
Old OpenCode shape:
```ts
// ProviderTransform.normalizeMessages(...)
if (model.api.id.toLowerCase().includes("deepseek")) {
assistant.content.push({ type: "reasoning", text: "" })
}
if (model.capabilities.interleaved?.field) {
msg.providerOptions.openaiCompatible[field] = reasoningText
}
```
Native shape: TODO.
Status: not ported yet. This should become provider-specific history shaping without exposing OpenAI-compatible reasoning internals globally.
8. Provider option namespacing
Old OpenCode shape:
```ts
// ProviderTransform.providerOptions(...)
if (model.api.npm === "@ai-sdk/gateway") return { gateway, [upstreamSlug]: rest }
if (model.api.npm === "@ai-sdk/azure") return { openai: options, azure: options }
return { [sdkKey(model.api.npm) ?? model.providerID]: options }
```
Native shape: TODO; the native OpenCode bridge currently falls back when prepared provider options are non-empty.
Status: not ported yet. These options are deployment/provider specific and should remain outside the common request model.
9. Model-specific reasoning defaults
Old OpenCode shape:
```ts
// ProviderTransform.options(...) and variants(...)
result["thinkingConfig"] = { includeThoughts: true }
result["enable_thinking"] = true
result["reasoningSummary"] = "auto"
result["include"] = ["reasoning.encrypted_content"]
```
Native shape: partly represented by `request.reasoning`; provider-native defaults are still TODO.
Status: not fully ported. Some models need native knobs that do not belong in the universal request shape.
## Terrace 6: Compare Designs
AI SDK has an excellent use-site shape.
```ts
openai("gpt-4o-mini")
openai.chat("gpt-4o-mini")
createOpenAICompatible({ baseURL })("gpt-4o-mini")
```
This package keeps the use-site shape familiar.
```ts
OpenAI.model("gpt-4o-mini", { apiKey })
OpenAI.chat("gpt-4o-mini", { apiKey })
OpenAICompatible.model("gpt-4o-mini", { provider, baseURL, apiKey })
```
The difference is below the public API.
| Concern | AI SDK | This package |
| --- | --- | --- |
| Use site | Provider creates runnable model object. | Provider creates a runnable model handle backed by serializable `ModelRef`. |
| Provider implementation | Usually provider-package-specific language model classes. | Protocol, endpoint, auth, framing, and patches are separate axes. |
| OpenAI-compatible reuse | Dedicated OpenAI-compatible implementation. | Reuses `OpenAIChat.protocol` with different deployment axes. |
| Debug/replay/parity | Mostly hidden behind provider implementation. | Exposed through request lowering, patches, adapters, and events. |
The tradeoff is intentional. The public API should feel small. The internals should be inspectable enough for OpenCode to preserve provider parity, replay HTTP, diff native payloads, and migrate provider-by-provider without cloning whole adapter classes.
### OpenCode Provider Loading
OpenCode's current AI SDK path is more dynamic than this package's native path.
```txt
OpenCode config/models.dev
-> model.api.npm
-> import or install AI SDK provider package
-> create provider SDK
-> sdk.languageModel(...) / sdk.responses(...) / sdk.chat(...)
```
That is why OpenCode can point at many AI SDK provider packages without this repo shipping a native adapter for each one.
The `@opencode-ai/llm` native path currently works in two modes:
| Mode | How it works | Good for |
| --- | --- | --- |
| In-process model helper | `OpenAI.model(...)`, `OpenAICompatible.model(...)`, or a third-party helper returns a model handle bound to an adapter. | Library users and code that imports the provider package directly. |
| Explicit adapter registry | `LLMClient.make({ adapters: [...] })` maps revived `ModelRef.protocol` values to shipped adapters. | OpenCode config/models.dev bridges, tests, request replay, serialized models. |
So OpenCode native integration is not “import any AI SDK provider package and it just works” yet. Today it supports protocols/providers that the OpenCode bridge can map to known native model helpers and adapters, plus generic OpenAI-compatible deployments. A config-defined provider with `@ai-sdk/openai-compatible` can map to `openai-compatible-chat`; a brand-new protocol needs a native adapter and bridge mapping.
The core package is now open enough for external protocols: `ProtocolID` is just a string, so a third-party package can define `Protocol.define(...)`, `Adapter.make(...)`, and a model helper without changing this package. To make OpenCode load those from config the same way it loads AI SDK packages, we would add an explicit native-provider loader/registry analogous to the AI SDK `model.api.npm` loader.
packages/llm/ARCHITECTURE.use-site-to-internals.md
-336
View File
@@ -1,336 +0,0 @@
# LLM Architecture
This package has one public shape:
```ts
const model = OpenAI.model("gpt-4o-mini", { apiKey })
const response = yield * LLM.generate({ model, prompt: "Say hello." })
```
Everything below explains how that stays simple while still supporting OpenAI, Anthropic, Gemini, Bedrock, OpenRouter, Azure, local OpenAI-compatible gateways, provider quirks, hosted tools, cache hints, and request replay.
Read from top to bottom. Stop when the next section is deeper than your task requires.
| Section | Use it when... |
| ------------------------------- | ------------------------------------------------------------------------------- |
| 1. The API You Use | You are writing application code or examples. |
| 2. What A Model Reference Means | You need to understand why provider, model, and protocol are separate. |
| 3. What Happens At Runtime | You are debugging what happens after `LLM.generate`. |
| 4. How Providers Are Built | You are wiring a new deployment or protocol. |
| 5. How Quirks Are Handled | You are preserving provider-specific behavior without polluting common schemas. |
| 6. Why This Design | You are relating this to AI SDK or OpenCode's current provider stack. |
## 1. The API You Use
Most code should live here.
```ts
import { Effect, Layer } from "effect"
import { LLM, RequestExecutor } from "@opencode-ai/llm"
import { OpenAI } from "@opencode-ai/llm/providers"
const model = OpenAI.model("gpt-4o-mini", {
apiKey: Bun.env.OPENAI_API_KEY,
})
const program = Effect.gen(function* () {
const response = yield* LLM.generate({
model,
prompt: "Say hello.",
})
console.log(response.text)
}).pipe(Effect.provide(Layer.mergeAll(LLM.layer({ providers: [OpenAI] }), RequestExecutor.defaultLayer)))
```
The public rule is:
```txt
provider helper -> model reference -> LLM.generate / LLM.stream
```
Provider helpers should feel boring at use sites.
```ts
OpenAI.model("gpt-4o-mini", { apiKey })
Anthropic.model("claude-3-5-sonnet-latest", { apiKey })
Google.model("gemini-2.0-flash", { apiKey })
OpenRouter.model("openai/gpt-4o-mini", { apiKey })
OpenAICompatible.model("gpt-4o-mini", {
name: "local-gateway",
baseURL: "http://localhost:11434/v1",
})
```
For OpenAI, `OpenAI.model(...)` means Responses. Use `OpenAI.chat(...)` only when you specifically need Chat Completions.
<details>
<summary>What this section hides</summary>
The call site does not name adapters, protocols, endpoints, auth, framing, patches, provider payloads, or stream parsers.
Those are runtime concerns. They should be inspectable and composable, but not required for normal use.
</details>
## 2. What A Model Reference Means
A model reference is a route card. It says which model to call, which provider owns the deployment, and which wire protocol can talk to it.
```ts
OpenAI.model("gpt-4o-mini", { apiKey })
-> provider: openai
-> protocol: openai-responses
-> id: gpt-4o-mini
OpenRouter.model("openai/gpt-4o-mini", { apiKey })
-> provider: openrouter
-> protocol: openai-compatible-chat
-> id: openai/gpt-4o-mini
OpenAICompatible.model("gpt-4o-mini", { name: "local-gateway", baseURL })
-> provider: local-gateway
-> protocol: openai-compatible-chat
-> id: gpt-4o-mini
```
This split is the core design choice.
| Concept | Question it answers |
| -------------------------------------------- | ------------------------------------------------------------ |
| `provider` | Who is the deployment or product surface? |
| `protocol` | Which request/response shape should the runtime use? |
| `id` | Which model/deployment id should be sent? |
| `baseURL` | Where should HTTP go? |
| `apiKey`, `headers`, `queryParams`, `native` | What deployment-specific transport data is needed? |
| `capabilities`, `limits` | What normalized features and constraints should callers see? |
Provider identity and wire protocol often differ. OpenRouter is not OpenAI, but many OpenRouter models speak enough OpenAI Chat shape to reuse the OpenAI Chat protocol.
<details>
<summary>Conceptual ModelRef shape</summary>
```ts
type ModelRef = {
id: ModelID
provider: ProviderID
protocol: ProtocolID
baseURL?: string
apiKey?: string
headers?: Record<string, string>
queryParams?: Record<string, string>
capabilities: ModelCapabilities
limits: ModelLimits
native?: Record<string, unknown>
}
```
`ModelRef` is not a provider client. It does not send requests. It is the stable, serializable description of what should be called.
</details>
## 3. What Happens At Runtime
At runtime, every request follows the same path.
```txt
LLM.generate({ model, prompt })
-> LLM.request(...)
-> LLMClient
-> adapter selected by model.protocol
-> provider-native payload
-> HttpClientRequest
-> RequestExecutor
-> provider response stream
-> LLMEvent stream
-> LLMResponse
```
The high-level API hides that pipeline.
```ts
const response =
yield *
LLM.generate({
model: OpenAI.model("gpt-4o-mini", { apiKey }),
prompt: "Say hello.",
})
```
The lower-level runtime sees this shape.
```ts
const request = LLM.request({
model,
prompt: "Say hello.",
})
const client = LLMClient.make({
adapters: [OpenAIResponses.adapter, OpenAIChat.adapter],
patches: ProviderPatch.defaults,
})
const response = yield * client.generate(request)
```
<details>
<summary>Adapter pipeline</summary>
The adapter is selected by `request.model.protocol`.
```ts
const adapter = adapters.get(request.model.protocol)
const draft = adapter.prepare(request)
const patched = applyTargetPatches(draft)
const target = adapter.validate(patched)
const http = adapter.toHttp(target)
const response = yield * RequestExecutor.execute(http)
const events = adapter.parse(response)
```
`generate` collects the same `LLMEvent` stream that `stream` exposes incrementally.
</details>
## 4. How Providers Are Built
Provider behavior is split across reusable layers instead of one large provider class.
```txt
Provider helper
creates ModelRef values
Provider module
exports adapters and helper constructors
Adapter
composes Protocol + Endpoint + Auth + Framing
Protocol
owns provider-native request and stream semantics
```
The composition rule is:
```txt
Adapter = Protocol + Endpoint + Auth + Framing
```
OpenAI Chat is a normal adapter composition.
```ts
export const adapter = Adapter.make({
id: "openai-chat",
protocol: OpenAIChat.protocol,
endpoint: Endpoint.baseURL({
default: "https://api.openai.com/v1",
path: "/chat/completions",
}),
auth: Auth.openAI,
framing: Framing.sse,
})
```
OpenAI-compatible Chat is the same protocol with different deployment axes.
```txt
OpenAI-compatible Chat adapter
= OpenAIChat.protocol
+ required baseURL endpoint
+ bearer auth
+ SSE framing
```
That is why these can share implementation without pretending they are the same provider.
```ts
OpenAI.chat("gpt-4o-mini", { apiKey })
OpenRouter.model("openai/gpt-4o-mini", { apiKey })
OpenAICompatible.model("gpt-4o-mini", { name: "local-gateway", baseURL })
```
<details>
<summary>Layer responsibilities</summary>
| Layer | Owns |
| --------------- | ----------------------------------------------------------------------------------------- |
| Provider helper | Public constructor, defaults, provider identity, model capabilities, limits. |
| Provider module | Exported adapters and helpers passed to `LLM.layer({ providers })`. |
| Adapter | Runtime registration and composition. |
| Protocol | Request lowering, payload schema, chunk schema, stream state machine. |
| Endpoint | URL construction, base URL, path, query params, deployment routing. |
| Auth | Bearer tokens, API-key headers, SigV4, future IAM/AAD signing. |
| Framing | Bytes to frames before protocol parsing, usually SSE. |
</details>
<details>
<summary>When to add what</summary>
| Need | Add |
| -------------------------------------------------------- | ----------------------------------------- |
| A new hosted product speaks an existing protocol | Provider helper plus adapter composition. |
| A provider has a unique request/response shape | New protocol plus adapter composition. |
| A provider has the same protocol but different auth | Reuse protocol, add auth axis. |
| A provider has the same protocol but different URL rules | Reuse protocol, add endpoint axis. |
| A provider streams non-SSE frames | Reuse or add protocol, add framing axis. |
| A model needs a one-off body tweak | Patch, not a common schema field. |
</details>
## 5. How Quirks Are Handled
Patches are named, traceable provider/model transformations inspired by OpenCode's existing `ProviderTransform` layer.
Use a patch when behavior is real but not universal enough to belong in the common request schema.
```txt
cache.prompt-hints
anthropic.scrub-tool-call-ids
target.openai-chat.include-usage
```
Each patch has an id, phase, predicate, and reason. Applied patches appear in `patchTrace`.
Patches are not a routing mechanism. Adapter selection happens from the original `request.model`; request patches may change payload details, but changing `model.provider`, `model.id`, or `model.protocol` is rejected. If a call needs a different provider, model, or protocol, construct a different model handle before building the request.
The rule is:
```txt
Common request shape stays small.
Provider quirks stay named and auditable.
Model routing stays explicit at the call site.
```
Good patch candidates include cache hint lowering, model-specific reasoning fields, OpenAI-compatible message cleanup, hosted-tool shape differences, metadata extraction, and provider option namespacing.
Bad patch candidates are behaviors that every provider supports the same way. Those belong in the common request model.
## 6. Why This Design
AI SDK has an excellent use-site shape.
```ts
openai("gpt-4o-mini")
openai.chat("gpt-4o-mini")
createOpenAICompatible({ baseURL })("gpt-4o-mini")
```
This package keeps the use-site shape familiar.
```ts
OpenAI.model("gpt-4o-mini", { apiKey })
OpenAI.chat("gpt-4o-mini", { apiKey })
OpenAICompatible.model("gpt-4o-mini", { name, baseURL, apiKey })
```
The difference is below the public API.
| Concern | AI SDK | This package |
| ----------------------- | --------------------------------------------------------- | ----------------------------------------------------------------- |
| Use site | Provider creates runnable model object. | Provider creates `ModelRef`; `LLM` runtime runs it. |
| Provider implementation | Usually provider-package-specific language model classes. | Protocol, endpoint, auth, framing, and patches are separate axes. |
| OpenAI-compatible reuse | Dedicated OpenAI-compatible implementation. | Reuses `OpenAIChat.protocol` with different deployment axes. |
| Debug/replay/parity | Mostly hidden behind provider implementation. | Exposed through request lowering, patches, adapters, and events. |
The tradeoff is intentional. The public API should feel small. The internals should be inspectable enough for OpenCode to preserve provider parity, replay HTTP, diff native payloads, and migrate provider-by-provider without cloning whole adapter classes.
packages/llm/PROPOSAL.openai-compatible-wrappers.md
-231
View File
@@ -1,231 +0,0 @@
# Proposal: OpenAI-Compatible Thin Wrappers
## Summary
Keep `OpenAICompatibleChat` as the shared implementation for providers that expose `/chat/completions`, but distinguish three levels of provider support:
| Level | Use When | Example |
| --- | --- | --- |
| Profile | Provider only needs `provider`, `baseURL`, and capabilities. | DeepSeek text/tool basics, TogetherAI, Cerebras, Fireworks. |
| Thin wrapper | Provider speaks OpenAI Chat shape but needs named options, patches, capability defaults, metadata extraction, or provider-defined tools. | Mistral, Groq, Perplexity. |
| Dedicated protocol | Request lowering or stream parsing stops being OpenAI Chat-compatible. | Not justified for these providers yet. |
The important rule: do not clone `OpenAIChat.protocol` for provider wrappers unless cassettes prove the wire format has diverged. A thin wrapper should reuse the shared protocol and adapter machinery, then add only provider policy.
## Current Shape
Today the generic adapter is already deep and reusable:
```ts
// src/provider/openai-compatible-chat.ts
export const adapter = Adapter.make({
id: "openai-compatible-chat",
protocol: OpenAIChat.protocol,
protocolId: "openai-compatible-chat",
endpoint: Endpoint.baseURL({
path: "/chat/completions",
required: "OpenAI-compatible Chat requires a baseURL",
}),
framing: Framing.sse,
})
```
Provider profiles are data:
```ts
// src/provider/openai-compatible-profile.ts
export const profiles = {
baseten: { provider: "baseten", baseURL: "https://inference.baseten.co/v1" },
cerebras: { provider: "cerebras", baseURL: "https://api.cerebras.ai/v1" },
deepinfra: { provider: "deepinfra", baseURL: "https://api.deepinfra.com/v1/openai" },
deepseek: { provider: "deepseek", baseURL: "https://api.deepseek.com/v1" },
fireworks: { provider: "fireworks", baseURL: "https://api.fireworks.ai/inference/v1" },
openrouter: { provider: "openrouter", baseURL: "https://openrouter.ai/api/v1" },
togetherai: { provider: "togetherai", baseURL: "https://api.together.xyz/v1" },
}
```
Current direct call site:
```ts
const model = OpenAICompatibleChat.deepseek({
id: "deepseek-chat",
apiKey: process.env.DEEPSEEK_API_KEY,
})
const llm = LLMClient.make({ adapters: [OpenAICompatibleChat.adapter] })
```
Current generic call site:
```ts
const model = OpenAICompatible.model("moonshot-v1-8k", {
provider: "moonshot",
baseURL: "https://api.moonshot.ai/v1",
apiKey: process.env.MOONSHOT_API_KEY,
})
const llm = LLMClient.make({ adapters: OpenAICompatible.adapters })
```
Current OpenCode bridge shape:
```ts
OpenAICompatible.model("deepseek-chat", {
provider: "deepseek",
baseURL: OpenAICompatibleProfiles.profiles.deepseek.baseURL,
apiKey,
})
// provider: "deepseek", protocol: "openai-compatible-chat"
```
Current default patches already contain provider-specific OpenAI-compatible policy:
```ts
ProviderPatch.scrubMistralToolIds
ProviderPatch.repairMistralToolResultUserSequence
ProviderPatch.addDeepSeekEmptyReasoning
ProviderPatch.moveOpenAICompatibleReasoningToNative
ProviderPatch.sanitizeMoonshotToolSchema
ProviderPatch.addOpenAICompatibleModalities
```
That is the right direction, but Mistral/Groq/Perplexity need a named home if they grow more than one or two patch entries.
## AI SDK Comparison
AI SDK has a generic `@ai-sdk/openai-compatible` provider, but it does not implement Mistral, Groq, Perplexity, or xAI chat by simply configuring that generic provider.
| Provider | AI SDK Shape | Why It Is Not Generic Only |
| --- | --- | --- |
| Mistral | Dedicated `MistralChatLanguageModel`. | `safe_prompt`, document limits, structured-output defaults, strict JSON schema, and special tool-choice mapping. |
| Groq | Dedicated `GroqChatLanguageModel`. | `reasoning_format`, `reasoning_effort`, `service_tier`, `parallel_tool_calls`, and provider-defined `browser_search`. |
| Perplexity | Dedicated `PerplexityLanguageModel`. | Citations, images, citation token usage, search query usage, provider option passthrough. |
| xAI | Dedicated `XaiChatLanguageModel`. | Search parameters, reasoning effort, xAI-specific tools/options; AI SDK only reuses OpenAI-compatible for xAI image generation. |
The lesson is not “copy AI SDK and create full dedicated adapters.” The lesson is that these providers have real named policy. In this package, named policy should start as thin wrappers over `OpenAICompatibleChat`.
## Proposed Shape
A thin wrapper is a provider-local module that reuses the common OpenAI-compatible adapter and protocol, then exports provider-specific model helpers, adapters, and patches.
Example Mistral wrapper:
```ts
// src/provider/mistral.ts
export const profile = {
provider: "mistral",
baseURL: "https://api.mistral.ai/v1",
capabilities: capabilities({ tools: { calls: true, streamingInput: true } }),
} satisfies OpenAICompatibleProfile
export const model = (input: ProviderFamilyModelInput) =>
OpenAICompatibleChat.profileModel(profile, input)
export const chat = model
export const patches = [
ProviderPatch.scrubMistralToolIds,
ProviderPatch.repairMistralToolResultUserSequence,
mistralToolChoicePatch,
mistralStructuredOutputPatch,
]
export const adapters = [
OpenAICompatibleChat.adapter.withPatches([mistralIncludeUsage]),
]
export * as Mistral from "./mistral"
```
The direct call site becomes named and discoverable:
```ts
const model = Mistral.chat({
id: "mistral-large-latest",
apiKey: process.env.MISTRAL_API_KEY,
})
const llm = LLMClient.make({
adapters: Mistral.adapters,
patches: ProviderPatch.defaults,
})
```
The existing generic call site still works for unwrapped providers:
```ts
const model = OpenAICompatible.model("some-model", {
provider: "some-provider",
baseURL: "https://api.some-provider.test/v1",
apiKey,
})
```
OpenCode bridge call sites become clearer:
```ts
Mistral.chat({
id: "mistral-large-latest",
apiKey,
})
// provider: "mistral", protocol: "openai-compatible-chat"
// baseURL defaults to "https://api.mistral.ai/v1"
```
## Provider Recommendations
| Provider | Today | Proposed Next Step | Reason |
| --- | --- | --- | --- |
| DeepSeek | Profile plus default reasoning patches. | Keep profile for now. | Current cassettes cover basic text; policy is still small and shared. |
| TogetherAI | Profile. | Keep profile. | No named provider policy yet beyond base URL. |
| Mistral | No profile helper yet, but default Mistral patches exist. | Add thin wrapper. | Policy already exists and AI SDK has enough Mistral-specific behavior to justify a named home. |
| Groq | No profile helper yet. | Start as profile or thin wrapper with only base URL; promote when reasoning/browser-search lands. | Basic OpenAI-compatible flow should work, but provider-defined tools and reasoning options need a wrapper. |
| Perplexity | No profile helper yet. | Add thin wrapper if citations/sources matter; otherwise start as profile for text only. | The value of Perplexity is source/search metadata, not just text. |
| xAI/Grok | Model helper currently points to `openai-responses`. | Keep separate from generic profiles. | xAI search/reasoning behavior is provider policy, and AI SDK treats chat as dedicated. |
## Why This Is Better Than Adding More Profiles Only
Profiles are excellent for base URL defaults. They become muddy when they need provider policy:
```ts
profiles.mistral = {
provider: "mistral",
baseURL: "https://api.mistral.ai/v1",
patches: [...], // not a profile anymore
options: {...}, // starts becoming a provider module
metadata: extract..., // definitely not profile data
}
```
Keeping profiles as data preserves their simplicity. Thin wrappers are where behavior belongs.
## Why This Is Better Than Dedicated Protocols Now
A dedicated protocol would duplicate the OpenAI Chat payload schema, message lowering, SSE framing, tool-call parsing, usage mapping, and finish mapping before we know those providers require it.
Thin wrappers keep one source of truth:
```ts
OpenAIChat.protocol
-> OpenAICompatibleChat.adapter
-> Mistral/Groq/Perplexity wrapper policy
```
If a recorded cassette later shows a provider emits incompatible stream chunks, that is the moment to split the protocol.
## Implementation Plan
1. Add `src/provider/mistral.ts` as the first thin wrapper because Mistral policy already exists in `ProviderPatch.defaults`.
2. Add Mistral to exports and model-helper bridge tests.
3. Add a recorded Mistral text cassette and tool cassette.
4. Only then decide whether Mistral needs payload patches for tool-choice or structured-output behavior.
5. Add Groq as a profile first, unless we immediately implement reasoning/browser-search options.
6. Add Perplexity as a thin wrapper when source/citation events or metadata are modeled.
## Open Questions
- Should provider wrapper modules export `adapters` or rely on callers using `OpenAICompatible.adapters`?
- Should wrapper-specific patches be included in `ProviderPatch.defaults`, or should wrappers export a `patches` list for explicit opt-in?
- Do Perplexity citations become common `source` events/content, provider-native metadata, or both?
- Should xAI continue routing to `openai-responses`, or should we add an xAI Chat wrapper when we add xAI cassettes?
packages/llm/PROPOSAL.patch-pipeline.md
-444
View File
@@ -1,444 +0,0 @@
# Proposal: Patch Pipeline
## Summary
Patch behaviour is currently split between the generic patch primitives in `src/patch.ts` and the request compilation flow in `src/adapter.ts`. This proposal introduces a patch pipeline module that owns the patch lifecycle in one place.
The pipeline is created once by `LLMClient.make(...)` with the client patch set. Each request then flows through that same pipeline instance. Adapter-local payload patches are still supplied per selected Adapter because they vary by route.
The goal is to make patch ordering, context refresh, route invariants, tool-schema handling, payload patching, stream patching, and trace assembly one deep module instead of implicit knowledge inside `LLMClient.compile(...)`.
## Current Shape
Patch definitions are small values:
```ts
// src/patch.ts
export interface Patch<A> {
readonly id: string
readonly phase: PatchPhase
readonly reason: string
readonly order?: number
readonly when: (context: PatchContext) => boolean
readonly apply: (value: A, context: PatchContext) => A
}
```
`Patch.plan(...)` handles one phase:
```ts
export function plan<A>(input: {
readonly phase: PatchPhase
readonly context: PatchContext
readonly patches: ReadonlyArray<Patch<A>>
}): PatchPlan<A> {
const patches = input.patches
.filter((patch) => patch.phase === input.phase && patch.when(input.context))
.toSorted((left, right) => (left.order ?? 0) - (right.order ?? 0) || left.id.localeCompare(right.id))
return {
phase: input.phase,
patches,
trace: patches.map((patch) => new PatchTrace({ id: patch.id, phase: patch.phase, reason: patch.reason })),
apply: (value) => patches.reduce((next, patch) => patch.apply(next, input.context), value),
}
}
```
The lifecycle is embedded in `LLMClient.compile(...)`:
```ts
const requestPlan = plan({ phase: "request", context: context({ request }), patches: registry.request })
const requestAfterRequestPatches = requestPlan.apply(request)
yield* ensureSameRoute(request.model, requestAfterRequestPatches.model)
const promptPlan = plan({ phase: "prompt", context: context({ request: requestAfterRequestPatches }), patches: registry.prompt })
const requestBeforeToolPatches = promptPlan.apply(requestAfterRequestPatches)
yield* ensureSameRoute(request.model, requestBeforeToolPatches.model)
const toolSchemaPlan = plan({ phase: "tool-schema", context: context({ request: requestBeforeToolPatches }), patches: registry.toolSchema })
const patchedRequest = requestBeforeToolPatches.tools.length === 0 || toolSchemaPlan.patches.length === 0
? requestBeforeToolPatches
: new LLMRequest({ ...requestBeforeToolPatches, tools: requestBeforeToolPatches.tools.map(toolSchemaPlan.apply) })
const candidate = yield* adapter.prepare(patchedRequest)
const payloadPlan = plan({ phase: "payload", context: context({ request: patchedRequest }), patches: [...adapter.patches, ...registry.payload] })
const payload = yield* adapter.validate(payloadPlan.apply(candidate))
const patchTrace = [...requestPlan.trace, ...promptPlan.trace, ...toolSchemaPlan.trace, ...payloadPlan.trace]
```
Stream patches are another single-phase plan later in `stream(...)`:
```ts
const streamPlan = plan({ phase: "stream", context: context({ request: compiled.request }), patches: registry.stream })
const events = compiled.adapter.parse(response, { request: compiled.request, patchTrace: compiled.patchTrace })
return streamPlan.patches.length === 0 ? events : events.pipe(Stream.map(streamPlan.apply))
```
## Current Patch Phase Usage
The runtime supports five phases today:
- `request`
- `prompt`
- `tool-schema`
- `payload`
- `stream`
Built-in default provider policy currently uses only `prompt` through `ProviderPatch.defaults`.
Built-in provider modules use `payload` for opt-in adapter-local patches such as `OpenAIChat.includeUsage` and `OpenAICompatibleChat.includeUsage`.
`request`, `tool-schema`, and `stream` are real runtime seams, but today they are used by tests and consumers rather than by default package policy.
That is still enough to justify one lifecycle module. The runtime already has all five seams; the problem is that their ordering and interactions are owned by `LLMClient` instead of by a patch pipeline.
## Problem
`Patch.plan(...)` is shallow. Its Interface is almost as complex as its Implementation: callers still choose the phase, build the context, remember ordering semantics, apply the plan, stitch traces, and decide when the context must be refreshed.
The deep behaviour is not in the patch module. It is spread across `LLMClient.compile(...)`:
- Adapter selection happens against the original request before request-shaped patches run.
- Request patches must run before prompt patches.
- Prompt patches must see the request after request patches.
- Request and prompt patches must not reroute `model.provider`, `model.id`, or `model.protocol`.
- Tool-schema patches apply to every tool definition, but only when tools exist and patches matched.
- Tool-schema trace appears once per matched patch, not once per tool.
- Payload patches run after Adapter lowering because they speak provider-native payload shape.
- Adapter-local payload patches and client registry payload patches are combined, then ordered by patch `order` and `id`.
- Adapter validation runs after payload patches, but validation logic remains owned by the Adapter.
- Trace order must match lifecycle order.
- Stream patches run after Adapter parsing, but use the compiled request as context.
This hurts locality. A bug in patch ordering or context refresh requires reading `src/patch.ts`, `src/adapter.ts`, provider patches, and tests. The rules are not discoverable from the patch Interface.
The deletion test shows the problem. Deleting `Patch.plan(...)` would not remove much complexity; callers could inline the filter/sort/reduce. Deleting the lifecycle code in `LLMClient.compile(...)` would make the complexity reappear anywhere requests need to be compiled correctly. That lifecycle is the module earning its keep, but it does not have its own seam.
## Proposed Shape
Introduce a patch pipeline module that closes over the client patch set once:
```ts
const pipeline = PatchPipeline.make(options.patches)
```
`PatchPipeline.make(...)` accepts the same patch inputs `LLMClient` accepts today:
```ts
PatchPipeline.make(options.patches)
PatchPipeline.make(ProviderPatch.defaults)
PatchPipeline.make(Patch.registry([...]))
```
The pipeline instance is immutable and reused for each request handled by that `LLMClient`.
```ts
export interface PatchPipeline {
readonly patchRequest: (request: LLMRequest) => Effect.Effect<PatchedRequest, LLMError>
readonly patchPayload: <Payload>(input: PatchPayloadInput<Payload>) => Effect.Effect<PatchedPayload<Payload>, LLMError>
readonly patchStreamEvents: (input: PatchStreamInput) => Stream.Stream<LLMEvent, LLMError>
}
```
The names should stay patch-focused. Avoid `prepareRequest` and `preparePayload` because `LLMClient.prepare`, `Adapter.prepare`, and Protocol lowering already use prepare terminology.
One possible state shape:
```ts
export interface PatchedRequest {
readonly original: LLMRequest
readonly request: LLMRequest
readonly trace: ReadonlyArray<PatchTrace>
}
export interface PatchPayloadInput<Payload> {
readonly state: PatchedRequest
readonly payload: Payload
readonly adapterPatches: ReadonlyArray<Patch<Payload>>
readonly validatePayload: (payload: Payload) => Effect.Effect<Payload, LLMError>
}
export interface PatchedPayload<Payload> {
readonly request: LLMRequest
readonly payload: Payload
readonly trace: ReadonlyArray<PatchTrace>
}
```
Then `LLMClient.compile(...)` becomes routing plus Adapter orchestration:
```ts
const pipeline = PatchPipeline.make(options.patches)
const compile = Effect.fn("LLM.compile")(function* (request: LLMRequest) {
const adapter = adapters.get(request.model.protocol) ?? modelAdapters.get(request.model)
if (!adapter) return yield* noAdapter(request.model)
const patchedRequest = yield* pipeline.patchRequest(request)
const candidate = yield* adapter.prepare(patchedRequest.request)
const patchedPayload = yield* pipeline.patchPayload({
state: patchedRequest,
payload: candidate,
adapterPatches: adapter.patches,
validatePayload: adapter.validate,
})
const http = yield* adapter.toHttp(patchedPayload.payload, {
request: patchedPayload.request,
patchTrace: patchedPayload.trace,
})
return {
request: patchedPayload.request,
adapter,
payload: patchedPayload.payload,
http,
patchTrace: patchedPayload.trace,
}
})
```
Stream patching also moves behind the same module, but only after Adapter parsing:
```ts
const events = compiled.adapter.parse(response, {
request: compiled.request,
patchTrace: compiled.patchTrace,
})
return pipeline.patchStreamEvents({
request: compiled.request,
events,
})
```
This is the important cleanup: `LLMClient` no longer hand-assembles phase plans, context refresh, route protection, payload patch ordering, validation timing, stream patch mapping, or patch trace concatenation.
## Performance And Simplicity
This design should be at least as performant as the current shape, and likely a little better, because patches generally live at client construction time rather than changing per request.
Today, every request rebuilds phase plans:
```ts
plan({ phase: "request", context, patches: registry.request })
plan({ phase: "prompt", context, patches: registry.prompt })
plan({ phase: "tool-schema", context, patches: registry.toolSchema })
plan({ phase: "payload", context, patches: [...adapter.patches, ...registry.payload] })
```
Each plan filters and sorts its phase patches. That cost is tiny compared with an LLM request, but it is still repeated work and repeated code.
The patch pipeline can precompile the client-level patch set once:
```ts
const pipeline = PatchPipeline.make(options.patches)
```
At construction time, the pipeline can:
- Normalize `undefined`, a patch array, or a `PatchRegistry` into one internal shape.
- Group patches by phase.
- Sort each client-level phase by `order` and `id` once.
- Store empty-phase fast paths so requests with no patches avoid allocation-heavy plan construction.
Per request, the pipeline still must evaluate `when(context)` predicates because predicates depend on the current request, model, protocol, metadata, tools, and provider. That part cannot be safely precompiled away unless a future patch type declares itself unconditional.
Payload patches are slightly different because adapter-local payload patches vary by selected Adapter. Keep the first version simple:
```ts
pipeline.patchPayload({
state,
payload,
adapterPatches: adapter.patches,
validatePayload: adapter.validate,
})
```
The pipeline can combine already-sorted client payload patches with adapter patches and apply the same ordering rule. If payload patch counts ever become large, the pipeline can cache the sorted merged payload patch list in a `WeakMap` keyed by the Adapter or by the adapter patch array. That is an internal Implementation optimization; the Interface does not need to expose it.
The important simplicity win is bigger than the micro-performance win. `LLMClient` would stop describing the patch algorithm in five places. The pipeline becomes a reusable compiled patch lifecycle: one small Interface, one place to optimize, one place to test.
## What The Module Owns
The patch pipeline module should own:
- Normalizing `PatchRegistry | ReadonlyArray<AnyPatch> | undefined` into a registry.
- Building fresh `PatchContext` after each request-shaped phase.
- Running request patches before prompt patches.
- Enforcing that request-shaped patches do not change `model.provider`, `model.id`, or `model.protocol`.
- Running tool-schema patches against every tool definition only when tools exist and patches matched.
- Emitting tool-schema trace once per matched patch, not once per tool.
- Combining request, prompt, tool-schema, and payload traces in lifecycle order.
- Combining adapter-local payload patches with client registry payload patches and applying the shared patch ordering rule.
- Invoking Adapter payload validation after payload patches.
- Applying stream patches to parsed `LLMEvent` streams with the compiled request context.
It should not own:
- Adapter lookup.
- Protocol lowering via `adapter.prepare(...)`.
- Payload validation Implementation.
- HTTP request construction.
- Provider-specific patch definitions.
- Provider stream parsing.
Those remain behind the Adapter, Protocol, Endpoint, Auth, Framing, ProviderPatch, and RequestExecutor modules.
## How This Cleans Up Code Elsewhere
`src/adapter.ts` gets smaller and more navigable:
- `normalizeRegistry(...)` moves out.
- `ensureSameRoute(...)` moves out.
- `compile(...)` stops constructing four separate plans.
- `compile(...)` stops manually refreshing contexts.
- `compile(...)` stops manually deciding when tool-schema traces count.
- `compile(...)` stops manually concatenating patch traces.
- `stream(...)` stops manually planning stream patches.
`src/patch.ts` becomes clearer:
- Patch constructors and predicates remain the primitive Interface.
- `plan(...)` can stay as an internal or low-level single-phase helper.
- Lifecycle semantics move to `src/patch-pipeline.ts` instead of being implied by Adapter tests.
Provider patch modules stay focused:
- `ProviderPatch.defaults` remains a list of provider facts.
- Provider-specific patches do not need to know lifecycle ordering.
- Adapter-local payload patches keep living on the selected Adapter.
Tests get better locality:
- Patch primitive tests stay in `patch.test.ts`.
- Patch lifecycle tests move to `patch-pipeline.test.ts`.
- Adapter tests keep only Adapter responsibilities and one end-to-end smoke test that `LLMClient` invokes the pipeline.
## Why This Is Deepening
The patch pipeline would be a deeper module because a small Interface hides a larger amount of behaviour.
Current Interface:
```ts
plan({ phase, context, patches }).apply(value)
```
That Interface is shallow because the caller must know the lifecycle.
Proposed Interface:
```ts
const pipeline = PatchPipeline.make(options.patches)
const request = yield* pipeline.patchRequest(input)
const payload = yield* pipeline.patchPayload({ state: request, payload, adapterPatches, validatePayload })
const events = pipeline.patchStreamEvents({ request: payload.request, events })
```
That Interface is deeper because callers get ordering, context refresh, route protection, tool-schema handling, payload patch composition, validation timing, stream mapping, and trace assembly without knowing each step.
## Principles
### Module
Today, the real patch lifecycle is an unnamed module embedded in `LLMClient.compile(...)`. Naming it as a patch pipeline module gives it one Interface and one Implementation.
### Interface
The Interface becomes the test surface. Tests should ask what the pipeline guarantees: request patches run before prompt patches, contexts refresh, route changes fail, payload patches trace after tool-schema patches, validation runs after payload patches, and stream patches see the compiled request.
### Depth
The module becomes deep because callers learn a small lifecycle Interface instead of the full phase choreography. More behaviour sits behind less required knowledge.
### Seam
The seam moves from scattered calls to `plan(...)` into the patch pipeline Interface. The existing patch Interface remains the seam where provider-specific patch behaviour enters the lifecycle.
### Adapter
Provider-specific patches are Adapters at the patch seam: each concrete patch satisfies the patch Interface. Adapter-local payload patches remain local to the selected Adapter, but the pipeline owns how those patches combine with client registry payload patches.
### Leverage
Callers get more leverage because `LLMClient`, tests, and future request-compilation paths can reuse one lifecycle. A fix to context refresh or route protection pays back everywhere.
### Locality
Maintainers get more locality because patch bugs concentrate in the patch pipeline Implementation. Provider patches can stay focused on provider facts instead of lifecycle rules.
### Deletion Test
Deleting the current `plan(...)` helper removes only a small filter/sort/reduce. Deleting the proposed patch pipeline would make lifecycle complexity reappear in `LLMClient`, tests, and any future compilation path. That means the proposed module earns its keep.
### One Adapter = Hypothetical Seam, Two Adapters = Real Seam
This proposal does not add a speculative seam with fake alternative implementations. It deepens an existing real seam: many provider patches already satisfy the patch Interface, and adapter-local plus client registry payload patches already vary across providers and call sites. The missing piece is locality for the lifecycle that applies those Adapters.
## Benefits
Locality improves because lifecycle rules live in one module instead of being embedded in request compilation.
Leverage improves because every provider patch and every client path gets the same ordering, trace, validation timing, and route-invariant behaviour.
Tests improve because the patch pipeline Interface becomes the test surface. Instead of constructing fake protocols, fake adapters, fake framing, and scripted HTTP flows to verify patch lifecycle behaviour, tests can exercise the lifecycle directly.
Useful tests:
- Adapter selection happens before request patches.
- Request patches run before prompt patches.
- Prompt patch predicates see the request after request patches.
- Request-shaped patches cannot change `model.provider`, `model.id`, or `model.protocol`.
- Tool-schema patches are skipped when there are no tools.
- Tool-schema traces appear only when tool-schema patches ran.
- Tool-schema trace appears once per matched patch, not once per tool.
- Adapter payload patches and client registry payload patches follow the shared patch ordering rule.
- Payload validation runs after payload patches.
- Stream patches see the compiled request, not the original request.
- Pipeline construction accepts `undefined`, a patch array, or a `PatchRegistry`.
## What Not To Do Yet
Do not change the public patch definition shape unless the pipeline proves it needs a missing field.
Do not create a full plugin system for patch ordering.
Do not move provider-specific patch logic into the pipeline.
Do not make payload patch typing more ambitious in this step; payload patches are already typed at adapter construction sites and erased in the registry.
Do not move Adapter lookup, Protocol lowering, HTTP construction, or stream parsing into the pipeline.
Do not change provider behaviour while extracting the lifecycle.
## Migration Plan
1. Add `src/patch-pipeline.ts` with the lifecycle Implementation and focused tests.
2. Keep `Patch.plan(...)` public during migration and use it internally inside the pipeline.
3. Move `normalizeRegistry(...)` and `ensureSameRoute(...)` from `src/adapter.ts` into the pipeline module.
4. Add `patchRequest(...)` that runs request, prompt, and tool-schema phases and returns a carried request state.
5. Add `patchPayload(...)` that applies adapter-local payload patches, client registry payload patches, Adapter validation, and returns a carried payload state with combined trace.
6. Add `patchStreamEvents(...)` that applies stream patches to parsed `LLMEvent` streams.
7. Add `test/patch-pipeline.test.ts` with lifecycle tests before changing `LLMClient`.
8. Replace handwritten phase choreography in `LLMClient.compile(...)` and `LLMClient.stream(...)` with the pipeline.
9. Keep one adapter-level smoke test proving `LLMClient` invokes patches end-to-end.
10. Move or delete adapter-level lifecycle tests that are now covered by patch pipeline tests.
11. Decide later whether `Patch.plan(...)` remains public or becomes internal.
## Open Questions
Should `Patch.plan(...)` remain public as a low-level primitive, or should the patch pipeline become the only exported lifecycle Interface?
Should stream patches be part of the same pipeline module from the first extraction, or should the first extraction focus only on request-to-payload compilation?
Should the pipeline return one combined trace array, or should it preserve phase-grouped traces internally for better debugging while exposing one ordered trace to callers?
Should route protection apply only after request and prompt phases, or should the pipeline also assert that payload and stream phases cannot observe changed route state?
Should payload patch ordering keep the current global `order`/`id` rule across adapter-local and client registry patches, or should adapter-local payload patches get an explicit ordering band before client registry payload patches?
## Recommendation
Do this before adding more provider-specific patches. The current shape is already correct enough to extract safely, and the next set of provider quirks will make patch ordering and conversation-shape rules more important. A patch pipeline module would turn implicit lifecycle knowledge into a deep Interface with better locality, better leverage, and a clearer test surface.
packages/llm/TODO.provider-transform-parity.md
+19 -19
View File
@@ -2,22 +2,22 @@
This tracks OpenCode behavior from `packages/opencode/src/provider/transform.ts` that is not fully represented in `packages/llm` yet.
Patches are the right seam when the behavior is a provider/model quirk that mutates request history, tool schemas, target bodies, or stream events. Do not add fields to the common request model just to carry one provider's native option.
Transforms are the right seam when the behavior is a provider/model quirk that mutates request history, tool schemas, adapter-owned payload bodies, or stream events. Do not add fields to the common request model just to carry one provider's native option.
## Ported Or Covered
- Empty Anthropic/Bedrock content cleanup: `ProviderPatch.removeEmptyAnthropicContent`.
- Claude tool id scrub: `ProviderPatch.scrubClaudeToolIds`.
- Mistral/Devstral tool id scrub: `ProviderPatch.scrubMistralToolIds`.
- Anthropic assistant `tool_use` ordering repair: `ProviderPatch.repairAnthropicToolUseOrder`.
- Mistral `tool -> user` sequence repair: `ProviderPatch.repairMistralToolResultUserSequence`.
- DeepSeek empty reasoning replay: `ProviderPatch.addDeepSeekEmptyReasoning` plus OpenAI-compatible native `reasoning_content` lowering.
- OpenAI-compatible reasoning history replay: `ProviderPatch.moveOpenAICompatibleReasoningToNative`.
- Unsupported user media fallback: `ProviderPatch.unsupportedMediaFallback`.
- Moonshot/Kimi schema sanitizer: `ProviderPatch.sanitizeMoonshotToolSchema`.
- Prompt cache hint placement: `ProviderPatch.cachePromptHints`.
- Empty Anthropic/Bedrock content cleanup: `ProviderTransform.removeEmptyAnthropicContent`.
- Claude tool id scrub: `ProviderTransform.scrubClaudeToolIds`.
- Mistral/Devstral tool id scrub: `ProviderTransform.scrubMistralToolIds`.
- Anthropic assistant `tool_use` ordering repair: `ProviderTransform.repairAnthropicToolUseOrder`.
- Mistral `tool -> user` sequence repair: `ProviderTransform.repairMistralToolResultUserSequence`.
- DeepSeek empty reasoning replay: `ProviderTransform.addDeepSeekEmptyReasoning` plus OpenAI-compatible native `reasoning_content` lowering.
- OpenAI-compatible reasoning history replay: `ProviderTransform.moveOpenAICompatibleReasoningToNative`.
- Unsupported user media fallback: `ProviderTransform.unsupportedMediaFallback`.
- Moonshot/Kimi schema sanitizer: `ProviderTransform.sanitizeMoonshotToolSchema`.
- Prompt cache hint placement: `ProviderTransform.cachePromptHints`.
- Gemini schema sanitizer/projector: handled inside `Gemini.protocol` because Gemini has a distinct schema dialect.
- OpenAI Chat/OpenAI-compatible streaming usage: adapter-local payload patches.
- OpenAI Chat/OpenAI-compatible streaming usage: adapter-local payload transforms.
## Not Fully Ported
@@ -36,7 +36,7 @@ Native status:
Likely shape:
- Payload patches for provider-native body knobs when the adapter payload has a real field.
- Adapter-local payload transforms for provider-native body knobs when the adapter payload has a real field.
- Bridge-level lowering for opaque OpenCode provider options until each option has a typed native destination.
### `options(...)` Defaults
@@ -61,7 +61,7 @@ Native status:
Likely shape:
- Adapter-local payload patches where the payload schema can express the option.
- Adapter-local payload transforms where the payload schema can express the option.
- New payload fields only when the provider actually accepts them.
- Avoid a generic `providerOptions` escape hatch unless the bridge still needs temporary fallback behavior.
@@ -81,7 +81,7 @@ Native status:
Likely shape:
- Keep the common intent small.
- Add provider/model payload patches that translate `request.reasoning` into each adapter payload's native fields.
- Add adapter-local payload transforms that translate `request.reasoning` into each adapter payload's native fields.
- Add tests per provider family because invalid reasoning fields are common provider rejection causes.
### Sampling Defaults
@@ -100,7 +100,7 @@ Native status:
Likely shape:
- Request or payload patches that fill unset generation fields for specific models.
- Runtime request transforms or adapter-local payload transforms that fill unset generation fields for specific models.
- Add `topK` only when enough adapters support it or when a specific adapter target needs it.
### Small Model Options
@@ -118,7 +118,7 @@ Native status:
Likely shape:
- First define how OpenCode marks a request as small in `LLMRequest` or bridge metadata.
- Then use payload patches keyed on that marker and provider/model.
- Then use adapter-local payload transforms keyed on that marker and provider/model.
### Interleaved Reasoning Field Variants
@@ -135,12 +135,12 @@ Native status:
Likely shape:
- Store the chosen field in model profile/native metadata.
- A prompt patch moves common reasoning parts into that provider-native field.
- A prompt transform moves common reasoning parts into that provider-native field.
- The OpenAI-compatible payload schema/lowerer emits the selected field.
## Suggested Order
1. Add payload patches for high-confidence OpenAI/OpenAI-compatible defaults that already have payload fields.
1. Add adapter-local payload transforms for high-confidence OpenAI/OpenAI-compatible defaults that already have payload fields.
2. Add provider-family reasoning mapping tests before porting more variants.
3. Define the bridge marker for “small” requests before implementing `smallOptions` parity.
4. Keep provider option namespacing in the bridge until individual native destinations are known.
packages/llm/TOUR.md
+76 -62
View File
@@ -34,7 +34,7 @@ The public `LLM` namespace lives in [`src/llm.ts`](./src/llm.ts).
Read these pieces first:
- `LLM.make` builds a runtime from providers, adapters, and patches.
- `LLM.make` builds a runtime from providers, adapters, and transforms.
- `LLM.layer` provides that runtime as an Effect service.
- `LLM.generate` and `LLM.stream` are thin service calls.
- `LLM.request` turns ergonomic input into canonical schema classes.
@@ -48,11 +48,11 @@ The key design choice is that the public request model is provider-neutral. Prov
Before following one request through the runtime, name the main concepts:
- `LLMRequest`: the canonical provider-neutral request. This is what callers build and what patches/protocols read.
- `LLMRequest`: the canonical provider-neutral request. This is what callers build and what transforms/protocols read.
- `ModelRef`: the selected model plus routing metadata. `model.adapter` chooses the runnable adapter route; `model.protocol` records the wire protocol semantics.
- `Protocol`: the wire-format brain. It converts `LLMRequest` into a provider-native payload and parses provider-native stream chunks back into `LLMEvent`s.
- `Adapter`: the runnable deployment. It combines one `Protocol` with an `Endpoint`, `Auth`, `Framing`, headers, and adapter-local payload patches.
- `PatchPipeline`: the tweak layer. It can rewrite the canonical request before lowering, rewrite tool schemas, rewrite the provider payload after lowering, or rewrite normalized stream events.
- `Adapter`: the runnable deployment. It combines one `Protocol` with an `Endpoint`, `Auth`, `Framing`, headers, and adapter-local payload transforms.
- `TransformPipeline`: the rewrite layer. Runtime transforms touch only common IR; adapter-local transforms touch native payloads.
- `RequestExecutor`: the transport boundary. It sends an `HttpClientRequest` and returns an `HttpClientResponse`.
- `LLMEvent`: the normalized stream output. Every provider eventually emits the same event vocabulary.
@@ -74,18 +74,18 @@ The runtime pipeline is concentrated in [`src/adapter.ts`](./src/adapter.ts).
The important functions are:
- `Adapter.model`, which binds a user-facing model helper to the adapter that can run it.
- `LLMClient.make`, which selects an adapter, applies patches, builds the payload, sends HTTP, and parses the response.
- `LLMClient.make`, which selects an adapter, applies transforms, builds the payload, sends HTTP, and parses the response.
- `Adapter.make`, which composes protocol semantics with endpoint, auth, and framing.
At runtime, the flow is easier to read as a sequence of value transformations:
At runtime, the flow is easier to read as a sequence of value transformations. There are two levels to keep separate:
- The main request path: caller input becomes a provider HTTP request, then normalized events.
- The parser zoom-in: `adapter.parse(...)` hides response framing, chunk decoding, and stream state.
The snippet below is pseudo-code. It shows resolved values at each boundary, not the `Effect` wrappers used by the implementation.
```ts
type Payload = OpenAIChatPayload
type Frame = string
type Chunk = OpenAIChatChunk
type State = OpenAIChatStreamState
// -----------------------------------------------------------------------------
// Stage 1: Caller Forms A Canonical Request
@@ -106,16 +106,19 @@ const request: LLMRequest = LLM.request(input)
// Stage 2: Caller Hands The Request To The Client
// -----------------------------------------------------------------------------
// The caller hands that request to the client. Normal callers use streaming or
// collected responses; lower-level tests can inspect the compiled request.
// The caller hands that request to the client and chooses one exit path:
// inspect the compiled request, stream events, or collect a final response.
const client: LLMClient = LLMClient.make({ adapters: [OpenAIChat.adapter] })
// Alternative A: compile without sending HTTP. Useful for request-shape tests.
// LLMRequest -> PreparedRequestOf<Payload>
const prepared: PreparedRequestOf<Payload> = client.prepare<Payload>(request)
// Alternative B: send HTTP and expose normalized stream events.
// LLMRequest -> Stream<LLMEvent>
const streamed: Stream.Stream<LLMEvent, LLMError> = client.stream(request)
// Alternative C: send HTTP and collect those same events into one response.
// LLMRequest -> LLMResponse
const generated: LLMResponse = client.generate(request)
@@ -123,10 +126,10 @@ const generated: LLMResponse = client.generate(request)
// Stage 3: Client Compiles The Request
// -----------------------------------------------------------------------------
// Internally, all three client methods start by compiling the request.
// PatchPipeline is the named tweak layer: it applies route-specific request,
// prompt, tool-schema, payload, and stream rewrites.
const patchPipeline: PatchPipeline = PatchPipeline.make(ProviderPatch.defaults)
// Internally, all three alternatives start by compiling the request.
// TransformPipeline is the named rewrite layer. Runtime transforms only touch
// canonical/common IR: request, prompt, tool-schema, and stream events.
const transformPipeline: TransformPipeline = TransformPipeline.make(ProviderTransform.defaults)
// The client selects the runnable adapter from the explicit registry keyed by
// `request.model.adapter`. The model-bound adapter is a fallback for models
@@ -135,15 +138,15 @@ const adapter: Adapter<Payload> = resolveAdapter(request.model)
// This first pipeline call only handles pre-lowering rewrites: whole-request
// policy, prompt/message cleanup, and tool schema cleanup.
// LLMRequest -> PatchedRequest
const patchedRequest: PatchedRequest = patchPipeline.patchRequest(request)
// LLMRequest -> TransformedRequest
const transformedRequest: TransformedRequest = transformPipeline.transformRequest(request)
// Adapter.toPayload is the protocol conversion boundary.
// PatchedRequest.request -> provider-native Payload
// TransformedRequest.request -> provider-native Payload
// It builds the JSON body shape for this API family, but does not choose a URL,
// add auth, encode JSON, or send HTTP.
// OpenAI Chat example output:
const draftPayload: Payload = adapter.toPayload(patchedRequest.request)
const draftPayload: Payload = adapter.toPayload(transformedRequest.request)
// {
// model: "gpt-4o-mini",
// messages: [
@@ -153,16 +156,14 @@ const draftPayload: Payload = adapter.toPayload(patchedRequest.request)
// stream: true,
// }
// This second pipeline call handles post-lowering payload rewrites. The same
// step validates the final provider-native JSON shape with `adapter.payloadSchema`.
// `PatchedPayload<Payload>` is not a different wire shape; it is the pipeline
// result envelope: { request, payload }. The inner `payload` is still the
// provider-native `Payload`.
// PatchedRequest + Payload -> PatchedPayload<Payload>
const payloadStep: PatchedPayload<Payload> = patchPipeline.patchPayload({
state: patchedRequest,
// Adapter-local payload transforms run after protocol lowering. They are the
// only transforms allowed to touch provider-native payloads, because the adapter
// owns the `Payload` type. The same step validates the final payload schema.
// TransformedRequest + Payload -> TransformedPayload<Payload>
const payloadStep: TransformedPayload<Payload> = transformPipeline.transformPayload({
state: transformedRequest,
payload: draftPayload,
adapterPatches: adapter.patches,
adapterTransforms: adapter.transforms,
schema: adapter.payloadSchema,
})
@@ -192,25 +193,34 @@ const events: Stream.Stream<LLMEvent, LLMError> = adapter.parse(httpResponse, {
request: payloadStep.request,
})
// Internally, Adapter.make builds `parse` from Framing + Protocol chunk decoding
// + Protocol.process. Those pieces have their own concrete types:
// ◆ Zoom in: what Adapter.parse hides ◆
// Adapter.make builds `parse` from Framing + protocol chunk decoding +
// Protocol.process. Those pieces have their own concrete types:
type Frame = string // One transport-framed item, before provider Schema decoding.
type Chunk = OpenAIChatChunk // One provider-native stream object, after Schema decoding.
type State = OpenAIChatStreamState // Parser memory needed across streamed chunks.
const protocol: Protocol<Payload, Frame, Chunk, State> = OpenAIChat.protocol
const framing: Framing<Frame> = Framing.sse
// Framing converts response bytes into protocol frames.
// SSE providers produce JSON strings. Bedrock produces AWS event-stream objects.
// Framing is the transport-to-protocol boundary. It splits raw response bytes
// into frames: the smallest complete response units the transport can deliver.
// For SSE, one frame is usually one `data:` string. For Bedrock, one frame is
// one AWS event-stream message object. A frame is not trusted provider data yet.
// Stream<Uint8Array> -> Stream<Frame>
const frames: Stream.Stream<Frame, ProviderChunkError> = framing.frame(httpResponse.stream)
// The chunk Schema decodes each frame into provider-native chunk objects.
// The chunk Schema turns one frame into one typed provider chunk. This is where
// transport output becomes provider-native data: OpenAIChatChunk,
// AnthropicMessagesChunk, GeminiChunk, and so on.
// Frame -> Chunk
const decodeChunk: (frame: Frame) => Effect.Effect<Chunk, ProviderChunkError> = (frame) =>
Schema.decodeUnknownEffect(protocol.chunk)(frame).pipe(Effect.mapError(() => chunkError(adapter.id, frame)))
const chunks: Stream.Stream<Chunk, ProviderChunkError> = frames.pipe(Stream.mapEffect(decodeChunk))
// Protocol.process is the stream parser state machine.
// It converts provider-native chunks into common LLMEvents.
// Protocol.process is the stream parser state machine. `State` carries whatever
// memory this API needs between chunks, such as partial text or tool arguments.
// State + Chunk -> State + ReadonlyArray<LLMEvent>
const initialState: State = protocol.initial()
const eventBatches: Stream.Stream<ReadonlyArray<LLMEvent>, ProviderChunkError> = chunks.pipe(
@@ -221,6 +231,10 @@ const eventBatches: Stream.Stream<ReadonlyArray<LLMEvent>, ProviderChunkError> =
// Stream<ReadonlyArray<LLMEvent>> -> Stream<LLMEvent>
const eventsFromInternals: Stream.Stream<LLMEvent, LLMError> = eventBatches.pipe(Stream.flatMap(Stream.fromIterable))
// ◇ Zoom out: back to the client lifecycle ◇
// From here on, the client no longer cares about frames, chunks, or parser
// state. It only has the normalized event stream returned by `adapter.parse(...)`.
// -----------------------------------------------------------------------------
// Stage 6: Client Exposes Or Collects Events
// -----------------------------------------------------------------------------
@@ -236,9 +250,9 @@ The important translation points are:
- `LLM.request(input)` turns ergonomic caller input into canonical `LLMRequest`.
- `client.prepare(request)`, `client.stream(request)`, and `client.generate(request)` hand the canonical request to the lower-level runtime.
- `patchPipeline.patchRequest(request)` applies request, prompt, and tool-schema patches.
- `adapter.toPayload(patchedRequest.request)` turns canonical `LLMRequest` into provider-native payload.
- `patchPipeline.patchPayload(...)` applies payload patches and validates with `adapter.payloadSchema`.
- `transformPipeline.transformRequest(request)` applies request, prompt, and tool-schema transforms.
- `adapter.toPayload(transformedRequest.request)` turns canonical `LLMRequest` into provider-native payload.
- `transformPipeline.transformPayload(...)` applies adapter-local payload transforms and validates with `adapter.payloadSchema`.
- `adapter.toHttp(payload, context)` turns provider-native payload into `HttpClientRequest`.
- `Framing` turns response bytes into protocol frames.
- `protocol.chunk` turns frames into provider-native chunks.
@@ -281,7 +295,7 @@ interface Protocol<Payload, Frame, Chunk, State> {
Read those generics as the parser pipeline:
- `Payload`: the provider-native JSON body after request conversion and payload patches.
- `Payload`: the provider-native JSON body after request conversion and adapter-local payload transforms.
- `Frame`: one response unit after byte framing, such as an SSE `data:` string or a Bedrock event-stream object.
- `Chunk`: the provider-native stream chunk after Schema decoding one frame.
- `State`: the accumulator needed to turn a sequence of chunks into common events.
@@ -330,7 +344,7 @@ interface Adapter<Payload> {
readonly id: string
readonly protocol: ProtocolID
readonly payloadSchema: Schema.Codec<Payload, unknown>
readonly patches: ReadonlyArray<Patch<Payload>>
readonly transforms: ReadonlyArray<Transform<Payload, "payload">>
readonly toPayload: (request: LLMRequest) => Effect.Effect<Payload, LLMError>
readonly toHttp: (
payload: Payload,
@@ -426,49 +440,49 @@ Examples:
Provider helpers should usually not contain stream parsing, JSON decoding, or protocol details. They set provider identity, defaults, capabilities, deployment options, and adapter registrations.
## 8. Patches Keep Provider Quirks Out Of Common Schemas
## 8. Transforms Keep Provider Quirks Out Of Common Schemas
The patch system keeps one-off provider/model quirks from leaking into `LLMRequest`.
The transform system keeps one-off provider/model quirks from leaking into `LLMRequest`.
This is not a substitute for putting the right behavior in a protocol. If Anthropic Messages always lowers a common feature the same way, that belongs in `anthropic-messages.ts`. A patch is for behavior that is conditional on provider, model, deployment, or caller policy: the same protocol shape is mostly right, but one route needs a small, inspectable rewrite.
This is not a substitute for putting the right behavior in a protocol. If Anthropic Messages always lowers a common feature the same way, that belongs in `anthropic-messages.ts`. A transform is for behavior that is conditional on provider, model, deployment, or caller policy: the same protocol shape is mostly right, but one route needs a small, inspectable rewrite.
That is why the pipeline exists. OpenCode already had a provider-transform layer because real providers reject or require little differences that are not worth baking into the common request model. The package keeps that idea, but makes each tweak named, phase-scoped, typed, ordered, and predicate-gated.
Start here:
- Patch types and constructors: [`src/patch.ts`](./src/patch.ts)
- Patch execution pipeline: [`src/patch-pipeline.ts`](./src/patch-pipeline.ts)
- Default provider patch registry: [`src/provider-patch.ts`](./src/provider-patch.ts)
- Provider-local patch example, OpenAI Chat include usage: [`src/protocols/openai-chat.ts`](./src/protocols/openai-chat.ts)
- Provider-specific wrapper patch, OpenRouter options: [`src/providers/openrouter.ts`](./src/providers/openrouter.ts)
- Transform types and constructors: [`src/transform.ts`](./src/transform.ts)
- Transform execution pipeline: [`src/transform-pipeline.ts`](./src/transform-pipeline.ts)
- Default provider transform registry: [`src/provider-transform.ts`](./src/provider-transform.ts)
- Adapter-local transform example, OpenAI Chat include usage: [`src/protocols/openai-chat.ts`](./src/protocols/openai-chat.ts)
- Provider-specific wrapper transform, OpenRouter options: [`src/providers/openrouter.ts`](./src/providers/openrouter.ts)
The pipeline has five phases:
```ts
type PatchPhase = "request" | "prompt" | "tool-schema" | "payload" | "stream"
type TransformPhase = "request" | "prompt" | "tool-schema" | "payload" | "stream"
```
The phases used today are:
- `prompt`: rewrite message history before protocol lowering.
- `tool-schema`: rewrite tool JSON Schema before protocol lowering.
- `payload`: rewrite the provider-native payload after lowering and before HTTP encoding.
- `payload`: adapter-local only; rewrite the provider-native payload after lowering and before HTTP encoding.
The phases available but not heavily used today are:
- `request`: reserved for whole-request policy before prompt/tool-schema patches.
- `request`: reserved for whole-request policy before prompt/tool-schema transforms.
- `stream`: reserved for normalized event rewrites after provider parsing.
There are two patch sources because they solve different problems:
There are two transform sources because they solve different problems:
- Adapter-local patches belong to one adapter's wire format. They are payload-only today, because the adapter owns `Payload`. Use them for things like `includeUsage` or OpenRouter payload options.
- Runtime/default patches are cross-adapter policy. They can run before lowering, so they can clean the canonical request, prompt history, or tool schemas before any protocol turns them into provider-native JSON.
- Adapter-local transforms belong to one adapter's wire format. They are payload-only today, because the adapter owns `Payload`. Use them for things like `includeUsage` or OpenRouter payload options.
- Runtime/default transforms are cross-adapter policy. They never touch provider-native payloads; they only clean the canonical request, prompt history, tool schemas, or normalized events.
If every tweak lived on adapters, cross-cutting behavior would either be duplicated across many adapters or hidden inside protocols where callers cannot turn it off. If every tweak were global, adapter-owned wire details would become too detached from the adapter that understands the payload. The split keeps protocol semantics stable, adapter quirks close to adapters, and runtime policy configurable at `LLM.make(...)` / `LLMClient.make(...)`.
If every tweak lived on adapters, cross-cutting behavior would either be duplicated across many adapters or hidden inside protocols where callers cannot turn it off. If payload tweaks were global, runtime code could mutate native payloads it does not own. The split keeps protocol semantics stable, adapter payload quirks close to adapters, and runtime policy configurable at `LLM.make(...)` / `LLMClient.make(...)`.
Default patches are enabled by `LLM.make(...)` through `ProviderPatch.defaults`. Direct `LLMClient.make(...)` callers opt in by passing `patches`, or by using adapters that include adapter-local payload patches.
Default transforms are enabled by `LLM.make(...)` through `ProviderTransform.defaults`. Direct `LLMClient.make(...)` callers opt in by passing `transforms`, or by using adapters that include adapter-local payload transforms.
Today the default provider patches do concrete work:
Today the default provider transforms do concrete work:
- Anthropic and Bedrock: remove empty text/reasoning content that those APIs reject.
- Claude: scrub tool call IDs to Claude's accepted character set.
@@ -479,14 +493,14 @@ Today the default provider patches do concrete work:
- Moonshot/Kimi: sanitize tool JSON Schema shapes the provider rejects.
- Prompt caching: mark cache-capable providers' first system parts and last message text blocks with ephemeral cache hints.
Adapter-local payload patches are used where the quirk is specific to one adapter deployment:
Adapter-local payload transforms are used where the quirk is specific to one adapter deployment:
- OpenAI Chat and OpenAI-compatible Chat: `includeUsage` adds `stream_options.include_usage` so streaming responses include the final usage chunk.
- OpenRouter: `applyOptions` lifts `usage`, `reasoning`, and `prompt_cache_key` model options into the OpenRouter Chat payload.
The important idea is that payload patches operate after protocol lowering but before payload validation and HTTP encoding. That gives providers a typed place to add `stream_options`, OpenRouter routing options, or other native fields without expanding the common request model for every provider.
The important idea is that payload transforms operate after protocol lowering but before payload validation and HTTP encoding. They are adapter-local only, which gives providers a typed place to add `stream_options`, OpenRouter routing options, or other native fields without giving runtime/global policy access to private payload shapes.
The tests to read are [`test/patch.test.ts`](./test/patch.test.ts), [`test/patch-pipeline.test.ts`](./test/patch-pipeline.test.ts), and [`test/adapter.test.ts`](./test/adapter.test.ts).
The tests to read are [`test/transform.test.ts`](./test/transform.test.ts), [`test/transform-pipeline.test.ts`](./test/transform-pipeline.test.ts), and [`test/adapter.test.ts`](./test/adapter.test.ts).
## 9. Tools Are Typed End To End
@@ -598,7 +612,7 @@ The package gets several useful properties from this shape:
- Simple use site from `LLM.generate`, provider model helpers, and `LLM.request` constructors.
- Provider code reuse from separating `Protocol`, `Endpoint`, `Auth`, and `Framing`.
- Native wire visibility because payload and chunk schemas stay close to lowering/parsing code.
- Safe provider quirks because patches transform provider payloads after lowering but before validation.
- Safe provider quirks because adapter-local transforms rewrite provider payloads after lowering but before validation.
- Common UI/runtime events because every provider parser emits `LLMEvent`s.
- Tool-loop portability because `ToolRuntime` consumes common tool events instead of provider-specific streams.
- Fast parser tests from `fixedResponse`, `dynamicResponse`, and `scriptedResponses`.
@@ -619,7 +633,7 @@ For a provider-composition demo:
1. Open [`src/protocols/openai-chat.ts`](./src/protocols/openai-chat.ts).
2. Open [`src/protocols/openai-compatible-chat.ts`](./src/protocols/openai-compatible-chat.ts).
3. Compare `OpenAIChat.protocol` reuse with a different adapter id and endpoint.
4. Open [`src/providers/openrouter.ts`](./src/providers/openrouter.ts) to show provider-specific options layered as a patch.
4. Open [`src/providers/openrouter.ts`](./src/providers/openrouter.ts) to show provider-specific options layered as an adapter-local transform.
5. Open [`src/providers/openai-compatible-profile.ts`](./src/providers/openai-compatible-profile.ts) to show family metadata and defaults.
For a testing demo:
packages/llm/package.json
+1 -1
View File
@@ -16,7 +16,7 @@
"./providers/*": "./src/providers/*.ts",
"./protocols": "./src/protocols.ts",
"./protocols/*": "./src/protocols/*.ts",
"./provider-patch": "./src/provider-patch.ts",
"./provider-transform": "./src/provider-transform.ts",
"./*": "./src/*.ts"
},
"devDependencies": {
packages/llm/script/setup-recording-env.ts
+1 -1
View File
@@ -6,7 +6,7 @@ import * as prompts from "@clack/prompts"
import { AwsV4Signer } from "aws4fetch"
import { Config, ConfigProvider, Effect, FileSystem, PlatformError, Redacted } from "effect"
import { FetchHttpClient, HttpClient, HttpClientRequest, type HttpClientResponse } from "effect/unstable/http"
import { ProviderShared } from "../src/protocols/shared"
import * as ProviderShared from "../src/protocols/shared"
type Provider = {
readonly id: string
packages/llm/src/adapter.ts
+29 -29
View File
@@ -4,9 +4,9 @@ import type { Auth } from "./auth"
import { bearer as authBearer } from "./auth"
import { type Endpoint, render as renderEndpoint } from "./endpoint"
import { RequestExecutor } from "./executor"
import type { AnyPatch, Patch, PatchInput, PatchRegistry } from "./patch"
import { payload as payloadPatch } from "./patch"
import { PatchPipeline } from "./patch-pipeline"
import type { AnyRuntimeTransform, Transform, TransformInput, TransformRegistry } from "./transform"
import { payload as payloadTransform } from "./transform"
import { TransformPipeline } from "./transform-pipeline"
import type { Framing } from "./framing"
import type { Protocol } from "./protocol"
import * as ProviderShared from "./protocols/shared"
@@ -37,7 +37,7 @@ export interface Adapter<Payload> {
readonly id: string
readonly protocol: ProtocolID
readonly payloadSchema: Schema.Codec<Payload, unknown>
readonly patches: ReadonlyArray<Patch<Payload>>
readonly transforms: ReadonlyArray<Transform<Payload, "payload">>
readonly toPayload: (request: LLMRequest) => Effect.Effect<Payload, LLMError>
readonly toHttp: (
payload: Payload,
@@ -49,13 +49,13 @@ export interface Adapter<Payload> {
) => Stream.Stream<LLMEvent, LLMError>
}
export type AdapterInput<Payload> = Omit<Adapter<Payload>, "patches"> & {
readonly patches?: ReadonlyArray<Patch<Payload>>
export type AdapterInput<Payload> = Omit<Adapter<Payload>, "transforms"> & {
readonly transforms?: ReadonlyArray<Transform<Payload, "payload">>
}
export interface AdapterDefinition<Payload> extends Adapter<Payload> {
readonly patch: (id: string, input: PatchInput<Payload>) => Patch<Payload>
readonly withPatches: (patches: ReadonlyArray<Patch<Payload>>) => AdapterDefinition<Payload>
readonly transform: (id: string, input: TransformInput<Payload>) => Transform<Payload, "payload">
readonly withTransforms: (transforms: ReadonlyArray<Transform<Payload, "payload">>) => AdapterDefinition<Payload>
}
// Adapter registries intentionally erase payload generics after the typed
@@ -167,7 +167,7 @@ export const preserveModelBinding = <Model extends ModelRef>(source: ModelRef, t
export interface LLMClient {
/**
* Compile a request through the adapter pipeline (patches, toPayload,
* Compile a request through the adapter pipeline (transforms, toPayload,
* protocol payload validation, toHttp) without sending it. Returns the
* prepared request including the provider-native payload.
*
@@ -183,14 +183,14 @@ export interface LLMClient {
export interface ClientOptions {
readonly adapters?: ReadonlyArray<AnyAdapter>
readonly patches?: PatchRegistry | ReadonlyArray<AnyPatch>
readonly transforms?: TransformRegistry | ReadonlyArray<AnyRuntimeTransform>
}
const noAdapter = (model: ModelRef) =>
new NoAdapterError({ adapter: model.adapter, protocol: model.protocol, provider: model.provider, model: model.id })
export interface MakeInput<Payload, Frame, Chunk, State> {
/** Adapter id used in registry lookup, error messages, and patch namespaces. */
/** Adapter id used in registry lookup, error messages, and transform namespaces. */
readonly id: string
/** Semantic API contract — owns lowering, payload schema, and parsing. */
readonly protocol: Protocol<Payload, Frame, Chunk, State>
@@ -208,8 +208,8 @@ export interface MakeInput<Payload, Frame, Chunk, State> {
readonly framing: Framing<Frame>
/** Static / per-request headers added before `auth` runs. */
readonly headers?: (input: { readonly request: LLMRequest }) => Record<string, string>
/** Provider patches that target this adapter (e.g. include-usage). */
readonly patches?: ReadonlyArray<Patch<Payload>>
/** Provider transforms that target this adapter payload (e.g. include-usage). */
readonly transforms?: ReadonlyArray<Transform<Payload, "payload">>
}
/**
@@ -220,7 +220,7 @@ export interface MakeInput<Payload, Frame, Chunk, State> {
* - `Auth` — how do I authenticate it?
* - `Framing` — how do I cut the response stream into protocol frames?
*
* Plus optional `headers` and `patches` for cross-cutting deployment concerns
* Plus optional `headers` and `transforms` for cross-cutting deployment concerns
* (provider version pins, per-deployment quirks).
*
* This is the canonical adapter constructor. If a new adapter does not fit
@@ -273,18 +273,18 @@ export function make<Payload, Frame, Chunk, State>(
onHalt: protocol.onHalt,
})
const patches = input.patches ?? []
const transforms = input.transforms ?? []
return {
id: input.id,
protocol: protocol.id,
payloadSchema: protocol.payload,
patches,
transforms,
toPayload: protocol.toPayload,
toHttp,
parse,
patch: (id, patchInput) => payloadPatch(`${input.id}.${id}`, patchInput),
withPatches: (next) => make({ ...input, patches: [...patches, ...next] }),
transform: (id, transformInput) => payloadTransform(`${input.id}.${id}`, transformInput),
withTransforms: (next) => make({ ...input, transforms: [...transforms, ...next] }),
}
}
@@ -294,29 +294,29 @@ export function make<Payload, Frame, Chunk, State>(
* but does not execute transport.
*/
const makeClient = (options: ClientOptions): LLMClient => {
const pipeline = PatchPipeline.make(options.patches)
const pipeline = TransformPipeline.make(options.transforms)
const adapters = new Map((options.adapters ?? []).map((adapter) => [adapter.id, adapter] as const))
const compile = Effect.fn("LLM.compile")(function* (request: LLMRequest) {
const adapter = adapters.get(request.model.adapter) ?? modelAdapters.get(request.model)
if (!adapter) return yield* noAdapter(request.model)
const patchedRequest = yield* pipeline.patchRequest(request)
const candidate = yield* adapter.toPayload(patchedRequest.request)
const patchedPayload = yield* pipeline.patchPayload({
state: patchedRequest,
const transformedRequest = yield* pipeline.transformRequest(request)
const candidate = yield* adapter.toPayload(transformedRequest.request)
const transformedPayload = yield* pipeline.transformPayload({
state: transformedRequest,
payload: candidate,
adapterPatches: adapter.patches,
adapterTransforms: adapter.transforms,
schema: adapter.payloadSchema,
})
const http = yield* adapter.toHttp(patchedPayload.payload, {
request: patchedPayload.request,
const http = yield* adapter.toHttp(transformedPayload.payload, {
request: transformedPayload.request,
})
return {
request: patchedPayload.request,
request: transformedPayload.request,
adapter,
payload: patchedPayload.payload,
payload: transformedPayload.payload,
http,
}
})
@@ -341,7 +341,7 @@ const makeClient = (options: ClientOptions): LLMClient => {
const events = compiled.adapter.parse(response, { request: compiled.request })
return pipeline.patchStreamEvents({ request: compiled.request, events })
return pipeline.transformStreamEvents({ request: compiled.request, events })
}),
)
packages/llm/src/endpoint.ts
+2 -2
View File
@@ -1,5 +1,5 @@
import { Effect } from "effect"
import { ProviderShared } from "./protocols/shared"
import * as ProviderShared from "./protocols/shared"
import type { LLMError, LLMRequest } from "./schema"
export interface EndpointInput<Payload> {
@@ -13,7 +13,7 @@ export type EndpointPart<Payload> = string | ((input: EndpointInput<Payload>) =>
* Declarative URL construction for one adapter.
*
* `Endpoint` is the deployment-side answer to "where does this request go?".
* `render(...)` interprets this data after request/payload patches, so dynamic
* `render(...)` interprets this data after request/payload transforms, so dynamic
* pieces can read the final `LLMRequest` and validated provider payload.
*/
export interface Endpoint<Payload> {
packages/llm/src/executor.ts
+7 -1
View File
@@ -1,5 +1,11 @@
import { Cause, Context, Effect, Layer } from "effect"
import { FetchHttpClient, HttpClient, HttpClientError, HttpClientRequest, HttpClientResponse } from "effect/unstable/http"
import {
FetchHttpClient,
HttpClient,
HttpClientError,
HttpClientRequest,
HttpClientResponse,
} from "effect/unstable/http"
import { ProviderRequestError, TransportError, type LLMError } from "./schema"
export interface Interface {
packages/llm/src/framing.ts
+1 -1
View File
@@ -1,5 +1,5 @@
import type { Stream } from "effect"
import { ProviderShared } from "./protocols/shared"
import * as ProviderShared from "./protocols/shared"
import type { ProviderChunkError } from "./schema"
/**
packages/llm/src/index.ts
+20 -21
View File
@@ -15,7 +15,7 @@ export type {
ModelRefInput,
} from "./adapter"
export * from "./executor"
export * from "./patch"
export * from "./transform"
export * from "./schema"
export * from "./tool-runtime"
export { Tool, ToolFailure, toDefinitions, tool } from "./tool"
@@ -31,34 +31,33 @@ export type { Framing as FramingDef } from "./framing"
export type { Protocol as ProtocolDef } from "./protocol"
export * as LLM from "./llm"
export * as ProviderPatch from "./provider-patch"
export * as ProviderTransform from "./provider-transform"
export * as Providers from "./providers"
export * as Protocols from "./protocols"
export type { CapabilitiesInput } from "./llm"
// Provider facades are the normal user-facing entrypoints. Prefer importing
// them from `@opencode-ai/llm/providers` in application code.
export { AmazonBedrock } from "./providers/amazon-bedrock"
export { Anthropic } from "./providers/anthropic"
export { Azure } from "./providers/azure"
export { Google } from "./providers/google"
export { GitHubCopilot } from "./providers/github-copilot"
export { OpenAI } from "./providers/openai"
export { OpenAICompatible } from "./providers/openai-compatible"
export { OpenRouter } from "./providers/openrouter"
export { XAI } from "./providers/xai"
export * as AmazonBedrock from "./providers/amazon-bedrock"
export * as Anthropic from "./providers/anthropic"
export * as Azure from "./providers/azure"
export * as Google from "./providers/google"
export * as GitHubCopilot from "./providers/github-copilot"
export * as OpenAI from "./providers/openai"
export * as OpenAICompatible from "./providers/openai-compatible"
export * as OpenRouter from "./providers/openrouter"
export * as XAI from "./providers/xai"
// Protocol modules expose low-level adapters, protocols, and payload types for
// tests, custom clients, and provider authors. Prefer
// `@opencode-ai/llm/protocols` for new advanced imports.
export { AnthropicMessages } from "./protocols/anthropic-messages"
export { BedrockConverse } from "./protocols/bedrock-converse"
export { Gemini } from "./protocols/gemini"
export { OpenAIChat } from "./protocols/openai-chat"
export { OpenAICompatibleChat } from "./protocols/openai-compatible-chat"
export { OpenAIResponses } from "./protocols/openai-responses"
export * as AnthropicMessages from "./protocols/anthropic-messages"
export * as BedrockConverse from "./protocols/bedrock-converse"
export * as Gemini from "./protocols/gemini"
export * as OpenAIChat from "./protocols/openai-chat"
export * as OpenAICompatibleChat from "./protocols/openai-compatible-chat"
export * as OpenAIResponses from "./protocols/openai-responses"
// OpenAI-compatible metadata helpers are shared by provider facades and
// advanced routing code; they are not standalone runnable providers.
export { OpenAICompatibleFamily } from "./providers/openai-compatible-family"
export { OpenAICompatibleProfiles } from "./providers/openai-compatible-profile"
// OpenAI-compatible profile metadata is shared by provider facades and advanced
// routing code; it is not a standalone runnable provider.
export * as OpenAICompatibleProfiles from "./providers/openai-compatible-profile"
packages/llm/src/llm.ts
+3 -3
View File
@@ -11,7 +11,7 @@ import {
type ModelRefInput,
} from "./adapter"
import type { RequestExecutor } from "./executor"
import { ProviderPatch } from "./provider-patch"
import { ProviderTransform } from "./provider-transform"
import { type Tools } from "./tool"
import { ToolRuntime, type RunOptions } from "./tool-runtime"
import {
@@ -37,7 +37,7 @@ export interface Provider {
export interface MakeOptions {
readonly providers?: ReadonlyArray<Provider>
readonly adapters?: ClientOptions["adapters"]
readonly patches?: ClientOptions["patches"]
readonly transforms?: ClientOptions["transforms"]
}
export type StreamWithToolsInput<T extends Tools> = Omit<RequestInput, "tools"> & Omit<RunOptions<T>, "request">
@@ -52,7 +52,7 @@ export class Service extends Context.Service<Service, Runtime>()("@opencode/LLM"
const clientOptions = (options: MakeOptions): ClientOptions => ({
adapters: [...(options.providers ?? []).flatMap((provider) => provider.adapters), ...(options.adapters ?? [])],
patches: options.patches ?? ProviderPatch.defaults,
transforms: options.transforms ?? ProviderTransform.defaults,
})
const requestOf = (input: LLMRequest | RequestInput) => input instanceof LLMRequest ? input : request(input)
packages/llm/src/patch-pipeline.ts
-115
View File
@@ -1,115 +0,0 @@
import { Effect, Schema, Stream } from "effect"
import type { AnyPatch, Patch, PatchRegistry } from "./patch"
import { context, emptyRegistry, plan, registry as makePatchRegistry } from "./patch"
import * as ProviderShared from "./protocols/shared"
import {
InvalidRequestError,
LLMRequest,
type LLMError,
type LLMEvent,
type ModelRef,
} from "./schema"
export interface PatchedRequest {
readonly request: LLMRequest
}
export interface PatchPayloadInput<Payload> {
readonly state: PatchedRequest
readonly payload: Payload
readonly adapterPatches: ReadonlyArray<Patch<Payload>>
readonly schema: Schema.Codec<Payload, unknown>
}
export interface PatchedPayload<Payload> {
readonly request: LLMRequest
readonly payload: Payload
}
export interface PatchStreamInput {
readonly request: LLMRequest
readonly events: Stream.Stream<LLMEvent, LLMError>
}
export interface PatchPipeline {
readonly patchRequest: (request: LLMRequest) => Effect.Effect<PatchedRequest, LLMError>
readonly patchPayload: <Payload>(input: PatchPayloadInput<Payload>) => Effect.Effect<PatchedPayload<Payload>, LLMError>
readonly patchStreamEvents: (input: PatchStreamInput) => Stream.Stream<LLMEvent, LLMError>
}
const normalizeRegistry = (patches: PatchRegistry | ReadonlyArray<AnyPatch> | undefined): PatchRegistry => {
if (!patches) return emptyRegistry
if ("request" in patches) return patches
return makePatchRegistry(patches)
}
const ensureSameRoute = (original: ModelRef, next: ModelRef) =>
Effect.gen(function* () {
if (
next.provider === original.provider &&
next.id === original.id &&
next.adapter === original.adapter &&
next.protocol === original.protocol
) return
return yield* new InvalidRequestError({
message: `Patches cannot change model routing (${original.provider}/${original.id}/${original.adapter}/${original.protocol} -> ${next.provider}/${next.id}/${next.adapter}/${next.protocol})`,
})
})
export const make = (patches?: PatchRegistry | ReadonlyArray<AnyPatch>): PatchPipeline => {
const registry = normalizeRegistry(patches)
const patchRequest = Effect.fn("PatchPipeline.patchRequest")(function* (request: LLMRequest) {
const requestPlan = plan({ phase: "request", context: context({ request }), patches: registry.request })
const requestAfterRequestPatches = requestPlan.apply(request)
yield* ensureSameRoute(request.model, requestAfterRequestPatches.model)
const promptPlan = plan({
phase: "prompt",
context: context({ request: requestAfterRequestPatches }),
patches: registry.prompt,
})
const requestBeforeToolPatches = promptPlan.apply(requestAfterRequestPatches)
yield* ensureSameRoute(request.model, requestBeforeToolPatches.model)
const toolSchemaPlan = requestBeforeToolPatches.tools.length === 0
? undefined
: plan({ phase: "tool-schema", context: context({ request: requestBeforeToolPatches }), patches: registry.toolSchema })
const hasToolSchemaPatches = toolSchemaPlan !== undefined && toolSchemaPlan.patches.length > 0
const patchedRequest = hasToolSchemaPatches
? new LLMRequest({
...requestBeforeToolPatches,
tools: requestBeforeToolPatches.tools.map(toolSchemaPlan.apply),
})
: requestBeforeToolPatches
return {
request: patchedRequest,
}
})
const patchPayload = Effect.fn("PatchPipeline.patchPayload")(function* <Payload>(input: PatchPayloadInput<Payload>) {
const payloadPlan = plan({
phase: "payload",
context: context({ request: input.state.request }),
patches: [...input.adapterPatches, ...(registry.payload as ReadonlyArray<Patch<Payload>>)],
})
const payload = yield* ProviderShared.validateWith(Schema.decodeUnknownEffect(input.schema))(
payloadPlan.apply(input.payload),
)
return {
request: input.state.request,
payload,
}
})
const patchStreamEvents = (input: PatchStreamInput) => {
const streamPlan = plan({ phase: "stream", context: context({ request: input.request }), patches: registry.stream })
if (streamPlan.patches.length === 0) return input.events
return input.events.pipe(Stream.map(streamPlan.apply))
}
return { patchRequest, patchPayload, patchStreamEvents }
}
export * as PatchPipeline from "./patch-pipeline"
packages/llm/src/patch.ts
-152
View File
@@ -1,152 +0,0 @@
import type { AdapterID, LLMEvent, LLMRequest, ModelRef, PatchPhase, ProtocolID, ToolDefinition } from "./schema"
export interface PatchContext {
readonly request: LLMRequest
readonly model: ModelRef
readonly adapter: ModelRef["adapter"]
readonly protocol: ModelRef["protocol"]
}
export interface Patch<A> {
readonly id: string
readonly phase: PatchPhase
readonly reason: string
readonly order?: number
readonly when: (context: PatchContext) => boolean
readonly apply: (value: A, context: PatchContext) => A
}
export interface AnyPatch {
readonly id: string
readonly phase: PatchPhase
readonly reason: string
readonly order?: number
readonly when: (context: PatchContext) => boolean
readonly apply: (value: never, context: PatchContext) => unknown
}
export interface PatchInput<A> {
readonly reason: string
readonly order?: number
readonly when?: PatchPredicate | ((context: PatchContext) => boolean)
readonly apply: (value: A, context: PatchContext) => A
}
export interface PatchPredicate {
(context: PatchContext): boolean
readonly and: (...predicates: ReadonlyArray<PatchPredicate>) => PatchPredicate
readonly or: (...predicates: ReadonlyArray<PatchPredicate>) => PatchPredicate
readonly not: () => PatchPredicate
}
export interface PatchPlan<A> {
readonly phase: PatchPhase
readonly patches: ReadonlyArray<Patch<A>>
readonly apply: (value: A) => A
}
export interface PatchRegistry {
readonly request: ReadonlyArray<Patch<LLMRequest>>
readonly prompt: ReadonlyArray<Patch<LLMRequest>>
readonly toolSchema: ReadonlyArray<Patch<ToolDefinition>>
readonly payload: ReadonlyArray<Patch<unknown>>
readonly stream: ReadonlyArray<Patch<LLMEvent>>
}
export const emptyRegistry: PatchRegistry = {
request: [],
prompt: [],
toolSchema: [],
payload: [],
stream: [],
}
export const predicate = (run: (context: PatchContext) => boolean): PatchPredicate => {
const self = Object.assign(run, {
and: (...predicates: ReadonlyArray<PatchPredicate>) =>
predicate((context) => self(context) && predicates.every((item) => item(context))),
or: (...predicates: ReadonlyArray<PatchPredicate>) =>
predicate((context) => self(context) || predicates.some((item) => item(context))),
not: () => predicate((context) => !self(context)),
})
return self
}
export const Model = {
provider: (provider: string) => predicate((context) => context.model.provider === provider),
adapter: (adapter: AdapterID) => predicate((context) => context.adapter === adapter),
protocol: (protocol: ProtocolID) => predicate((context) => context.protocol === protocol),
id: (id: string) => predicate((context) => context.model.id === id),
idIncludes: (value: string) => predicate((context) => context.model.id.toLowerCase().includes(value.toLowerCase())),
}
export const make = <A>(id: string, phase: PatchPhase, input: PatchInput<A>): Patch<A> => ({
id,
phase,
reason: input.reason,
order: input.order,
when: input.when ?? (() => true),
apply: input.apply,
})
export const request = (id: string, input: PatchInput<LLMRequest>) => make(`request.${id}`, "request", input)
export const prompt = (id: string, input: PatchInput<LLMRequest>) => make(`prompt.${id}`, "prompt", input)
export const toolSchema = (id: string, input: PatchInput<ToolDefinition>) => make(`schema.${id}`, "tool-schema", input)
export const payload = <A>(id: string, input: PatchInput<A>) => make(`payload.${id}`, "payload", input)
export const stream = (id: string, input: PatchInput<LLMEvent>) => make(`stream.${id}`, "stream", input)
export function registry(patches: ReadonlyArray<AnyPatch>): PatchRegistry {
return {
request: patches.filter((patch): patch is Patch<LLMRequest> => patch.phase === "request"),
prompt: patches.filter((patch): patch is Patch<LLMRequest> => patch.phase === "prompt"),
toolSchema: patches.filter((patch): patch is Patch<ToolDefinition> => patch.phase === "tool-schema"),
payload: patches.filter((patch) => patch.phase === "payload") as unknown as ReadonlyArray<Patch<unknown>>,
stream: patches.filter((patch): patch is Patch<LLMEvent> => patch.phase === "stream"),
}
}
export function context(input: {
readonly request: LLMRequest
}): PatchContext {
return {
request: input.request,
model: input.request.model,
adapter: input.request.model.adapter,
protocol: input.request.model.protocol,
}
}
export function plan<A>(input: {
readonly phase: PatchPhase
readonly context: PatchContext
readonly patches: ReadonlyArray<Patch<A>>
}): PatchPlan<A> {
const patches = input.patches
.filter((patch) => patch.phase === input.phase && patch.when(input.context))
.toSorted((left, right) => (left.order ?? 0) - (right.order ?? 0) || left.id.localeCompare(right.id))
return {
phase: input.phase,
patches,
apply: (value) => patches.reduce((next, patch) => patch.apply(next, input.context), value),
}
}
export function mergeRegistries(registries: ReadonlyArray<PatchRegistry>): PatchRegistry {
return registries.reduce(
(merged, registry) => ({
request: [...merged.request, ...registry.request],
prompt: [...merged.prompt, ...registry.prompt],
toolSchema: [...merged.toolSchema, ...registry.toolSchema],
payload: [...merged.payload, ...registry.payload],
stream: [...merged.stream, ...registry.stream],
}),
emptyRegistry,
)
}
export * as Patch from "./patch"
packages/llm/src/protocol.ts
+1 -1
View File
@@ -25,7 +25,7 @@ import type { LLMError, LLMEvent, LLMRequest, ProtocolID, ProviderChunkError } f
*
* The four type parameters reflect the pipeline:
*
* - `Payload` — provider-native request payload candidate. Payload patches can
* - `Payload` — provider-native request payload candidate. Payload transforms can
* transform this value, then `Adapter.make(...)` validates and
* JSON-encodes it with `payload`.
* - `Frame` — one unit of the framed response stream. SSE: a JSON data
packages/llm/src/protocols/bedrock-converse.ts
+1 -1
View File
@@ -719,7 +719,7 @@ export const adapter = Adapter.make({
protocol,
endpoint: Endpoint.baseURL<BedrockConversePayload>({
// Bedrock's URL embeds the region in the host and the validated modelId
// in the path. We reach into the payload after payload patches so the URL
// in the path. We reach into the payload after payload transforms so the URL
// matches the body that gets signed.
default: ({ request }) => `https://bedrock-runtime.${region(request)}.amazonaws.com`,
path: ({ payload }) => `/model/${encodeURIComponent(payload.modelId)}/converse-stream`,
packages/llm/src/protocols/gemini.ts
+1 -1
View File
@@ -165,7 +165,7 @@ const isRecord = ProviderShared.isRecord
// allowlist (e.g. `additionalProperties`, `$ref`) is silently dropped.
//
// Sanitize runs first, then project. Both passes live here so the adapter
// owns the full transformation; consumers don't need to register a patch.
// owns the full transformation; consumers don't need to register a transform.
const SCHEMA_INTENT_KEYS = [
"type",
packages/llm/src/protocols/openai-chat.ts
+5 -5
View File
@@ -238,7 +238,7 @@ const lowerMessages = Effect.fn("OpenAIChat.lowerMessages")(function* (request:
const toPayload = Effect.fn("OpenAIChat.toPayload")(function* (request: LLMRequest) {
// `toPayload` returns the provider payload only. Endpoint, auth, framing,
// patches, validation, and HTTP execution are all composed by `Adapter.make`.
// transforms, validation, and HTTP execution are all composed by `Adapter.make`.
return {
model: request.model.id,
messages: yield* lowerMessages(request),
@@ -366,7 +366,7 @@ export const adapter = Adapter.make({
})
// =============================================================================
// Model Helper And Patches
// Model Helper And Transforms
// =============================================================================
export const model = Adapter.model(adapter, {
// `Adapter.model` creates a user-facing model factory bound to this adapter.
@@ -376,9 +376,9 @@ export const model = Adapter.model(adapter, {
capabilities: capabilities({ tools: { calls: true, streamingInput: true } }),
})
export const includeUsage = adapter.patch("include-usage", {
// Adapter-local patches are named payload transforms. They are inspectable in
// patch traces and cannot reroute the request to another model/protocol.
export const includeUsage = adapter.transform("include-usage", {
// Adapter-local transforms are named payload rewrites. They cannot reroute
// the request to another model/protocol.
reason: "request final usage chunk from OpenAI Chat streaming responses",
apply: (payload) => ({
...payload,
packages/llm/src/protocols/openai-compatible-chat.ts
+1 -1
View File
@@ -33,7 +33,7 @@ export const model = Adapter.model<OpenAICompatibleChatModelInput>(adapter, {
capabilities: capabilities({ tools: { calls: true, streamingInput: true } }),
})
export const includeUsage = adapter.patch("include-usage", {
export const includeUsage = adapter.transform("include-usage", {
reason: "request final usage chunk from OpenAI-compatible Chat streaming responses",
apply: (payload) => ({
...payload,
packages/llm/src/provider-patch.ts → packages/llm/src/provider-transform.ts
+13 -13
View File
@@ -1,4 +1,4 @@
import { Model, Patch, predicate } from "./patch"
import { Model, Transform, predicate } from "./transform"
import { CacheHint } from "./schema"
import type { ContentPart, JsonSchema, LLMRequest, Message, ToolDefinition } from "./schema"
@@ -38,7 +38,7 @@ const rewriteToolIds = (request: LLMRequest, scrub: (id: string) => string): LLM
}),
})
export const removeEmptyAnthropicContent = Patch.prompt("anthropic.remove-empty-content", {
export const removeEmptyAnthropicContent = Transform.prompt("anthropic.remove-empty-content", {
reason: "remove empty text/reasoning blocks for providers that reject empty content",
when: Model.provider("anthropic").or(Model.provider("bedrock"), Model.provider("amazon-bedrock")),
apply: (request) => ({
@@ -50,19 +50,19 @@ export const removeEmptyAnthropicContent = Patch.prompt("anthropic.remove-empty-
}),
})
export const scrubClaudeToolIds = Patch.prompt("anthropic.scrub-tool-call-ids", {
export const scrubClaudeToolIds = Transform.prompt("anthropic.scrub-tool-call-ids", {
reason: "Claude tool_use ids only accept alphanumeric, underscore, and dash characters",
when: Model.idIncludes("claude"),
apply: (request) => rewriteToolIds(request, (id) => id.replace(/[^a-zA-Z0-9_-]/g, "_")),
})
export const scrubMistralToolIds = Patch.prompt("mistral.scrub-tool-call-ids", {
export const scrubMistralToolIds = Transform.prompt("mistral.scrub-tool-call-ids", {
reason: "Mistral tool call ids must be short alphanumeric identifiers",
when: Model.provider("mistral").or(Model.idIncludes("mistral"), Model.idIncludes("devstral")),
apply: (request) => rewriteToolIds(request, (id) => id.replace(/[^a-zA-Z0-9]/g, "").slice(0, 9).padEnd(9, "0")),
})
export const repairAnthropicToolUseOrder = Patch.prompt("anthropic.repair-tool-use-order", {
export const repairAnthropicToolUseOrder = Transform.prompt("anthropic.repair-tool-use-order", {
reason: "Anthropic rejects assistant turns where tool_use blocks are followed by non-tool content",
when: Model.provider("anthropic").or(Model.provider("google-vertex-anthropic"), Model.idIncludes("claude")),
apply: (request) => ({
@@ -80,7 +80,7 @@ export const repairAnthropicToolUseOrder = Patch.prompt("anthropic.repair-tool-u
}),
})
export const repairMistralToolResultUserSequence = Patch.prompt("mistral.repair-tool-user-sequence", {
export const repairMistralToolResultUserSequence = Transform.prompt("mistral.repair-tool-user-sequence", {
reason: "Mistral rejects tool messages followed immediately by user messages",
when: Model.provider("mistral").or(Model.idIncludes("mistral"), Model.idIncludes("devstral")),
apply: (request) => ({
@@ -93,7 +93,7 @@ export const repairMistralToolResultUserSequence = Patch.prompt("mistral.repair-
}),
})
export const addDeepSeekEmptyReasoning = Patch.prompt("deepseek.empty-reasoning-replay", {
export const addDeepSeekEmptyReasoning = Transform.prompt("deepseek.empty-reasoning-replay", {
reason: "DeepSeek expects assistant history to carry reasoning_content, even when empty",
when: Model.idIncludes("deepseek"),
apply: (request) => ({
@@ -115,7 +115,7 @@ export const addDeepSeekEmptyReasoning = Patch.prompt("deepseek.empty-reasoning-
}),
})
export const moveOpenAICompatibleReasoningToNative = Patch.prompt("openai-compatible.reasoning-native-field", {
export const moveOpenAICompatibleReasoningToNative = Transform.prompt("openai-compatible.reasoning-native-field", {
reason: "OpenAI-compatible reasoning providers replay reasoning in provider-native assistant fields",
when: Model.adapter("openai-compatible-chat"),
apply: (request) => ({
@@ -139,7 +139,7 @@ export const moveOpenAICompatibleReasoningToNative = Patch.prompt("openai-compat
}),
})
export const unsupportedMediaFallback = Patch.prompt("capabilities.unsupported-media-fallback", {
export const unsupportedMediaFallback = Transform.prompt("capabilities.unsupported-media-fallback", {
reason: "turn unsupported user media into model-visible error text instead of provider request failures",
apply: (request) => ({
...request,
@@ -161,7 +161,7 @@ export const unsupportedMediaFallback = Patch.prompt("capabilities.unsupported-m
}),
})
export const sanitizeMoonshotToolSchema = Patch.toolSchema("moonshot.schema", {
export const sanitizeMoonshotToolSchema = Transform.toolSchema("moonshot.schema", {
reason: "Moonshot/Kimi rejects $ref sibling keywords and tuple-style array items",
when: Model.provider("moonshotai").or(Model.idIncludes("kimi")),
apply: (tool): ToolDefinition => ({
@@ -170,7 +170,7 @@ export const sanitizeMoonshotToolSchema = Patch.toolSchema("moonshot.schema", {
}),
})
// Single shared CacheHint instance — the cache patch reuses this one object
// Single shared CacheHint instance — the cache transform reuses this one object
// across every marked part. Adapters lower CacheHint structurally
// (`cache?.type === "ephemeral"`) so reference equality is incidental, but
// keeping a class instance preserves any consumer that checks
@@ -192,7 +192,7 @@ const withCacheOnLastText = (content: ReadonlyArray<ContentPart>): ReadonlyArray
// this a no-op for adapters that don't advertise prompt-level caching, so
// non-cache providers (OpenAI Responses, Gemini, OpenAI-compatible Chat)
// are unaffected.
export const cachePromptHints = Patch.prompt("cache.prompt-hints", {
export const cachePromptHints = Transform.prompt("cache.prompt-hints", {
reason: "mark first 2 system parts and last 2 messages with ephemeral cache hints on cache-capable adapters",
when: predicate((context) => context.model.capabilities.cache?.prompt === true),
apply: (request) => ({
@@ -221,4 +221,4 @@ export const defaults = [
cachePromptHints,
]
export * as ProviderPatch from "./provider-patch"
export * as ProviderTransform from "./provider-transform"
packages/llm/src/providers/amazon-bedrock.ts
+2 -3
View File
@@ -1,5 +1,6 @@
import { Adapter, type AdapterModelInput } from "../adapter"
import { BedrockConverse, type BedrockCredentials } from "../protocols/bedrock-converse"
import * as BedrockConverse from "../protocols/bedrock-converse"
import type { BedrockCredentials } from "../protocols/bedrock-converse"
export type ModelOptions = Omit<AdapterModelInput, "id"> & {
readonly apiKey?: string
@@ -22,5 +23,3 @@ export const model = (modelID: string, options: ModelOptions = {}) => {
native: BedrockConverse.nativeCredentials(options.native, credentials),
})
}
export * as AmazonBedrock from "./amazon-bedrock"
packages/llm/src/providers/anthropic.ts
+2 -5
View File
@@ -1,10 +1,7 @@
import { AnthropicMessages, type AnthropicMessagesModelInput } from "../protocols/anthropic-messages"
import * as AnthropicMessages from "../protocols/anthropic-messages"
import type { AnthropicMessagesModelInput } from "../protocols/anthropic-messages"
export const adapters = [AnthropicMessages.adapter]
export const model = (id: string, options: Omit<AnthropicMessagesModelInput, "id"> = {}) =>
AnthropicMessages.model({ ...options, id })
export const messages = model
export * as Anthropic from "./anthropic"
packages/llm/src/providers/azure.ts
+2 -4
View File
@@ -1,8 +1,8 @@
import { Adapter } from "../adapter"
import type { ModelInput } from "../llm"
import { ProviderID } from "../schema"
import { OpenAIChat } from "../protocols/openai-chat"
import { OpenAIResponses } from "../protocols/openai-responses"
import * as OpenAIChat from "../protocols/openai-chat"
import * as OpenAIResponses from "../protocols/openai-responses"
export const id = ProviderID.make("azure")
@@ -36,5 +36,3 @@ export const model = (modelID: string, options: ModelOptions = {}) => {
},
})
}
export * as Azure from "./azure"
packages/llm/src/providers/github-copilot.ts
+2 -4
View File
@@ -1,8 +1,8 @@
import { Adapter } from "../adapter"
import type { ModelInput } from "../llm"
import { ProviderID } from "../schema"
import { OpenAIChat } from "../protocols/openai-chat"
import { OpenAIResponses } from "../protocols/openai-responses"
import * as OpenAIChat from "../protocols/openai-chat"
import * as OpenAIResponses from "../protocols/openai-responses"
export const id = ProviderID.make("github-copilot")
@@ -23,5 +23,3 @@ export const model = (modelID: string, options: ModelOptions = {}) => {
const create = shouldUseResponsesApi(modelID) ? responsesModel : chatModel
return create({ ...options, id: modelID })
}
export * as GitHubCopilot from "./github-copilot"
packages/llm/src/providers/google.ts
+2 -5
View File
@@ -1,10 +1,7 @@
import { Gemini, type GeminiModelInput } from "../protocols/gemini"
import * as Gemini from "../protocols/gemini"
import type { GeminiModelInput } from "../protocols/gemini"
export const adapters = [Gemini.adapter]
export const model = (id: string, options: Omit<GeminiModelInput, "id"> = {}) =>
Gemini.model({ ...options, id })
export const gemini = model
export * as Google from "./google"
packages/llm/src/providers/openai-compatible-family.ts
-7
View File
@@ -1,7 +0,0 @@
import { byProvider, profiles, type OpenAICompatibleProfile } from "./openai-compatible-profile"
export type ProviderFamily = OpenAICompatibleProfile
export const families = profiles
export { byProvider }
export * as OpenAICompatibleFamily from "./openai-compatible-family"
packages/llm/src/providers/openai-compatible-profile.ts
-2
View File
@@ -21,5 +21,3 @@ export const profiles = {
export const byProvider: Record<string, OpenAICompatibleProfile> = Object.fromEntries(
Object.values(profiles).map((profile) => [profile.provider, profile]),
)
export * as OpenAICompatibleProfiles from "./openai-compatible-profile"
packages/llm/src/providers/openai-compatible.ts
+35 -3
View File
@@ -1,10 +1,16 @@
import { ProviderID } from "../schema"
import { OpenAICompatibleChat, type OpenAICompatibleChatModelInput } from "../protocols/openai-compatible-chat"
import * as OpenAICompatibleChat from "../protocols/openai-compatible-chat"
import type { OpenAICompatibleChatModelInput } from "../protocols/openai-compatible-chat"
import { profiles, type OpenAICompatibleProfile } from "./openai-compatible-profile"
export type ModelOptions = Omit<OpenAICompatibleChatModelInput, "id" | "provider"> & {
readonly provider: string
}
export type FamilyModelOptions = Omit<OpenAICompatibleChatModelInput, "id" | "provider" | "baseURL"> & {
readonly baseURL?: string
}
export const adapters = [OpenAICompatibleChat.adapter]
export const model = (id: string, options: ModelOptions) => {
@@ -15,6 +21,32 @@ export const model = (id: string, options: ModelOptions) => {
})
}
export const chat = model
const profileBaseURL = (profile: OpenAICompatibleProfile, options: FamilyModelOptions) => {
const baseURL = options.baseURL ?? profile.baseURL
if (baseURL) return baseURL
throw new Error(`OpenAI-compatible profile ${profile.provider} requires a baseURL`)
}
export * as OpenAICompatible from "./openai-compatible"
export const profileModel = (profile: OpenAICompatibleProfile, id: string, options: FamilyModelOptions = {}) =>
OpenAICompatibleChat.model({
...options,
id,
provider: profile.provider,
baseURL: profileBaseURL(profile, options),
capabilities: options.capabilities ?? profile.capabilities,
})
const define = (profile: OpenAICompatibleProfile) => ({
id: profile.provider,
adapters,
model: (id: string, options: FamilyModelOptions = {}) => profileModel(profile, id, options),
})
export const baseten = define(profiles.baseten)
export const cerebras = define(profiles.cerebras)
export const deepinfra = define(profiles.deepinfra)
export const deepseek = define(profiles.deepseek)
export const fireworks = define(profiles.fireworks)
export const groq = define(profiles.groq)
export const togetherai = define(profiles.togetherai)
export const xai = define(profiles.xai)
packages/llm/src/providers/openai.ts
+4 -4
View File
@@ -1,5 +1,7 @@
import { OpenAIChat, type OpenAIChatModelInput } from "../protocols/openai-chat"
import { OpenAIResponses, type OpenAIResponsesModelInput } from "../protocols/openai-responses"
import * as OpenAIChat from "../protocols/openai-chat"
import type { OpenAIChatModelInput } from "../protocols/openai-chat"
import * as OpenAIResponses from "../protocols/openai-responses"
import type { OpenAIResponsesModelInput } from "../protocols/openai-responses"
export const adapters = [OpenAIResponses.adapter, OpenAIChat.adapter]
@@ -10,5 +12,3 @@ export const chat = (id: string, options: Omit<OpenAIChatModelInput, "id"> = {})
OpenAIChat.model({ ...options, id })
export const model = responses
export * as OpenAI from "./openai"
packages/llm/src/providers/openrouter.ts
+3 -3
View File
@@ -3,7 +3,7 @@ import { Adapter, type AdapterModelInput } from "../adapter"
import { Endpoint } from "../endpoint"
import { Framing } from "../framing"
import { capabilities } from "../llm"
import { payload as payloadPatch } from "../patch"
import { payload as payloadTransform } from "../transform"
import { Protocol } from "../protocol"
import * as OpenAICompatibleProfiles from "./openai-compatible-profile"
import * as OpenAIChat from "../protocols/openai-chat"
@@ -55,7 +55,7 @@ const nativeOptions = (options: ModelOptions) => {
return { ...options.native, openrouter }
}
export const applyOptions = payloadPatch<OpenRouterPayload>("openrouter.options", {
export const applyOptions = payloadTransform<OpenRouterPayload>("openrouter.options", {
reason: "apply OpenRouter provider options to the Chat payload",
when: (context) => context.model.provider === profile.provider && Object.keys(payloadOptions(context.model.native?.openrouter)).length > 0,
apply: (payload, context) => {
@@ -70,7 +70,7 @@ export const adapter = Adapter.make({
protocol,
endpoint: Endpoint.baseURL({ default: profile.baseURL, path: "/chat/completions" }),
framing: Framing.sse,
patches: [applyOptions],
transforms: [applyOptions],
})
export const adapters = [adapter]
packages/llm/src/providers/xai.ts
+2 -4
View File
@@ -1,7 +1,7 @@
import { Adapter } from "../adapter"
import type { ModelInput } from "../llm"
import { OpenAICompatibleProfiles } from "./openai-compatible-profile"
import { OpenAIResponses } from "../protocols/openai-responses"
import * as OpenAICompatibleProfiles from "./openai-compatible-profile"
import * as OpenAIResponses from "../protocols/openai-responses"
export type ModelOptions = Omit<ModelInput, "id" | "provider" | "protocol">
@@ -15,5 +15,3 @@ export const model = (modelID: string, options: ModelOptions = {}) =>
id: modelID,
baseURL: options.baseURL ?? OpenAICompatibleProfiles.profiles.xai.baseURL,
})
export * as XAI from "./xai"
packages/llm/src/schema.ts
+2 -2
View File
@@ -23,8 +23,8 @@ export const ReasoningEfforts = ["none", "minimal", "low", "medium", "high", "xh
export const ReasoningEffort = Schema.Literals(ReasoningEfforts)
export type ReasoningEffort = Schema.Schema.Type<typeof ReasoningEffort>
export const PatchPhase = Schema.Literals(["request", "prompt", "tool-schema", "payload", "stream"])
export type PatchPhase = Schema.Schema.Type<typeof PatchPhase>
export const TransformPhase = Schema.Literals(["request", "prompt", "tool-schema", "payload", "stream"])
export type TransformPhase = Schema.Schema.Type<typeof TransformPhase>
export const MessageRole = Schema.Literals(["user", "assistant", "tool"])
export type MessageRole = Schema.Schema.Type<typeof MessageRole>
packages/llm/src/transform-pipeline.ts
+115
View File
@@ -0,0 +1,115 @@
import { Effect, Schema, Stream } from "effect"
import type { AnyRuntimeTransform, Transform, TransformRegistry } from "./transform"
import { context, emptyRegistry, plan, registry as makeTransformRegistry } from "./transform"
import * as ProviderShared from "./protocols/shared"
import {
InvalidRequestError,
LLMRequest,
type LLMError,
type LLMEvent,
type ModelRef,
} from "./schema"
export interface TransformedRequest {
readonly request: LLMRequest
}
export interface TransformPayloadInput<Payload> {
readonly state: TransformedRequest
readonly payload: Payload
readonly adapterTransforms: ReadonlyArray<Transform<Payload, "payload">>
readonly schema: Schema.Codec<Payload, unknown>
}
export interface TransformedPayload<Payload> {
readonly request: LLMRequest
readonly payload: Payload
}
export interface TransformStreamInput {
readonly request: LLMRequest
readonly events: Stream.Stream<LLMEvent, LLMError>
}
export interface TransformPipeline {
readonly transformRequest: (request: LLMRequest) => Effect.Effect<TransformedRequest, LLMError>
readonly transformPayload: <Payload>(input: TransformPayloadInput<Payload>) => Effect.Effect<TransformedPayload<Payload>, LLMError>
readonly transformStreamEvents: (input: TransformStreamInput) => Stream.Stream<LLMEvent, LLMError>
}
const normalizeRegistry = (transforms: TransformRegistry | ReadonlyArray<AnyRuntimeTransform> | undefined): TransformRegistry => {
if (!transforms) return emptyRegistry
if ("request" in transforms) return transforms
return makeTransformRegistry(transforms)
}
const ensureSameRoute = (original: ModelRef, next: ModelRef) =>
Effect.gen(function* () {
if (
next.provider === original.provider &&
next.id === original.id &&
next.adapter === original.adapter &&
next.protocol === original.protocol
) return
return yield* new InvalidRequestError({
message: `Transforms cannot change model routing (${original.provider}/${original.id}/${original.adapter}/${original.protocol} -> ${next.provider}/${next.id}/${next.adapter}/${next.protocol})`,
})
})
export const make = (transforms?: TransformRegistry | ReadonlyArray<AnyRuntimeTransform>): TransformPipeline => {
const registry = normalizeRegistry(transforms)
const transformRequest = Effect.fn("TransformPipeline.transformRequest")(function* (request: LLMRequest) {
const requestPlan = plan({ phase: "request", context: context({ request }), transforms: registry.request })
const requestAfterRequestTransforms = requestPlan.apply(request)
yield* ensureSameRoute(request.model, requestAfterRequestTransforms.model)
const promptPlan = plan({
phase: "prompt",
context: context({ request: requestAfterRequestTransforms }),
transforms: registry.prompt,
})
const requestBeforeToolTransforms = promptPlan.apply(requestAfterRequestTransforms)
yield* ensureSameRoute(request.model, requestBeforeToolTransforms.model)
const toolSchemaPlan = requestBeforeToolTransforms.tools.length === 0
? undefined
: plan({ phase: "tool-schema", context: context({ request: requestBeforeToolTransforms }), transforms: registry.toolSchema })
const hasToolSchemaTransforms = toolSchemaPlan !== undefined && toolSchemaPlan.transforms.length > 0
const transformedRequest = hasToolSchemaTransforms
? new LLMRequest({
...requestBeforeToolTransforms,
tools: requestBeforeToolTransforms.tools.map(toolSchemaPlan.apply),
})
: requestBeforeToolTransforms
return {
request: transformedRequest,
}
})
const transformPayload = Effect.fn("TransformPipeline.transformPayload")(function* <Payload>(input: TransformPayloadInput<Payload>) {
const payloadPlan = plan({
phase: "payload",
context: context({ request: input.state.request }),
transforms: input.adapterTransforms,
})
const payload = yield* ProviderShared.validateWith(Schema.decodeUnknownEffect(input.schema))(
payloadPlan.apply(input.payload),
)
return {
request: input.state.request,
payload,
}
})
const transformStreamEvents = (input: TransformStreamInput) => {
const streamPlan = plan({ phase: "stream", context: context({ request: input.request }), transforms: registry.stream })
if (streamPlan.transforms.length === 0) return input.events
return input.events.pipe(Stream.map(streamPlan.apply))
}
return { transformRequest, transformPayload, transformStreamEvents }
}
export * as TransformPipeline from "./transform-pipeline"
packages/llm/src/transform.ts
+154
View File
@@ -0,0 +1,154 @@
import type { AdapterID, LLMEvent, LLMRequest, ModelRef, ProtocolID, ToolDefinition, TransformPhase } from "./schema"
export interface TransformContext {
readonly request: LLMRequest
readonly model: ModelRef
readonly adapter: ModelRef["adapter"]
readonly protocol: ModelRef["protocol"]
}
export interface Transform<A, Phase extends TransformPhase = TransformPhase> {
readonly id: string
readonly phase: Phase
readonly reason: string
readonly order?: number
readonly when: (context: TransformContext) => boolean
readonly apply: (value: A, context: TransformContext) => A
}
export interface AnyTransform {
readonly id: string
readonly phase: TransformPhase
readonly reason: string
readonly order?: number
readonly when: (context: TransformContext) => boolean
readonly apply: (value: never, context: TransformContext) => unknown
}
export type AnyRuntimeTransform =
| Transform<LLMRequest, "request">
| Transform<LLMRequest, "prompt">
| Transform<ToolDefinition, "tool-schema">
| Transform<LLMEvent, "stream">
export interface TransformInput<A> {
readonly reason: string
readonly order?: number
readonly when?: TransformPredicate | ((context: TransformContext) => boolean)
readonly apply: (value: A, context: TransformContext) => A
}
export interface TransformPredicate {
(context: TransformContext): boolean
readonly and: (...predicates: ReadonlyArray<TransformPredicate>) => TransformPredicate
readonly or: (...predicates: ReadonlyArray<TransformPredicate>) => TransformPredicate
readonly not: () => TransformPredicate
}
export interface TransformPlan<A> {
readonly phase: TransformPhase
readonly transforms: ReadonlyArray<Transform<A>>
readonly apply: (value: A) => A
}
export interface TransformRegistry {
readonly request: ReadonlyArray<Transform<LLMRequest, "request">>
readonly prompt: ReadonlyArray<Transform<LLMRequest, "prompt">>
readonly toolSchema: ReadonlyArray<Transform<ToolDefinition, "tool-schema">>
readonly stream: ReadonlyArray<Transform<LLMEvent, "stream">>
}
export const emptyRegistry: TransformRegistry = {
request: [],
prompt: [],
toolSchema: [],
stream: [],
}
export const predicate = (run: (context: TransformContext) => boolean): TransformPredicate => {
const self = Object.assign(run, {
and: (...predicates: ReadonlyArray<TransformPredicate>) =>
predicate((context) => self(context) && predicates.every((item) => item(context))),
or: (...predicates: ReadonlyArray<TransformPredicate>) =>
predicate((context) => self(context) || predicates.some((item) => item(context))),
not: () => predicate((context) => !self(context)),
})
return self
}
export const Model = {
provider: (provider: string) => predicate((context) => context.model.provider === provider),
adapter: (adapter: AdapterID) => predicate((context) => context.adapter === adapter),
protocol: (protocol: ProtocolID) => predicate((context) => context.protocol === protocol),
id: (id: string) => predicate((context) => context.model.id === id),
idIncludes: (value: string) => predicate((context) => context.model.id.toLowerCase().includes(value.toLowerCase())),
}
export const make = <A, Phase extends TransformPhase>(id: string, phase: Phase, input: TransformInput<A>): Transform<A, Phase> => ({
id,
phase,
reason: input.reason,
order: input.order,
when: input.when ?? (() => true),
apply: input.apply,
})
export const request = (id: string, input: TransformInput<LLMRequest>) => make(`request.${id}`, "request", input)
export const prompt = (id: string, input: TransformInput<LLMRequest>) => make(`prompt.${id}`, "prompt", input)
export const toolSchema = (id: string, input: TransformInput<ToolDefinition>) => make(`schema.${id}`, "tool-schema", input)
export const payload = <A>(id: string, input: TransformInput<A>) => make(`payload.${id}`, "payload", input)
export const stream = (id: string, input: TransformInput<LLMEvent>) => make(`stream.${id}`, "stream", input)
export function registry(transforms: ReadonlyArray<AnyRuntimeTransform>): TransformRegistry {
return {
request: transforms.filter((transform): transform is Transform<LLMRequest, "request"> => transform.phase === "request"),
prompt: transforms.filter((transform): transform is Transform<LLMRequest, "prompt"> => transform.phase === "prompt"),
toolSchema: transforms.filter((transform): transform is Transform<ToolDefinition, "tool-schema"> => transform.phase === "tool-schema"),
stream: transforms.filter((transform): transform is Transform<LLMEvent, "stream"> => transform.phase === "stream"),
}
}
export function context(input: {
readonly request: LLMRequest
}): TransformContext {
return {
request: input.request,
model: input.request.model,
adapter: input.request.model.adapter,
protocol: input.request.model.protocol,
}
}
export function plan<A>(input: {
readonly phase: TransformPhase
readonly context: TransformContext
readonly transforms: ReadonlyArray<Transform<A>>
}): TransformPlan<A> {
const transforms = input.transforms
.filter((transform) => transform.phase === input.phase && transform.when(input.context))
.toSorted((left, right) => (left.order ?? 0) - (right.order ?? 0) || left.id.localeCompare(right.id))
return {
phase: input.phase,
transforms,
apply: (value) => transforms.reduce((next, transform) => transform.apply(next, input.context), value),
}
}
export function mergeRegistries(registries: ReadonlyArray<TransformRegistry>): TransformRegistry {
return registries.reduce(
(merged, registry) => ({
request: [...merged.request, ...registry.request],
prompt: [...merged.prompt, ...registry.prompt],
toolSchema: [...merged.toolSchema, ...registry.toolSchema],
stream: [...merged.stream, ...registry.stream],
}),
emptyRegistry,
)
}
export * as Transform from "./transform"
packages/llm/test/adapter.test.ts
+8 -8
View File
@@ -2,7 +2,7 @@ import { describe, expect } from "bun:test"
import { Effect, Schema, Stream } from "effect"
import { Endpoint, LLM, Protocol } from "../src"
import { Adapter, LLMClient } from "../src/adapter"
import { Patch } from "../src/patch"
import { Transform } from "../src/transform"
import type { FramingDef } from "../src"
import type { ModelRef } from "../src/schema"
import { testEffect } from "./lib/effect"
@@ -115,13 +115,13 @@ const echoLayer = dynamicResponse(({ text, respond }) =>
const it = testEffect(echoLayer)
describe("llm adapter", () => {
it.effect("prepare applies payload patches", () =>
it.effect("prepare applies payload transforms", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.make({
adapters: [
fake.withPatches([
fake.patch("include-usage", {
reason: "fake payload patch",
fake.withTransforms([
fake.transform("include-usage", {
reason: "fake payload transform",
apply: (payload) => ({ ...payload, includeUsage: true }),
}),
]),
@@ -183,12 +183,12 @@ describe("llm adapter", () => {
}),
)
it.effect("stream patches transform raised events", () =>
it.effect("stream transforms rewrite raised events", () =>
Effect.gen(function* () {
const llm = LLMClient.make({
adapters: [fake],
patches: [
Patch.stream("test.uppercase", {
transforms: [
Transform.stream("test.uppercase", {
reason: "uppercase text deltas",
apply: (event) => (event.type === "text-delta" ? { ...event, text: event.text.toUpperCase() } : event),
}),
packages/llm/test/provider/anthropic-messages.recorded.test.ts
+6 -6
View File
@@ -1,6 +1,6 @@
import { describe, expect } from "bun:test"
import { Effect } from "effect"
import { LLM, ProviderPatch, ProviderRequestError, type PreparedRequestOf } from "../../src"
import { LLM, ProviderRequestError, ProviderTransform, type PreparedRequestOf } from "../../src"
import type { AnthropicMessagesPayload } from "../../src/protocols/anthropic-messages"
import { LLMClient } from "../../src/adapter"
import * as AnthropicMessages from "../../src/protocols/anthropic-messages"
@@ -33,7 +33,7 @@ const recorded = recordedTests({
options: { requestHeaders: ["content-type", "anthropic-version"] },
})
const anthropic = LLMClient.make({ adapters: [AnthropicMessages.adapter] })
const anthropicWithPatches = LLMClient.make({ adapters: [AnthropicMessages.adapter], patches: ProviderPatch.defaults })
const anthropicWithTransforms = LLMClient.make({ adapters: [AnthropicMessages.adapter], transforms: ProviderTransform.defaults })
const malformedToolOrderRequest = LLM.request({
id: "recorded_anthropic_malformed_tool_order",
@@ -78,7 +78,7 @@ describe("Anthropic Messages recorded", () => {
}),
)
recorded.effect.with("rejects malformed assistant tool order without patch", { tags: ["tool", "sad-path"] }, () =>
recorded.effect.with("rejects malformed assistant tool order without transform", { tags: ["tool", "sad-path"] }, () =>
Effect.gen(function* () {
const error = yield* anthropic.generate(malformedToolOrderRequest).pipe(Effect.flip)
@@ -88,10 +88,10 @@ describe("Anthropic Messages recorded", () => {
}),
)
recorded.effect.with("accepts malformed assistant tool order with default patch", { tags: ["tool"] }, () =>
recorded.effect.with("accepts malformed assistant tool order with default transform", { tags: ["tool"] }, () =>
Effect.gen(function* () {
const prepared: PreparedRequestOf<AnthropicMessagesPayload> = yield* anthropicWithPatches.prepare<AnthropicMessagesPayload>(malformedToolOrderRequest)
const response = yield* anthropicWithPatches.generate(malformedToolOrderRequest)
const prepared: PreparedRequestOf<AnthropicMessagesPayload> = yield* anthropicWithTransforms.prepare<AnthropicMessagesPayload>(malformedToolOrderRequest)
const response = yield* anthropicWithTransforms.generate(malformedToolOrderRequest)
expect(prepared.payload.messages.slice(0, 2)).toMatchObject([
{ role: "assistant", content: [{ type: "text", text: "I will check the weather." }] },
packages/llm/test/provider/anthropic-messages.test.ts
+1 -1
View File
@@ -2,7 +2,7 @@ import { describe, expect } from "bun:test"
import { Effect, Layer } from "effect"
import { CacheHint, LLM, ProviderRequestError } from "../../src"
import { LLMClient } from "../../src/adapter"
import { AnthropicMessages } from "../../src/protocols/anthropic-messages"
import * as AnthropicMessages from "../../src/protocols/anthropic-messages"
import { testEffect } from "../lib/effect"
import { fixedResponse } from "../lib/http"
import { sseEvents } from "../lib/sse"
packages/llm/test/provider/bedrock-converse.test.ts
+1 -1
View File
@@ -4,7 +4,7 @@ import { describe, expect } from "bun:test"
import { Effect, Layer } from "effect"
import { CacheHint, LLM } from "../../src"
import { LLMClient } from "../../src/adapter"
import { BedrockConverse } from "../../src/protocols/bedrock-converse"
import * as BedrockConverse from "../../src/protocols/bedrock-converse"
import { testEffect } from "../lib/effect"
import { fixedResponse } from "../lib/http"
import { eventSummary, expectWeatherToolLoop, runWeatherToolLoop, weatherTool, weatherToolLoopRequest, weatherToolName } from "../recorded-scenarios"
packages/llm/test/provider/gemini.recorded.test.ts
+1 -1
View File
@@ -2,7 +2,7 @@ import { describe, expect } from "bun:test"
import { Effect } from "effect"
import { LLM } from "../../src"
import { LLMClient } from "../../src/adapter"
import { Gemini } from "../../src/protocols/gemini"
import * as Gemini from "../../src/protocols/gemini"
import { eventSummary, textRequest, weatherToolName, weatherToolRequest } from "../recorded-scenarios"
import { recordedTests } from "../recorded-test"
packages/llm/test/provider/gemini.test.ts
+1 -1
View File
@@ -2,7 +2,7 @@ import { describe, expect } from "bun:test"
import { Effect, Layer } from "effect"
import { LLM, ProviderChunkError } from "../../src"
import { LLMClient } from "../../src/adapter"
import { Gemini } from "../../src/protocols/gemini"
import * as Gemini from "../../src/protocols/gemini"
import { testEffect } from "../lib/effect"
import { fixedResponse } from "../lib/http"
import { sseEvents, sseRaw } from "../lib/sse"
packages/llm/test/provider/openai-chat-tool-loop.recorded.test.ts
+1 -1
View File
@@ -2,7 +2,7 @@ import { describe, expect } from "bun:test"
import { Effect, Stream } from "effect"
import { LLM } from "../../src"
import { LLMClient } from "../../src/adapter"
import { OpenAIChat } from "../../src/protocols/openai-chat"
import * as OpenAIChat from "../../src/protocols/openai-chat"
import { ToolRuntime } from "../../src/tool-runtime"
import { eventSummary, weatherRuntimeTool } from "../recorded-scenarios"
import { recordedTests } from "../recorded-test"
packages/llm/test/provider/openai-chat.recorded.test.ts
+2 -2
View File
@@ -2,7 +2,7 @@ import { describe, expect } from "bun:test"
import { Effect } from "effect"
import { LLM } from "../../src"
import { LLMClient } from "../../src/adapter"
import { OpenAIChat } from "../../src/protocols/openai-chat"
import * as OpenAIChat from "../../src/protocols/openai-chat"
import { eventSummary, textRequest, weatherToolName, weatherToolRequest } from "../recorded-scenarios"
import { recordedTests } from "../recorded-test"
@@ -37,7 +37,7 @@ const recorded = recordedTests({
requires: ["OPENAI_API_KEY"],
})
const openai = LLMClient.make({ adapters: [OpenAIChat.adapter] })
const openaiWithUsage = LLMClient.make({ adapters: [OpenAIChat.adapter.withPatches([OpenAIChat.includeUsage])] })
const openaiWithUsage = LLMClient.make({ adapters: [OpenAIChat.adapter.withTransforms([OpenAIChat.includeUsage])] })
describe("OpenAI Chat recorded", () => {
recorded.effect("streams text", () =>
packages/llm/test/provider/openai-chat.test.ts
+1 -1
View File
@@ -46,7 +46,7 @@ describe("OpenAI Chat adapter", () => {
// typed to the adapter's native shape — the assertions below read field
// names without `unknown` casts.
const prepared = yield* LLMClient.make({
adapters: [OpenAIChat.adapter.withPatches([OpenAIChat.includeUsage])],
adapters: [OpenAIChat.adapter.withTransforms([OpenAIChat.includeUsage])],
}).prepare<OpenAIChat.OpenAIChatPayload>(request)
const _typed: { readonly model: string; readonly stream: true } = prepared.payload
packages/llm/test/provider/openai-compatible-chat.recorded.test.ts
+8 -12
View File
@@ -2,28 +2,26 @@ import { describe, expect } from "bun:test"
import { Effect } from "effect"
import { LLM } from "../../src"
import { LLMClient } from "../../src/adapter"
import { OpenAICompatibleChat } from "../../src/protocols/openai-compatible-chat"
import { OpenRouter } from "../../src/providers/openrouter"
import * as OpenAICompatible from "../../src/providers/openai-compatible"
import * as OpenAICompatibleChat from "../../src/protocols/openai-compatible-chat"
import * as OpenRouter from "../../src/providers/openrouter"
import { expectFinish, expectWeatherToolCall, expectWeatherToolLoop, runWeatherToolLoop, textRequest, weatherToolLoopRequest, weatherToolRequest } from "../recorded-scenarios"
import { recordedTests } from "../recorded-test"
const deepseekModel = OpenAICompatibleChat.deepseek({
id: "deepseek-chat",
const deepseekModel = OpenAICompatible.deepseek.model("deepseek-chat", {
apiKey: process.env.DEEPSEEK_API_KEY ?? "fixture",
})
const deepseekRequest = textRequest({ id: "recorded_deepseek_text", model: deepseekModel })
const togetherModel = OpenAICompatibleChat.togetherai({
id: "meta-llama/Llama-3.3-70B-Instruct-Turbo",
const togetherModel = OpenAICompatible.togetherai.model("meta-llama/Llama-3.3-70B-Instruct-Turbo", {
apiKey: process.env.TOGETHER_AI_API_KEY ?? "fixture",
})
const togetherRequest = textRequest({ id: "recorded_togetherai_text", model: togetherModel })
const togetherToolRequest = weatherToolRequest({ id: "recorded_togetherai_tool_call", model: togetherModel })
const groqModel = OpenAICompatibleChat.groq({
id: "llama-3.3-70b-versatile",
const groqModel = OpenAICompatible.groq.model("llama-3.3-70b-versatile", {
apiKey: process.env.GROQ_API_KEY ?? "fixture",
})
@@ -45,13 +43,11 @@ const openrouterOpus47Model = OpenRouter.model("anthropic/claude-opus-4.7", {
apiKey: process.env.OPENROUTER_API_KEY ?? "fixture",
})
const xaiModel = OpenAICompatibleChat.xai({
id: "grok-3-mini",
const xaiModel = OpenAICompatible.xai.model("grok-3-mini", {
apiKey: process.env.XAI_API_KEY ?? "fixture",
})
const xaiFlagshipModel = OpenAICompatibleChat.xai({
id: "grok-4.3",
const xaiFlagshipModel = OpenAICompatible.xai.model("grok-4.3", {
apiKey: process.env.XAI_API_KEY ?? "fixture",
})
packages/llm/test/provider/openai-compatible-chat.test.ts
+1 -1
View File
@@ -193,7 +193,7 @@ describe("OpenAI-compatible Chat adapter", () => {
it.effect("posts to the configured compatible endpoint and parses text usage", () =>
Effect.gen(function* () {
const response = yield* LLMClient.make({
adapters: [OpenAICompatibleChat.adapter.withPatches([OpenAICompatibleChat.includeUsage])],
adapters: [OpenAICompatibleChat.adapter.withTransforms([OpenAICompatibleChat.includeUsage])],
})
.generate(request)
.pipe(
packages/llm/test/provider/openai-responses.recorded.test.ts
+1 -1
View File
@@ -2,7 +2,7 @@ import { describe, expect } from "bun:test"
import { Effect } from "effect"
import { LLM } from "../../src"
import { LLMClient } from "../../src/adapter"
import { OpenAIResponses } from "../../src/protocols/openai-responses"
import * as OpenAIResponses from "../../src/protocols/openai-responses"
import { expectFinish, expectWeatherToolCall, expectWeatherToolLoop, runWeatherToolLoop, weatherTool, weatherToolLoopRequest, weatherToolName } from "../recorded-scenarios"
import { recordedTests } from "../recorded-test"
packages/llm/test/provider/openai-responses.test.ts
+1 -1
View File
@@ -3,7 +3,7 @@ import { Effect, Layer } from "effect"
import { HttpClientRequest } from "effect/unstable/http"
import { LLM, ProviderRequestError } from "../../src"
import { LLMClient } from "../../src/adapter"
import { OpenAIResponses } from "../../src/protocols/openai-responses"
import * as OpenAIResponses from "../../src/protocols/openai-responses"
import { testEffect } from "../lib/effect"
import { dynamicResponse, fixedResponse } from "../lib/http"
import { sseEvents } from "../lib/sse"
packages/llm/test/patch-pipeline.test.ts → packages/llm/test/transform-pipeline.test.ts
+47 -53
View File
@@ -1,8 +1,8 @@
import { describe, expect, test } from "bun:test"
import { Effect, Schema, Stream } from "effect"
import { LLM } from "../src"
import { Patch } from "../src/patch"
import { PatchPipeline } from "../src/patch-pipeline"
import { Transform } from "../src/transform"
import { TransformPipeline } from "../src/transform-pipeline"
import type { LLMRequest, ModelRef, ToolDefinition } from "../src/schema"
const request = LLM.request({
@@ -36,23 +36,23 @@ const updateToolDefinition = (tool: ToolDefinition, patch: Partial<ToolDefinitio
...patch,
})
describe("llm patch pipeline", () => {
test("patches request, prompt, and tool-schema phases in order", () => {
describe("llm transform pipeline", () => {
test("transforms request, prompt, and tool-schema phases in order", () => {
const result = Effect.runSync(
PatchPipeline.make([
Patch.request("test.id", {
TransformPipeline.make([
Transform.request("test.id", {
reason: "rewrite request id",
apply: (request) => LLM.updateRequest(request, { id: "req_patched" }),
}),
Patch.prompt("test.message", {
Transform.prompt("test.message", {
reason: "rewrite prompt text",
apply: mapText(() => "patched"),
}),
Patch.toolSchema("test.description", {
Transform.toolSchema("test.description", {
reason: "rewrite tool description",
apply: (tool) => updateToolDefinition(tool, { description: "patched tool" }),
}),
]).patchRequest(
]).transformRequest(
LLM.updateRequest(request, {
tools: [{ name: "lookup", description: "original", inputSchema: {} }],
}),
@@ -64,25 +64,25 @@ describe("llm patch pipeline", () => {
expect(result.request.tools[0]?.description).toBe("patched tool")
})
test("prompt predicates see request patches", () => {
test("prompt predicates see request transforms", () => {
const result = Effect.runSync(
PatchPipeline.make([
Patch.request("mark-request", {
TransformPipeline.make([
Transform.request("mark-request", {
reason: "mark request before prompt phase",
apply: (request) => LLM.updateRequest(request, { metadata: { ...request.metadata, promptPatchEnabled: true } }),
}),
Patch.prompt("rewrite-only-when-marked", {
Transform.prompt("rewrite-only-when-marked", {
reason: "rewrite prompt text only after request marker",
when: (ctx) => ctx.request.metadata?.promptPatchEnabled === true,
apply: mapText((text) => `rewrote-${text}`),
}),
]).patchRequest(request),
]).transformRequest(request),
)
expect(result.request.messages[0]?.content).toEqual([{ type: "text", text: "rewrote-hello" }])
})
test("rejects request-shaped patches that change model routing", () => {
test("rejects request-shaped transforms that change model routing", () => {
const changedRoutes = [
{ provider: "other-provider" },
{ id: "other-model" },
@@ -91,39 +91,39 @@ describe("llm patch pipeline", () => {
for (const patch of changedRoutes) {
const error = Effect.runSync(
PatchPipeline.make([
Patch.request("route", {
TransformPipeline.make([
Transform.request("route", {
reason: "attempt to rewrite route",
apply: (request) => LLM.updateRequest(request, { model: updateModel(request.model, patch) }),
}),
]).patchRequest(request).pipe(Effect.flip),
]).transformRequest(request).pipe(Effect.flip),
)
expect(error.message).toContain("Patches cannot change model routing")
expect(error.message).toContain("Transforms cannot change model routing")
}
})
test("skips tool-schema patches when there are no tools", () => {
test("skips tool-schema transforms when there are no tools", () => {
const result = Effect.runSync(
PatchPipeline.make([
Patch.toolSchema("test.description", {
TransformPipeline.make([
Transform.toolSchema("test.description", {
reason: "rewrite tool description",
apply: (tool) => updateToolDefinition(tool, { description: "patched tool" }),
}),
]).patchRequest(request),
]).transformRequest(request),
)
expect(result.request.tools).toEqual([])
})
test("applies tool-schema patches to every tool", () => {
test("applies tool-schema transforms to every tool", () => {
const result = Effect.runSync(
PatchPipeline.make([
Patch.toolSchema("test.description", {
TransformPipeline.make([
Transform.toolSchema("test.description", {
reason: "rewrite tool description",
apply: (tool) => updateToolDefinition(tool, { description: `patched ${tool.name}` }),
}),
]).patchRequest(
]).transformRequest(
LLM.updateRequest(request, {
tools: [
{ name: "first", description: "original", inputSchema: {} },
@@ -136,49 +136,43 @@ describe("llm patch pipeline", () => {
expect(result.request.tools.map((tool) => tool.description)).toEqual(["patched first", "patched second"])
})
test("patches payloads before validation", () => {
const pipeline = PatchPipeline.make([
Patch.payload("client", {
reason: "client payload patch",
order: 2,
apply: (payload: { readonly value: string }) => ({ value: `${payload.value}|client` }),
}),
])
const state = Effect.runSync(pipeline.patchRequest(request))
test("adapter-local payload transforms run before validation", () => {
const pipeline = TransformPipeline.make()
const state = Effect.runSync(pipeline.transformRequest(request))
const result = Effect.runSync(
pipeline.patchPayload({
pipeline.transformPayload({
state,
payload: { value: "start" },
adapterPatches: [
Patch.payload("adapter", {
reason: "adapter payload patch",
adapterTransforms: [
Transform.payload("adapter", {
reason: "adapter payload transform",
order: 1,
apply: (payload: { readonly value: string }) => ({ value: `${payload.value}|adapter` }),
}),
],
schema: Schema.Struct({ value: Schema.Literal("start|adapter|client") }),
schema: Schema.Struct({ value: Schema.Literal("start|adapter") }),
}),
)
expect(result.payload).toEqual({ value: "start|adapter|client" })
expect(result.payload).toEqual({ value: "start|adapter" })
})
test("patches stream events with the compiled request context", () => {
const pipeline = PatchPipeline.make([
Patch.request("mark-request", {
test("transforms stream events with the compiled request context", () => {
const pipeline = TransformPipeline.make([
Transform.request("mark-request", {
reason: "mark request before stream phase",
apply: (request) => LLM.updateRequest(request, { metadata: { ...request.metadata, streamPatchEnabled: true } }),
}),
Patch.stream("uppercase", {
Transform.stream("uppercase", {
reason: "uppercase when compiled request is marked",
when: (ctx) => ctx.request.metadata?.streamPatchEnabled === true,
apply: (event) => (event.type === "text-delta" ? { ...event, text: event.text.toUpperCase() } : event),
}),
])
const patched = Effect.runSync(pipeline.patchRequest(request))
const transformed = Effect.runSync(pipeline.transformRequest(request))
const events = Effect.runSync(
pipeline.patchStreamEvents({
request: patched.request,
pipeline.transformStreamEvents({
request: transformed.request,
events: Stream.fromIterable([{ type: "text-delta", text: "hello" }]),
}).pipe(Stream.runCollect),
)
@@ -186,14 +180,14 @@ describe("llm patch pipeline", () => {
expect(Array.from(events)).toEqual([{ type: "text-delta", text: "HELLO" }])
})
test("accepts a prebuilt patch registry", () => {
test("accepts a prebuilt transform registry", () => {
const result = Effect.runSync(
PatchPipeline.make(Patch.registry([
Patch.prompt("test.message", {
TransformPipeline.make(Transform.registry([
Transform.prompt("test.message", {
reason: "rewrite prompt text",
apply: mapText(() => "patched"),
}),
])).patchRequest(request),
])).transformRequest(request),
)
expect(result.request.messages[0]?.content).toEqual([{ type: "text", text: "patched" }])
packages/llm/test/patch.test.ts → packages/llm/test/transform.test.ts
+31 -32
View File
@@ -1,7 +1,7 @@
import { describe, expect, test } from "bun:test"
import { Effect } from "effect"
import { AnthropicMessages, LLM, LLMClient, OpenAICompatible, OpenAICompatibleChat, ProviderPatch } from "../src"
import { Model, Patch, context, plan } from "../src/patch"
import { AnthropicMessages, LLM, LLMClient, OpenAICompatible, OpenAICompatibleChat, ProviderTransform } from "../src"
import { Model, Transform, context, plan } from "../src/transform"
const request = LLM.request({
id: "req_1",
@@ -13,24 +13,23 @@ const request = LLM.request({
prompt: "hi",
})
describe("llm patch", () => {
describe("llm transform", () => {
test("constructors prefix ids and registry groups by phase", () => {
const prompt = Patch.prompt("mistral.test", {
const prompt = Transform.prompt("mistral.test", {
reason: "test prompt",
when: Model.provider("mistral"),
apply: (request) => request,
})
const payload = Patch.payload("fake.test", {
const payload = Transform.payload("fake.test", {
reason: "test payload",
apply: (draft: { value: number }) => draft,
})
const registry = Patch.registry([prompt, payload])
const registry = Transform.registry([prompt])
expect(prompt.id).toBe("prompt.mistral.test")
expect(payload.id).toBe("payload.fake.test")
expect(registry.prompt).toEqual([prompt])
expect(registry.payload.map((item) => item.id)).toEqual([payload.id])
})
test("predicates compose", () => {
@@ -42,30 +41,30 @@ describe("llm patch", () => {
})
test("plan filters, sorts, and applies deterministically", () => {
const patches = [
Patch.prompt("b", {
const transforms = [
Transform.prompt("b", {
reason: "second alphabetically",
order: 1,
apply: (request) => ({ ...request, metadata: { ...request.metadata, b: true } }),
}),
Patch.prompt("a", {
Transform.prompt("a", {
reason: "first alphabetically",
order: 1,
apply: (request) => ({ ...request, metadata: { ...request.metadata, a: true } }),
}),
Patch.prompt("skip", {
Transform.prompt("skip", {
reason: "not selected",
when: Model.provider("anthropic"),
apply: (request) => ({ ...request, metadata: { ...request.metadata, skip: true } }),
}),
]
const output = plan({ phase: "prompt", context: context({ request }), patches }).apply(request)
const output = plan({ phase: "prompt", context: context({ request }), transforms }).apply(request)
expect(output.metadata).toEqual({ a: true, b: true })
})
test("provider patch examples remove empty Anthropic content", () => {
test("provider transform examples remove empty Anthropic content", () => {
const input = LLM.request({
id: "anthropic_empty",
model: LLM.model({ id: "claude-sonnet", provider: "anthropic", protocol: "anthropic-messages" }),
@@ -78,7 +77,7 @@ describe("llm patch", () => {
const output = plan({
phase: "prompt",
context: context({ request: input }),
patches: [ProviderPatch.removeEmptyAnthropicContent],
transforms: [ProviderTransform.removeEmptyAnthropicContent],
}).apply(input)
expect(output.system).toEqual([])
@@ -86,7 +85,7 @@ describe("llm patch", () => {
expect(output.messages[0]?.content).toEqual([{ type: "text", text: "hello" }])
})
test("provider patch examples scrub model-specific tool call ids", () => {
test("provider transform examples scrub model-specific tool call ids", () => {
const input = LLM.request({
id: "mistral_tool_ids",
model: LLM.model({ id: "devstral-small", provider: "mistral", protocol: "openai-chat" }),
@@ -98,7 +97,7 @@ describe("llm patch", () => {
const output = plan({
phase: "prompt",
context: context({ request: input }),
patches: [ProviderPatch.scrubMistralToolIds],
transforms: [ProviderTransform.scrubMistralToolIds],
}).apply(input)
expect(output.messages[0]?.content[0]).toMatchObject({ type: "tool-call", id: "callbadva" })
@@ -119,7 +118,7 @@ describe("llm patch", () => {
const output = plan({
phase: "prompt",
context: context({ request: input }),
patches: [ProviderPatch.repairAnthropicToolUseOrder],
transforms: [ProviderTransform.repairAnthropicToolUseOrder],
}).apply(input)
expect(output.messages).toHaveLength(2)
@@ -139,7 +138,7 @@ describe("llm patch", () => {
const output = plan({
phase: "prompt",
context: context({ request: input }),
patches: [ProviderPatch.repairMistralToolResultUserSequence],
transforms: [ProviderTransform.repairMistralToolResultUserSequence],
}).apply(input)
expect(output.messages.map((message) => message.role)).toEqual(["tool", "assistant", "user"])
@@ -155,7 +154,7 @@ describe("llm patch", () => {
const output = plan({
phase: "prompt",
context: context({ request: input }),
patches: [ProviderPatch.addDeepSeekEmptyReasoning],
transforms: [ProviderTransform.addDeepSeekEmptyReasoning],
}).apply(input)
expect(output.messages[0]?.content).toEqual([{ type: "text", text: "answer" }])
@@ -173,7 +172,7 @@ describe("llm patch", () => {
const output = plan({
phase: "prompt",
context: context({ request: input }),
patches: [ProviderPatch.unsupportedMediaFallback],
transforms: [ProviderTransform.unsupportedMediaFallback],
}).apply(input)
expect(output.messages[0]?.content).toEqual([
@@ -205,7 +204,7 @@ describe("llm patch", () => {
const output = plan({
phase: "tool-schema",
context: context({ request: input }),
patches: [ProviderPatch.sanitizeMoonshotToolSchema],
transforms: [ProviderTransform.sanitizeMoonshotToolSchema],
}).apply(input.tools[0])
expect(output.inputSchema.properties).toEqual({
@@ -214,9 +213,9 @@ describe("llm patch", () => {
})
})
test("default patches compile invalid Anthropic tool-use ordering into valid payload order", () => {
test("default transforms compile invalid Anthropic tool-use ordering into valid payload order", () => {
const prepared = Effect.runSync(
LLMClient.make({ adapters: [AnthropicMessages.adapter], patches: ProviderPatch.defaults }).prepare(
LLMClient.make({ adapters: [AnthropicMessages.adapter], transforms: ProviderTransform.defaults }).prepare(
LLM.request({
id: "anthropic_default_tool_order",
model: AnthropicMessages.model({ id: "claude-sonnet" }),
@@ -238,9 +237,9 @@ describe("llm patch", () => {
})
})
test("default patches compile DeepSeek reasoning replay into OpenAI-compatible native field", () => {
test("default transforms compile DeepSeek reasoning replay into OpenAI-compatible native field", () => {
const prepared = Effect.runSync(
LLMClient.make({ adapters: [OpenAICompatibleChat.adapter], patches: ProviderPatch.defaults }).prepare(
LLMClient.make({ adapters: [OpenAICompatibleChat.adapter], transforms: ProviderTransform.defaults }).prepare(
LLM.request({
id: "deepseek_default_reasoning",
model: OpenAICompatible.deepseek.model("deepseek-reasoner"),
@@ -266,11 +265,11 @@ describe("llm patch", () => {
capabilities: LLM.capabilities({ cache: { prompt: true, contentBlocks: true } }),
})
const runCachePatch = (input: ReturnType<typeof LLM.request>) =>
const runCacheTransform = (input: ReturnType<typeof LLM.request>) =>
plan({
phase: "prompt",
context: context({ request: input }),
patches: [ProviderPatch.cachePromptHints],
transforms: [ProviderTransform.cachePromptHints],
}).apply(input)
test("marks first 2 system parts with an ephemeral cache hint", () => {
@@ -280,7 +279,7 @@ describe("llm patch", () => {
system: ["First", "Second", "Third"].map(LLM.system),
prompt: "hello",
})
const output = runCachePatch(input)
const output = runCacheTransform(input)
expect(output.system).toHaveLength(3)
expect(output.system[0]).toMatchObject({ text: "First", cache: { type: "ephemeral" } })
@@ -299,7 +298,7 @@ describe("llm patch", () => {
LLM.user([{ type: "text", text: "m2" }]),
],
})
const output = runCachePatch(input)
const output = runCacheTransform(input)
expect(output.messages).toHaveLength(3)
// First message untouched.
@@ -322,7 +321,7 @@ describe("llm patch", () => {
]),
],
})
const output = runCachePatch(input)
const output = runCacheTransform(input)
const content = output.messages[0].content
expect(content[0]).toMatchObject({ type: "text", text: "calling tool", cache: { type: "ephemeral" } })
@@ -337,7 +336,7 @@ describe("llm patch", () => {
LLM.toolMessage({ id: "call_1", name: "lookup", result: { ok: true } }),
],
})
const output = runCachePatch(input)
const output = runCacheTransform(input)
expect(output.messages[0].content[0]).toMatchObject({ type: "tool-result", id: "call_1" })
// No text part to mark, so the content array is identity-equal — the
@@ -357,7 +356,7 @@ describe("llm patch", () => {
system: ["A", "B"].map(LLM.system),
messages: [LLM.user([{ type: "text", text: "hi" }])],
})
const output = runCachePatch(input)
const output = runCacheTransform(input)
// Every text part should be free of cache hints.
for (const part of output.system) expect(part.cache).toBeUndefined()
packages/opencode/src/provider/llm-bridge.ts
+2 -3
View File
@@ -7,7 +7,6 @@ import {
LLM,
OpenAI,
OpenAICompatible,
OpenAICompatibleChat,
OpenAICompatibleProfiles,
ReasoningEfforts,
XAI,
@@ -124,11 +123,11 @@ const openAICompatibleModel: ProviderModel = (input, options) => {
const resolvedBaseURL = baseURL(input, options, profile?.baseURL)
if (!resolvedBaseURL) return undefined
const modelOptions = sharedOptions(input, options, {
protocol: "openai-compatible-chat",
protocol: "openai-chat",
baseURL: resolvedBaseURL,
capabilities: profile?.capabilities,
})
if (profile) return OpenAICompatibleChat.profileModel(profile, { ...modelOptions, id: String(input.model.api.id) })
if (profile) return OpenAICompatible.profileModel(profile, String(input.model.api.id), modelOptions)
return OpenAICompatible.model(String(input.model.api.id), { ...modelOptions, provider, baseURL: resolvedBaseURL })
}
packages/opencode/src/provider/provider.ts
+1 -1
View File
@@ -25,7 +25,7 @@ import { InstanceState } from "@/effect/instance-state"
import { AppFileSystem } from "@opencode-ai/core/filesystem"
import { isRecord } from "@/util/record"
import { optionalOmitUndefined, withStatics } from "@/util/schema"
import { GitHubCopilot } from "@opencode-ai/llm/providers/github-copilot"
import * as GitHubCopilot from "@opencode-ai/llm/providers/github-copilot"
import * as ProviderTransform from "./transform"
import { ModelID, ProviderID } from "./schema"
packages/opencode/src/session/llm-native.ts
+3 -3
View File
@@ -239,9 +239,9 @@ export const request = Effect.fn("LLMNative.request")(function* (input: RequestI
}
const headers = { ...model.headers, ...input.headers }
const requestModel = Object.keys(headers).length === 0 ? model : LLM.model({ ...model, headers })
// Cache hints, tool-id scrubbing, and other adapter-aware patches live in
// `@opencode-ai/llm`'s `ProviderPatch` registry. Callers wire them in at
// `client({ adapters, patches: ProviderPatch.defaults })` time so the
// Cache hints, tool-id scrubbing, and other adapter-aware transforms live in
// `@opencode-ai/llm`'s `ProviderTransform` registry. Callers wire them in at
// `client({ adapters, transforms: ProviderTransform.defaults })` time so the
// bridge stays focused on shape conversion.
return LLM.request({
id: input.id,
packages/opencode/src/session/llm.ts
+2 -2
View File
@@ -14,7 +14,7 @@ import {
OpenAIChat,
OpenAICompatibleChat,
OpenAIResponses,
ProviderPatch,
ProviderTransform as LLMProviderTransform,
RequestExecutor,
type ProtocolID,
} from "@opencode-ai/llm"
@@ -509,7 +509,7 @@ const live: Layer.Layer<
const nativeClient = LLMClient.make({
adapters: NATIVE_ADAPTERS,
patches: ProviderPatch.defaults,
transforms: LLMProviderTransform.defaults,
})
const runNative = Effect.fn("LLM.runNative")(function* (input: StreamRequest, prepared: PreparedStream) {
packages/opencode/test/provider/llm-bridge.test.ts
+2 -2
View File
@@ -100,7 +100,7 @@ describe("ProviderLLMBridge", () => {
expect(ref).toMatchObject({
id: "meta-llama/Llama-3.3-70B-Instruct-Turbo",
provider: "togetherai",
protocol: "openai-compatible-chat",
protocol: "openai-chat",
baseURL: "https://api.together.xyz/v1",
apiKey: "together-key",
})
@@ -171,7 +171,7 @@ describe("ProviderLLMBridge", () => {
})
expect(ref).toMatchObject({
protocol: "openai-compatible-chat",
protocol: "openai-chat",
baseURL: "https://custom.cerebras.test/v1",
apiKey: "cerebras-key",
headers: {
packages/opencode/test/session/llm-native-stream.test.ts
+4 -4
View File
@@ -7,7 +7,7 @@ import {
OpenAIChat,
OpenAICompatibleChat,
OpenAIResponses,
ProviderPatch,
ProviderTransform,
RequestExecutor,
} from "@opencode-ai/llm"
import { Effect, Layer, Ref, Schema, Stream } from "effect"
@@ -127,7 +127,7 @@ describe("LLMNative stream wire-up (audit gap #4 phase 1)", () => {
messages: [userMessage(mdl, userID, [userPart(userID, "Say hello.")])],
})
const client = LLMClient.make({ adapters, patches: ProviderPatch.defaults })
const client = LLMClient.make({ adapters, transforms: ProviderTransform.defaults })
const map = LLMNativeEvents.mapper()
const body = sseBody([
@@ -245,7 +245,7 @@ describe("LLMNative stream wire-up (audit gap #4 phase 1)", () => {
{ type: "message_stop" },
])
const client = LLMClient.make({ adapters, patches: ProviderPatch.defaults })
const client = LLMClient.make({ adapters, transforms: ProviderTransform.defaults })
const map = LLMNativeEvents.mapper()
const events = yield* LLMNativeTools.runWithTools({
@@ -322,7 +322,7 @@ describe("LLMNative stream wire-up (audit gap #4 phase 1)", () => {
tools: [lookupTool],
})
const prepared = yield* LLMClient.make({ adapters, patches: ProviderPatch.defaults }).prepare(llmRequest)
const prepared = yield* LLMClient.make({ adapters, transforms: ProviderTransform.defaults }).prepare(llmRequest)
expect(prepared.payload).toMatchObject({
tools: [
{
packages/opencode/test/session/llm-native.test.ts
+7 -7
View File
@@ -1,5 +1,5 @@
import { describe, expect } from "bun:test"
import { AnthropicMessages, BedrockConverse, Gemini, LLMClient, OpenAICompatibleChat, OpenAIResponses, ProviderPatch } from "@opencode-ai/llm"
import { AnthropicMessages, BedrockConverse, Gemini, LLMClient, OpenAICompatibleChat, OpenAIResponses, ProviderTransform } from "@opencode-ai/llm"
import { Cause, Effect, Exit, Layer, Schema } from "effect"
import { ModelID, ProviderID } from "../../src/provider/schema"
import { LLMNative } from "../../src/session/llm-native"
@@ -894,7 +894,7 @@ describe("LLMNative.request", () => {
}))
// Cache hint policy. The bridge produces a hint-free `LLMRequest`; the
// `ProviderPatch.cachePromptHints` patch (loaded in `ProviderPatch.defaults`)
// `ProviderTransform.cachePromptHints` transform (loaded in `ProviderTransform.defaults`)
// marks first-2 system parts and last-2 messages with ephemeral cache
// hints when the model advertises `capabilities.cache.prompt`. Adapters
// then lower the hints to the provider-specific marker — `cache_control`
@@ -931,7 +931,7 @@ describe("LLMNative.request", () => {
})
const prepared = yield* LLMClient.make({
adapters: [AnthropicMessages.adapter],
patches: ProviderPatch.defaults,
transforms: ProviderTransform.defaults,
}).prepare(request)
expect(prepared.payload).toMatchObject({
@@ -956,7 +956,7 @@ describe("LLMNative.request", () => {
})
const prepared = yield* LLMClient.make({
adapters: [AnthropicMessages.adapter],
patches: ProviderPatch.defaults,
transforms: ProviderTransform.defaults,
}).prepare(request)
expect(prepared.payload).toMatchObject({
@@ -983,7 +983,7 @@ describe("LLMNative.request", () => {
})
const prepared = yield* LLMClient.make({
adapters: [BedrockConverse.adapter],
patches: ProviderPatch.defaults,
transforms: ProviderTransform.defaults,
}).prepare(request)
expect(prepared.payload).toMatchObject({
@@ -1011,7 +1011,7 @@ describe("LLMNative.request", () => {
})
const prepared = yield* LLMClient.make({
adapters: [OpenAIResponses.adapter],
patches: ProviderPatch.defaults,
transforms: ProviderTransform.defaults,
}).prepare(request)
// The serialized OpenAI Responses payload has no cache concept; the
@@ -1090,7 +1090,7 @@ describe("LLMNative.request", () => {
})
const prepared = yield* LLMClient.make({
adapters: [AnthropicMessages.adapter],
patches: ProviderPatch.defaults,
transforms: ProviderTransform.defaults,
}).prepare(request)
expect(prepared.payload).toMatchObject({