refactor(llm): streamline adapter model handles
This commit is contained in:
Kit Langton
@@ -42,7 +42,7 @@ const program = Effect.gen(function* () {
|
||||
console.log(response.text)
|
||||
}).pipe(
|
||||
Effect.provide(Layer.mergeAll(
|
||||
LLM.layer({ providers: [OpenAI] }),
|
||||
LLM.layer(),
|
||||
RequestExecutor.defaultLayer,
|
||||
)),
|
||||
)
|
||||
@@ -51,7 +51,7 @@ const program = Effect.gen(function* () {
|
||||
The public rule is:
|
||||
|
||||
```txt
|
||||
provider helper -> model reference -> LLM.generate / LLM.stream
|
||||
provider helper -> model handle -> LLM.generate / LLM.stream
|
||||
```
|
||||
|
||||
Provider helpers should feel boring at use sites.
|
||||
@@ -103,7 +103,7 @@ This split is the core design choice.
|
||||
| Concept | Question it answers |
|
||||
| --- | --- |
|
||||
| `provider` | Who is the deployment or product surface? |
|
||||
| `protocol` | Which request/response shape should the runtime use? |
|
||||
| `protocol` | Which request/response shape should the runtime use? This is an open string so custom providers can add new protocol ids. |
|
||||
| `id` | Which model/deployment id should be sent? |
|
||||
| `baseURL` | Where should HTTP go? |
|
||||
| `apiKey`, `headers`, `queryParams`, `native` | What deployment-specific transport data is needed? |
|
||||
@@ -129,7 +129,7 @@ type ModelRef = {
|
||||
}
|
||||
```
|
||||
|
||||
`ModelRef` is not a provider client. It does not send requests. It is the stable, serializable description of what should be called.
|
||||
`ModelRef` is the stable, serializable description of what should be called. Provider helpers also bind an in-memory adapter to the returned model handle so direct call sites do not need to manually register adapters; serialized copies fall back to `model.protocol` registry lookup.
|
||||
</details>
|
||||
|
||||
## Terrace 3: Follow A Request
|
||||
@@ -140,7 +140,7 @@ At runtime, the flow is a staircase.
|
||||
LLM.generate({ model, prompt })
|
||||
-> LLM.request(...)
|
||||
-> LLMClient
|
||||
-> adapter selected by model.protocol
|
||||
-> adapter from the model handle, or explicit registry fallback
|
||||
-> provider-native target payload
|
||||
-> HttpClientRequest
|
||||
-> RequestExecutor
|
||||
@@ -167,7 +167,7 @@ const request = LLM.request({
|
||||
})
|
||||
|
||||
const client = LLMClient.make({
|
||||
adapters: [OpenAIResponses.adapter, OpenAIChat.adapter],
|
||||
adapters: [],
|
||||
patches: ProviderPatch.defaults,
|
||||
})
|
||||
|
||||
@@ -177,10 +177,10 @@ const response = yield* client.generate(request)
|
||||
<details>
|
||||
<summary>Adapter pipeline</summary>
|
||||
|
||||
The adapter is selected by `request.model.protocol`.
|
||||
Explicit adapters passed to `LLMClient.make(...)` win first. If no explicit adapter matches, the adapter bound to the in-memory model handle is used. If the model was serialized and revived, `LLMClient` falls back to the explicit registry keyed by `request.model.protocol`.
|
||||
|
||||
```ts
|
||||
const adapter = adapters.get(request.model.protocol)
|
||||
const adapter = adapters.get(request.model.protocol) ?? modelAdapters.get(request.model)
|
||||
const candidate = adapter.prepare(request)
|
||||
const patched = applyTargetPatches(candidate)
|
||||
const target = adapter.validate(patched)
|
||||
@@ -196,22 +196,22 @@ const events = adapter.parse(response)
|
||||
|
||||
Keeping the current names, an `Adapter` is the runnable implementation for one registered request route.
|
||||
|
||||
It is selected by `model.protocol`, not by `model.provider`.
|
||||
It is selected from the model handle when the provider helper created the model in the same process. Explicit adapter registration overrides that default and remains the fallback for revived models, OpenCode config bridges, and low-level tests.
|
||||
|
||||
```ts
|
||||
const adapters = new Map(
|
||||
options.adapters.map((source) => [source.runtime.protocol, source.runtime] as const),
|
||||
options.adapters.map((adapter) => [adapter.protocol, adapter] as const),
|
||||
)
|
||||
|
||||
const adapter = adapters.get(request.model.protocol)
|
||||
const adapter = adapters.get(request.model.protocol) ?? modelAdapters.get(request.model)
|
||||
```
|
||||
|
||||
That means `protocol` currently has two jobs:
|
||||
That means `protocol` has two jobs only in fallback paths:
|
||||
|
||||
| Job | Example |
|
||||
| --- | --- |
|
||||
| Describes the wire API shape | `openai-responses`, `anthropic-messages`, `gemini`. |
|
||||
| Selects the runtime adapter | `LLMClient` looks up `adapters.get(request.model.protocol)`. |
|
||||
| Selects the adapter after serialization | `LLMClient` looks up `adapters.get(request.model.protocol)`. |
|
||||
|
||||
The adapter then owns the full compile/run boundary for that selected route.
|
||||
|
||||
@@ -241,7 +241,7 @@ So the current relationship is:
|
||||
|
||||
```txt
|
||||
ModelRef.protocol
|
||||
-> selects Adapter
|
||||
-> selects Adapter after serialization / registry lookup
|
||||
-> Adapter composes Protocol + Endpoint + Auth + Framing
|
||||
-> Adapter compiles the request and parses the response
|
||||
```
|
||||
@@ -269,7 +269,7 @@ Provider behavior is split across reusable layers instead of one large provider
|
||||
|
||||
```txt
|
||||
Provider helper
|
||||
creates ModelRef values
|
||||
creates model handles backed by ModelRef values
|
||||
|
||||
Provider module
|
||||
exports adapters and helper constructors
|
||||
@@ -325,8 +325,8 @@ OpenAICompatible.model("gpt-4o-mini", { provider: "local-gateway", baseURL })
|
||||
|
||||
| Layer | Owns |
|
||||
| --- | --- |
|
||||
| Provider helper | Public constructor, defaults, provider identity, model capabilities, limits. |
|
||||
| Provider module | Exported adapters and helpers passed to `LLM.layer({ providers })`. |
|
||||
| Provider helper | Public constructor, defaults, provider identity, model capabilities, limits, in-process adapter binding. |
|
||||
| Provider module | Exported adapters and helpers for explicit registry fallback. |
|
||||
| Adapter | Runtime registration and composition. |
|
||||
| Protocol | Request lowering, target schema, chunk schema, stream state machine. |
|
||||
| Endpoint | URL construction, base URL, path, query params, deployment routing. |
|
||||
@@ -394,9 +394,34 @@ The difference is below the public API.
|
||||
|
||||
| Concern | AI SDK | This package |
|
||||
| --- | --- | --- |
|
||||
| Use site | Provider creates runnable model object. | Provider creates `ModelRef`; `LLM` runtime runs it. |
|
||||
| Use site | Provider creates runnable model object. | Provider creates a runnable model handle backed by serializable `ModelRef`. |
|
||||
| Provider implementation | Usually provider-package-specific language model classes. | Protocol, endpoint, auth, framing, and patches are separate axes. |
|
||||
| OpenAI-compatible reuse | Dedicated OpenAI-compatible implementation. | Reuses `OpenAIChat.protocol` with different deployment axes. |
|
||||
| Debug/replay/parity | Mostly hidden behind provider implementation. | Exposed through request lowering, patches, adapters, and events. |
|
||||
|
||||
The tradeoff is intentional. The public API should feel small. The internals should be inspectable enough for OpenCode to preserve provider parity, replay HTTP, diff native payloads, and migrate provider-by-provider without cloning whole adapter classes.
|
||||
|
||||
### OpenCode Provider Loading
|
||||
|
||||
OpenCode's current AI SDK path is more dynamic than this package's native path.
|
||||
|
||||
```txt
|
||||
OpenCode config/models.dev
|
||||
-> model.api.npm
|
||||
-> import or install AI SDK provider package
|
||||
-> create provider SDK
|
||||
-> sdk.languageModel(...) / sdk.responses(...) / sdk.chat(...)
|
||||
```
|
||||
|
||||
That is why OpenCode can point at many AI SDK provider packages without this repo shipping a native adapter for each one.
|
||||
|
||||
The `@opencode-ai/llm` native path currently works in two modes:
|
||||
|
||||
| Mode | How it works | Good for |
|
||||
| --- | --- | --- |
|
||||
| In-process model helper | `OpenAI.model(...)`, `OpenAICompatible.model(...)`, or a third-party helper returns a model handle bound to an adapter. | Library users and code that imports the provider package directly. |
|
||||
| Explicit adapter registry | `LLMClient.make({ adapters: [...] })` maps revived `ModelRef.protocol` values to shipped adapters. | OpenCode config/models.dev bridges, tests, request replay, serialized models. |
|
||||
|
||||
So OpenCode native integration is not “import any AI SDK provider package and it just works” yet. Today it supports the protocols/providers we can resolve to known native adapters, plus generic OpenAI-compatible deployments. A config-defined provider with `@ai-sdk/openai-compatible` can resolve to `openai-compatible-chat`; a brand-new protocol needs a native adapter and resolver mapping.
|
||||
|
||||
The core package is now open enough for external protocols: `ProtocolID` is just a string, so a third-party package can define `Protocol.define(...)`, `Adapter.fromProtocol(...)`, and a model helper without changing this package. To make OpenCode load those from config the same way it loads AI SDK packages, we would add an explicit native-provider loader/registry analogous to the AI SDK `model.api.npm` loader.
|
||||
|
||||
@@ -96,10 +96,9 @@ const FakeTarget = Schema.Struct({
|
||||
type FakeTarget = Schema.Schema.Type<typeof FakeTarget>
|
||||
|
||||
const FakeProtocol = Protocol.define<FakeTarget, string, string, void>({
|
||||
// ProtocolID is a closed union in this package. A real new provider protocol
|
||||
// would add its own id there; this tutorial reuses `openai-chat` so the fake
|
||||
// provider can compile without changing production protocol ids.
|
||||
id: "openai-chat",
|
||||
// Protocol ids are open strings, so external packages can define their own
|
||||
// protocols without changing this package.
|
||||
id: "fake-echo",
|
||||
target: FakeTarget,
|
||||
prepare: (request) =>
|
||||
Effect.succeed({
|
||||
@@ -129,16 +128,19 @@ const FakeAdapter = Adapter.fromProtocol({
|
||||
framing: Framing.sse,
|
||||
})
|
||||
|
||||
// A provider module exports adapters plus model helpers. The model helper sets
|
||||
// provider identity and the protocol id used for adapter lookup.
|
||||
// A provider module exports a model helper. The model helper sets provider
|
||||
// identity, protocol id, and the adapter that can run this in-memory model
|
||||
// handle. Serialized / revived models can still use explicit provider adapters.
|
||||
const FakeEcho = {
|
||||
adapters: [FakeAdapter],
|
||||
model: (id: string) =>
|
||||
LLM.model({
|
||||
id,
|
||||
provider: "fake-echo",
|
||||
protocol: "openai-chat",
|
||||
}),
|
||||
Adapter.bindModel(
|
||||
LLM.model({
|
||||
id,
|
||||
provider: "fake-echo",
|
||||
protocol: "fake-echo",
|
||||
}),
|
||||
FakeAdapter,
|
||||
),
|
||||
}
|
||||
|
||||
// `prepare` compiles through patches, protocol lowering, validation, endpoint,
|
||||
@@ -152,7 +154,7 @@ const inspectFakeProvider = Effect.gen(function* () {
|
||||
console.log("\n== fake provider prepare ==")
|
||||
console.log("adapter:", prepared.adapter)
|
||||
console.log("target:", Formatter.formatJson(prepared.target, { space: 2 }))
|
||||
}).pipe(Effect.provide(LLM.layer({ providers: [FakeEcho] })))
|
||||
}).pipe(Effect.provide(LLM.layer()))
|
||||
|
||||
// Provide the LLM runtime and the HTTP request executor once. The default path
|
||||
// sends one live generate call and one local fake-provider prepare call.
|
||||
@@ -163,6 +165,6 @@ const program = Effect.gen(function* () {
|
||||
yield* inspectFakeProvider
|
||||
// yield* streamText
|
||||
// yield* streamWithTools
|
||||
}).pipe(Effect.provide(Layer.mergeAll(LLM.layer({ providers: [OpenAI] }), RequestExecutor.defaultLayer)))
|
||||
}).pipe(Effect.provide(Layer.mergeAll(LLM.layer(), RequestExecutor.defaultLayer)))
|
||||
|
||||
Effect.runPromise(program)
|
||||
|
||||
+39
-45
@@ -21,49 +21,50 @@ import type {
|
||||
} from "./schema"
|
||||
import { LLMRequest as LLMRequestSchema, LLMResponse, NoAdapterError, PreparedRequest as PreparedRequestSchema } from "./schema"
|
||||
|
||||
interface RuntimeAdapter {
|
||||
readonly id: string
|
||||
readonly protocol: ProtocolID
|
||||
readonly patches: ReadonlyArray<Patch<unknown>>
|
||||
readonly prepare: (request: LLMRequest) => Effect.Effect<unknown, LLMError>
|
||||
readonly validate: (draft: unknown) => Effect.Effect<unknown, LLMError>
|
||||
readonly toHttp: (target: unknown, context: HttpContext) => Effect.Effect<HttpClientRequest.HttpClientRequest, LLMError>
|
||||
readonly parse: (response: HttpClientResponse.HttpClientResponse, context: HttpContext) => Stream.Stream<LLMEvent, LLMError>
|
||||
}
|
||||
|
||||
interface RuntimeAdapterSource {
|
||||
readonly runtime: RuntimeAdapter
|
||||
}
|
||||
|
||||
export interface HttpContext {
|
||||
readonly request: LLMRequest
|
||||
readonly patchTrace: ReadonlyArray<PatchTrace>
|
||||
}
|
||||
|
||||
export interface Adapter<Draft, Target> {
|
||||
export interface Adapter<Target> {
|
||||
readonly id: string
|
||||
readonly protocol: ProtocolID
|
||||
readonly patches: ReadonlyArray<Patch<Draft>>
|
||||
readonly prepare: (request: LLMRequest) => Effect.Effect<Draft, LLMError>
|
||||
readonly validate: (draft: Draft) => Effect.Effect<Target, LLMError>
|
||||
readonly patches: ReadonlyArray<Patch<Target>>
|
||||
readonly prepare: (request: LLMRequest) => Effect.Effect<Target, LLMError>
|
||||
readonly validate: (target: Target) => Effect.Effect<Target, LLMError>
|
||||
readonly toHttp: (target: Target, context: HttpContext) => Effect.Effect<HttpClientRequest.HttpClientRequest, LLMError>
|
||||
readonly parse: (response: HttpClientResponse.HttpClientResponse, context: HttpContext) => Stream.Stream<LLMEvent, LLMError>
|
||||
}
|
||||
|
||||
export interface AdapterInput<Draft, Target> {
|
||||
readonly id: string
|
||||
readonly protocol: ProtocolID
|
||||
readonly patches?: ReadonlyArray<Patch<Draft>>
|
||||
readonly prepare: (request: LLMRequest) => Effect.Effect<Draft, LLMError>
|
||||
readonly validate: (draft: Draft) => Effect.Effect<Target, LLMError>
|
||||
readonly toHttp: (target: Target, context: HttpContext) => Effect.Effect<HttpClientRequest.HttpClientRequest, LLMError>
|
||||
readonly parse: (response: HttpClientResponse.HttpClientResponse, context: HttpContext) => Stream.Stream<LLMEvent, LLMError>
|
||||
export type AdapterInput<Target> = Omit<Adapter<Target>, "patches"> & {
|
||||
readonly patches?: ReadonlyArray<Patch<Target>>
|
||||
}
|
||||
|
||||
export interface AdapterDefinition<Draft, Target> extends Adapter<Draft, Target> {
|
||||
readonly runtime: RuntimeAdapter
|
||||
readonly patch: (id: string, input: PatchInput<Draft>) => Patch<Draft>
|
||||
readonly withPatches: (patches: ReadonlyArray<Patch<Draft>>) => AdapterDefinition<Draft, Target>
|
||||
export interface AdapterDefinition<Target> extends Adapter<Target> {
|
||||
readonly patch: (id: string, input: PatchInput<Target>) => Patch<Target>
|
||||
readonly withPatches: (patches: ReadonlyArray<Patch<Target>>) => AdapterDefinition<Target>
|
||||
}
|
||||
|
||||
// Adapter registries intentionally erase target generics after the typed
|
||||
// adapter is constructed. This keeps normal call sites on `OpenAIChat.adapter`
|
||||
// instead of leaking a separate runtime-adapter wrapper.
|
||||
// oxlint-disable-next-line typescript-eslint/no-explicit-any
|
||||
export type AnyAdapter = AdapterDefinition<any>
|
||||
|
||||
const modelAdapters = new WeakMap<ModelRef, AnyAdapter>()
|
||||
|
||||
export const bindModel = <Model extends ModelRef>(model: Model, adapter: AnyAdapter): Model => {
|
||||
if (model.protocol !== adapter.protocol) {
|
||||
throw new Error(`Cannot bind ${adapter.id} adapter (${adapter.protocol}) to ${model.provider}/${model.id} (${model.protocol})`)
|
||||
}
|
||||
modelAdapters.set(model, adapter)
|
||||
return model
|
||||
}
|
||||
|
||||
export const preserveModelBinding = <Model extends ModelRef>(source: ModelRef, target: Model): Model => {
|
||||
const adapter = modelAdapters.get(source)
|
||||
if (!adapter) return target
|
||||
return bindModel(target, adapter)
|
||||
}
|
||||
|
||||
export interface LLMClient {
|
||||
@@ -85,7 +86,7 @@ export interface LLMClient {
|
||||
}
|
||||
|
||||
export interface ClientOptions {
|
||||
readonly adapters: ReadonlyArray<RuntimeAdapterSource>
|
||||
readonly adapters?: ReadonlyArray<AnyAdapter>
|
||||
readonly patches?: PatchRegistry | ReadonlyArray<AnyPatch>
|
||||
}
|
||||
|
||||
@@ -108,16 +109,11 @@ const normalizeRegistry = (patches: PatchRegistry | ReadonlyArray<AnyPatch> | un
|
||||
* canonical path is `Adapter.fromProtocol(...)`. New adapters should start
|
||||
* there and prove they need otherwise before reaching for this.
|
||||
*/
|
||||
export function unsafe<Draft, Target>(input: AdapterInput<Draft, Target>): AdapterDefinition<Draft, Target> {
|
||||
const build = (patches: ReadonlyArray<Patch<Draft>>): AdapterDefinition<Draft, Target> => ({
|
||||
export function unsafe<Target>(input: AdapterInput<Target>): AdapterDefinition<Target> {
|
||||
const build = (patches: ReadonlyArray<Patch<Target>>): AdapterDefinition<Target> => ({
|
||||
id: input.id,
|
||||
protocol: input.protocol,
|
||||
patches,
|
||||
get runtime() {
|
||||
// Runtime registry erases adapter draft/target generics after validation.
|
||||
// oxlint-disable-next-line typescript-eslint/no-unsafe-type-assertion
|
||||
return this as unknown as RuntimeAdapter
|
||||
},
|
||||
prepare: input.prepare,
|
||||
validate: input.validate,
|
||||
toHttp: input.toHttp,
|
||||
@@ -175,7 +171,7 @@ export interface FromProtocolInput<Target, Frame, Chunk, State> {
|
||||
*/
|
||||
export function fromProtocol<Target, Frame, Chunk, State>(
|
||||
input: FromProtocolInput<Target, Frame, Chunk, State>,
|
||||
): AdapterDefinition<Target, Target> {
|
||||
): AdapterDefinition<Target> {
|
||||
const auth = input.auth ?? authBearer
|
||||
const protocol = input.protocol
|
||||
const validateTarget = ProviderShared.validateWith(Schema.decodeUnknownEffect(protocol.target))
|
||||
@@ -233,12 +229,10 @@ export function fromProtocol<Target, Frame, Chunk, State>(
|
||||
|
||||
const makeClient = (options: ClientOptions): LLMClient => {
|
||||
const registry = normalizeRegistry(options.patches)
|
||||
const adapters = new Map(
|
||||
options.adapters.map((source) => [source.runtime.protocol, source.runtime] as const),
|
||||
)
|
||||
const adapters = new Map((options.adapters ?? []).map((adapter) => [adapter.protocol, adapter] as const))
|
||||
|
||||
const compile = Effect.fn("LLM.compile")(function* (request: LLMRequest) {
|
||||
const adapter = adapters.get(request.model.protocol)
|
||||
const adapter = adapters.get(request.model.protocol) ?? modelAdapters.get(request.model)
|
||||
if (!adapter) return yield* noAdapter(request.model)
|
||||
|
||||
const requestPlan = plan({
|
||||
@@ -266,13 +260,13 @@ const makeClient = (options: ClientOptions): LLMClient => {
|
||||
tools: requestBeforeToolPatches.tools.map(toolSchemaPlan.apply),
|
||||
})
|
||||
const patchContext = context({ request: patchedRequest })
|
||||
const draft = yield* adapter.prepare(patchedRequest)
|
||||
const candidate = yield* adapter.prepare(patchedRequest)
|
||||
const targetPlan = plan({
|
||||
phase: "target",
|
||||
context: patchContext,
|
||||
patches: [...adapter.patches, ...registry.target],
|
||||
})
|
||||
const target = yield* adapter.validate(targetPlan.apply(draft))
|
||||
const target = yield* adapter.validate(targetPlan.apply(candidate))
|
||||
const targetPatchTrace = [
|
||||
...requestPlan.trace,
|
||||
...promptPlan.trace,
|
||||
|
||||
@@ -35,6 +35,7 @@ export { GitHubCopilot } from "./provider/github-copilot"
|
||||
export { OpenAIChat } from "./provider/openai-chat"
|
||||
export { OpenAICompatibleChat } from "./provider/openai-compatible-chat"
|
||||
export { OpenAICompatibleFamily } from "./provider/openai-compatible-family"
|
||||
export { OpenAICompatibleProfiles } from "./provider/openai-compatible-profile"
|
||||
export { OpenAIResponses } from "./provider/openai-responses"
|
||||
export { ProviderResolver } from "./provider-resolver"
|
||||
export { OpenAI } from "./provider/openai"
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { Context, Effect, Layer, Stream } from "effect"
|
||||
import { LLMClient, type ClientOptions } from "./adapter"
|
||||
import { LLMClient, preserveModelBinding, type AnyAdapter, type ClientOptions } from "./adapter"
|
||||
import type { RequestExecutor } from "./executor"
|
||||
import { ProviderPatch } from "./provider/patch"
|
||||
import { type Tools } from "./tool"
|
||||
@@ -29,7 +29,7 @@ import {
|
||||
import type { LLMError, PreparedRequestOf } from "./schema"
|
||||
|
||||
export interface Provider {
|
||||
readonly adapters: ClientOptions["adapters"]
|
||||
readonly adapters: ReadonlyArray<AnyAdapter>
|
||||
}
|
||||
|
||||
export interface MakeOptions {
|
||||
@@ -50,15 +50,13 @@ export interface Runtime {
|
||||
export class Service extends Context.Service<Service, Runtime>()("@opencode/LLM") {}
|
||||
|
||||
const clientOptions = (options: MakeOptions): ClientOptions => ({
|
||||
adapters: [...(options.adapters ?? []), ...(options.providers ?? []).flatMap((provider) => provider.adapters)].filter(
|
||||
(source, index, all) => all.findIndex((item) => item.runtime.protocol === source.runtime.protocol) === index,
|
||||
),
|
||||
adapters: [...(options.providers ?? []).flatMap((provider) => provider.adapters), ...(options.adapters ?? [])],
|
||||
patches: options.patches ?? ProviderPatch.defaults,
|
||||
})
|
||||
|
||||
const requestOf = (input: LLMRequest | RequestInput) => input instanceof LLMRequest ? input : request(input)
|
||||
|
||||
export const make = (options: MakeOptions): Runtime => {
|
||||
export const make = (options: MakeOptions = {}): Runtime => {
|
||||
const client = LLMClient.make(clientOptions(options))
|
||||
return {
|
||||
prepare: (input) => client.prepare(requestOf(input)),
|
||||
@@ -71,7 +69,7 @@ export const make = (options: MakeOptions): Runtime => {
|
||||
}
|
||||
}
|
||||
|
||||
export const layer = (options: MakeOptions): Layer.Layer<Service> =>
|
||||
export const layer = (options: MakeOptions = {}): Layer.Layer<Service> =>
|
||||
Layer.succeed(Service, Service.of(make(options)))
|
||||
|
||||
export const prepare = <Target = unknown>(input: LLMRequest | RequestInput) =>
|
||||
@@ -253,7 +251,7 @@ export const requestInput = (input: LLMRequest): RequestInput => ({
|
||||
|
||||
export const request = (input: RequestInput) => {
|
||||
const { system: requestSystem, prompt, messages, tools, toolChoice: requestToolChoice, generation: requestGeneration, ...rest } = input
|
||||
return new LLMRequest({
|
||||
const result = new LLMRequest({
|
||||
...rest,
|
||||
system: systemParts(requestSystem),
|
||||
messages: [...(messages?.map(message) ?? []), ...(prompt === undefined ? [] : [user(prompt)])],
|
||||
@@ -261,6 +259,8 @@ export const request = (input: RequestInput) => {
|
||||
toolChoice: requestToolChoice ? toolChoice(requestToolChoice) : undefined,
|
||||
generation: generation(requestGeneration),
|
||||
})
|
||||
preserveModelBinding(input.model, result.model)
|
||||
return result
|
||||
}
|
||||
|
||||
export const updateRequest = (input: LLMRequest, patch: Partial<RequestInput>) =>
|
||||
|
||||
@@ -513,16 +513,19 @@ export const adapter = Adapter.fromProtocol({
|
||||
})
|
||||
|
||||
export const model = (input: AnthropicMessagesModelInput) =>
|
||||
llmModel({
|
||||
...input,
|
||||
provider: "anthropic",
|
||||
protocol: "anthropic-messages",
|
||||
capabilities: input.capabilities ?? capabilities({
|
||||
output: { reasoning: true },
|
||||
tools: { calls: true, streamingInput: true },
|
||||
cache: { prompt: true, contentBlocks: true },
|
||||
reasoning: { efforts: ["low", "medium", "high", "xhigh", "max"], summaries: false, encryptedContent: true },
|
||||
Adapter.bindModel(
|
||||
llmModel({
|
||||
...input,
|
||||
provider: "anthropic",
|
||||
protocol: "anthropic-messages",
|
||||
capabilities: input.capabilities ?? capabilities({
|
||||
output: { reasoning: true },
|
||||
tools: { calls: true, streamingInput: true },
|
||||
cache: { prompt: true, contentBlocks: true },
|
||||
reasoning: { efforts: ["low", "medium", "high", "xhigh", "max"], summaries: false, encryptedContent: true },
|
||||
}),
|
||||
}),
|
||||
})
|
||||
adapter,
|
||||
)
|
||||
|
||||
export * as AnthropicMessages from "./anthropic-messages"
|
||||
|
||||
@@ -1,11 +1,8 @@
|
||||
import { EventStreamCodec } from "@smithy/eventstream-codec"
|
||||
import { fromUtf8, toUtf8 } from "@smithy/util-utf8"
|
||||
import { AwsV4Signer } from "aws4fetch"
|
||||
import { Effect, Option, Schema, Stream } from "effect"
|
||||
import { Effect, Option, Schema } from "effect"
|
||||
import { Adapter } from "../adapter"
|
||||
import { Auth } from "../auth"
|
||||
import { Endpoint } from "../endpoint"
|
||||
import type { Framing } from "../framing"
|
||||
import { capabilities, model as llmModel, type ModelInput } from "../llm"
|
||||
import { Protocol } from "../protocol"
|
||||
import {
|
||||
@@ -15,11 +12,11 @@ import {
|
||||
type LLMEvent,
|
||||
type LLMRequest,
|
||||
type MediaPart,
|
||||
type ProviderChunkError,
|
||||
type ToolCallPart,
|
||||
type ToolDefinition,
|
||||
type ToolResultPart,
|
||||
} from "../schema"
|
||||
import { BedrockEventStream } from "./bedrock-event-stream"
|
||||
import { ProviderShared } from "./shared"
|
||||
|
||||
const ADAPTER = "bedrock-converse"
|
||||
@@ -679,87 +676,7 @@ const processChunk = (state: ParserState, chunk: BedrockChunk) =>
|
||||
return [state, []] as const
|
||||
})
|
||||
|
||||
// Bedrock streams responses using the AWS event stream binary protocol — each
|
||||
// frame is `[length:4][headers-length:4][prelude-crc:4][headers][payload][crc:4]`.
|
||||
// We use `@smithy/eventstream-codec` to validate framing and CRCs, then
|
||||
// reconstruct the JSON wrapping by `:event-type` so the chunk schema can match.
|
||||
const eventCodec = new EventStreamCodec(toUtf8, fromUtf8)
|
||||
const utf8 = new TextDecoder()
|
||||
|
||||
// Cursor-tracking buffer state. Bytes accumulate in `buffer`; `offset` is the
|
||||
// read position. Reading by `subarray` is zero-copy. We only allocate a fresh
|
||||
// buffer when (a) a new network chunk arrives and we need to append, or (b)
|
||||
// the consumed prefix is more than half the buffer (compaction).
|
||||
interface FrameBufferState {
|
||||
readonly buffer: Uint8Array
|
||||
readonly offset: number
|
||||
}
|
||||
|
||||
const initialFrameBuffer: FrameBufferState = { buffer: new Uint8Array(0), offset: 0 }
|
||||
|
||||
const appendChunk = (state: FrameBufferState, chunk: Uint8Array): FrameBufferState => {
|
||||
const remaining = state.buffer.length - state.offset
|
||||
// Compact: drop the consumed prefix and append the new chunk in one alloc.
|
||||
// This bounds buffer growth to at most one network chunk past the live
|
||||
// window, regardless of stream length.
|
||||
const next = new Uint8Array(remaining + chunk.length)
|
||||
next.set(state.buffer.subarray(state.offset), 0)
|
||||
next.set(chunk, remaining)
|
||||
return { buffer: next, offset: 0 }
|
||||
}
|
||||
|
||||
const consumeFrames = (state: FrameBufferState, chunk: Uint8Array) =>
|
||||
Effect.gen(function* () {
|
||||
let cursor = appendChunk(state, chunk)
|
||||
const out: object[] = []
|
||||
while (cursor.buffer.length - cursor.offset >= 4) {
|
||||
const view = cursor.buffer.subarray(cursor.offset)
|
||||
const totalLength = new DataView(view.buffer, view.byteOffset, view.byteLength).getUint32(0, false)
|
||||
if (view.length < totalLength) break
|
||||
|
||||
const decoded = yield* Effect.try({
|
||||
try: () => eventCodec.decode(view.subarray(0, totalLength)),
|
||||
catch: (error) =>
|
||||
ProviderShared.chunkError(
|
||||
ADAPTER,
|
||||
`Failed to decode Bedrock Converse event-stream frame: ${
|
||||
error instanceof Error ? error.message : String(error)
|
||||
}`,
|
||||
),
|
||||
})
|
||||
cursor = { buffer: cursor.buffer, offset: cursor.offset + totalLength }
|
||||
|
||||
if (decoded.headers[":message-type"]?.value !== "event") continue
|
||||
const eventType = decoded.headers[":event-type"]?.value
|
||||
if (typeof eventType !== "string") continue
|
||||
const payload = utf8.decode(decoded.body)
|
||||
if (!payload) continue
|
||||
// The AWS event stream pads short payloads with a `p` field. Drop it
|
||||
// before handing the object to the chunk schema. JSON decode goes
|
||||
// through the shared Schema-driven codec to satisfy the package rule
|
||||
// against ad-hoc `JSON.parse` calls.
|
||||
const parsed = (yield* ProviderShared.parseJson(
|
||||
ADAPTER,
|
||||
payload,
|
||||
"Failed to parse Bedrock Converse event-stream payload",
|
||||
)) as Record<string, unknown>
|
||||
delete parsed.p
|
||||
out.push({ [eventType]: parsed })
|
||||
}
|
||||
return [cursor, out] as const
|
||||
})
|
||||
|
||||
/**
|
||||
* AWS event-stream framing for Bedrock Converse. Each frame is decoded by
|
||||
* `@smithy/eventstream-codec` (length + header + payload + CRC) and rewrapped
|
||||
* under its `:event-type` header so the chunk schema can match the JSON
|
||||
* payload directly. Reusable for any AWS service that wraps JSON payloads in
|
||||
* event-stream frames keyed by `:event-type`.
|
||||
*/
|
||||
const framing: Framing<object> = {
|
||||
id: "aws-event-stream",
|
||||
frame: (bytes) => bytes.pipe(Stream.mapAccumEffect(() => initialFrameBuffer, consumeFrames)),
|
||||
}
|
||||
const framing = BedrockEventStream.framing(ADAPTER)
|
||||
|
||||
// If a stream ends after `messageStop` but before `metadata` (rare but
|
||||
// possible on truncated transports), still surface a terminal finish.
|
||||
@@ -803,25 +720,28 @@ export const adapter = Adapter.fromProtocol({
|
||||
|
||||
export const model = (input: BedrockConverseModelInput) => {
|
||||
const { credentials, ...rest } = input
|
||||
return llmModel({
|
||||
...rest,
|
||||
provider: "bedrock",
|
||||
protocol: "bedrock-converse",
|
||||
capabilities:
|
||||
input.capabilities ??
|
||||
capabilities({
|
||||
output: { reasoning: true },
|
||||
tools: { calls: true, streamingInput: true },
|
||||
cache: { prompt: true, contentBlocks: true },
|
||||
}),
|
||||
native: credentials
|
||||
? {
|
||||
...input.native,
|
||||
aws_credentials: credentials,
|
||||
aws_region: credentials.region,
|
||||
}
|
||||
: input.native,
|
||||
})
|
||||
return Adapter.bindModel(
|
||||
llmModel({
|
||||
...rest,
|
||||
provider: "bedrock",
|
||||
protocol: "bedrock-converse",
|
||||
capabilities:
|
||||
input.capabilities ??
|
||||
capabilities({
|
||||
output: { reasoning: true },
|
||||
tools: { calls: true, streamingInput: true },
|
||||
cache: { prompt: true, contentBlocks: true },
|
||||
}),
|
||||
native: credentials
|
||||
? {
|
||||
...input.native,
|
||||
aws_credentials: credentials,
|
||||
aws_region: credentials.region,
|
||||
}
|
||||
: input.native,
|
||||
}),
|
||||
adapter,
|
||||
)
|
||||
}
|
||||
|
||||
export * as BedrockConverse from "./bedrock-converse"
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
import { EventStreamCodec } from "@smithy/eventstream-codec"
|
||||
import { fromUtf8, toUtf8 } from "@smithy/util-utf8"
|
||||
import { Effect, Stream } from "effect"
|
||||
import type { Framing } from "../framing"
|
||||
import { ProviderShared } from "./shared"
|
||||
|
||||
// Bedrock streams responses using the AWS event stream binary protocol — each
|
||||
// frame is `[length:4][headers-length:4][prelude-crc:4][headers][payload][crc:4]`.
|
||||
// We use `@smithy/eventstream-codec` to validate framing and CRCs, then
|
||||
// reconstruct the JSON wrapping by `:event-type` so the chunk schema can match.
|
||||
const eventCodec = new EventStreamCodec(toUtf8, fromUtf8)
|
||||
const utf8 = new TextDecoder()
|
||||
|
||||
// Cursor-tracking buffer state. Bytes accumulate in `buffer`; `offset` is the
|
||||
// read position. Reading by `subarray` is zero-copy. We only allocate a fresh
|
||||
// buffer when a new network chunk arrives and we need to append.
|
||||
interface FrameBufferState {
|
||||
readonly buffer: Uint8Array
|
||||
readonly offset: number
|
||||
}
|
||||
|
||||
const initialFrameBuffer: FrameBufferState = { buffer: new Uint8Array(0), offset: 0 }
|
||||
|
||||
const appendChunk = (state: FrameBufferState, chunk: Uint8Array): FrameBufferState => {
|
||||
const remaining = state.buffer.length - state.offset
|
||||
// Compact: drop the consumed prefix and append the new chunk in one alloc.
|
||||
// This bounds buffer growth to at most one network chunk past the live
|
||||
// window, regardless of stream length.
|
||||
const next = new Uint8Array(remaining + chunk.length)
|
||||
next.set(state.buffer.subarray(state.offset), 0)
|
||||
next.set(chunk, remaining)
|
||||
return { buffer: next, offset: 0 }
|
||||
}
|
||||
|
||||
const consumeFrames = (adapter: string) => (state: FrameBufferState, chunk: Uint8Array) =>
|
||||
Effect.gen(function* () {
|
||||
let cursor = appendChunk(state, chunk)
|
||||
const out: object[] = []
|
||||
while (cursor.buffer.length - cursor.offset >= 4) {
|
||||
const view = cursor.buffer.subarray(cursor.offset)
|
||||
const totalLength = new DataView(view.buffer, view.byteOffset, view.byteLength).getUint32(0, false)
|
||||
if (view.length < totalLength) break
|
||||
|
||||
const decoded = yield* Effect.try({
|
||||
try: () => eventCodec.decode(view.subarray(0, totalLength)),
|
||||
catch: (error) =>
|
||||
ProviderShared.chunkError(
|
||||
adapter,
|
||||
`Failed to decode Bedrock Converse event-stream frame: ${
|
||||
error instanceof Error ? error.message : String(error)
|
||||
}`,
|
||||
),
|
||||
})
|
||||
cursor = { buffer: cursor.buffer, offset: cursor.offset + totalLength }
|
||||
|
||||
if (decoded.headers[":message-type"]?.value !== "event") continue
|
||||
const eventType = decoded.headers[":event-type"]?.value
|
||||
if (typeof eventType !== "string") continue
|
||||
const payload = utf8.decode(decoded.body)
|
||||
if (!payload) continue
|
||||
// The AWS event stream pads short payloads with a `p` field. Drop it
|
||||
// before handing the object to the chunk schema. JSON decode goes
|
||||
// through the shared Schema-driven codec to satisfy the package rule
|
||||
// against ad-hoc `JSON.parse` calls.
|
||||
const parsed = (yield* ProviderShared.parseJson(
|
||||
adapter,
|
||||
payload,
|
||||
"Failed to parse Bedrock Converse event-stream payload",
|
||||
)) as Record<string, unknown>
|
||||
delete parsed.p
|
||||
out.push({ [eventType]: parsed })
|
||||
}
|
||||
return [cursor, out] as const
|
||||
})
|
||||
|
||||
/**
|
||||
* AWS event-stream framing for Bedrock Converse. Each frame is decoded by
|
||||
* `@smithy/eventstream-codec` (length + header + payload + CRC) and rewrapped
|
||||
* under its `:event-type` header so the chunk schema can match the JSON
|
||||
* payload directly.
|
||||
*/
|
||||
export const framing = (adapter: string): Framing<object> => ({
|
||||
id: "aws-event-stream",
|
||||
frame: (bytes) => bytes.pipe(Stream.mapAccumEffect(() => initialFrameBuffer, consumeFrames(adapter))),
|
||||
})
|
||||
|
||||
export * as BedrockEventStream from "./bedrock-event-stream"
|
||||
@@ -469,16 +469,19 @@ export const adapter = Adapter.fromProtocol({
|
||||
})
|
||||
|
||||
export const model = (input: GeminiModelInput) =>
|
||||
llmModel({
|
||||
...input,
|
||||
provider: "google",
|
||||
protocol: "gemini",
|
||||
capabilities: input.capabilities ?? capabilities({
|
||||
input: { image: true, audio: true, video: true, pdf: true },
|
||||
output: { reasoning: true },
|
||||
tools: { calls: true },
|
||||
reasoning: { efforts: ["minimal", "low", "medium", "high", "xhigh", "max"] },
|
||||
Adapter.bindModel(
|
||||
llmModel({
|
||||
...input,
|
||||
provider: "google",
|
||||
protocol: "gemini",
|
||||
capabilities: input.capabilities ?? capabilities({
|
||||
input: { image: true, audio: true, video: true, pdf: true },
|
||||
output: { reasoning: true },
|
||||
tools: { calls: true },
|
||||
reasoning: { efforts: ["minimal", "low", "medium", "high", "xhigh", "max"] },
|
||||
}),
|
||||
}),
|
||||
})
|
||||
adapter,
|
||||
)
|
||||
|
||||
export * as Gemini from "./gemini"
|
||||
|
||||
@@ -347,12 +347,15 @@ export const adapter = Adapter.fromProtocol({
|
||||
})
|
||||
|
||||
export const model = (input: OpenAIChatModelInput) =>
|
||||
llmModel({
|
||||
...input,
|
||||
provider: "openai",
|
||||
protocol: "openai-chat",
|
||||
capabilities: input.capabilities ?? capabilities({ tools: { calls: true, streamingInput: true } }),
|
||||
})
|
||||
Adapter.bindModel(
|
||||
llmModel({
|
||||
...input,
|
||||
provider: "openai",
|
||||
protocol: "openai-chat",
|
||||
capabilities: input.capabilities ?? capabilities({ tools: { calls: true, streamingInput: true } }),
|
||||
}),
|
||||
adapter,
|
||||
)
|
||||
|
||||
export const includeUsage = adapter.patch("include-usage", {
|
||||
reason: "request final usage chunk from OpenAI Chat streaming responses",
|
||||
|
||||
@@ -3,7 +3,7 @@ import { Endpoint } from "../endpoint"
|
||||
import { Framing } from "../framing"
|
||||
import { capabilities, model as llmModel, type ModelInput } from "../llm"
|
||||
import { OpenAIChat } from "./openai-chat"
|
||||
import { families, type ProviderFamily } from "./openai-compatible-family"
|
||||
import { families, type OpenAICompatibleProfile } from "./openai-compatible-profile"
|
||||
|
||||
const ADAPTER = "openai-compatible-chat"
|
||||
|
||||
@@ -38,30 +38,39 @@ export const adapter = Adapter.fromProtocol({
|
||||
})
|
||||
|
||||
export const model = (input: OpenAICompatibleChatModelInput) =>
|
||||
llmModel({
|
||||
...input,
|
||||
protocol: "openai-compatible-chat",
|
||||
capabilities: input.capabilities ?? capabilities({ tools: { calls: true, streamingInput: true } }),
|
||||
})
|
||||
Adapter.bindModel(
|
||||
llmModel({
|
||||
...input,
|
||||
protocol: "openai-compatible-chat",
|
||||
capabilities: input.capabilities ?? capabilities({ tools: { calls: true, streamingInput: true } }),
|
||||
}),
|
||||
adapter,
|
||||
)
|
||||
|
||||
const familyModel = (family: ProviderFamily, input: ProviderFamilyModelInput) =>
|
||||
const profileBaseURL = (profile: OpenAICompatibleProfile, input: ProviderFamilyModelInput) => {
|
||||
const baseURL = input.baseURL ?? profile.baseURL
|
||||
if (baseURL) return baseURL
|
||||
throw new Error(`OpenAI-compatible profile ${profile.provider} requires a baseURL`)
|
||||
}
|
||||
|
||||
export const profileModel = (profile: OpenAICompatibleProfile, input: ProviderFamilyModelInput) =>
|
||||
model({
|
||||
...input,
|
||||
provider: family.provider,
|
||||
baseURL: input.baseURL ?? family.baseURL,
|
||||
provider: profile.provider,
|
||||
baseURL: profileBaseURL(profile, input),
|
||||
})
|
||||
|
||||
export const baseten = (input: ProviderFamilyModelInput) => familyModel(families.baseten, input)
|
||||
export const baseten = (input: ProviderFamilyModelInput) => profileModel(families.baseten, input)
|
||||
|
||||
export const cerebras = (input: ProviderFamilyModelInput) => familyModel(families.cerebras, input)
|
||||
export const cerebras = (input: ProviderFamilyModelInput) => profileModel(families.cerebras, input)
|
||||
|
||||
export const deepinfra = (input: ProviderFamilyModelInput) => familyModel(families.deepinfra, input)
|
||||
export const deepinfra = (input: ProviderFamilyModelInput) => profileModel(families.deepinfra, input)
|
||||
|
||||
export const deepseek = (input: ProviderFamilyModelInput) => familyModel(families.deepseek, input)
|
||||
export const deepseek = (input: ProviderFamilyModelInput) => profileModel(families.deepseek, input)
|
||||
|
||||
export const fireworks = (input: ProviderFamilyModelInput) => familyModel(families.fireworks, input)
|
||||
export const fireworks = (input: ProviderFamilyModelInput) => profileModel(families.fireworks, input)
|
||||
|
||||
export const togetherai = (input: ProviderFamilyModelInput) => familyModel(families.togetherai, input)
|
||||
export const togetherai = (input: ProviderFamilyModelInput) => profileModel(families.togetherai, input)
|
||||
|
||||
export const includeUsage = adapter.patch("include-usage", {
|
||||
reason: "request final usage chunk from OpenAI-compatible Chat streaming responses",
|
||||
|
||||
@@ -1,36 +1,10 @@
|
||||
import { ProviderResolver } from "../provider-resolver"
|
||||
import { families, familyByProvider, familyResolver, resolveFamily } from "./openai-compatible-profile"
|
||||
import type { OpenAICompatibleProfile } from "./openai-compatible-profile"
|
||||
|
||||
export interface ProviderFamily {
|
||||
readonly provider: string
|
||||
readonly baseURL: string
|
||||
}
|
||||
|
||||
export const families = {
|
||||
baseten: { provider: "baseten", baseURL: "https://inference.baseten.co/v1" },
|
||||
cerebras: { provider: "cerebras", baseURL: "https://api.cerebras.ai/v1" },
|
||||
deepinfra: { provider: "deepinfra", baseURL: "https://api.deepinfra.com/v1/openai" },
|
||||
deepseek: { provider: "deepseek", baseURL: "https://api.deepseek.com/v1" },
|
||||
fireworks: { provider: "fireworks", baseURL: "https://api.fireworks.ai/inference/v1" },
|
||||
togetherai: { provider: "togetherai", baseURL: "https://api.together.xyz/v1" },
|
||||
} as const satisfies Record<string, ProviderFamily>
|
||||
|
||||
export const byProvider: Record<string, ProviderFamily> = Object.fromEntries(
|
||||
Object.values(families).map((family) => [family.provider, family]),
|
||||
)
|
||||
|
||||
const resolutions = Object.fromEntries(
|
||||
Object.values(families).map((family) => [
|
||||
family.provider,
|
||||
ProviderResolver.make(family.provider, "openai-compatible-chat", { baseURL: family.baseURL }),
|
||||
]),
|
||||
)
|
||||
|
||||
export const resolve = (provider: string) =>
|
||||
resolutions[provider] ?? ProviderResolver.make(provider, "openai-compatible-chat")
|
||||
|
||||
export const resolver = ProviderResolver.define({
|
||||
id: ProviderResolver.make("openai-compatible", "openai-compatible-chat").provider,
|
||||
resolve: (input) => resolve(input.providerID),
|
||||
})
|
||||
export type ProviderFamily = OpenAICompatibleProfile
|
||||
export const byProvider = familyByProvider
|
||||
export const resolve = resolveFamily
|
||||
export const resolver = familyResolver
|
||||
export { families }
|
||||
|
||||
export * as OpenAICompatibleFamily from "./openai-compatible-family"
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
import type { CapabilitiesInput } from "../llm"
|
||||
import { ProviderResolver, type ProviderResolution } from "../provider-resolver"
|
||||
|
||||
export interface OpenAICompatibleProfile {
|
||||
readonly provider: string
|
||||
readonly baseURL?: string
|
||||
readonly capabilities?: CapabilitiesInput
|
||||
readonly resolver?: Partial<Omit<ProviderResolution, "provider" | "protocol">>
|
||||
}
|
||||
|
||||
export const families = {
|
||||
baseten: { provider: "baseten", baseURL: "https://inference.baseten.co/v1" },
|
||||
cerebras: { provider: "cerebras", baseURL: "https://api.cerebras.ai/v1" },
|
||||
deepinfra: { provider: "deepinfra", baseURL: "https://api.deepinfra.com/v1/openai" },
|
||||
deepseek: { provider: "deepseek", baseURL: "https://api.deepseek.com/v1" },
|
||||
fireworks: { provider: "fireworks", baseURL: "https://api.fireworks.ai/inference/v1" },
|
||||
togetherai: { provider: "togetherai", baseURL: "https://api.together.xyz/v1" },
|
||||
} as const satisfies Record<string, OpenAICompatibleProfile>
|
||||
|
||||
export const profiles = {
|
||||
...families,
|
||||
openrouter: { provider: "openrouter", baseURL: "https://openrouter.ai/api/v1" },
|
||||
} as const satisfies Record<string, OpenAICompatibleProfile>
|
||||
|
||||
export const familyByProvider: Record<string, OpenAICompatibleProfile> = Object.fromEntries(
|
||||
Object.values(families).map((profile) => [profile.provider, profile]),
|
||||
)
|
||||
|
||||
export const byProvider: Record<string, OpenAICompatibleProfile> = Object.fromEntries(
|
||||
Object.values(profiles).map((profile) => [profile.provider, profile]),
|
||||
)
|
||||
|
||||
export const resolution = (profile: OpenAICompatibleProfile) =>
|
||||
ProviderResolver.make(profile.provider, "openai-compatible-chat", {
|
||||
baseURL: profile.baseURL,
|
||||
capabilities: profile.capabilities,
|
||||
...profile.resolver,
|
||||
})
|
||||
|
||||
export const resolve = (provider: string) => {
|
||||
const profile = byProvider[provider]
|
||||
if (profile) return resolution(profile)
|
||||
return ProviderResolver.make(provider, "openai-compatible-chat")
|
||||
}
|
||||
|
||||
export const resolveFamily = (provider: string) => {
|
||||
const profile = familyByProvider[provider]
|
||||
if (profile) return resolution(profile)
|
||||
return ProviderResolver.make(provider, "openai-compatible-chat")
|
||||
}
|
||||
|
||||
export const resolverFor = (profile: OpenAICompatibleProfile) =>
|
||||
ProviderResolver.define({
|
||||
id: ProviderResolver.make(profile.provider, "openai-compatible-chat").provider,
|
||||
resolve: () => resolution(profile),
|
||||
})
|
||||
|
||||
export const resolver = ProviderResolver.define({
|
||||
id: ProviderResolver.make("openai-compatible", "openai-compatible-chat").provider,
|
||||
resolve: (input) => resolve(input.providerID),
|
||||
})
|
||||
|
||||
export const familyResolver = ProviderResolver.define({
|
||||
id: ProviderResolver.make("openai-compatible", "openai-compatible-chat").provider,
|
||||
resolve: (input) => resolveFamily(input.providerID),
|
||||
})
|
||||
|
||||
export * as OpenAICompatibleProfiles from "./openai-compatible-profile"
|
||||
@@ -383,11 +383,14 @@ export const adapter = Adapter.fromProtocol({
|
||||
})
|
||||
|
||||
export const model = (input: OpenAIResponsesModelInput) =>
|
||||
llmModel({
|
||||
...input,
|
||||
provider: "openai",
|
||||
protocol: "openai-responses",
|
||||
capabilities: input.capabilities ?? capabilities({ tools: { calls: true, streamingInput: true } }),
|
||||
})
|
||||
Adapter.bindModel(
|
||||
llmModel({
|
||||
...input,
|
||||
provider: "openai",
|
||||
protocol: "openai-responses",
|
||||
capabilities: input.capabilities ?? capabilities({ tools: { calls: true, streamingInput: true } }),
|
||||
}),
|
||||
adapter,
|
||||
)
|
||||
|
||||
export * as OpenAIResponses from "./openai-responses"
|
||||
|
||||
@@ -1,24 +1,25 @@
|
||||
import { ProviderResolver } from "../provider-resolver"
|
||||
import { OpenAICompatible, type ModelOptions as OpenAICompatibleModelOptions } from "./openai-compatible"
|
||||
import { OpenAICompatibleProfiles } from "./openai-compatible-profile"
|
||||
|
||||
const baseURL = "https://openrouter.ai/api/v1"
|
||||
export const profile = OpenAICompatibleProfiles.profiles.openrouter
|
||||
|
||||
export type ModelOptions = Omit<OpenAICompatibleModelOptions, "provider" | "baseURL"> & {
|
||||
readonly baseURL?: string
|
||||
}
|
||||
|
||||
export const resolver = ProviderResolver.fixed("openrouter", "openai-compatible-chat", {
|
||||
baseURL,
|
||||
})
|
||||
export const resolver = OpenAICompatibleProfiles.resolverFor(profile)
|
||||
|
||||
export const adapters = OpenAICompatible.adapters
|
||||
|
||||
export const model = (id: string, options: ModelOptions = {}) =>
|
||||
OpenAICompatible.model(id, {
|
||||
export const model = (id: string, options: ModelOptions = {}) => {
|
||||
const baseURL = options.baseURL ?? profile.baseURL
|
||||
if (!baseURL) throw new Error("OpenRouter requires a baseURL")
|
||||
return OpenAICompatible.model(id, {
|
||||
...options,
|
||||
provider: "openrouter",
|
||||
baseURL: options.baseURL ?? baseURL,
|
||||
provider: profile.provider,
|
||||
baseURL,
|
||||
})
|
||||
}
|
||||
|
||||
export const chat = model
|
||||
|
||||
|
||||
@@ -169,14 +169,14 @@ export const invalidRequest = (message: string) => new InvalidRequestError({ mes
|
||||
|
||||
/**
|
||||
* Build a `validate` step from a Schema decoder. Replaces the per-adapter
|
||||
* lambda body `(draft) => decode(draft).pipe(Effect.mapError((e) =>
|
||||
* lambda body `(target) => decode(target).pipe(Effect.mapError((e) =>
|
||||
* invalid(e.message)))`. Any decode error is translated into
|
||||
* `InvalidRequestError` carrying the original parse-error message.
|
||||
*/
|
||||
export const validateWith =
|
||||
<A, I, E extends { readonly message: string }>(decode: (input: I) => Effect.Effect<A, E>) =>
|
||||
(draft: I) =>
|
||||
decode(draft).pipe(Effect.mapError((error) => invalidRequest(error.message)))
|
||||
(target: I) =>
|
||||
decode(target).pipe(Effect.mapError((error) => invalidRequest(error.message)))
|
||||
|
||||
/**
|
||||
* Build an HTTP POST with a JSON body. Sets `content-type: application/json`
|
||||
|
||||
@@ -6,14 +6,7 @@ import { Schema } from "effect"
|
||||
* the runtime registry keys lookups by it. The implementation type itself is
|
||||
* `Protocol` (see `protocol.ts`).
|
||||
*/
|
||||
export const ProtocolID = Schema.Literals([
|
||||
"openai-chat",
|
||||
"openai-compatible-chat",
|
||||
"openai-responses",
|
||||
"anthropic-messages",
|
||||
"gemini",
|
||||
"bedrock-converse",
|
||||
])
|
||||
export const ProtocolID = Schema.String
|
||||
export type ProtocolID = Schema.Schema.Type<typeof ProtocolID>
|
||||
|
||||
export const ModelID = Schema.String.pipe(Schema.brand("LLM.ModelID"))
|
||||
|
||||
@@ -53,7 +53,7 @@ const mapText = (fn: (text: string) => string) => (request: LLMRequest): LLMRequ
|
||||
const Json = Schema.fromJsonString(Schema.Unknown)
|
||||
const encodeJson = Schema.encodeSync(Json)
|
||||
|
||||
type FakeDraft = {
|
||||
type FakeTarget = {
|
||||
readonly body: string
|
||||
readonly includeUsage?: boolean
|
||||
}
|
||||
@@ -80,10 +80,10 @@ const raiseChunk = (chunk: FakeChunk): import("../src/schema").LLMEvent =>
|
||||
? { type: "request-finish", reason: chunk.reason }
|
||||
: { type: "text-delta", text: chunk.text }
|
||||
|
||||
const fake = Adapter.unsafe<FakeDraft, FakeDraft>({
|
||||
const fake = Adapter.unsafe<FakeTarget>({
|
||||
id: "fake",
|
||||
protocol: "openai-chat",
|
||||
validate: (draft) => Effect.succeed(draft),
|
||||
validate: (target) => Effect.succeed(target),
|
||||
prepare: (request) =>
|
||||
Effect.succeed({
|
||||
body: [
|
||||
@@ -113,7 +113,7 @@ const fake = Adapter.unsafe<FakeDraft, FakeDraft>({
|
||||
),
|
||||
})
|
||||
|
||||
const gemini = Adapter.unsafe<FakeDraft, FakeDraft>({
|
||||
const gemini = Adapter.unsafe<FakeTarget>({
|
||||
...fake,
|
||||
id: "gemini-fake",
|
||||
protocol: "gemini",
|
||||
@@ -140,7 +140,7 @@ describe("llm adapter", () => {
|
||||
fake.withPatches([
|
||||
fake.patch("include-usage", {
|
||||
reason: "fake target patch",
|
||||
apply: (draft) => ({ ...draft, includeUsage: true }),
|
||||
apply: (target) => ({ ...target, includeUsage: true }),
|
||||
}),
|
||||
]),
|
||||
],
|
||||
@@ -172,6 +172,33 @@ describe("llm adapter", () => {
|
||||
}),
|
||||
)
|
||||
|
||||
it.effect("falls back to adapter bound to model", () =>
|
||||
Effect.gen(function* () {
|
||||
const prepared = yield* LLMClient.make({ adapters: [] }).prepare(
|
||||
LLM.updateRequest(request, {
|
||||
model: Adapter.bindModel(updateModel(request.model, { protocol: "gemini" }), gemini),
|
||||
}),
|
||||
)
|
||||
|
||||
expect(prepared.adapter).toBe("gemini-fake")
|
||||
}),
|
||||
)
|
||||
|
||||
it.effect("explicit adapters override provider adapters", () =>
|
||||
Effect.gen(function* () {
|
||||
const override = Adapter.unsafe<FakeTarget>({
|
||||
...fake,
|
||||
id: "fake-override",
|
||||
prepare: () => Effect.succeed({ body: "override" }),
|
||||
})
|
||||
|
||||
const prepared = yield* LLM.make({ providers: [{ adapters: [fake] }], adapters: [override] }).prepare(request)
|
||||
|
||||
expect(prepared.adapter).toBe("fake-override")
|
||||
expect(prepared.target).toEqual({ body: "override" })
|
||||
}),
|
||||
)
|
||||
|
||||
it.effect("request, prompt, and tool-schema patches run before adapter prepare", () =>
|
||||
Effect.gen(function* () {
|
||||
const prepared = yield* LLMClient.make({
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { describe, expect, test } from "bun:test"
|
||||
import { Azure, GitHubCopilot, OpenAI, OpenAICompatibleFamily, ProviderResolver } from "../src"
|
||||
import { Azure, GitHubCopilot, OpenAI, OpenAICompatibleFamily, OpenAICompatibleProfiles, OpenRouter, ProviderResolver } from "../src"
|
||||
|
||||
describe("provider resolver", () => {
|
||||
test("fixed providers resolve protocol and auth defaults", () => {
|
||||
@@ -30,6 +30,18 @@ describe("provider resolver", () => {
|
||||
baseURL: "https://api.together.xyz/v1",
|
||||
auth: "key",
|
||||
})
|
||||
expect(OpenAICompatibleProfiles.resolve("deepseek")).toMatchObject({
|
||||
provider: "deepseek",
|
||||
protocol: "openai-compatible-chat",
|
||||
baseURL: "https://api.deepseek.com/v1",
|
||||
auth: "key",
|
||||
})
|
||||
expect(OpenRouter.resolver.resolve(ProviderResolver.input("openai/gpt-4o-mini", "openrouter", {}))).toMatchObject({
|
||||
provider: "openrouter",
|
||||
protocol: "openai-compatible-chat",
|
||||
baseURL: "https://openrouter.ai/api/v1",
|
||||
auth: "key",
|
||||
})
|
||||
})
|
||||
|
||||
test("Azure resolves resource URLs and API-version query params", () => {
|
||||
|
||||
@@ -35,16 +35,16 @@ describe("llm schema", () => {
|
||||
expect(decoded.messages[0]?.content[0]?.type).toBe("text")
|
||||
})
|
||||
|
||||
test("rejects invalid protocol", () => {
|
||||
expect(() =>
|
||||
Schema.decodeUnknownSync(LLMRequest)({
|
||||
model: { ...model, protocol: "bogus" },
|
||||
system: [],
|
||||
messages: [],
|
||||
tools: [],
|
||||
generation: {},
|
||||
}),
|
||||
).toThrow()
|
||||
test("accepts custom protocol ids", () => {
|
||||
const decoded = Schema.decodeUnknownSync(LLMRequest)({
|
||||
model: { ...model, protocol: "custom-protocol" },
|
||||
system: [],
|
||||
messages: [],
|
||||
tools: [],
|
||||
generation: {},
|
||||
})
|
||||
|
||||
expect(decoded.model.protocol).toBe("custom-protocol")
|
||||
})
|
||||
|
||||
test("rejects invalid event type", () => {
|
||||
|
||||
@@ -489,7 +489,8 @@ const live: Layer.Layer<
|
||||
// (the AI SDK `messages` array isn't enough — the LLM-native bridge
|
||||
// needs the typed parts).
|
||||
// - The bridge can route the model to one of the protocols listed in
|
||||
// `NATIVE_PROTOCOLS` (today: Anthropic only).
|
||||
// `NATIVE_PROTOCOLS`. The adapter registry is broader than this
|
||||
// allowlist so we can enable providers incrementally.
|
||||
// - If tools are present, the caller supplied a native tool definition
|
||||
// for every AI SDK tool key so the native path can dispatch them.
|
||||
//
|
||||
|
||||
Reference in New Issue
Block a user