admin/AirCoding
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix: logo 右半部分从 CODING 改为 CODE

Browse Source 
去掉难以正确渲染的 N 和 G 字母，右半部分简化为 CODE（4 字母），
与左半部分 AIR 组合为 AIR CODE。
This commit is contained in:
airlongdian
2026-06-14 09:54:53 +08:00
commit c4f9fe109e
5757 changed files with 1170016 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

321
packages/llm/AGENTS.md Normal file
View File

@@ -0,0 +1,321 @@
# LLM Package Guide
## Effect
- Prefer `HttpClient.HttpClient` / `HttpClientResponse.HttpClientResponse` over web `fetch` / `Response` at package boundaries.
- Use `Stream.Stream` for streaming data flow. Avoid ad hoc async generators or manual web reader loops unless an Effect `Stream` API cannot model the behavior.
- Use Effect Schema codecs for JSON encode/decode (`Schema.fromJsonString(...)`) instead of direct `JSON.parse` / `JSON.stringify` in implementation code.
- In `Effect.gen`, yield yieldable errors directly (`return yield* new MyError(...)`) instead of `Effect.fail(new MyError(...))`.
- Use `Effect.void` instead of `Effect.succeed(undefined)` when the successful value is intentionally void.
## Conventions
Per-type constructors live on the type, not as top-level re-exports. Use `Message.system(...)`, `Message.user(...)`, `Message.assistant(...)`, `Message.tool(...)`, `Model.make(...)`, `ToolDefinition.make(...)`, `ToolCallPart.make(...)`, `ToolResultPart.make(...)`, `ToolChoice.make(...)`, `ToolChoice.named(...)`, `SystemPart.make(...)`, and `GenerationOptions.make(...)` directly. The top-level `LLM` namespace is reserved for request-shaped call APIs: `LLM.request`, `LLM.generate`, `LLM.stream`, `LLM.updateRequest`, and `LLM.generateObject`. Two ways to construct the same thing is one too many.
## Tests
- Use `testEffect(...)` from `test/lib/effect.ts` for tests requiring Effect layers.
- Keep provider tests fixture-first. Live provider calls must stay behind `RECORD=true` and required API-key checks.
## Architecture
This package is an Effect Schema-first LLM core. The Schema classes in `src/schema/` are the canonical runtime data model. Convenience functions in `src/llm.ts` are thin constructors that return those same Schema class instances; they should improve callsites without creating a second model.
Primary in-repo integration point:
- `packages/opencode/src/session/llm.ts` is the session-owned orchestration layer that decides whether a request uses AI SDK or this package's native route runtime.
- `packages/opencode/src/session/llm/native-request.ts` is the lowering adapter from opencode's session/AI SDK-shaped data into this package's `LLMRequest` model.
- `packages/opencode/src/session/llm/native-runtime.ts` is the execution adapter that calls raw `LLMClient.stream(request)` and bridges one provider turn of opencode tool calls through this package's typed dispatcher.
- `packages/opencode/src/session/llm/ai-sdk.ts` keeps the default AI SDK path compatible by converting AI SDK stream parts into this package's shared `LLMEvent`s.
Keep this package independent of session concerns. Session auth, permissions, plugins, telemetry headers, and runtime selection belong in `packages/opencode/src/session/llm.ts` and its local adapters.
### Request Flow
The intended callsite is:
```ts
const request = LLM.request({
model: OpenAI.configure({ apiKey }).responses("gpt-4o-mini"),
system: "You are concise.",
prompt: "Say hello.",
})
const response = yield * LLMClient.generate(request)
```
`LLM.request(...)` builds an `LLMRequest`. `LLMClient.generate(...)` reads the executable route carried by `request.model.route`, builds the provider-native body, asks the route's transport for a real `HttpClientRequest.HttpClientRequest`, sends it through `RequestExecutor.Service`, parses the provider stream into common `LLMEvent`s, and finally returns an `LLMResponse`.
Use `LLMClient.stream(request)` when callers want incremental `LLMEvent`s. Use `LLMClient.generate(request)` when callers want those same events collected into an `LLMResponse`. Use `LLMClient.prepare<Body>(request)` to compile a request through the route pipeline without sending it — the optional `Body` type argument narrows `.body` to the route's native shape (e.g. `prepare<OpenAIChatBody>(...)` returns a `PreparedRequestOf<OpenAIChatBody>`). The runtime body is identical; the generic is a type-level assertion.
Filter or narrow `LLMEvent` streams with `LLMEvent.is.*` (camelCase guards, e.g. `events.filter(LLMEvent.is.toolCall)`). The kebab-case `LLMEvent.guards["tool-call"]` form also works but prefer `is.*` in new code.
### Routes
A route is the registered, runnable composition of four orthogonal pieces:
- **`Protocol`** (`src/route/protocol.ts`) — semantic API contract. Owns request body construction (`body.from`), the body schema (`body.schema`), the streaming-event schema (`stream.event`), and the event-to-`LLMEvent` state machine (`stream.step`). `Route.make(...)` validates and JSON-encodes the body from `body.schema` and decodes frames with `stream.event`. Examples: `OpenAIChat.protocol`, `OpenAIResponses.protocol`, `AnthropicMessages.protocol`, `Gemini.protocol`, `BedrockConverse.protocol`.
- **`Endpoint`** (`src/route/endpoint.ts`) — URL construction. The host, path, and route query live on the endpoint. `Endpoint.path("/chat/completions", { baseURL })` is the common case; pass a function for paths that embed the model id or a body field (e.g. `Endpoint.path(({ body }) => `/model/${body.modelId}/converse-stream`)`).
- **`Auth`** (`src/route/auth.ts`) — per-request transport authentication. Provider facades configure credentials onto the route before model selection, usually via `Auth.bearer(apiKey)` or `Auth.header(name, apiKey)`. Routes that need per-request signing (Bedrock SigV4, future Vertex IAM, Azure AAD) implement `Auth` as a function that signs the body and merges signed headers into the result.
- **`Framing`** (`src/route/framing.ts`) — bytes → frames. SSE (`Framing.sse`) is shared; Bedrock keeps its AWS event-stream framing as a typed `Framing<object>` value alongside its protocol.
Compose them via `Route.make(...)`:
```ts
export const route = Route.make({
id: "openai-chat",
provider: "openai",
protocol: OpenAIChat.protocol,
endpoint: Endpoint.path("/chat/completions", {
baseURL: "https://api.openai.com/v1",
}),
auth: Auth.bearer(),
framing: Framing.sse,
})
```
Route defaults are request-shaping defaults such as `headers`, `limits`, `generation`, `providerOptions`, and `http`. Endpoint host/query belongs on the route endpoint. Selected `Model` values carry only model id, provider id, and the configured route value. Model capability/catalog metadata lives outside this package; protocol support is enforced by request lowering and typed `LLMError`s.
The four-axis decomposition is the reason DeepSeek, TogetherAI, Cerebras, Baseten, Fireworks, and DeepInfra all reuse `OpenAIChat.protocol` verbatim — each provider deployment is a 5-15 line `Route.make(...)` call instead of a 300-400 line route clone. Bug fixes in one protocol propagate to every consumer of that protocol in a single commit.
When a provider ships a non-HTTP transport (OpenAI's WebSocket Responses backend, hypothetical bidirectional streaming APIs), the seam is `Transport` — `WebSocketTransport.jsonTransport.with(...)` constructs an IO template whose `prepare` receives the route endpoint/auth at compile time, builds a WebSocket URL and message, and whose `frames` yields decoded text from the socket. Same protocol and endpoint source, different transport.
### URL Construction
`Endpoint` owns `{ baseURL, path, query }`. Each protocol route includes a canonical endpoint when the provider has one (e.g. `https://api.openai.com/v1`); provider helpers override endpoint fields by configuring the route before selecting a model. Routes that have no canonical URL (OpenAI-compatible Chat, GitHub Copilot) require configuration before execution.
For providers where the URL is derived from typed inputs (Azure resource name, Bedrock region), the provider helper configures the route endpoint before calling `.model(...)`. Use `AtLeastOne<T>` from `route/auth-options.ts` for inputs that accept either of two derivation paths (Azure: `resourceName` or `baseURL`).
### Provider Facades
Provider-facing APIs are configured facades over route values. Endpoint/auth/resource/API-version setup happens before model selection, and model selectors accept only a model or deployment id:
```ts
const openai = OpenAI.configure({ apiKey, baseURL })
const model = openai.responses("gpt-4o-mini")
const azure = Azure.configure({ resourceName, apiKey, apiVersion: "v1" })
const deployment = azure.responses("my-deployment")
const gateway = CloudflareAIGateway.configure({ accountId, gatewayId, gatewayApiKey, apiKey })
const proxied = gateway.model("openai/gpt-4o-mini")
```
Keep provider facades small and explicit:
- Use branded `ProviderID.make(...)` and `ModelID.make(...)` where ids are constructed directly.
- Use `model` for the default API path and named methods for provider-native alternatives such as OpenAI `responses`, `responsesWebSocket`, and `chat`.
- Put provider-specific setup on `.configure(...)`; do not add `model(id, overrides)` as a duplicate construction path.
- Export lower-level `routes` arrays separately only when advanced internal wiring needs them.
- Prefer `apiKey` as provider-specific sugar and `auth` as the explicit override; keep them mutually exclusive in provider option types with `ProviderAuthOption`.
- Resolve `apiKey` → `Auth` with `AuthOptions.bearer(options, "<PROVIDER>_API_KEY")` (it honors an explicit `auth` override and falls back to `Auth.config(envVar)` so missing keys surface a typed `Authentication` error rather than a runtime crash).
- Use separate top-level facades for products with different required setup, such as `CloudflareAIGateway` and `CloudflareWorkersAI`.
`Provider.make(...)` remains available for simple static provider definitions, but new built-in providers should prefer plain configured facades unless a helper removes real duplication without adding runtime behavior.
### Folder layout
```
packages/llm/src/
schema/ canonical Schema model, split by concern
ids.ts branded IDs, literal types, ProviderMetadata
options.ts Generation/Provider/Http options, Limits, Model, cache policy
messages.ts content parts, Message, ToolDefinition, LLMRequest
events.ts Usage, individual events, LLMEvent, PreparedRequest, LLMResponse
errors.ts error reasons, LLMError, ToolFailure
index.ts barrel
llm.ts request constructors and convenience helpers
route/
index.ts @opencode-ai/llm/route advanced barrel
client.ts Route.make + LLMClient.prepare/stream/generate
executor.ts RequestExecutor service + transport error mapping
protocol.ts Protocol type + Protocol.make
endpoint.ts Endpoint type + Endpoint.path
auth.ts Auth type + Auth.bearer / Auth.apiKeyHeader / Auth.passthrough
auth-options.ts ProviderAuthOption shape, AuthOptions.bearer, AtLeastOne helper
framing.ts Framing type + Framing.sse
transport/ transport implementations
index.ts Transport type + HttpTransport / WebSocketTransport namespaces
http.ts HttpTransport.httpJson — POST + framing
websocket.ts WebSocketTransport.json + WebSocketExecutor service
protocols/
shared.ts ProviderShared toolkit used inside protocol impls
openai-chat.ts protocol + route (compose OpenAIChat.protocol)
openai-responses.ts
anthropic-messages.ts
gemini.ts
bedrock-converse.ts
bedrock-event-stream.ts framing for AWS event-stream binary frames
openai-compatible-chat.ts route that reuses OpenAIChat.protocol, no canonical URL
utils/ per-protocol helpers (auth, cache, media, tool-stream, ...)
providers/
openai-compatible.ts generic compatible helper + family model helpers
openai-compatible-profile.ts family defaults (deepseek, togetherai, ...)
azure.ts / amazon-bedrock.ts / cloudflare.ts / github-copilot.ts / google.ts / xai.ts / openai.ts / anthropic.ts / openrouter.ts
tool.ts typed tool() helper
tool-runtime.ts narrow one-call typed tool dispatcher
```
The dependency arrow points down: `providers/*.ts` files import protocol routes and auth-option utilities; protocol modules import `endpoint`, `auth`, `framing`, and transport pieces. Protocols do not import provider facades. Lower-level modules know nothing about provider catalog metadata.
### Shared protocol helpers
`ProviderShared` exports a small toolkit used inside protocol implementations to keep them focused on provider-native shapes:
- `joinText(parts)` — joins an array of `TextPart` (or anything with a `.text`) with newlines. Use this anywhere a protocol flattens text content into a single string for a provider field.
- `parseToolInput(route, name, raw)` — Schema-decodes a tool-call argument string with the canonical "Invalid JSON input for `<route>` tool call `<name>`" error message. Treats empty input as `{}`.
- `parseJson(route, raw, message)` — generic JSON-via-Schema decode for non-tool bodies.
- `eventError(route, message, ...)` — typed `InvalidProviderOutput` constructor for stream-time decode failures.
- `validateWith(decoder)` — maps Schema decode errors to `InvalidRequest`. `Route.make(...)` uses this for body validation; lower-level routes can reuse it.
- `matchToolChoice(provider, choice, branches)` — branches over `LLMRequest["toolChoice"]` for provider-specific lowering.
If you find yourself copying a 3-to-5-line snippet between two protocols, lift it into `ProviderShared` next to these helpers rather than duplicating.
### Chronological System Updates
`LLMRequest.system` is the initial privileged prompt that applies ahead of the conversation. `Message.system(...)` is a separate, provider-neutral chronological operator update inside `LLMRequest.messages`; it applies only from its position in history onward and accepts text content only.
Native chronological system messages are route/model-specific. Anthropic Messages lowers them natively for Claude Opus 4.8 (`claude-opus-4-8`). Other routes and models intentionally lower the update in place into ordinary user-compatible text using this stable escaped representation:
```text
<system-update>
...
</system-update>
```
The wrapped-user fallback preserves ordering while visibly lowering authority. Never silently pass a raw chronological `role: "system"` through a route that might reject it. Do not insert raw retrieved documents, tool output, or web content into privileged chronological system updates; keep untrusted content in ordinary user/tool channels.
### Tools
Tool loops are represented in common messages and events:
```ts
const call = ToolCallPart.make({ id: "call_1", name: "lookup", input: { query: "weather" } })
const result = Message.tool({ id: "call_1", name: "lookup", result: { forecast: "sunny" } })
const followUp = LLM.request({
model,
messages: [Message.user("Weather?"), Message.assistant([call]), result],
})
```
Routes lower these into provider-native assistant tool-call messages and tool-result messages. Streaming providers should emit `tool-input-delta` events while arguments arrive, then a final `tool-call` event with parsed input.
### Tool dispatch
`LLM.stream(request)` and `LLM.generate(request)` each run exactly one provider turn. Add tool schemas to `request.tools` with `Tool.toDefinitions(tools)`. When a caller wants the package's typed one-call execution behavior, pass each canonical local `tool-call` event to `ToolRuntime.dispatch(tools, call)`.
```ts
const get_weather = tool({
description: "Get current weather for a city",
parameters: Schema.Struct({ city: Schema.String }),
success: Schema.Struct({ temperature: Schema.Number, condition: Schema.String }),
execute: ({ city }) =>
Effect.gen(function* () {
// city: string — typed from parameters Schema
const data = yield* WeatherApi.fetch(city)
return { temperature: data.temp, condition: data.cond }
// return type checked against success Schema
}),
})
const tools = { get_weather, get_time, ... }
const events = yield* LLM.stream(
LLM.updateRequest(request, { tools: Tool.toDefinitions(tools) }),
).pipe(Stream.runCollect)
const call = Array.from(events).find(LLMEvent.is.toolCall)
if (call && !call.providerExecuted) {
const dispatched = yield* ToolRuntime.dispatch(tools, call)
// Persist call + dispatched.result, then construct the next request explicitly.
}
```
The dispatcher:
- On `tool-call`: looks up the named tool, decodes input against `parameters` Schema, dispatches to the typed `execute`, encodes the result against `success` Schema, and returns canonical `tool-result` events.
- Does not stream providers, construct Session events, schedule fibers, append history, count steps, or continue model rounds.
- Leaves persistence and continuation to the enclosing product flow.
Handler dependencies (services, permissions, plugin hooks, abort handling) are closed over by the consumer at tool-construction time. Build the tools record inside an `Effect.gen` once and reuse it across many dispatches.
Errors must be expressed as `ToolFailure`. The runtime catches it and emits a `tool-error` event, then a `tool-result` of `type: "error"`, so the model can self-correct on the next step. Anything that is not a `ToolFailure` is treated as a defect and fails the stream. Three recoverable error paths produce `tool-error` events:
- The model called an unknown tool name.
- Input failed the `parameters` Schema.
- The handler returned a `ToolFailure`.
Provider-defined / hosted tools (Anthropic `web_search` / `code_execution` / `web_fetch`, OpenAI Responses `web_search_call` / `file_search_call` / `code_interpreter_call` / `mcp_call` / `local_shell_call` / `image_generation_call` / `computer_use_call`) pass through the runtime untouched:
- Routes surface the model's call as a `tool-call` event with `providerExecuted: true`, and the provider's result as a matching `tool-result` event with `providerExecuted: true`.
- Callers detect `providerExecuted` on `tool-call` and **skip local dispatch** — no handler is invoked and no `tool-error` is raised for "unknown tool". The provider already executed it.
- Callers that continue should retain both events in explicit history when the protocol requires it. Anthropic encodes them back as `server_tool_use` + `web_search_tool_result` (or `code_execution_tool_result` / `web_fetch_tool_result`) blocks; OpenAI Responses callers typically use `previous_response_id` instead of resending hosted-tool items.
Add provider-defined tools to `request.tools` (no runtime entry needed). The matching route must know how to lower the tool definition into the provider-native shape; right now Anthropic accepts `web_search` / `code_execution` / `web_fetch` and OpenAI Responses accepts the hosted tool names listed above.
## Protocol File Style
Protocol files should look self-similar. Provider quirks belong behind named helpers so a new route can be reviewed by comparing the same sections across files.
### Section order
Use this order for every protocol module:
1. Public model input
2. Request body schema
3. Streaming event schema
4. Parser state
5. Request body construction (`fromRequest`)
6. Stream parsing (`step` and per-event handlers)
7. Protocol and route
8. Protocol route export
### Rules
- Keep protocol files focused on the protocol. Move provider-specific projection, signing, media normalization, or other bulky transformations into `src/protocols/utils/*`.
- Use `Effect.fn("Provider.fromRequest")` for request body construction entrypoints. Use `Effect.fn(...)` for event handlers that yield effects; keep purely synchronous handlers as plain functions returning a `StepResult` that the dispatcher lifts via `Effect.succeed(...)`.
- Parser state owns terminal information. The state machine records finish reason, usage, and pending tool calls; emit one terminal `finish` event (or `provider-error`) for each completed response. If a provider splits reason and usage across events, merge them in parser state before flushing.
- Emit exactly one terminal `finish` event for a completed response, normally after a matching `step-finish`. Use `stream.terminal` to stop reading when the provider has a completion sentinel; use `stream.onHalt` when the final event must be flushed after the framed stream ends.
- Use shared helpers for repeated protocol policy such as text joining, usage totals, JSON parsing, and tool-call accumulation. `ToolStream` (`protocols/utils/tool-stream.ts`) accumulates streamed tool-call arguments uniformly.
- Make intentional provider differences explicit in helper names or comments. If two protocol files differ visually, the reason should be obvious from the names.
- Prefer dispatched per-event handlers (`onMessageStart`, `onContentBlockDelta`, ...) called from a small top-level `step` switch over a long if-chain. The dispatcher keeps the event surface visible at a glance.
- Keep tests in the same conceptual order as the protocol: basic prepare, tools prepare, unsupported lowering, text/usage parsing, tool streaming, finish reasons, provider errors.
### Review checklist
- Can the file be skimmed side-by-side with `openai-chat.ts` without hunting for equivalent sections?
- Are provider quirks named, isolated, and covered by focused tests?
- Does request body construction validate unsupported common content at the protocol boundary?
- Does stream parsing emit stable common events without leaking provider event order to callers?
- Does `toolChoice: "none"` behavior read as intentional?
## Recording Tests
Recorded tests use one cassette file per scenario. A cassette holds an ordered array of `{ request, response }` interactions, so multi-step flows (tool loops, retries, polling) record into a single file. Use `recordedTests({ prefix, requires })` and let the helper derive cassette names from test names:
```ts
const recorded = recordedTests({ prefix: "openai-chat", requires: ["OPENAI_API_KEY"] })
recorded.effect("streams text", () =>
Effect.gen(function* () {
// test body
}),
)
```
Replay is the default. `RECORD=true` records fresh cassettes and requires the listed env vars. Cassettes are written as pretty-printed JSON so multi-interaction diffs stay reviewable.
Pass `provider`, `protocol`, and optional `tags` to `recordedTests(...)` / `recorded.effect.with(...)` so cassettes carry searchable metadata. Use recorded-test filters to replay or record a narrow subset without rewriting a whole file:
- `RECORDED_PROVIDER=openai` matches tests tagged with `provider:openai`; comma-separated values are allowed.
- `RECORDED_PREFIX=openai-chat` matches cassette groups by `recordedTests({ prefix })`; comma-separated values are allowed.
- `RECORDED_TAGS=tool` requires all listed tags to be present, e.g. `RECORDED_TAGS=provider:togetherai,tool`.
- `RECORDED_TEST="streams text"` matches by test name, kebab-case test id, or cassette path.
Filters apply in replay and record mode. Combine them with `RECORD=true` when refreshing only one provider or scenario.
**Binary response bodies.** Most providers stream text (SSE, JSON). The recorder treats known textual media types (`text/*`, JSON/XML structured types, JavaScript, forms, YAML, and SVG) as text and stores every other response as base64 with `bodyEncoding: "base64"`. This preserves binary formats such as AWS event-stream frames without a lossy UTF-8 round trip.
**Matching strategy.** Replay walks the cassette in record order via an internal cursor: the Nth runtime request is served by the Nth recorded interaction, and each one is validated by comparing method, URL, allow-listed headers, and the canonical JSON body. This handles tool loops (each round's request differs as history grows) and retry/polling scenarios (successive byte-identical requests with different responses) uniformly. If a test reorders its requests, re-record the cassette. `scriptedResponses` (in `test/lib/http.ts`) is the deterministic counterpart for tests that don't need a live provider; it scripts response bodies in order without reading from disk.
Do not blanket re-record an entire test file when adding one cassette. `RECORD=true` rewrites every recorded case that runs, and provider streams contain volatile IDs, timestamps, fingerprints, and obfuscation fields. Prefer deleting the one cassette you intend to refresh, or run a focused test pattern that only registers the scenario you want to record. Keep stable existing cassettes unchanged unless their request shape or expected behavior changed.

131
packages/llm/README.md Normal file
View File

@@ -0,0 +1,131 @@
# @opencode-ai/llm
Schema-first LLM core for opencode. One typed request, response, event, and tool language; provider quirks live in adapters, not in calling code.
```ts
import { Effect } from "effect"
import { LLM, LLMClient } from "@opencode-ai/llm"
import { OpenAI } from "@opencode-ai/llm/providers"
const model = OpenAI.configure({ apiKey: process.env.OPENAI_API_KEY }).responses("gpt-4o-mini")
const request = LLM.request({
model,
system: "You are concise.",
prompt: "Say hello in one short sentence.",
generation: { maxTokens: 40 },
})
const program = Effect.gen(function* () {
const response = yield* LLMClient.generate(request)
console.log(response.text)
})
```
Run `LLMClient.stream(request)` instead of `generate` when you want incremental `LLMEvent`s. The event stream is provider-neutral — same shape across OpenAI Chat, OpenAI Responses, Anthropic Messages, Gemini, Bedrock Converse, and any OpenAI-compatible deployment.
## Public API
- **`LLM.request({...})`** — build a provider-neutral `LLMRequest`. Accepts ergonomic inputs (`system: string`, `prompt: string`) that normalize into the canonical Schema classes.
- **`LLM.generate` / `LLM.stream`** — re-exported from `LLMClient` for one-import use.
- **`Message.user(...)` / `Message.assistant(...)` / `Message.tool(...)`** — message constructors from the canonical schema model.
- **`Model.make(...)` / `ToolCallPart.make(...)` / `ToolResultPart.make(...)` / `ToolDefinition.make(...)`** — model and tool-related constructors from the canonical schema model.
- **`LLMClient.prepare(request)`** — compile a request through protocol body construction, validation, and HTTP preparation without sending. Useful for inspection and testing.
- **`LLMEvent.is.*`** — typed guards (`is.textDelta`, `is.toolCall`, `is.finish`, …) for filtering streams.
## Caching
Prompt caching is **on by default**. Every `LLMRequest` resolves to `cache: "auto"` unless the caller opts out with `cache: "none"`. Each protocol translates `CacheHint`s to its wire format (`cache_control` on Anthropic, `cachePoint` on Bedrock; OpenAI and Gemini do implicit caching server-side and don't need inline markers — auto is a no-op there).
### Auto placement
`"auto"` places three breakpoints — last tool definition, last system part, latest user message. The last-user-message boundary is the load-bearing detail: in a tool-use loop, a single user turn expands into many assistant/tool round-trips, all sharing that prefix. Caching at that boundary lets every intra-turn API call hit.
The math justifies the default: Anthropic's 5-minute cache write is 1.25× base, read is 0.1×, so a single reuse within 5 minutes already wins. One-shot completions below the per-model minimum-cacheable-token threshold silently no-op on the wire, so the worst case is harmless.
### Opting out
```ts
LLM.request({
model,
system,
prompt: "one-off question",
cache: "none",
})
```
### Granular policy
```ts
cache: {
tools?: boolean,
system?: boolean,
messages?: "latest-user-message" | "latest-assistant" | { tail: number },
ttlSeconds?: number, // ≥ 3600 → 1h on Anthropic/Bedrock; else 5m
}
```
### Manual hints
Inline `CacheHint` on any text / system / tool / tool-result part overrides automatic placement. The auto policy preserves manual hints; it only fills gaps.
```ts
LLM.request({
model,
system: [
{ type: "text", text: "stable system prompt", cache: { type: "ephemeral" } },
],
...
})
```
### Provider behavior table
| Protocol | `cache: "auto"` |
| ----------------------- | ------------------------------------------------------------------------- |
| Anthropic Messages | emits up to 3 `cache_control` markers (4-breakpoint cap enforced) |
| Bedrock Converse | emits up to 3 `cachePoint` blocks (4-breakpoint cap enforced) |
| OpenAI Chat / Responses | no-op (implicit caching above 1024 tokens) |
| Gemini | no-op (implicit caching on 2.5+; explicit `CachedContent` is out-of-band) |
Normalized cache usage is read back into `response.usage.cacheReadInputTokens` and `cacheWriteInputTokens` across every provider.
## Providers
Provider facades configure endpoint/auth/deployment details first, then expose model selectors that take only a model or deployment id. The selected model carries the executable route value used at runtime.
```ts
import { OpenAI, CloudflareAIGateway } from "@opencode-ai/llm/providers"
const openai = OpenAI.configure({ apiKey: process.env.OPENAI_API_KEY }).responses("gpt-4o-mini")
const gateway = CloudflareAIGateway.configure({
accountId: process.env.CLOUDFLARE_ACCOUNT_ID,
gatewayApiKey: process.env.CLOUDFLARE_API_TOKEN,
}).model("workers-ai/@cf/meta/llama-3.1-8b-instruct")
```
Included providers: OpenAI, Anthropic, Google (Gemini), Amazon Bedrock, Azure OpenAI, Cloudflare AI Gateway, Cloudflare Workers AI, GitHub Copilot, OpenRouter, xAI, plus generic OpenAI-compatible helpers for DeepSeek, Cerebras, Groq, Fireworks, Together, etc.
## Provider options & HTTP overlays
Three escape hatches in order of stability:
1. **`generation`** — portable knobs (`maxTokens`, `temperature`, `topP`, `topK`, penalties, seed, stop).
2. **`providerOptions: { <provider>: {...} }`** — typed-at-the-facade provider-specific knobs (OpenAI `promptCacheKey`, Anthropic `thinking`, Gemini `thinkingConfig`, OpenRouter routing).
3. **`http: { body, headers, query }`** — last-resort serializable overlays merged into the final HTTP request. Reach for this only when a stable typed path doesn't yet exist.
Route/provider defaults are overridden by request-level values for each axis.
## Routes
Adding a new model or deployment is usually 5-15 lines using `Route.make({ protocol, endpoint, auth, framing, ... })`. The route owns endpoint/auth/framing and the protocol owns body construction plus stream parsing. Transports are reusable IO templates that receive route endpoint/auth at compile time. Capability/catalog metadata lives outside this low-level package; unsupported request shapes fail during protocol lowering. See `AGENTS.md` for the architectural detail.
## Effect
This package is built on Effect. Public methods return `Effect` or `Stream`; provide `LLMClient.layer` for runtime dispatch and import the provider/protocol modules for the routes you use. The example at `example/tutorial.ts` is a runnable walkthrough.
## See also
- `AGENTS.md` — architecture, route construction, contributor guide
- `example/tutorial.ts` — runnable end-to-end walkthrough
- `test/provider/*.test.ts` — fixture-first protocol tests; `*.recorded.test.ts` files cover live cassettes

591
packages/llm/example/call-sites.md Normal file
View File

@@ -0,0 +1,591 @@
# LLM Call Site Sketches
Scratchpad for examples first, abstractions second. Current direction: routes
execute, provider facades organize configured route sets, and models carry route
values directly.
## Conversation Summary
Kit and Aidan want provider-specific LLM behavior to move out of opencode's AI
SDK transform path and into `packages/llm` where possible. The goal is not a big
generic transform layer; the goal is small composable route definitions backed by
recorded golden tests.
Things to keep testing against:
- Cache placement: `cache: "auto"`, manual cache breakpoints, provider cache usage.
- Images: golden image tests for providers/protocols that claim image support.
- Reasoning: canonical reasoning parts/events versus provider-native knobs.
- Auth: bearer, custom headers, multiple credentials, query auth, SigV4, OAuth, no auth.
- OpenAI-compatible providers: DeepSeek, Together, Groq, Alibaba/DashScope, custom routers.
- Provider switching: stale signatures, encrypted reasoning, provider metadata, incompatible parts.
- Error quality: typed errors instead of generic SDK/server failures.
## Final Guide: Routes Execute, Providers Organize
Do not introduce a first-class `Deployment` abstraction unless it gains real
semantics. Provider facades are ergonomic configured route groups, not execution
registries. The executable/composable thing is still a route. Do not make route
construction publish to a global registry; models should carry their route value
directly.
Keep durable identity separate from runtime capability:
- Durable identity is small serializable data like `{ providerID, modelID }` for
config, sessions, logs, and catalogs.
- Runtime capability is a `Model` with a route value, protocol, transport, auth,
and defaults. It is allowed to contain functions and schemas.
- If persisted identity needs to become executable, resolve it through an app
boundary first. Do not make `LLMRequest` recover behavior from a global route
side table.
Keep unconfigured behavior values as values, not factories. A transport like
`HttpTransport.sseJson` should be a reusable immutable value. Use a function only
when the caller supplies options or when construction needs fresh state.
Use constants to remove repetition before inventing abstractions. Provider ids
are branded once per provider facade and reused across routes; a plain exported
object is enough for the provider-facing API unless a helper earns its keep by
removing repeated route projection.
Expose default configured provider instances, and put provider-specific setup on
`.configure(...)`. Model selectors stay pure: `model(id)`, `responses(id)`,
`chat(id)`, etc. Endpoint/auth/resource/api-version configuration happens before
model selection, not as a second argument to model selection.
Use provider/product facades consistently:
- One coherent provider/product config surface gets one top-level facade.
- APIs/model kinds that share that config are methods on the facade.
- Different products with different required config get separate top-level
facades, not a shared namespace with unrelated children.
- Default facades are exposed only when concrete defaults or lazy env/credential
defaults make the facade valid.
Examples:
```ts
OpenAI.responses("gpt-4o")
OpenAI.chat("gpt-4o")
OpenAI.responsesWebSocket("gpt-4o")
Azure.configure({ resourceName, apiKey }).responses("my-deployment")
AmazonBedrock.configure({ region, credentials }).model("anthropic.claude-3-5-sonnet-20241022-v2:0")
CloudflareAIGateway.configure({ accountId, gatewayId, gatewayApiKey, apiKey }).model("openai/gpt-4o")
CloudflareWorkersAI.configure({ accountId, apiKey }).model("@cf/meta/llama-3.1-8b-instruct")
OpenAICompatible.configure({
provider: "custom",
baseURL: "https://custom.example/v1",
auth: Auth.bearer(apiKey),
}).model("custom-model")
```
Standardize the provider facade contract before abstracting construction. A
plain object is enough at first; add a helper only if repeated route projection
starts hiding the real provider-specific config.
`Route.with(...)` patch semantics should be boring and explicit:
- Omitted fields inherit from the original route.
- `endpoint` patches merge with the existing endpoint, so overriding `baseURL`
keeps the existing `path`.
- `endpoint.query` merges by default; later values win.
- `auth` replaces.
- `headers` merge by default; undefined values are omitted.
- `id` is optional in patches. Route ids are diagnostic/provider API labels, not
global runtime registry keys.
1. **Route**
- route id
- provider id
- protocol
- body schema
- body builder
- stream event schema
- parser/state machine
- transport
- method / IO shape
- framing
- request preparation
- constants when unconfigured; functions only when configured
- endpoint
- base URL
- static path
- body/model-derived path
- query params
- auth
- bearer
- custom header
- multiple credentials
- SigV4
- none
- defaults
- headers
- generation defaults
- provider options
- limits
2. **Provider Facade**
- default configured provider instance
- provider-specific `.configure(...)`
- plain object/function facade over one or more routes
- top-level export only when it represents one coherent config surface
- no passive `Provider.make(...)` wrapper unless it gains runtime behavior
3. **Model Selector**
- route/provider-owned selector
- accepts model id only
- returns executable models
- does not accept endpoint/auth/deployment overrides
4. **Model**
- model id
- route value
- provider id
- configured route value at selection time
5. **LLM Request**
- model
- messages/tools
- generation/cache/reasoning/response-format options
- request-level HTTP overlays for per-request headers/query/body additions,
not provider endpoint/auth reconfiguration
6. **Compile**
- read route from model
- merge route defaults and request overrides
- build final URL from route endpoint
- apply auth from the configured route
- build body with protocol
- execute with transport and parse with protocol
## Provider Facade Shape
The provider abstraction is a facade over configured routes, not the runtime
execution mechanism:
```ts
type ProviderFacade<APIs, Config> = {
readonly id: ProviderID
readonly model: (id: string) => Model
readonly configure: (input?: Config) => ProviderFacade<APIs, Config>
} & APIs
```
Manual construction is fine and should be the default until duplication earns a
helper:
```ts
export const OpenAI = {
id: openAIProvider,
model: openAIResponses.model,
responses: openAIResponses.model,
chat: openAIChat.model,
configure: configureOpenAI,
} satisfies ProviderFacade<
{
responses: (id: string) => Model
chat: (id: string) => Model
},
OpenAIConfig
>
```
If several providers repeat the same projection from route values to model
methods, the helper can stay deliberately tiny:
```ts
const configureOpenAI = (input: OpenAIConfig = {}) =>
Provider.define({
id: openAIProvider,
routes: {
responses: openAIResponses.with(openAIConfig(input)),
chat: openAIChat.with(openAIConfig(input)),
},
default: "responses",
configure: configureOpenAI,
})
export const OpenAI = configureOpenAI()
```
`Provider.define(...)` would only project route methods and preserve types:
```ts
OpenAI.model("gpt-4o")
OpenAI.responses("gpt-4o")
OpenAI.chat("gpt-4o")
OpenAI.configure({ apiKey }).responses("gpt-4o")
```
It must not register routes, select routes dynamically, or participate in
execution. Execution still reads the route value carried by the model.
## Ideal Call Sites
Define concrete routes for a native provider, then project them through a
provider facade:
```ts
const openAIProvider = ProviderID.make("openai")
const openAIResponses = Route.make({
id: "openai-responses",
provider: openAIProvider,
protocol: OpenAIResponses.protocol,
transport: HttpTransport.sseJson,
endpoint: {
baseURL: "https://api.openai.com/v1",
path: "/responses",
},
auth: Auth.envBearer("OPENAI_API_KEY"),
})
const openAIChat = Route.make({
id: "openai-chat",
provider: openAIProvider,
protocol: OpenAIChat.protocol,
transport: HttpTransport.sseJson,
endpoint: {
baseURL: "https://api.openai.com/v1",
path: "/chat/completions",
},
auth: Auth.envBearer("OPENAI_API_KEY"),
})
const openAIResponsesWebSocket = openAIResponses.with({
id: "openai-responses-websocket",
transport: WebSocketTransport.json,
})
const openAIConfig = (input: OpenAIConfig) => ({
endpoint: input.endpoint,
auth: input.auth ?? (input.apiKey ? Auth.bearer(input.apiKey) : undefined),
headers: {
"OpenAI-Organization": input.organization,
"OpenAI-Project": input.project,
},
})
const configureOpenAI = (input: OpenAIConfig = {}) => {
const responses = openAIResponses.with(openAIConfig(input))
const responsesWebSocket = openAIResponsesWebSocket.with(openAIConfig(input))
const chat = openAIChat.with(openAIConfig(input))
return {
id: openAIProvider,
responses: responses.model,
responsesWebSocket: responsesWebSocket.model,
chat: chat.model,
model: responses.model,
configure: configureOpenAI,
}
}
export const OpenAI = configureOpenAI()
```
Specialize it functionally for concrete providers:
```ts
const deepSeekProvider = ProviderID.make("deepseek")
const deepseekChat = openAIChat.with({
id: "deepseek-chat",
provider: deepSeekProvider,
endpoint: {
baseURL: "https://api.deepseek.com/v1",
},
auth: Auth.envBearer("DEEPSEEK_API_KEY"),
})
const configureDeepSeek = (input: OpenAICompatibleConfig = {}) => {
const route = deepseekChat.with({
endpoint: input.endpoint,
auth: input.auth ?? (input.apiKey ? Auth.bearer(input.apiKey) : undefined),
})
return {
id: deepSeekProvider,
model: route.model,
configure: configureDeepSeek,
}
}
export const DeepSeek = {
id: deepSeekProvider,
model: deepseekChat.model,
configure: configureDeepSeek,
}
```
Provider-specific configuration happens before model selection:
```ts
const deepseek = DeepSeek.configure({
endpoint: {
baseURL: "https://proxy.example.com/v1",
},
auth: Auth.bearer(apiKey),
})
const model = deepseek.model("deepseek-chat")
```
Final request call site stays boring:
```ts
const response =
yield *
LLM.generate(
LLM.request({
model: DeepSeek.model("deepseek-chat"),
prompt: "Hello.",
}),
)
```
HTTP versus WebSocket is represented as named route selectors, not as model or
request overrides. Same protocol, different transport, different route:
```ts
OpenAI.responses("gpt-4o")
OpenAI.responsesWebSocket("gpt-4o")
```
The client should not require a different public layer just because a selected
route uses WebSocket. Use one `LLMClient.layer` with HTTP and WebSocket runtime
capabilities available; routes that do not need WebSocket simply never touch it.
If a WebSocket route is selected in an environment without WebSocket support,
fail with a typed transport configuration error.
Azure is a route specialization with auth/path/default changes plus input
mapping. The public API configures the Azure resource once, then selects
deployment ids with pure model selectors:
```ts
const azureProvider = ProviderID.make("azure")
const azureResponses = openAIResponses.with({
id: "azure-openai-responses",
provider: azureProvider,
auth: Auth.envHeader("api-key", "AZURE_OPENAI_API_KEY"),
})
const configureAzure = (input: AzureConfig = {}) => {
const route = azureResponses.with({
endpoint: {
baseURL:
input.baseURL ??
Endpoint.envBaseURL(
"AZURE_RESOURCE_NAME",
(resourceName) => `https://${resourceName}.openai.azure.com/openai/v1`,
),
query: { "api-version": input.apiVersion ?? "v1" },
},
auth: input.apiKey ? Auth.header("api-key", input.apiKey) : Auth.envHeader("api-key", "AZURE_OPENAI_API_KEY"),
})
return {
id: azureProvider,
model: route.model,
responses: route.model,
configure: configureAzure,
}
}
export const Azure = configureAzure()
const azure = Azure.configure({
resourceName: "my-resource",
apiVersion: "v1",
})
const model = azure.responses("my-deployment")
```
Default provider facades are only valid when required configuration has a lazy
default source. `Azure.responses("my-deployment")` can be valid if endpoint
resolution reads `AZURE_RESOURCE_NAME` lazily and fails with a typed
configuration error when missing. If a provider has no sensible lazy default,
do not expose a default model selector; expose only a configured entrypoint.
Cloudflare AI Gateway and Workers AI are separate product facades because their
configuration surfaces differ. Do not make a root `Cloudflare.configure(...)`
pretend there is one coherent Cloudflare provider configuration:
```ts
const cloudflareProvider = ProviderID.make("cloudflare-ai-gateway")
const cloudflareOpenAIChat = openAIChat.with({
id: "cloudflare-ai-gateway-openai-chat",
provider: cloudflareProvider,
auth: Auth.bearerHeader("cf-aig-authorization").andThen(Auth.bearer()),
})
const configureCloudflareAIGateway = (input: CloudflareAIGatewayConfig) => {
const route = cloudflareOpenAIChat.with({
endpoint: {
baseURL: `https://gateway.ai.cloudflare.com/v1/${input.accountId}/${input.gatewayId}/openai`,
},
auth: Auth.bearerHeader("cf-aig-authorization", input.gatewayApiKey).andThen(Auth.bearer(input.apiKey)),
})
return {
id: cloudflareProvider,
model: (modelID: string) => route.model({ id: modelID }),
configure: configureCloudflareAIGateway,
}
}
export const CloudflareAIGateway = {
id: cloudflareProvider,
configure: configureCloudflareAIGateway,
}
const gateway = CloudflareAIGateway.configure({
accountId: "account",
gatewayId: "gateway",
gatewayApiKey,
apiKey,
})
const model = gateway.model("openai/gpt-4o")
```
If a Cloudflare product gains a full lazy env default, it can expose a direct
selector too. Until then, omitting `CloudflareAIGateway.model(...)` makes missing
account/gateway configuration unrepresentable.
opencode's dynamic runtime should construct executable models at its app
boundary instead of exposing a giant unstructured public model constructor or a
generic dynamic resolver:
```ts
const model =
providerID === "azure"
? Azure.configure(resolvedAzureConfig).responses(apiModelID)
: endpoint.websocket
? OpenAI.responsesWebSocket(apiModelID)
: OpenAI.responses(apiModelID)
```
That boundary can branch on durable config/catalog metadata and call typed
provider APIs directly. Transport selection belongs there too: map metadata like
`endpoint.websocket` to `OpenAI.responsesWebSocket(apiModelID)`; otherwise use
the normal `OpenAI.responses(apiModelID)` route. The client runtime only executes
the route carried by the model.
## Competitive Shape
This follows the strongest parts of adjacent libraries:
- AI SDK: configured provider instances expose provider-specific model methods.
- Effect AI: executable models carry provider requirements and can be resolved by
an app boundary.
- LiteLLM/opencode config: dynamic `providerID/modelID` branching belongs at the
app boundary, not in the typed public provider API or a global runtime
resolver.
- LangChain/LlamaIndex: constructor-style config plus model id is convenient,
but we avoid making model selection also configure endpoint/auth.
The chosen split is:
```txt
Route = execution mechanics
Provider facade = configured route group
Model = selected executable model carrying route value
App boundary = explicit durable-config -> typed-provider call
```
## What This Removes
- No `Provider.make(...)` as a core abstraction.
- No `Provider.make(...)` wrapper just to bind an id to model functions. Use a
branded provider id constant and a plain exported provider facade.
- No `Deployment.define(...)` unless future examples force it.
- No global route registry as the normal execution path.
- No import side effects required before a model can execute.
- No duplicate `provider.id` object when selected models already carry provider
id.
- No `model(id, overrides)` escape hatch. Model selection takes the model id;
endpoint/auth/deployment customization happens by configuring the route first.
- No transport override on model/request. HTTP SSE versus WebSocket is a named
route selector such as `responses` versus `responsesWebSocket`.
- No separate public `LLMClient.layerWithWebSocket`. The runtime should expose one
client layer with the available transport capabilities.
- No executable `ModelRef`. The executable handle is `Model`; durable model
identity stays separate and cannot execute on its own.
## Implementation Todo
- [x] Replace the current executable `ModelRef` with `Model`.
- [x] Change `Model.route` to carry a route value, not a `RouteID` string.
- [ ] Keep a separate durable model identity type for persisted/session/catalog
data, likely `{ providerID, modelID }`, and make it clear that it cannot
execute without resolver context.
- [x] Change route model selectors so `route.model(id)` returns an executable
model with the route value attached, not a globally registered route id.
- [x] Remove the standalone `Route.model(route, defaults, mapInput)` helper;
configured route instances own model selection.
- [x] Remove endpoint/auth escape hatches from route model selection; callers must
configure endpoint/auth through `route.with(...)` or provider facades before
calling `.model(...)`.
- [x] Remove request-shaping defaults from `Model`; selected models now carry only
id, provider, and configured route while defaults live on routes or requests.
- [x] Rework `LLMClient.prepare` / `stream` / `generate` to read
`request.model.route` directly instead of calling `registeredRoute(...)`.
- [x] Remove `Route.make(...)` global registration from the normal execution
path; keep route ids only as diagnostics/provider API labels.
- [x] Model endpoint as `{ baseURL, path, query }` on routes, then remove the
current split where host/query live on the model and path lives in route
transport setup.
- [x] Define `Route.with(...)` with explicit patch semantics for endpoint merge,
query merge, header merge, auth replacement, and optional diagnostic id.
- [x] Make unconfigured transports reusable constants such as
`HttpTransport.sseJson`; keep transport functions only for configured/fresh
state construction.
- [x] Collapse the public WebSocket runtime split so one `LLMClient.layer`
exposes available transport capabilities and selected routes fail with typed
transport config errors when a required capability is missing.
- [x] Convert OpenAI provider APIs to provider-facade shape:
`OpenAI.configure(config).responses(id)`, `.chat(id)`, and
`.responsesWebSocket(id)`.
- [x] Convert Azure to a configured facade where resource/base URL/api version
setup happens before selecting deployment ids.
- [x] Split Cloudflare products into separate facades such as
`CloudflareAIGateway` and `CloudflareWorkersAI`; do not expose a shared root
config surface unless one product actually exists.
- [x] Migrate remaining built-in provider facades one at a time so configuration
happens before model selection and selectors accept only ids:
xAI, GitHub Copilot, OpenRouter, OpenAI-compatible families, Anthropic,
Google/Gemini, and Amazon Bedrock now use configured facades such as
`Provider.configure(options).model(id)` with named selectors where needed.
- [ ] Decide whether a tiny `Provider.define(...)` helper is warranted after two
or three provider conversions; start with plain objects if duplication is not
yet painful.
- [x] Update `packages/opencode/src/session/llm/native-request.ts` to construct
executable models at the session boundary with explicit provider facade
calls, mapping catalog metadata such as `endpoint.websocket` to the correct
named route selector.
- [ ] Update tests so direct route/provider tests assert route values are carried
by executable models, and opencode/native tests assert boundary-based route
selection.
- [ ] Remove compatibility exports or stale docs only after internal call sites
are migrated; do not keep duplicate constructor paths without an external
compatibility need.
## Open Questions
- Default facades with required setup: should providers like Azure and Bedrock
expose default model selectors only when all required setup has lazy env or
credential-chain defaults? If not, omit the default selector so missing config
is impossible at the type/API level.
- Lazy endpoint/auth values: should `Endpoint.envBaseURL(...)` and env-backed
auth produce typed configuration/authentication errors at compile/prepare time
or only when executing the transport?
- `Route.with(...)` clearing semantics: endpoint/query/header patches merge by
default, but what is the explicit way to remove an inherited value?
- Provider facade helper: keep plain objects until duplication hurts, or add a
tiny `Provider.define(...)` immediately to enforce shape and method projection?
- Auth shape: should auth stay as today's composable `Auth`, or split into an
auth placement/strategy and credential sources?
- Naming: is `baseURL` still the right endpoint field name, or should it be
`origin` / `urlPrefix` to clarify that route `path` is appended?

255
packages/llm/example/tutorial.ts Normal file
View File

@@ -0,0 +1,255 @@
import { Config, Effect, Formatter, Layer, Schema, Stream } from "effect"
import { LLM, LLMClient, Message, ProviderID, Tool, ToolRuntime } from "@opencode-ai/llm"
import { Route, Auth, Endpoint, Framing, Protocol, RequestExecutor, WebSocketExecutor } from "@opencode-ai/llm/route"
import { OpenAI } from "@opencode-ai/llm/providers"
/**
* A runnable walkthrough of the LLM package use-site API.
*
* Run from `packages/llm` with an OpenAI key in the environment:
*
* OPENAI_API_KEY=... bun example/tutorial.ts
*
* The file is intentionally written as a normal TypeScript program. You can
* hover imports and local values to see how the public API is typed.
*/
const apiKey = Config.redacted("OPENAI_API_KEY")
// 1. Pick a model. The provider helper records provider identity, protocol
// choice, capabilities, deployment options, authentication, and defaults.
const model = OpenAI.configure({
apiKey,
generation: { maxTokens: 160 },
providerOptions: {
openai: { store: false },
},
}).model("gpt-4o-mini")
// 2. Build a provider-neutral request. This is useful when reusing one request
// across generate and stream examples.
//
// Options can live on both the configured route/provider facade and the request:
//
// - `generation`: common controls such as max tokens, temperature, topP/topK,
// penalties, seed, and stop sequences.
// - `providerOptions`: namespaced provider-native behavior. For example,
// OpenAI cache keys and store behavior, Anthropic thinking, Gemini thinking
// config, or OpenRouter routing/reasoning.
// - `http`: last-resort serializable overlays for final request body, headers,
// and query params. Prefer typed `providerOptions` when a field is stable.
//
// Route/provider options are defaults. Request options override them for this call.
const request = LLM.request({
model,
system: "You are concise and practical.",
prompt: "Tell me a joke",
generation: { maxTokens: 80, temperature: 0.7 },
providerOptions: {
openai: { promptCacheKey: "tutorial-joke" },
},
})
// `http` is intentionally not needed for normal calls. This shows the shape for
// newly released provider fields before they deserve a typed provider option.
const rawOverlayExample = LLM.request({
model,
prompt: "Show the final HTTP overlay shape.",
http: {
body: { metadata: { example: "tutorial" } },
headers: { "x-opencode-tutorial": "1" },
query: { debug: "1" },
},
})
// 3. `generate` sends the request and collects the event stream into one
// response object. `response.text` is the collected text output.
const generateOnce = Effect.gen(function* () {
const response = yield* LLM.generate(request)
console.log("\n== generate ==")
console.log("generated text:", response.text)
console.log("usage", Formatter.formatJson(response.usage, { space: 2 }))
})
// 4. `stream` exposes provider output as common `LLMEvent`s for UIs that want
// incremental text, reasoning, tool input, usage, or finish events.
const streamText = LLM.stream(request).pipe(
Stream.tap((event) =>
Effect.sync(() => {
if (event.type === "text-delta") process.stdout.write(`\ntext: ${event.text}`)
if (event.type === "finish") process.stdout.write(`\nfinish: ${event.reason}\n`)
}),
),
Stream.runDrain,
)
// 5. Tools are typed with Effect Schema. Provider turns remain explicit:
// advertise definitions on the request, stream one turn, dispatch local calls,
// then persist/build follow-up history in the enclosing product flow.
const tools = {
get_weather: Tool.make({
description: "Get current weather for a city.",
parameters: Schema.Struct({ city: Schema.String }),
success: Schema.Struct({ forecast: Schema.String }),
execute: (input) => Effect.succeed({ forecast: `${input.city}: sunny, 72F` }),
}),
}
const streamWithTools = Effect.gen(function* () {
const request = LLM.request({
model,
prompt: "Use get_weather for San Francisco, then answer in one sentence.",
generation: { maxTokens: 80, temperature: 0 },
tools: Tool.toDefinitions(tools),
})
const events = Array.from(yield* LLM.stream(request).pipe(Stream.runCollect))
for (const event of events) {
if (event.type === "tool-call") console.log("tool call", event.name, event.input)
if (event.type === "text-delta") process.stdout.write(event.text)
if (event.type !== "tool-call" || event.providerExecuted) continue
const dispatched = yield* ToolRuntime.dispatch(tools, event)
console.log("tool result", event.name, dispatched.result)
// A durable agent would persist these messages before starting another
// raw model turn. This tutorial keeps the boundary visible instead.
const followUp = LLM.updateRequest(request, {
messages: [
...request.messages,
Message.assistant([event]),
Message.tool({ ...event, result: dispatched.result }),
],
})
console.log("follow-up history messages:", followUp.messages.length)
}
})
// 6. `generateObject` is the structured-output helper. It forces a synthetic
// tool call internally, so the same call site works across providers instead of
// depending on provider-specific JSON mode flags.
const WeatherReport = Schema.Struct({
city: Schema.String,
forecast: Schema.String,
highFahrenheit: Schema.Number,
})
const generateStructuredObject = Effect.gen(function* () {
const response = yield* LLM.generateObject({
model,
system: "Return only structured weather data.",
prompt: "Give me today's weather for San Francisco.",
schema: WeatherReport,
generation: { maxTokens: 120, temperature: 0 },
})
console.log("\n== generateObject ==")
console.log(Formatter.formatJson(response.object, { space: 2 }))
})
// If the shape is only known at runtime, pass raw JSON Schema instead. The
// `.object` type is `unknown`; callers that need static types should validate it.
const generateDynamicObject = LLM.generateObject({
model,
prompt: "Extract the city and forecast from: San Francisco is sunny.",
jsonSchema: {
type: "object",
properties: {
city: { type: "string" },
forecast: { type: "string" },
},
required: ["city", "forecast"],
},
})
// -----------------------------------------------------------------------------
// Part 2: provider composition with a fake provider
// -----------------------------------------------------------------------------
// A protocol is the provider-native API shape: common request -> body, response
// frames -> common events. This fake one turns text prompts into a JSON body
// and treats every SSE frame as output text.
const FakeBody = Schema.Struct({
model: Schema.String,
input: Schema.String,
})
type FakeBody = Schema.Schema.Type<typeof FakeBody>
const FakeProtocol = Protocol.make<FakeBody, string, string, void>({
// Protocol ids are open strings, so external packages can define their own
// protocols without changing this package.
id: "fake-echo",
body: {
schema: FakeBody,
from: (request) =>
Effect.succeed({
model: request.model.id,
input: request.messages
.flatMap((message) => message.content)
.filter((part) => part.type === "text")
.map((part) => part.text)
.join("\n"),
}),
},
stream: {
event: Schema.String,
initial: () => undefined,
step: (_, frame) => Effect.succeed([undefined, [{ type: "text-delta", id: "text-0", text: frame }]] as const),
onHalt: () => [{ type: "finish", reason: "stop" }],
},
})
// An route is the runnable binding for that protocol. It adds the deployment
// axes that the protocol deliberately does not know: URL, auth, and framing.
const FakeAdapter = Route.make({
id: "fake-echo",
provider: "fake-echo",
protocol: FakeProtocol,
endpoint: Endpoint.path("/v1/echo", { baseURL: "https://fake.local" }),
auth: Auth.passthrough,
framing: Framing.sse,
})
// A provider module exports a configured facade. Configuration happens before
// model selection; model selectors accept ids only.
const FakeEcho = {
id: ProviderID.make("fake-echo"),
configure: () => ({
id: ProviderID.make("fake-echo"),
model: (id: string) => FakeAdapter.model({ id }),
}),
}
// `LLMClient.prepare` is the lower-level inspection hook: it compiles through
// body conversion, validation, endpoint, auth, and HTTP construction without
// sending anything over the network.
const inspectFakeProvider = Effect.gen(function* () {
const prepared = yield* LLMClient.prepare(
LLM.request({
model: FakeEcho.configure().model("tiny-echo"),
prompt: "Show me the provider pipeline.",
}),
)
console.log("\n== fake provider prepare ==")
console.log("route:", prepared.route)
console.log("body:", Formatter.formatJson(prepared.body, { space: 2 }))
})
// Provide the LLM runtime and the HTTP request executor once. Keep one path
// enabled at a time so the tutorial can demonstrate generate, prepare, stream,
// or tool-loop behavior without spending tokens on every example.
const requestExecutorLayer = RequestExecutor.defaultLayer
const llmDeps = Layer.mergeAll(requestExecutorLayer, WebSocketExecutor.layer)
const llmClientLayer = LLMClient.layer.pipe(Layer.provide(llmDeps))
const program = Effect.gen(function* () {
// yield* generateOnce
// yield* inspectFakeProvider
// yield* LLMClient.prepare(rawOverlayExample).pipe(Effect.andThen((prepared) => Effect.sync(() => console.log(prepared.body))))
// yield* streamText
// yield* generateStructuredObject
// yield* generateDynamicObject.pipe(Effect.andThen((response) => Effect.sync(() => console.log(response.object))))
yield* streamWithTools
}).pipe(Effect.provide(Layer.mergeAll(llmDeps, llmClientLayer)))
Effect.runPromise(program)

51
packages/llm/package.json Normal file
View File

@@ -0,0 +1,51 @@
{
"$schema": "https://json.schemastore.org/package.json",
"version": "1.17.4",
"name": "@opencode-ai/llm",
"type": "module",
"license": "MIT",
"private": true,
"scripts": {
"setup:recording-env": "bun run script/setup-recording-env.ts",
"test": "bun test --timeout 30000 --only-failures",
"typecheck": "tsgo --noEmit"
},
"exports": {
".": "./src/index.ts",
"./route": "./src/route/index.ts",
"./provider": "./src/provider.ts",
"./providers": "./src/providers/index.ts",
"./providers/amazon-bedrock": "./src/providers/amazon-bedrock.ts",
"./providers/anthropic": "./src/providers/anthropic.ts",
"./providers/azure": "./src/providers/azure.ts",
"./providers/cloudflare": "./src/providers/cloudflare.ts",
"./providers/github-copilot": "./src/providers/github-copilot.ts",
"./providers/google": "./src/providers/google.ts",
"./providers/openai": "./src/providers/openai.ts",
"./providers/openai-compatible": "./src/providers/openai-compatible.ts",
"./providers/openai-compatible-profile": "./src/providers/openai-compatible-profile.ts",
"./providers/openrouter": "./src/providers/openrouter.ts",
"./providers/xai": "./src/providers/xai.ts",
"./protocols": "./src/protocols/index.ts",
"./protocols/anthropic-messages": "./src/protocols/anthropic-messages.ts",
"./protocols/bedrock-converse": "./src/protocols/bedrock-converse.ts",
"./protocols/gemini": "./src/protocols/gemini.ts",
"./protocols/openai-chat": "./src/protocols/openai-chat.ts",
"./protocols/openai-compatible-chat": "./src/protocols/openai-compatible-chat.ts",
"./protocols/openai-responses": "./src/protocols/openai-responses.ts"
},
"devDependencies": {
"@clack/prompts": "1.0.0-alpha.1",
"@effect/platform-node": "catalog:",
"@opencode-ai/http-recorder": "workspace:*",
"@tsconfig/bun": "catalog:",
"@types/bun": "catalog:",
"@typescript/native-preview": "catalog:"
},
"dependencies": {
"@smithy/eventstream-codec": "4.2.14",
"@smithy/util-utf8": "4.2.2",
"aws4fetch": "1.0.20",
"effect": "catalog:"
}
}

250
packages/llm/script/recording-cost-report.ts Normal file
View File

@@ -0,0 +1,250 @@
import * as fs from "node:fs/promises"
import * as path from "node:path"
const RECORDINGS_DIR = path.resolve(import.meta.dir, "..", "test", "fixtures", "recordings")
const MODELS_DEV_URL = "https://models.dev/api.json"
type JsonRecord = Record<string, unknown>
type Pricing = {
readonly input?: number
readonly output?: number
readonly cache_read?: number
readonly cache_write?: number
readonly reasoning?: number
}
type Usage = {
readonly inputTokens: number
readonly outputTokens: number
readonly cacheReadTokens: number
readonly cacheWriteTokens: number
readonly reasoningTokens: number
readonly reportedCost: number
}
type Row = Usage & {
readonly cassette: string
readonly provider: string
readonly model: string
readonly estimatedCost: number
readonly pricingSource: string
}
const isRecord = (value: unknown): value is JsonRecord =>
value !== null && typeof value === "object" && !Array.isArray(value)
const asNumber = (value: unknown) => (typeof value === "number" && Number.isFinite(value) ? value : 0)
const asString = (value: unknown) => (typeof value === "string" ? value : undefined)
const readJson = async (file: string) => JSON.parse(await Bun.file(file).text()) as unknown
const walk = async (dir: string): Promise<ReadonlyArray<string>> =>
(await fs.readdir(dir, { withFileTypes: true }))
.flatMap((entry) => {
const file = path.join(dir, entry.name)
return entry.isDirectory() ? [] : [file]
})
.concat(
...(await Promise.all(
(await fs.readdir(dir, { withFileTypes: true }))
.filter((entry) => entry.isDirectory())
.map((entry) => walk(path.join(dir, entry.name))),
)),
)
const providerFromUrl = (url: string) => {
if (url.includes("api.openai.com")) return "openai"
if (url.includes("api.anthropic.com")) return "anthropic"
if (url.includes("generativelanguage.googleapis.com")) return "google"
if (url.includes("bedrock")) return "amazon-bedrock"
if (url.includes("openrouter.ai")) return "openrouter"
if (url.includes("api.x.ai")) return "xai"
if (url.includes("api.groq.com")) return "groq"
if (url.includes("api.deepseek.com")) return "deepseek"
if (url.includes("api.together.xyz")) return "togetherai"
return "unknown"
}
const providerAliases: Record<string, ReadonlyArray<string>> = {
openai: ["openai"],
anthropic: ["anthropic"],
google: ["google"],
"amazon-bedrock": ["amazon-bedrock"],
openrouter: ["openrouter", "openai", "anthropic", "google"],
xai: ["xai"],
groq: ["groq"],
deepseek: ["deepseek"],
togetherai: ["togetherai"],
}
const modelAliases = (model: string) => [
model,
model.replace(/^models\//, ""),
model.replace(/-\d{8}$/, ""),
model.replace(/-\d{4}-\d{2}-\d{2}$/, ""),
model.replace(/-\d{4}-\d{2}-\d{2}$/, "").replace(/-\d{8}$/, ""),
model.replace(/^openai\//, ""),
model.replace(/^anthropic\//, ""),
model.replace(/^google\//, ""),
]
const pricingFor = (models: JsonRecord, provider: string, model: string) => {
for (const providerID of providerAliases[provider] ?? [provider]) {
const providerEntry = models[providerID]
if (!isRecord(providerEntry) || !isRecord(providerEntry.models)) continue
for (const modelID of modelAliases(model)) {
const modelEntry = providerEntry.models[modelID]
if (isRecord(modelEntry) && isRecord(modelEntry.cost))
return { pricing: modelEntry.cost as Pricing, source: `${providerID}/${modelID}` }
}
}
return { pricing: undefined, source: "missing" }
}
const estimateCost = (usage: Usage, pricing: Pricing | undefined) => {
if (!pricing) return 0
return (
(usage.inputTokens * (pricing.input ?? 0) +
usage.outputTokens * (pricing.output ?? 0) +
usage.cacheReadTokens * (pricing.cache_read ?? 0) +
usage.cacheWriteTokens * (pricing.cache_write ?? 0) +
usage.reasoningTokens * (pricing.reasoning ?? 0)) /
1_000_000
)
}
const emptyUsage = (): Usage => ({
inputTokens: 0,
outputTokens: 0,
cacheReadTokens: 0,
cacheWriteTokens: 0,
reasoningTokens: 0,
reportedCost: 0,
})
const addUsage = (a: Usage, b: Usage): Usage => ({
inputTokens: a.inputTokens + b.inputTokens,
outputTokens: a.outputTokens + b.outputTokens,
cacheReadTokens: a.cacheReadTokens + b.cacheReadTokens,
cacheWriteTokens: a.cacheWriteTokens + b.cacheWriteTokens,
reasoningTokens: a.reasoningTokens + b.reasoningTokens,
reportedCost: a.reportedCost + b.reportedCost,
})
const usageFromObject = (usage: unknown): Usage => {
if (!isRecord(usage)) return emptyUsage()
const promptDetails = isRecord(usage.prompt_tokens_details) ? usage.prompt_tokens_details : {}
const completionDetails = isRecord(usage.completion_tokens_details) ? usage.completion_tokens_details : {}
const inputDetails = isRecord(usage.input_tokens_details) ? usage.input_tokens_details : {}
const outputDetails = isRecord(usage.output_tokens_details) ? usage.output_tokens_details : {}
const cacheWriteTokens = asNumber(promptDetails.cache_write_tokens) + asNumber(inputDetails.cache_write_tokens)
return {
inputTokens: asNumber(usage.prompt_tokens) + asNumber(usage.input_tokens),
outputTokens: asNumber(usage.completion_tokens) + asNumber(usage.output_tokens),
cacheReadTokens: asNumber(promptDetails.cached_tokens) + asNumber(inputDetails.cached_tokens),
cacheWriteTokens,
reasoningTokens: asNumber(completionDetails.reasoning_tokens) + asNumber(outputDetails.reasoning_tokens),
reportedCost: asNumber(usage.cost),
}
}
const jsonPayloads = (body: string) =>
body
.split("\n")
.map((line) => line.trim())
.filter((line) => line.startsWith("data:"))
.map((line) => line.slice("data:".length).trim())
.filter((line) => line !== "" && line !== "[DONE]")
.flatMap((line) => {
try {
return [JSON.parse(line) as unknown]
} catch {
return []
}
})
const usageFromResponseBody = (body: string) =>
jsonPayloads(body).reduce<Usage>((usage, payload) => {
if (!isRecord(payload)) return usage
return addUsage(
usage,
addUsage(
usageFromObject(payload.usage),
usageFromObject(isRecord(payload.response) ? payload.response.usage : undefined),
),
)
}, emptyUsage())
const modelFromRequest = (request: unknown) => {
if (!isRecord(request)) return "unknown"
const requestBody = asString(request.body)
if (!requestBody) return "unknown"
try {
const body = JSON.parse(requestBody) as unknown
if (!isRecord(body)) return "unknown"
return asString(body.model) ?? "unknown"
} catch {
return "unknown"
}
}
const rowFor = (models: JsonRecord, file: string, cassette: unknown): Row | undefined => {
if (!isRecord(cassette) || !Array.isArray(cassette.interactions)) return undefined
const first = cassette.interactions.find(isRecord)
if (!first || !isRecord(first.request)) return undefined
const provider = providerFromUrl(asString(first.request.url) ?? "")
const model = modelFromRequest(first.request)
const usage = cassette.interactions.filter(isRecord).reduce<Usage>((total, interaction) => {
if (!isRecord(interaction.response)) return total
const responseBody = asString(interaction.response.body)
if (!responseBody) return total
return addUsage(total, usageFromResponseBody(responseBody))
}, emptyUsage())
const priced = pricingFor(models, provider, model)
return {
cassette: path.relative(RECORDINGS_DIR, file),
provider,
model,
...usage,
estimatedCost: estimateCost(usage, priced.pricing),
pricingSource: priced.source,
}
}
const money = (value: number) => (value === 0 ? "$0.000000" : `$${value.toFixed(6)}`)
const tokens = (value: number) => value.toLocaleString("en-US")
const models = (await (await fetch(MODELS_DEV_URL)).json()) as JsonRecord
const rows = (
await Promise.all(
(await walk(RECORDINGS_DIR))
.filter((file) => file.endsWith(".json"))
.map(async (file) => rowFor(models, file, await readJson(file))),
)
).filter((row): row is Row => row !== undefined)
const totals = rows.reduce(
(total, row) => ({
...addUsage(total, row),
estimatedCost: total.estimatedCost + row.estimatedCost,
}),
{ ...emptyUsage(), estimatedCost: 0 },
)
console.log("# Recording Cost Report")
console.log("")
console.log(`Pricing: ${MODELS_DEV_URL}`)
console.log(`Cassettes: ${rows.length}`)
console.log(`Reported cost: ${money(totals.reportedCost)}`)
console.log(`Estimated cost: ${money(totals.estimatedCost)}`)
console.log("")
console.log("| Provider | Model | Input | Output | Reasoning | Reported | Estimated | Pricing | Cassette |")
console.log("|---|---:|---:|---:|---:|---:|---:|---|---|")
for (const row of rows.toSorted((a, b) => b.reportedCost + b.estimatedCost - (a.reportedCost + a.estimatedCost))) {
if (row.inputTokens + row.outputTokens + row.reasoningTokens + row.reportedCost + row.estimatedCost === 0) continue
console.log(
`| ${row.provider} | ${row.model} | ${tokens(row.inputTokens)} | ${tokens(row.outputTokens)} | ${tokens(row.reasoningTokens)} | ${money(row.reportedCost)} | ${money(row.estimatedCost)} | ${row.pricingSource} | ${row.cassette} |`,
)
}

542
packages/llm/script/setup-recording-env.ts Normal file
View File

@@ -0,0 +1,542 @@
#!/usr/bin/env bun
import { NodeFileSystem } from "@effect/platform-node"
import * as path from "node:path"
import * as prompts from "@clack/prompts"
import { AwsV4Signer } from "aws4fetch"
import { Config, ConfigProvider, Effect, FileSystem, PlatformError, Redacted } from "effect"
import { FetchHttpClient, HttpClient, HttpClientRequest, type HttpClientResponse } from "effect/unstable/http"
import * as ProviderShared from "../src/protocols/shared"
import * as Cloudflare from "../src/providers/cloudflare"
type Provider = {
readonly id: string
readonly label: string
readonly tier: "core" | "canary" | "compatible" | "optional"
readonly note: string
readonly vars: ReadonlyArray<{
readonly name: string
readonly label?: string
readonly optional?: boolean
readonly secret?: boolean
}>
readonly validate?: (env: Env) => Effect.Effect<string | undefined, unknown, HttpClient.HttpClient>
}
type Env = Record<string, string>
const PROVIDERS: ReadonlyArray<Provider> = [
{
id: "openai",
label: "OpenAI",
tier: "core",
note: "Native OpenAI Chat / Responses recorded tests",
vars: [{ name: "OPENAI_API_KEY" }],
validate: (env) => validateBearer("https://api.openai.com/v1/models", Redacted.make(env.OPENAI_API_KEY)),
},
{
id: "anthropic",
label: "Anthropic",
tier: "core",
note: "Native Anthropic Messages recorded tests",
vars: [{ name: "ANTHROPIC_API_KEY" }],
validate: (env) =>
HttpClientRequest.get("https://api.anthropic.com/v1/models").pipe(
HttpClientRequest.setHeaders({
"anthropic-version": "2023-06-01",
"x-api-key": Redacted.value(Redacted.make(env.ANTHROPIC_API_KEY)),
}),
executeRequest,
),
},
{
id: "google",
label: "Google Gemini",
tier: "core",
note: "Native Gemini recorded tests",
vars: [{ name: "GOOGLE_GENERATIVE_AI_API_KEY" }],
validate: (env) =>
HttpClientRequest.get(
`https://generativelanguage.googleapis.com/v1beta/models?key=${encodeURIComponent(env.GOOGLE_GENERATIVE_AI_API_KEY)}`,
).pipe(executeRequest),
},
{
id: "bedrock",
label: "Amazon Bedrock",
tier: "core",
note: "Native Bedrock Converse recorded tests",
vars: [
{ name: "AWS_ACCESS_KEY_ID" },
{ name: "AWS_SECRET_ACCESS_KEY" },
{ name: "AWS_SESSION_TOKEN", optional: true },
{ name: "BEDROCK_RECORDING_REGION", optional: true },
{ name: "BEDROCK_MODEL_ID", optional: true },
],
validate: (env) => validateBedrock(env),
},
{
id: "groq",
label: "Groq",
tier: "canary",
note: "Fast OpenAI-compatible canary for text/tool streaming",
vars: [{ name: "GROQ_API_KEY" }],
validate: (env) => validateBearer("https://api.groq.com/openai/v1/models", Redacted.make(env.GROQ_API_KEY)),
},
{
id: "openrouter",
label: "OpenRouter",
tier: "canary",
note: "Router canary for OpenAI-compatible text/tool streaming",
vars: [{ name: "OPENROUTER_API_KEY" }],
validate: (env) =>
validateChat({
url: "https://openrouter.ai/api/v1/chat/completions",
token: Redacted.make(env.OPENROUTER_API_KEY),
model: "openai/gpt-4o-mini",
}),
},
{
id: "xai",
label: "xAI",
tier: "canary",
note: "OpenAI-compatible xAI chat endpoint",
vars: [{ name: "XAI_API_KEY" }],
validate: (env) => validateBearer("https://api.x.ai/v1/models", Redacted.make(env.XAI_API_KEY)),
},
{
id: "cloudflare-ai-gateway",
label: "Cloudflare AI Gateway",
tier: "canary",
note: "Cloudflare Unified/OpenAI-compatible gateway; supports provider/model ids like workers-ai/@cf/...",
vars: [
{ name: "CLOUDFLARE_ACCOUNT_ID", label: "Cloudflare account ID", secret: false },
{
name: "CLOUDFLARE_GATEWAY_ID",
label: "Cloudflare AI Gateway ID (defaults to default)",
optional: true,
secret: false,
},
{ name: "CLOUDFLARE_API_TOKEN", label: "Cloudflare AI Gateway token" },
],
validate: (env) =>
validateChat({
url: `${Cloudflare.aiGatewayBaseURL({
accountId: env.CLOUDFLARE_ACCOUNT_ID,
gatewayId: env.CLOUDFLARE_GATEWAY_ID || undefined,
})}/chat/completions`,
token: Redacted.make(envValue(env, Cloudflare.aiGatewayAuthEnvVars)),
tokenHeader: "cf-aig-authorization",
model: "workers-ai/@cf/meta/llama-3.1-8b-instruct",
}),
},
{
id: "cloudflare-workers-ai",
label: "Cloudflare Workers AI",
tier: "canary",
note: "Direct Workers AI OpenAI-compatible endpoint; supports model ids like @cf/meta/...",
vars: [
{ name: "CLOUDFLARE_ACCOUNT_ID", label: "Cloudflare account ID", secret: false },
{ name: "CLOUDFLARE_API_KEY", label: "Cloudflare Workers AI API token" },
],
validate: (env) =>
validateChat({
url: `${Cloudflare.workersAIBaseURL({ accountId: env.CLOUDFLARE_ACCOUNT_ID })}/chat/completions`,
token: Redacted.make(envValue(env, Cloudflare.workersAIAuthEnvVars)),
model: "@cf/meta/llama-3.1-8b-instruct",
}),
},
{
id: "deepseek",
label: "DeepSeek",
tier: "compatible",
note: "Existing OpenAI-compatible recorded tests",
vars: [{ name: "DEEPSEEK_API_KEY" }],
validate: (env) => validateBearer("https://api.deepseek.com/models", Redacted.make(env.DEEPSEEK_API_KEY)),
},
{
id: "togetherai",
label: "TogetherAI",
tier: "compatible",
note: "Existing OpenAI-compatible text/tool recorded tests",
vars: [{ name: "TOGETHER_AI_API_KEY" }],
validate: (env) => validateBearer("https://api.together.xyz/v1/models", Redacted.make(env.TOGETHER_AI_API_KEY)),
},
{
id: "mistral",
label: "Mistral",
tier: "optional",
note: "OpenAI-compatible bridge; native reasoning parity is follow-up work",
vars: [{ name: "MISTRAL_API_KEY" }],
validate: (env) => validateBearer("https://api.mistral.ai/v1/models", Redacted.make(env.MISTRAL_API_KEY)),
},
{
id: "perplexity",
label: "Perplexity",
tier: "optional",
note: "OpenAI-compatible bridge; citations/search metadata are follow-up work",
vars: [{ name: "PERPLEXITY_API_KEY" }],
validate: (env) => validateBearer("https://api.perplexity.ai/models", Redacted.make(env.PERPLEXITY_API_KEY)),
},
{
id: "venice",
label: "Venice",
tier: "optional",
note: "OpenAI-compatible bridge",
vars: [{ name: "VENICE_API_KEY" }],
validate: (env) => validateBearer("https://api.venice.ai/api/v1/models", Redacted.make(env.VENICE_API_KEY)),
},
{
id: "cerebras",
label: "Cerebras",
tier: "optional",
note: "OpenAI-compatible bridge",
vars: [{ name: "CEREBRAS_API_KEY" }],
validate: (env) => validateBearer("https://api.cerebras.ai/v1/models", Redacted.make(env.CEREBRAS_API_KEY)),
},
{
id: "deepinfra",
label: "DeepInfra",
tier: "optional",
note: "OpenAI-compatible bridge",
vars: [{ name: "DEEPINFRA_API_KEY" }],
validate: (env) =>
validateBearer("https://api.deepinfra.com/v1/openai/models", Redacted.make(env.DEEPINFRA_API_KEY)),
},
{
id: "fireworks",
label: "Fireworks",
tier: "optional",
note: "OpenAI-compatible bridge",
vars: [{ name: "FIREWORKS_API_KEY" }],
validate: (env) =>
validateBearer("https://api.fireworks.ai/inference/v1/models", Redacted.make(env.FIREWORKS_API_KEY)),
},
{
id: "baseten",
label: "Baseten",
tier: "optional",
note: "OpenAI-compatible bridge",
vars: [{ name: "BASETEN_API_KEY" }],
},
]
const args = process.argv.slice(2)
const hasFlag = (name: string) => args.includes(name)
const option = (name: string) => {
const index = args.indexOf(name)
if (index === -1) return undefined
return args[index + 1]
}
const envPath = path.resolve(process.cwd(), option("--env") ?? ".env.local")
const checkOnly = hasFlag("--check")
const providerOption = option("--providers")
const interactive = Boolean(process.stdin.isTTY && process.stdout.isTTY)
const envNames = Array.from(new Set(PROVIDERS.flatMap((provider) => provider.vars.map((item) => item.name))))
const providersForOption = (value: string | undefined) => {
if (!value || value === "recommended")
return PROVIDERS.filter((provider) => provider.tier === "core" || provider.tier === "canary")
if (value === "recorded") return PROVIDERS.filter((provider) => provider.tier !== "optional")
if (value === "all") return PROVIDERS
const ids = new Set(
value
.split(",")
.map((item) => item.trim())
.filter(Boolean),
)
return PROVIDERS.filter((provider) => ids.has(provider.id))
}
const chooseProviders = async () => {
if (providerOption) return providersForOption(providerOption)
return providersForOption("recommended")
}
const catchMissingFile = (error: PlatformError.PlatformError) => {
if (error.reason._tag === "NotFound") return Effect.succeed("")
return Effect.fail(error)
}
const readEnvFile = Effect.fn("RecordingEnv.readFile")(function* () {
const fileSystem = yield* FileSystem.FileSystem
return yield* fileSystem.readFileString(envPath).pipe(Effect.catch(catchMissingFile))
})
const readConfigString = (provider: ConfigProvider.ConfigProvider, name: string) =>
Config.string(name)
.parse(provider)
.pipe(
Effect.match({
onFailure: () => undefined,
onSuccess: (value) => value,
}),
)
const parseEnv = Effect.fn("RecordingEnv.parseEnv")(function* (contents: string) {
const provider = ConfigProvider.fromDotEnvContents(contents)
return Object.fromEntries(
(yield* Effect.forEach(envNames, (name) =>
readConfigString(provider, name).pipe(Effect.map((value) => [name, value] as const)),
)).filter((entry): entry is readonly [string, string] => entry[1] !== undefined),
)
})
const quote = (value: string) => JSON.stringify(value)
const status = (name: string, fileEnv: Env) => {
if (fileEnv[name]) return "file"
if (process.env[name]) return "shell"
return "missing"
}
const statusLine = (provider: Provider, fileEnv: Env) =>
[
`${provider.label} (${provider.tier})`,
provider.note,
...provider.vars.map((item) => {
const value = status(item.name, fileEnv)
const suffix = item.optional ? " optional" : ""
return ` ${value === "missing" ? "missing" : "set"} ${item.name}${suffix}${value === "shell" ? " (shell only)" : ""}`
}),
].join("\n")
const printStatus = (providers: ReadonlyArray<Provider>, fileEnv: Env) => {
prompts.note(providers.map((provider) => statusLine(provider, fileEnv)).join("\n\n"), `Recording env: ${envPath}`)
}
const exitIfCancel = <A>(value: A | symbol): A => {
if (!prompts.isCancel(value)) return value as A
prompts.cancel("Cancelled")
process.exit(130)
}
const upsertEnv = (contents: string, values: Env) => {
const names = Object.keys(values)
const seen = new Set<string>()
const lines = contents.split(/\r?\n/).map((line) => {
const match = line.match(/^\s*(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=/)
if (!match || !names.includes(match[1])) return line
seen.add(match[1])
return `${match[1]}=${quote(values[match[1]])}`
})
const missing = names.filter((name) => !seen.has(name))
if (missing.length === 0) return lines.join("\n").replace(/\n*$/, "\n")
const prefix = lines.join("\n").trimEnd()
const block = [
"",
"# Added by bun run setup:recording-env",
...missing.map((name) => `${name}=${quote(values[name])}`),
].join("\n")
return `${prefix}${block}\n`
}
const providerRequiredStatus = (provider: Provider, fileEnv: Env) => {
const required = requiredVars(provider)
if (required.some((item) => status(item.name, fileEnv) === "missing")) return "missing"
if (required.some((item) => status(item.name, fileEnv) === "shell")) return "set in shell"
return "already added"
}
const requiredVars = (provider: Provider) => provider.vars.filter((item) => !item.optional)
const promptVars = (provider: Provider) => provider.vars.filter((item) => !item.optional || item.secret === false)
const processEnv = (): Env =>
Object.fromEntries(Object.entries(process.env).filter((entry): entry is [string, string] => entry[1] !== undefined))
const envValue = (env: Env, names: ReadonlyArray<string>) => names.map((name) => env[name]).find(Boolean) ?? ""
const envWithValues = (fileEnv: Env, values: Env): Env => ({
...processEnv(),
...fileEnv,
...values,
})
const responseError = Effect.fn("RecordingEnv.responseError")(function* (
response: HttpClientResponse.HttpClientResponse,
) {
if (response.status >= 200 && response.status < 300) return undefined
const body = yield* response.text.pipe(Effect.catch(() => Effect.succeed("")))
return `${response.status}${body ? `: ${body.slice(0, 180)}` : ""}`
})
const executeRequest = Effect.fn("RecordingEnv.executeRequest")(function* (
request: HttpClientRequest.HttpClientRequest,
) {
const http = yield* HttpClient.HttpClient
return yield* http.execute(request).pipe(Effect.flatMap(responseError))
})
const validateBearer = (url: string, token: Redacted.Redacted<string>, headers: Record<string, string> = {}) =>
HttpClientRequest.get(url).pipe(
HttpClientRequest.setHeaders({ ...headers, authorization: `Bearer ${Redacted.value(token)}` }),
executeRequest,
)
const validateChat = (input: {
readonly url: string
readonly token: Redacted.Redacted<string>
readonly tokenHeader?: string
readonly model: string
readonly headers?: Record<string, string>
}) =>
ProviderShared.jsonPost({
url: input.url,
headers: { ...input.headers, [input.tokenHeader ?? "authorization"]: `Bearer ${Redacted.value(input.token)}` },
body: ProviderShared.encodeJson({
model: input.model,
messages: [{ role: "user", content: "Reply with exactly: ok" }],
max_tokens: 3,
temperature: 0,
}),
}).pipe(executeRequest)
const validateBedrock = (env: Env) =>
Effect.gen(function* () {
const request = yield* Effect.promise(() =>
new AwsV4Signer({
url: `https://bedrock.${env.BEDROCK_RECORDING_REGION || "us-east-1"}.amazonaws.com/foundation-models`,
method: "GET",
service: "bedrock",
region: env.BEDROCK_RECORDING_REGION || "us-east-1",
accessKeyId: env.AWS_ACCESS_KEY_ID,
secretAccessKey: env.AWS_SECRET_ACCESS_KEY,
sessionToken: env.AWS_SESSION_TOKEN || undefined,
}).sign(),
)
return yield* HttpClientRequest.get(request.url.toString()).pipe(
HttpClientRequest.setHeaders(Object.fromEntries(request.headers.entries())),
executeRequest,
)
})
const validateProvider = Effect.fn("RecordingEnv.validateProvider")(function* (provider: Provider, env: Env) {
return yield* (provider.validate?.(env) ?? Effect.succeed("no lightweight validator")).pipe(
Effect.catch((error) => {
if (error instanceof Error) return Effect.succeed(error.message)
return Effect.succeed(String(error))
}),
)
})
const validateProviders = Effect.fn("RecordingEnv.validateProviders")(function* (
providers: ReadonlyArray<Provider>,
env: Env,
) {
const spinner = prompts.spinner()
spinner.start("Validating credentials")
const results = yield* Effect.forEach(
providers,
(provider) => validateProvider(provider, env).pipe(Effect.map((error) => ({ provider, error }))),
{ concurrency: 4 },
)
spinner.stop("Validation complete")
prompts.note(
results
.map(
(result) =>
`${result.error ? "failed" : "ok"} ${result.provider.label}${result.error ? ` - ${result.error}` : ""}`,
)
.join("\n"),
"Credential validation",
)
})
const writeEnvFile = Effect.fn("RecordingEnv.writeFile")(function* (contents: string) {
const fileSystem = yield* FileSystem.FileSystem
yield* fileSystem.makeDirectory(path.dirname(envPath), { recursive: true })
yield* fileSystem.writeFileString(envPath, contents, { mode: 0o600 })
})
const prompt = <A>(run: () => Promise<A | symbol>) => Effect.promise(run).pipe(Effect.map(exitIfCancel))
const chooseConfigurableProviders = Effect.fn("RecordingEnv.chooseConfigurableProviders")(function* (
providers: ReadonlyArray<Provider>,
fileEnv: Env,
) {
const configurable = providers.filter((provider) => requiredVars(provider).length > 0)
const selected = yield* prompt<ReadonlyArray<string>>(() =>
prompts.multiselect({
message: "Select provider credentials to add or override",
options: configurable.map((provider) => ({
value: provider.id,
label: provider.label,
hint: `${providerRequiredStatus(provider, fileEnv)} - ${requiredVars(provider)
.map((item) => item.name)
.join(", ")}`,
})),
initialValues: configurable
.filter((provider) => providerRequiredStatus(provider, fileEnv) === "missing")
.map((provider) => provider.id),
}),
)
return configurable.filter((provider) => selected.includes(provider.id))
})
const promptEnvVar = (item: Provider["vars"][number]) =>
prompt<string>(() => {
const input = {
message: item.label ?? item.name,
validate: (input: string | undefined) => {
if (item.optional) return undefined
return !input || input.length === 0 ? "Leave blank by pressing Esc/cancel, or paste a value" : undefined
},
}
return item.secret === false ? prompts.text(input) : prompts.password(input)
})
const promptProviderValues = Effect.fn("RecordingEnv.promptProviderValues")(function* (
providers: ReadonlyArray<Provider>,
) {
const values: Env = {}
for (const provider of providers) {
prompts.log.info(`${provider.label}: ${provider.note}`)
for (const item of promptVars(provider)) {
if (values[item.name]) continue
const value = yield* promptEnvVar(item)
if (value !== "") values[item.name] = value
}
}
return values
})
const main = Effect.fn("RecordingEnv.main")(function* () {
prompts.intro("LLM recording credentials")
const contents = yield* readEnvFile()
const fileEnv = yield* parseEnv(contents)
const providers = yield* Effect.promise(() => chooseProviders())
printStatus(providers, fileEnv)
if (checkOnly) {
prompts.outro("Check complete")
return
}
if (!interactive) {
prompts.outro("Run this command in a terminal to enter credentials")
return
}
const selectedProviders = yield* chooseConfigurableProviders(providers, fileEnv)
const values = yield* promptProviderValues(selectedProviders)
if (Object.keys(values).length === 0) {
prompts.outro("No changes")
return
}
if (
interactive &&
(yield* prompt(() => prompts.confirm({ message: "Validate credentials before saving?", initialValue: true })))
) {
yield* validateProviders(selectedProviders, envWithValues(fileEnv, values))
}
yield* writeEnvFile(upsertEnv(contents, values))
prompts.log.success(
`Saved ${Object.keys(values).length} value${Object.keys(values).length === 1 ? "" : "s"} to ${envPath}`,
)
prompts.outro("Keep .env.local local. Store shared team credentials in a password manager or vault.")
})
await Effect.runPromise(main().pipe(Effect.provide(NodeFileSystem.layer), Effect.provide(FetchHttpClient.layer)))

111
packages/llm/src/cache-policy.ts Normal file
View File

@@ -0,0 +1,111 @@
// Apply an `LLMRequest.cache` policy by injecting `CacheHint`s onto the parts
// the policy designates. Runs once at compile time, before the per-protocol
// body builder, so the existing inline-hint lowering path handles the rest.
//
// The default `"auto"` shape places one breakpoint at the last tool definition,
// one at the last system part, and one at the latest user message. This
// matches what production agent harnesses (LangChain's caching middleware,
// kern-ai's 10x cost-reduction playbook) converge on for tool-use loops: the
// latest user message stays put while a single turn explodes into many
// assistant/tool round-trips, so caching at that boundary lets every
// intra-turn API call hit the prefix.
//
// Manual `cache: CacheHint` placements on individual parts are preserved —
// this function only fills gaps the caller left empty.
import { CacheHint, type CachePolicy, type CachePolicyObject } from "./schema/options"
import { LLMRequest, Message, ToolDefinition, type ContentPart } from "./schema/messages"
const AUTO: CachePolicyObject = {
tools: true,
system: true,
messages: "latest-user-message",
}
const NONE: CachePolicyObject = {}
// Resolution rules:
// - undefined → "auto" — caching is on by default. The math favors it:
// Anthropic 5m-cache write is 1.25x base, read is 0.1x,
// so a single reuse within 5 minutes already wins.
// - "auto" → tools + system + latest user msg.
// - "none" → no auto placement; manual `CacheHint`s still flow.
// - object form → exactly what the caller asked for.
const resolve = (policy: CachePolicy | undefined): CachePolicyObject => {
if (policy === undefined || policy === "auto") return AUTO
if (policy === "none") return NONE
return policy
}
// Protocols whose wire format ignores inline cache markers (OpenAI's implicit
// prefix caching, Gemini's implicit + out-of-band CachedContent). Skip the
// whole policy pass for these — emitting hints would be harmless but pointless.
const RESPECTS_INLINE_HINTS = new Set(["anthropic-messages", "bedrock-converse"])
const makeHint = (ttlSeconds: number | undefined): CacheHint =>
ttlSeconds !== undefined ? new CacheHint({ type: "ephemeral", ttlSeconds }) : new CacheHint({ type: "ephemeral" })
const markLastTool = (tools: ReadonlyArray<ToolDefinition>, hint: CacheHint): ReadonlyArray<ToolDefinition> => {
if (tools.length === 0) return tools
const last = tools.length - 1
if (tools[last]!.cache) return tools
return tools.map((tool, i) => (i === last ? new ToolDefinition({ ...tool, cache: hint }) : tool))
}
const markLastSystem = (system: LLMRequest["system"], hint: CacheHint): LLMRequest["system"] => {
if (system.length === 0) return system
const last = system.length - 1
if (system[last]!.cache) return system
return system.map((part, i) => (i === last ? { ...part, cache: hint } : part))
}
const lastIndexOfRole = (messages: ReadonlyArray<Message>, role: Message["role"]): number =>
messages.findLastIndex((m) => m.role === role)
// Mark the last text part of `messages[index]`. If no text part exists, mark
// the last content part regardless of type — that's the breakpoint position
// in tool-result-only messages too.
const markMessageAt = (messages: ReadonlyArray<Message>, index: number, hint: CacheHint): ReadonlyArray<Message> => {
if (index < 0 || index >= messages.length) return messages
const target = messages[index]!
if (target.content.length === 0) return messages
const lastTextIndex = target.content.findLastIndex((part) => part.type === "text")
const markAt = lastTextIndex >= 0 ? lastTextIndex : target.content.length - 1
const existing = target.content[markAt]!
if ("cache" in existing && existing.cache) return messages
const nextContent = target.content.map((part, i) => (i === markAt ? ({ ...part, cache: hint } as ContentPart) : part))
const next = new Message({ ...target, content: nextContent })
// Single pass over `messages`, substituting the one updated entry. Long
// conversations call this on every request, so avoid `.map()` here — its
// closure dispatch and identity copies show up in profiling.
const result = messages.slice()
result[index] = next
return result
}
const markMessages = (
messages: ReadonlyArray<Message>,
strategy: NonNullable<CachePolicyObject["messages"]>,
hint: CacheHint,
): ReadonlyArray<Message> => {
if (messages.length === 0) return messages
if (strategy === "latest-user-message") return markMessageAt(messages, lastIndexOfRole(messages, "user"), hint)
if (strategy === "latest-assistant") return markMessageAt(messages, lastIndexOfRole(messages, "assistant"), hint)
const start = Math.max(0, messages.length - strategy.tail)
let next = messages
for (let i = start; i < messages.length; i++) next = markMessageAt(next, i, hint)
return next
}
export const applyCachePolicy = (request: LLMRequest): LLMRequest => {
if (!RESPECTS_INLINE_HINTS.has(request.model.route.id)) return request
const policy = resolve(request.cache)
if (!policy.tools && !policy.system && !policy.messages) return request
const hint = makeHint(policy.ttlSeconds)
const tools = policy.tools ? markLastTool(request.tools, hint) : request.tools
const system = policy.system ? markLastSystem(request.system, hint) : request.system
const messages = policy.messages ? markMessages(request.messages, policy.messages, hint) : request.messages
if (tools === request.tools && system === request.system && messages === request.messages) return request
return LLMRequest.update(request, { tools, system, messages })
}

33
packages/llm/src/index.ts Normal file
View File

@@ -0,0 +1,33 @@
export { LLMClient } from "./route/client"
export { Auth } from "./route/auth"
export { Provider } from "./provider"
export { isContextOverflow, isContextOverflowFailure } from "./provider-error"
export type {
RouteModelInput,
RouteRoutedModelInput,
Interface as LLMClientShape,
Service as LLMClientService,
} from "./route/client"
export * from "./schema"
export { Tool, ToolFailure, toDefinitions } from "./tool"
export { ToolRuntime } from "./tool-runtime"
export type { DispatchResult as ToolDispatchResult, ToolSettlement } from "./tool-runtime"
export type {
AnyExecutableTool,
AnyTool,
ExecutableTool,
ExecutableTools,
Tool as ToolShape,
ToolExecute,
ToolExecuteContext,
ToolModelOutputInput,
Tools,
ToolSchema,
ToolToModelOutput,
} from "./tool"
export * as LLM from "./llm"
export type {
Definition as ProviderDefinition,
ModelFactory as ProviderModelFactory,
ModelOptions as ProviderModelOptions,
} from "./provider"

186
packages/llm/src/llm.ts Normal file
View File

@@ -0,0 +1,186 @@
import { Effect, JsonSchema, Schema } from "effect"
import { LLMClient } from "./route/client"
import {
GenerationOptions,
HttpOptions,
InvalidProviderOutputReason,
LLMError,
LLMEvent,
LLMRequest,
LLMResponse,
Message,
type ModelInput as SchemaModelInput,
SystemPart,
ToolChoice,
ToolDefinition,
type ContentPart,
ToolResultPart,
} from "./schema"
import { make as makeTool, toDefinitions, type ToolSchema } from "./tool"
export type ModelInput = SchemaModelInput
export type MessageInput = Message.Input
export type ToolChoiceInput = ToolChoice.Input
export type ToolChoiceMode = ToolChoice.Mode
export type ToolResultInput = Parameters<typeof ToolResultPart.make>[0]
/** Input accepted by `LLM.request`, normalized into the canonical `LLMRequest` class. */
export type RequestInput = Omit<
ConstructorParameters<typeof LLMRequest>[0],
"system" | "messages" | "tools" | "toolChoice" | "generation" | "http" | "providerOptions"
> & {
readonly system?: string | SystemPart | ReadonlyArray<SystemPart>
readonly prompt?: string | ContentPart | ReadonlyArray<ContentPart>
readonly messages?: ReadonlyArray<Message | MessageInput>
readonly tools?: ReadonlyArray<ToolDefinition.Input>
readonly toolChoice?: ToolChoiceInput
readonly generation?: GenerationOptions.Input
readonly providerOptions?: ConstructorParameters<typeof LLMRequest>[0]["providerOptions"]
readonly http?: HttpOptions.Input
}
export const generate = LLMClient.generate
export const stream = LLMClient.stream
export const requestInput = (input: LLMRequest): RequestInput => ({
...LLMRequest.input(input),
})
export const request = (input: RequestInput) => {
const {
system: requestSystem,
prompt,
messages,
tools,
toolChoice: requestToolChoice,
generation: requestGeneration,
providerOptions: requestProviderOptions,
http: requestHttp,
...rest
} = input
return new LLMRequest({
...rest,
system: SystemPart.content(requestSystem),
messages: [...(messages?.map(Message.make) ?? []), ...(prompt === undefined ? [] : [Message.user(prompt)])],
tools: tools?.map(ToolDefinition.make) ?? [],
toolChoice: requestToolChoice ? ToolChoice.make(requestToolChoice) : undefined,
generation: requestGeneration === undefined ? undefined : GenerationOptions.make(requestGeneration),
providerOptions: requestProviderOptions,
http: requestHttp === undefined ? undefined : HttpOptions.make(requestHttp),
})
}
export const updateRequest = (input: LLMRequest, patch: Partial<RequestInput>) =>
request({ ...requestInput(input), ...patch })
const GENERATE_OBJECT_TOOL_NAME = "generate_object"
const GENERATE_OBJECT_TOOL_DESCRIPTION = "Return the structured result by calling this tool."
type GenerateObjectBase = Omit<RequestInput, "tools" | "toolChoice" | "responseFormat">
export class GenerateObjectResponse<T> {
constructor(
readonly object: T,
readonly response: LLMResponse,
) {}
get events() {
return this.response.events
}
get usage() {
return this.response.usage
}
}
export interface GenerateObjectOptions<S extends ToolSchema<any>> extends GenerateObjectBase {
readonly schema: S
}
export interface GenerateObjectDynamicOptions extends GenerateObjectBase {
/** Raw JSON Schema object describing the expected output shape. */
readonly jsonSchema: JsonSchema.JsonSchema
}
const runGenerateObject = Effect.fn("LLM.generateObject")(function* (
options: GenerateObjectBase,
tool: ReturnType<typeof makeTool>,
) {
const baseRequest = request(options)
const generateRequest = LLMRequest.update(baseRequest, {
tools: toDefinitions({ [GENERATE_OBJECT_TOOL_NAME]: tool }),
toolChoice: ToolChoice.named(GENERATE_OBJECT_TOOL_NAME),
})
const response = yield* LLMClient.generate(generateRequest)
const call = response.toolCalls.find(
(event) => LLMEvent.is.toolCall(event) && event.name === GENERATE_OBJECT_TOOL_NAME,
)
if (!call || !LLMEvent.is.toolCall(call))
return yield* new LLMError({
module: "LLM",
method: "generateObject",
reason: new InvalidProviderOutputReason({
message: `generateObject: model did not call the forced \`${GENERATE_OBJECT_TOOL_NAME}\` tool`,
}),
})
const object = yield* tool._decode(call.input).pipe(
Effect.mapError(
(error) =>
new LLMError({
module: "LLM",
method: "generateObject",
reason: new InvalidProviderOutputReason({
message: `generateObject: tool input failed schema decode: ${error.message}`,
}),
}),
),
)
return new GenerateObjectResponse(object, response)
})
/**
* Run a model and decode its output against `schema`. Works on every protocol
* because it forces a synthetic tool call internally — provider-native JSON
* modes are intentionally avoided so behaviour is uniform.
*
* Two input modes:
*
* 1. `schema: EffectSchema<T>` — `.object` is decoded and typed as `T`.
* Decode failures surface as `LLMError`.
* 2. `jsonSchema: JsonSchema.JsonSchema` — `.object` is `unknown`. Use when
* the schema is only available at runtime (MCP, plugin manifests). Caller validates.
*/
export function generateObject<S extends ToolSchema<any>>(
options: GenerateObjectOptions<S>,
): Effect.Effect<GenerateObjectResponse<Schema.Schema.Type<S>>, LLMError>
export function generateObject(
options: GenerateObjectDynamicOptions,
): Effect.Effect<GenerateObjectResponse<unknown>, LLMError>
export function generateObject(options: GenerateObjectOptions<ToolSchema<any>> | GenerateObjectDynamicOptions) {
if ("schema" in options) {
const { schema, ...rest } = options
return runGenerateObject(
rest,
makeTool({
description: GENERATE_OBJECT_TOOL_DESCRIPTION,
parameters: schema,
success: Schema.Unknown as ToolSchema<unknown>,
execute: () => Effect.void,
}),
)
}
const { jsonSchema, ...rest } = options
return runGenerateObject(
rest,
makeTool({
description: GENERATE_OBJECT_TOOL_DESCRIPTION,
jsonSchema,
execute: () => Effect.void,
}),
)
}

845
packages/llm/src/protocols/anthropic-messages.ts Normal file
View File

@@ -0,0 +1,845 @@
import { Effect, Schema } from "effect"
import { Route } from "../route/client"
import { Auth } from "../route/auth"
import { Endpoint } from "../route/endpoint"
import { Framing } from "../route/framing"
import { Protocol } from "../route/protocol"
import {
LLMEvent,
Usage,
type CacheHint,
type FinishReason,
type LLMRequest,
type MediaPart,
type ProviderMetadata,
type ToolCallPart,
type ToolDefinition,
type ToolContent,
type ToolResultPart,
} from "../schema"
import { JsonObject, optionalArray, optionalNull, ProviderShared } from "./shared"
import { isContextOverflow } from "../provider-error"
import * as Cache from "./utils/cache"
import { Lifecycle } from "./utils/lifecycle"
import { ToolStream } from "./utils/tool-stream"
const ADAPTER = "anthropic-messages"
export const DEFAULT_BASE_URL = "https://api.anthropic.com/v1"
export const PATH = "/messages"
// =============================================================================
// Request Body Schema
// =============================================================================
const AnthropicCacheControl = Schema.Struct({
type: Schema.tag("ephemeral"),
ttl: Schema.optional(Schema.Literals(["5m", "1h"])),
})
const AnthropicTextBlock = Schema.Struct({
type: Schema.tag("text"),
text: Schema.String,
cache_control: Schema.optional(AnthropicCacheControl),
})
type AnthropicTextBlock = Schema.Schema.Type<typeof AnthropicTextBlock>
const AnthropicImageBlock = Schema.Struct({
type: Schema.tag("image"),
source: Schema.Struct({
type: Schema.tag("base64"),
media_type: Schema.String,
data: Schema.String,
}),
cache_control: Schema.optional(AnthropicCacheControl),
})
type AnthropicImageBlock = Schema.Schema.Type<typeof AnthropicImageBlock>
const AnthropicThinkingBlock = Schema.Struct({
type: Schema.tag("thinking"),
thinking: Schema.String,
signature: Schema.optional(Schema.String),
cache_control: Schema.optional(AnthropicCacheControl),
})
const AnthropicToolUseBlock = Schema.Struct({
type: Schema.tag("tool_use"),
id: Schema.String,
name: Schema.String,
input: Schema.Unknown,
cache_control: Schema.optional(AnthropicCacheControl),
})
type AnthropicToolUseBlock = Schema.Schema.Type<typeof AnthropicToolUseBlock>
const AnthropicServerToolUseBlock = Schema.Struct({
type: Schema.tag("server_tool_use"),
id: Schema.String,
name: Schema.String,
input: Schema.Unknown,
cache_control: Schema.optional(AnthropicCacheControl),
})
type AnthropicServerToolUseBlock = Schema.Schema.Type<typeof AnthropicServerToolUseBlock>
// Server tool result blocks: web_search_tool_result, code_execution_tool_result,
// and web_fetch_tool_result. The provider executes the tool and inlines the
// structured result into the assistant turn — there is no client tool_result
// round-trip. We round-trip the structured `content` payload as opaque JSON so
// the next request can echo it back when continuing the conversation.
const AnthropicServerToolResultType = Schema.Literals([
"web_search_tool_result",
"code_execution_tool_result",
"web_fetch_tool_result",
])
type AnthropicServerToolResultType = Schema.Schema.Type<typeof AnthropicServerToolResultType>
const AnthropicServerToolResultBlock = Schema.Struct({
type: AnthropicServerToolResultType,
tool_use_id: Schema.String,
content: Schema.Unknown,
cache_control: Schema.optional(AnthropicCacheControl),
})
type AnthropicServerToolResultBlock = Schema.Schema.Type<typeof AnthropicServerToolResultBlock>
// Anthropic accepts either a plain string or an ordered array of text/image
// blocks inside `tool_result.content`. The array form is required when a tool
// returns image bytes (screenshot, image search, etc.) so they can be passed
// to the model as proper image inputs instead of being JSON-stringified into
// the prompt — which silently inflates context by megabytes and can push the
// conversation over the model's token limit.
const AnthropicToolResultContent = Schema.Union([AnthropicTextBlock, AnthropicImageBlock])
const AnthropicToolResultBlock = Schema.Struct({
type: Schema.tag("tool_result"),
tool_use_id: Schema.String,
content: Schema.Union([Schema.String, Schema.Array(AnthropicToolResultContent)]),
is_error: Schema.optional(Schema.Boolean),
cache_control: Schema.optional(AnthropicCacheControl),
})
const AnthropicUserBlock = Schema.Union([AnthropicTextBlock, AnthropicImageBlock, AnthropicToolResultBlock])
type AnthropicUserBlock = Schema.Schema.Type<typeof AnthropicUserBlock>
const AnthropicAssistantBlock = Schema.Union([
AnthropicTextBlock,
AnthropicThinkingBlock,
AnthropicToolUseBlock,
AnthropicServerToolUseBlock,
AnthropicServerToolResultBlock,
])
type AnthropicAssistantBlock = Schema.Schema.Type<typeof AnthropicAssistantBlock>
type AnthropicToolResultBlock = Schema.Schema.Type<typeof AnthropicToolResultBlock>
const AnthropicMessage = Schema.Union([
Schema.Struct({ role: Schema.Literal("user"), content: Schema.Array(AnthropicUserBlock) }),
Schema.Struct({ role: Schema.Literal("assistant"), content: Schema.Array(AnthropicAssistantBlock) }),
Schema.Struct({ role: Schema.Literal("system"), content: Schema.Array(AnthropicTextBlock) }),
]).pipe(Schema.toTaggedUnion("role"))
type AnthropicMessage = Schema.Schema.Type<typeof AnthropicMessage>
const AnthropicTool = Schema.Struct({
name: Schema.String,
description: Schema.String,
input_schema: JsonObject,
cache_control: Schema.optional(AnthropicCacheControl),
})
type AnthropicTool = Schema.Schema.Type<typeof AnthropicTool>
const AnthropicToolChoice = Schema.Union([
Schema.Struct({ type: Schema.Literals(["auto", "any"]) }),
Schema.Struct({ type: Schema.tag("tool"), name: Schema.String }),
])
const AnthropicThinking = Schema.Struct({
type: Schema.tag("enabled"),
budget_tokens: Schema.Number,
})
const AnthropicBodyFields = {
model: Schema.String,
system: optionalArray(AnthropicTextBlock),
messages: Schema.Array(AnthropicMessage),
tools: optionalArray(AnthropicTool),
tool_choice: Schema.optional(AnthropicToolChoice),
stream: Schema.Literal(true),
max_tokens: Schema.Number,
temperature: Schema.optional(Schema.Number),
top_p: Schema.optional(Schema.Number),
top_k: Schema.optional(Schema.Number),
stop_sequences: optionalArray(Schema.String),
thinking: Schema.optional(AnthropicThinking),
}
const AnthropicMessagesBody = Schema.Struct(AnthropicBodyFields)
export type AnthropicMessagesBody = Schema.Schema.Type<typeof AnthropicMessagesBody>
const AnthropicUsage = Schema.Struct({
input_tokens: Schema.optional(Schema.Number),
output_tokens: Schema.optional(Schema.Number),
cache_creation_input_tokens: optionalNull(Schema.Number),
cache_read_input_tokens: optionalNull(Schema.Number),
})
type AnthropicUsage = Schema.Schema.Type<typeof AnthropicUsage>
const AnthropicStreamBlock = Schema.Struct({
type: Schema.String,
id: Schema.optional(Schema.String),
name: Schema.optional(Schema.String),
text: Schema.optional(Schema.String),
thinking: Schema.optional(Schema.String),
signature: Schema.optional(Schema.String),
input: Schema.optional(Schema.Unknown),
// *_tool_result blocks arrive whole as content_block_start (no streaming
// delta) with the structured payload in `content` and the originating
// server_tool_use id in `tool_use_id`.
tool_use_id: Schema.optional(Schema.String),
content: Schema.optional(Schema.Unknown),
})
const AnthropicStreamDelta = Schema.Struct({
type: Schema.optional(Schema.String),
text: Schema.optional(Schema.String),
thinking: Schema.optional(Schema.String),
partial_json: Schema.optional(Schema.String),
signature: Schema.optional(Schema.String),
stop_reason: optionalNull(Schema.String),
stop_sequence: optionalNull(Schema.String),
})
const AnthropicEvent = Schema.Struct({
type: Schema.String,
index: Schema.optional(Schema.Number),
message: Schema.optional(Schema.Struct({ usage: Schema.optional(AnthropicUsage) })),
content_block: Schema.optional(AnthropicStreamBlock),
delta: Schema.optional(AnthropicStreamDelta),
usage: Schema.optional(AnthropicUsage),
// `type` and `message` are both required per Anthropic's spec, but
// OpenAI-compatible proxies and gateway translations occasionally drop one
// or the other; mark them optional so a partial payload still parses and
// the parser can fall back to whichever field is populated.
error: Schema.optional(
Schema.Struct({ type: Schema.optional(Schema.String), message: Schema.optional(Schema.String) }),
),
})
type AnthropicEvent = Schema.Schema.Type<typeof AnthropicEvent>
interface ParserState {
readonly tools: ToolStream.State<number>
readonly usage?: Usage
readonly lifecycle: Lifecycle.State
}
const invalid = ProviderShared.invalidRequest
// =============================================================================
// Request Lowering
// =============================================================================
// Anthropic accepts at most 4 explicit cache_control breakpoints per request,
// across `tools`, `system`, and `messages`. Beyond the cap the API returns a
// 400 — so the lowering layer counts emitted markers and silently drops any
// that exceed it.
const ANTHROPIC_BREAKPOINT_CAP = 4
const EPHEMERAL_5M = { type: "ephemeral" as const }
const EPHEMERAL_1H = { type: "ephemeral" as const, ttl: "1h" as const }
const cacheControl = (breakpoints: Cache.Breakpoints, cache: CacheHint | undefined) => {
if (cache?.type !== "ephemeral" && cache?.type !== "persistent") return undefined
if (breakpoints.remaining <= 0) {
breakpoints.dropped += 1
return undefined
}
breakpoints.remaining -= 1
return Cache.ttlBucket(cache.ttlSeconds) === "1h" ? EPHEMERAL_1H : EPHEMERAL_5M
}
const anthropicMetadata = (metadata: Record<string, unknown>): ProviderMetadata => ({ anthropic: metadata })
const signatureFromMetadata = (metadata: ProviderMetadata | undefined): string | undefined => {
const anthropic = metadata?.anthropic
if (!ProviderShared.isRecord(anthropic)) return undefined
return typeof anthropic.signature === "string" ? anthropic.signature : undefined
}
const lowerTool = (breakpoints: Cache.Breakpoints, tool: ToolDefinition): AnthropicTool => ({
name: tool.name,
description: tool.description,
input_schema: tool.inputSchema,
cache_control: cacheControl(breakpoints, tool.cache),
})
const lowerToolChoice = (toolChoice: NonNullable<LLMRequest["toolChoice"]>) =>
ProviderShared.matchToolChoice("Anthropic Messages", toolChoice, {
auto: () => ({ type: "auto" as const }),
none: () => undefined,
required: () => ({ type: "any" as const }),
tool: (name) => ({ type: "tool" as const, name }),
})
const lowerToolCall = (part: ToolCallPart): AnthropicToolUseBlock => ({
type: "tool_use",
id: part.id,
name: part.name,
input: part.input,
})
const lowerServerToolCall = (part: ToolCallPart): AnthropicServerToolUseBlock => ({
type: "server_tool_use",
id: part.id,
name: part.name,
input: part.input,
})
// Server tool result blocks are typed by name. Anthropic ships three today;
// extend this list when new server tools land. The block content is the
// structured payload returned by the provider, which we round-trip as-is.
const serverToolResultType = (name: string): AnthropicServerToolResultType | undefined => {
if (name === "web_search") return "web_search_tool_result"
if (name === "code_execution") return "code_execution_tool_result"
if (name === "web_fetch") return "web_fetch_tool_result"
return undefined
}
const lowerServerToolResult = Effect.fn("AnthropicMessages.lowerServerToolResult")(function* (part: ToolResultPart) {
const wireType = serverToolResultType(part.name)
if (!wireType)
return yield* invalid(`Anthropic Messages does not know how to round-trip server tool result for ${part.name}`)
return { type: wireType, tool_use_id: part.id, content: part.result.value } satisfies AnthropicServerToolResultBlock
})
const lowerImage = Effect.fn("AnthropicMessages.lowerImage")(function* (part: MediaPart) {
const media = yield* ProviderShared.validateMedia(
"Anthropic Messages",
part,
new Set<string>(ProviderShared.IMAGE_MIMES),
)
return {
type: "image" as const,
source: {
type: "base64" as const,
media_type: media.mime,
data: media.base64,
},
} satisfies AnthropicImageBlock
})
// Tool results may carry structured text/images. Keep media as provider-native
// content instead of JSON-stringifying base64 into a prompt string.
const lowerToolResultContentItem = Effect.fn("AnthropicMessages.lowerToolResultContentItem")(function* (
item: ToolContent,
) {
if (item.type === "text") return { type: "text" as const, text: item.text } satisfies AnthropicTextBlock
const media = yield* ProviderShared.validateToolFile(
"Anthropic Messages",
item,
new Set<string>(ProviderShared.IMAGE_MIMES),
)
return {
type: "image" as const,
source: {
type: "base64" as const,
media_type: media.mime,
data: media.base64,
},
} satisfies AnthropicImageBlock
})
const lowerToolResultContent = Effect.fn("AnthropicMessages.lowerToolResultContent")(function* (part: ToolResultPart) {
// Text / json / error results stay as a string for backward compatibility
// with existing cassettes and provider expectations.
if (part.result.type !== "content") return ProviderShared.toolResultText(part)
// Preserve the narrowed array element type when compiled through a consumer package.
const content: ReadonlyArray<ToolContent> = part.result.value
return yield* Effect.forEach(content, lowerToolResultContentItem)
})
// Mid-conversation system messages are a native Claude API feature only for
// Opus 4.8. Other Anthropic models intentionally use the same visible wrapped-
// user fallback as non-Anthropic routes rather than sending a role they reject.
const supportsNativeSystemUpdates = (request: LLMRequest) => String(request.model.id) === "claude-opus-4-8"
const endsInServerToolUse = (message: LLMRequest["messages"][number]) => {
const last = message.content.at(-1)
return message.role === "assistant" && last?.type === "tool-call" && last.providerExecuted === true
}
const canUseNativeSystemUpdate = (messages: LLMRequest["messages"], index: number) => {
const previous = messages[index - 1]
const next = messages[index + 1]
return (
previous !== undefined &&
previous.role !== "system" &&
(previous.role === "user" || previous.role === "tool" || endsInServerToolUse(previous)) &&
next?.role !== "system" &&
(next === undefined || next.role === "assistant")
)
}
const splitsLocalToolResults = (messages: LLMRequest["messages"], index: number) => {
const pending = new Set<string>()
for (const message of messages.slice(0, index)) {
for (const part of message.content) {
if (message.role === "assistant" && part.type === "tool-call" && part.providerExecuted !== true)
pending.add(part.id)
if (message.role === "tool" && part.type === "tool-result") pending.delete(part.id)
}
}
return pending.size > 0
}
const lowerNativeSystemUpdate = Effect.fn("AnthropicMessages.lowerNativeSystemUpdate")(function* (
message: LLMRequest["messages"][number],
breakpoints: Cache.Breakpoints,
) {
const content = yield* ProviderShared.systemUpdateText("Anthropic Messages", message)
return {
role: "system" as const,
content: content.map((part) => ({
type: "text" as const,
text: part.text,
cache_control: cacheControl(breakpoints, part.cache),
})),
}
})
const lowerMessages = Effect.fn("AnthropicMessages.lowerMessages")(function* (
request: LLMRequest,
breakpoints: Cache.Breakpoints,
) {
const messages: AnthropicMessage[] = []
for (const [index, message] of request.messages.entries()) {
if (message.role === "system") {
if (splitsLocalToolResults(request.messages, index))
return yield* invalid("Anthropic Messages system updates cannot split a local tool call from its tool result")
if (supportsNativeSystemUpdates(request) && canUseNativeSystemUpdate(request.messages, index)) {
messages.push(yield* lowerNativeSystemUpdate(message, breakpoints))
continue
}
const part = yield* ProviderShared.wrappedSystemUpdate("Anthropic Messages", message)
const block = { type: "text" as const, text: part.text, cache_control: cacheControl(breakpoints, part.cache) }
const previous = messages.at(-1)
if (previous?.role === "user")
messages[messages.length - 1] = { role: "user", content: [...previous.content, block] }
else messages.push({ role: "user", content: [block] })
continue
}
if (message.role === "user") {
const content: AnthropicUserBlock[] = []
for (const part of message.content) {
if (part.type === "text") {
content.push({ type: "text", text: part.text, cache_control: cacheControl(breakpoints, part.cache) })
continue
}
if (part.type === "media") {
content.push(yield* lowerImage(part))
continue
}
return yield* ProviderShared.unsupportedContent("Anthropic Messages", "user", ["text", "media"])
}
messages.push({ role: "user", content })
continue
}
if (message.role === "assistant") {
const content: AnthropicAssistantBlock[] = []
for (const part of message.content) {
if (part.type === "text") {
content.push({ type: "text", text: part.text, cache_control: cacheControl(breakpoints, part.cache) })
continue
}
if (part.type === "reasoning") {
content.push({
type: "thinking",
thinking: part.text,
signature: part.encrypted ?? signatureFromMetadata(part.providerMetadata),
})
continue
}
if (part.type === "tool-call") {
content.push(part.providerExecuted ? lowerServerToolCall(part) : lowerToolCall(part))
continue
}
if (part.type === "tool-result" && part.providerExecuted) {
content.push(yield* lowerServerToolResult(part))
continue
}
return yield* invalid(
`Anthropic Messages assistant messages only support text, reasoning, and tool-call content for now`,
)
}
messages.push({ role: "assistant", content })
continue
}
const content: AnthropicToolResultBlock[] = []
for (const part of message.content) {
if (!ProviderShared.supportsContent(part, ["tool-result"]))
return yield* ProviderShared.unsupportedContent("Anthropic Messages", "tool", ["tool-result"])
content.push({
type: "tool_result",
tool_use_id: part.id,
content: yield* lowerToolResultContent(part),
is_error: part.result.type === "error" ? true : undefined,
cache_control: cacheControl(breakpoints, part.cache),
})
}
messages.push({ role: "user", content })
}
return messages
})
const anthropicOptions = (request: LLMRequest) => request.providerOptions?.anthropic
const lowerThinking = Effect.fn("AnthropicMessages.lowerThinking")(function* (request: LLMRequest) {
const thinking = anthropicOptions(request)?.thinking
if (!ProviderShared.isRecord(thinking) || thinking.type !== "enabled") return undefined
const budget =
typeof thinking.budgetTokens === "number"
? thinking.budgetTokens
: typeof thinking.budget_tokens === "number"
? thinking.budget_tokens
: undefined
if (budget === undefined) return yield* invalid("Anthropic thinking provider option requires budgetTokens")
return { type: "enabled" as const, budget_tokens: budget }
})
const fromRequest = Effect.fn("AnthropicMessages.fromRequest")(function* (request: LLMRequest) {
const toolChoice = request.toolChoice ? yield* lowerToolChoice(request.toolChoice) : undefined
const generation = request.generation
// Allocate the 4-breakpoint budget in invalidation order: tools → system →
// messages. Tools live highest in the cache hierarchy, so when callers
// over-mark we keep their tool hints and shed the message-tail ones first.
const breakpoints = Cache.newBreakpoints(ANTHROPIC_BREAKPOINT_CAP)
const tools =
request.tools.length === 0 || request.toolChoice?.type === "none"
? undefined
: request.tools.map((tool) => lowerTool(breakpoints, tool))
const system =
request.system.length === 0
? undefined
: request.system.map((part) => ({
type: "text" as const,
text: part.text,
cache_control: cacheControl(breakpoints, part.cache),
}))
const messages = yield* lowerMessages(request, breakpoints)
if (breakpoints.dropped > 0) {
yield* Effect.logWarning(
`Anthropic Messages: dropped ${breakpoints.dropped} cache breakpoint(s); the API allows at most ${ANTHROPIC_BREAKPOINT_CAP} per request.`,
)
}
return {
model: request.model.id,
system,
messages,
tools,
tool_choice: toolChoice,
stream: true as const,
max_tokens: generation?.maxTokens ?? request.model.route.defaults.limits?.output ?? 4096,
temperature: generation?.temperature,
top_p: generation?.topP,
top_k: generation?.topK,
stop_sequences: generation?.stop,
thinking: yield* lowerThinking(request),
}
})
// =============================================================================
// Stream Parsing
// =============================================================================
const mapFinishReason = (reason: string | null | undefined): FinishReason => {
if (reason === "end_turn" || reason === "stop_sequence" || reason === "pause_turn") return "stop"
if (reason === "max_tokens") return "length"
if (reason === "tool_use") return "tool-calls"
if (reason === "refusal") return "content-filter"
return "unknown"
}
// Anthropic reports the non-overlapping breakdown natively — its
// `input_tokens` is the *non-cached* count per the Messages API docs, with
// cache reads and writes as separate fields. We sum them to derive the
// inclusive `inputTokens` the rest of the contract expects. Extended
// thinking tokens are *not* broken out by Anthropic — they're billed as
// part of `output_tokens`, so `reasoningTokens` stays `undefined` and
// `outputTokens` carries the combined total.
const mapUsage = (usage: AnthropicUsage | undefined): Usage | undefined => {
if (!usage) return undefined
const nonCached = usage.input_tokens
const cacheRead = usage.cache_read_input_tokens ?? undefined
const cacheWrite = usage.cache_creation_input_tokens ?? undefined
const inputTokens = ProviderShared.sumTokens(nonCached, cacheRead, cacheWrite)
return new Usage({
inputTokens,
outputTokens: usage.output_tokens,
nonCachedInputTokens: nonCached,
cacheReadInputTokens: cacheRead,
cacheWriteInputTokens: cacheWrite,
totalTokens: ProviderShared.totalTokens(inputTokens, usage.output_tokens, undefined),
providerMetadata: { anthropic: usage },
})
}
// Anthropic emits usage on `message_start` and again on `message_delta` — the
// final delta carries the authoritative totals. Right-biased merge: each
// field prefers `right` when defined, falls back to `left`. `inputTokens` is
// recomputed from the merged breakdown so the inclusive total stays
// consistent with `nonCached + cacheRead + cacheWrite`.
const mergeUsage = (left: Usage | undefined, right: Usage | undefined) => {
if (!left) return right
if (!right) return left
const nonCachedInputTokens = right.nonCachedInputTokens ?? left.nonCachedInputTokens
const cacheReadInputTokens = right.cacheReadInputTokens ?? left.cacheReadInputTokens
const cacheWriteInputTokens = right.cacheWriteInputTokens ?? left.cacheWriteInputTokens
const inputTokens = ProviderShared.sumTokens(nonCachedInputTokens, cacheReadInputTokens, cacheWriteInputTokens)
const outputTokens = right.outputTokens ?? left.outputTokens
return new Usage({
inputTokens,
outputTokens,
nonCachedInputTokens,
cacheReadInputTokens,
cacheWriteInputTokens,
totalTokens: ProviderShared.totalTokens(inputTokens, outputTokens, undefined),
providerMetadata: {
anthropic: {
...left.providerMetadata?.["anthropic"],
...right.providerMetadata?.["anthropic"],
},
},
})
}
// Server tool result blocks come whole in `content_block_start` (no streaming
// delta sequence). We convert the payload to a `tool-result` event with
// `providerExecuted: true`. The runtime appends it to the assistant message
// for round-trip; downstream consumers can inspect `result.value` for the
// structured payload.
const SERVER_TOOL_RESULT_NAMES: Record<AnthropicServerToolResultType, string> = {
web_search_tool_result: "web_search",
code_execution_tool_result: "code_execution",
web_fetch_tool_result: "web_fetch",
}
const isServerToolResultType = (type: string): type is AnthropicServerToolResultType => type in SERVER_TOOL_RESULT_NAMES
const serverToolResultEvent = (block: NonNullable<AnthropicEvent["content_block"]>): LLMEvent | undefined => {
if (!block.type || !isServerToolResultType(block.type)) return undefined
const errorPayload =
typeof block.content === "object" && block.content !== null && "type" in block.content
? String((block.content as Record<string, unknown>).type)
: ""
const isError = errorPayload.endsWith("_tool_result_error")
return LLMEvent.toolResult({
id: block.tool_use_id ?? "",
name: SERVER_TOOL_RESULT_NAMES[block.type],
result: isError ? { type: "error", value: block.content } : { type: "json", value: block.content },
providerExecuted: true,
providerMetadata: anthropicMetadata({ blockType: block.type }),
})
}
type StepResult = readonly [ParserState, ReadonlyArray<LLMEvent>]
const NO_EVENTS: StepResult["1"] = []
const onMessageStart = (state: ParserState, event: AnthropicEvent): StepResult => {
const usage = mapUsage(event.message?.usage)
return [usage ? { ...state, usage: mergeUsage(state.usage, usage) } : state, NO_EVENTS]
}
const onContentBlockStart = (state: ParserState, event: AnthropicEvent): StepResult => {
const block = event.content_block
if (!block) return [state, NO_EVENTS]
if ((block.type === "tool_use" || block.type === "server_tool_use") && event.index !== undefined) {
const events: LLMEvent[] = []
const lifecycle = Lifecycle.stepStart(state.lifecycle, events)
return [
{
...state,
lifecycle,
tools: ToolStream.start(state.tools, event.index, {
id: block.id ?? String(event.index),
name: block.name ?? "",
providerExecuted: block.type === "server_tool_use",
}),
},
[...events, LLMEvent.toolInputStart({ id: block.id ?? String(event.index), name: block.name ?? "" })],
]
}
if (block.type === "text" && block.text) {
const events: LLMEvent[] = []
return [
{ ...state, lifecycle: Lifecycle.textDelta(state.lifecycle, events, `text-${event.index ?? 0}`, block.text) },
events,
]
}
if (block.type === "thinking" && block.thinking) {
const events: LLMEvent[] = []
return [
{
...state,
lifecycle: Lifecycle.reasoningDelta(state.lifecycle, events, `reasoning-${event.index ?? 0}`, block.thinking),
},
events,
]
}
const result = serverToolResultEvent(block)
if (!result) return [state, NO_EVENTS]
const events: LLMEvent[] = []
return [{ ...state, lifecycle: Lifecycle.stepStart(state.lifecycle, events) }, [...events, result]]
}
const onContentBlockDelta = Effect.fn("AnthropicMessages.onContentBlockDelta")(function* (
state: ParserState,
event: AnthropicEvent,
) {
const delta = event.delta
if (delta?.type === "text_delta" && delta.text) {
const events: LLMEvent[] = []
return [
{ ...state, lifecycle: Lifecycle.textDelta(state.lifecycle, events, `text-${event.index ?? 0}`, delta.text) },
events,
] satisfies StepResult
}
if (delta?.type === "thinking_delta" && delta.thinking) {
const events: LLMEvent[] = []
return [
{
...state,
lifecycle: Lifecycle.reasoningDelta(state.lifecycle, events, `reasoning-${event.index ?? 0}`, delta.thinking),
},
events,
] satisfies StepResult
}
if (delta?.type === "signature_delta" && delta.signature) {
const events: LLMEvent[] = []
return [
{
...state,
lifecycle: Lifecycle.reasoningEnd(
state.lifecycle,
events,
`reasoning-${event.index ?? 0}`,
anthropicMetadata({ signature: delta.signature }),
),
},
events,
] satisfies StepResult
}
if (delta?.type === "input_json_delta" && event.index !== undefined) {
if (!delta.partial_json) return [state, NO_EVENTS] satisfies StepResult
const result = ToolStream.appendExisting(
ADAPTER,
state.tools,
event.index,
delta.partial_json,
"Anthropic Messages tool argument delta is missing its tool call",
)
if (ToolStream.isError(result)) return yield* result
const events: LLMEvent[] = []
const lifecycle = result.events.length ? Lifecycle.stepStart(state.lifecycle, events) : state.lifecycle
events.push(...result.events)
return [{ ...state, lifecycle, tools: result.tools }, events] satisfies StepResult
}
return [state, NO_EVENTS] satisfies StepResult
})
const onContentBlockStop = Effect.fn("AnthropicMessages.onContentBlockStop")(function* (
state: ParserState,
event: AnthropicEvent,
) {
if (event.index === undefined) return [state, NO_EVENTS] satisfies StepResult
const result = yield* ToolStream.finish(ADAPTER, state.tools, event.index)
const events: LLMEvent[] = []
const resultEvents = result.events ?? []
const lifecycle = resultEvents.length
? Lifecycle.stepStart(state.lifecycle, events)
: Lifecycle.reasoningEnd(
Lifecycle.textEnd(state.lifecycle, events, `text-${event.index}`),
events,
`reasoning-${event.index}`,
)
events.push(...resultEvents)
return [{ ...state, lifecycle, tools: result.tools }, events] satisfies StepResult
})
const onMessageDelta = (state: ParserState, event: AnthropicEvent): StepResult => {
const usage = mergeUsage(state.usage, mapUsage(event.usage))
const events: LLMEvent[] = []
const lifecycle = Lifecycle.finish(state.lifecycle, events, {
reason: mapFinishReason(event.delta?.stop_reason),
usage,
providerMetadata: event.delta?.stop_sequence
? anthropicMetadata({ stopSequence: event.delta.stop_sequence })
: undefined,
})
return [{ ...state, lifecycle, usage }, events]
}
// Prefix `error.type` so overloads, rate limits, and quota errors are visible
// even when the provider message is generic or empty.
const providerErrorMessage = (event: AnthropicEvent): string => {
const type = event.error?.type
const message = event.error?.message
if (type && message) return `${type}: ${message}`
return message || type || "Anthropic Messages stream error"
}
const onError = (state: ParserState, event: AnthropicEvent): StepResult => [
state,
[
LLMEvent.providerError({
message: providerErrorMessage(event),
classification: isContextOverflow(event.error?.message ?? "") ? "context-overflow" : undefined,
}),
],
]
const step = (state: ParserState, event: AnthropicEvent) => {
if (event.type === "message_start") return Effect.succeed(onMessageStart(state, event))
if (event.type === "content_block_start") return Effect.succeed(onContentBlockStart(state, event))
if (event.type === "content_block_delta") return onContentBlockDelta(state, event)
if (event.type === "content_block_stop") return onContentBlockStop(state, event)
if (event.type === "message_delta") return Effect.succeed(onMessageDelta(state, event))
if (event.type === "error") return Effect.succeed(onError(state, event))
return Effect.succeed<StepResult>([state, NO_EVENTS])
}
// =============================================================================
// Protocol And Anthropic Route
// =============================================================================
/**
* The Anthropic Messages protocol — request body construction, body schema,
* and the streaming-event state machine. Used by native Anthropic Cloud and
* (once registered) Vertex Anthropic / Bedrock-hosted Anthropic passthrough.
*/
export const protocol = Protocol.make({
id: ADAPTER,
body: {
schema: AnthropicMessagesBody,
from: fromRequest,
},
stream: {
event: Protocol.jsonEvent(AnthropicEvent),
initial: () => ({ tools: ToolStream.empty<number>(), lifecycle: Lifecycle.initial() }),
step,
},
})
export const route = Route.make({
id: ADAPTER,
provider: "anthropic",
protocol,
endpoint: Endpoint.path(PATH, { baseURL: DEFAULT_BASE_URL }),
auth: Auth.none,
framing: Framing.sse,
headers: () => ({ "anthropic-version": "2023-06-01" }),
})
export * as AnthropicMessages from "./anthropic-messages"

664
packages/llm/src/protocols/bedrock-converse.ts Normal file
View File

@@ -0,0 +1,664 @@
import { Effect, Schema } from "effect"
import { Route } from "../route/client"
import { Endpoint } from "../route/endpoint"
import { Protocol } from "../route/protocol"
import {
LLMEvent,
Usage,
type CacheHint,
type FinishReason,
type LLMRequest,
type ProviderMetadata,
type ReasoningPart,
type ToolCallPart,
type ToolDefinition,
type ToolResultPart,
} from "../schema"
import { BedrockEventStream } from "./bedrock-event-stream"
import { isContextOverflow } from "../provider-error"
import { JsonObject, optionalArray, ProviderShared } from "./shared"
import { BedrockAuth } from "./utils/bedrock-auth"
import { BedrockCache } from "./utils/bedrock-cache"
import { BedrockMedia } from "./utils/bedrock-media"
import { Lifecycle } from "./utils/lifecycle"
import { ToolStream } from "./utils/tool-stream"
const ADAPTER = "bedrock-converse"
export type { Credentials as BedrockCredentials } from "./utils/bedrock-auth"
// =============================================================================
// Request Body Schema
// =============================================================================
const BedrockTextBlock = Schema.Struct({
text: Schema.String,
})
type BedrockTextBlock = Schema.Schema.Type<typeof BedrockTextBlock>
const BedrockToolUseBlock = Schema.Struct({
toolUse: Schema.Struct({
toolUseId: Schema.String,
name: Schema.String,
input: Schema.Unknown,
}),
})
type BedrockToolUseBlock = Schema.Schema.Type<typeof BedrockToolUseBlock>
const BedrockToolResultContentItem = Schema.Union([
Schema.Struct({ text: Schema.String }),
Schema.Struct({ json: Schema.Unknown }),
BedrockMedia.ImageBlock,
])
const BedrockToolResultBlock = Schema.Struct({
toolResult: Schema.Struct({
toolUseId: Schema.String,
content: Schema.Array(BedrockToolResultContentItem),
status: Schema.optional(Schema.Literals(["success", "error"])),
}),
})
type BedrockToolResultBlock = Schema.Schema.Type<typeof BedrockToolResultBlock>
const BedrockReasoningBlock = Schema.Struct({
reasoningContent: Schema.Struct({
reasoningText: Schema.optional(
Schema.Struct({
text: Schema.String,
signature: Schema.optional(Schema.String),
}),
),
}),
})
const BedrockUserBlock = Schema.Union([
BedrockTextBlock,
BedrockMedia.ImageBlock,
BedrockMedia.DocumentBlock,
BedrockToolResultBlock,
BedrockCache.CachePointBlock,
])
type BedrockUserBlock = Schema.Schema.Type<typeof BedrockUserBlock>
const BedrockAssistantBlock = Schema.Union([
BedrockTextBlock,
BedrockReasoningBlock,
BedrockToolUseBlock,
BedrockCache.CachePointBlock,
])
type BedrockAssistantBlock = Schema.Schema.Type<typeof BedrockAssistantBlock>
const BedrockMessage = Schema.Union([
Schema.Struct({ role: Schema.Literal("user"), content: Schema.Array(BedrockUserBlock) }),
Schema.Struct({ role: Schema.Literal("assistant"), content: Schema.Array(BedrockAssistantBlock) }),
]).pipe(Schema.toTaggedUnion("role"))
type BedrockMessage = Schema.Schema.Type<typeof BedrockMessage>
const BedrockSystemBlock = Schema.Union([BedrockTextBlock, BedrockCache.CachePointBlock])
type BedrockSystemBlock = Schema.Schema.Type<typeof BedrockSystemBlock>
const BedrockToolSpec = Schema.Struct({
toolSpec: Schema.Struct({
name: Schema.String,
description: Schema.String,
inputSchema: Schema.Struct({
json: JsonObject,
}),
}),
})
type BedrockToolSpec = Schema.Schema.Type<typeof BedrockToolSpec>
const BedrockTool = Schema.Union([BedrockToolSpec, BedrockCache.CachePointBlock])
type BedrockTool = Schema.Schema.Type<typeof BedrockTool>
const BedrockToolChoice = Schema.Union([
Schema.Struct({ auto: Schema.Struct({}) }),
Schema.Struct({ any: Schema.Struct({}) }),
Schema.Struct({ tool: Schema.Struct({ name: Schema.String }) }),
])
const BedrockBodyFields = {
modelId: Schema.String,
messages: Schema.Array(BedrockMessage),
system: optionalArray(BedrockSystemBlock),
inferenceConfig: Schema.optional(
Schema.Struct({
maxTokens: Schema.optional(Schema.Number),
temperature: Schema.optional(Schema.Number),
topP: Schema.optional(Schema.Number),
stopSequences: optionalArray(Schema.String),
}),
),
toolConfig: Schema.optional(
Schema.Struct({
tools: Schema.Array(BedrockTool),
toolChoice: Schema.optional(BedrockToolChoice),
}),
),
additionalModelRequestFields: Schema.optional(JsonObject),
}
const BedrockConverseBody = Schema.Struct(BedrockBodyFields)
export type BedrockConverseBody = Schema.Schema.Type<typeof BedrockConverseBody>
const BedrockUsageSchema = Schema.Struct({
inputTokens: Schema.optional(Schema.Number),
outputTokens: Schema.optional(Schema.Number),
totalTokens: Schema.optional(Schema.Number),
cacheReadInputTokens: Schema.optional(Schema.Number),
cacheWriteInputTokens: Schema.optional(Schema.Number),
})
type BedrockUsageSchema = Schema.Schema.Type<typeof BedrockUsageSchema>
// Streaming event shape — the AWS event stream wraps each JSON payload by its
// `:event-type` header (e.g. `messageStart`, `contentBlockDelta`). We
// reconstruct that wrapping in `decodeFrames` below so the event schema can
// stay a plain discriminated record.
const BedrockEvent = Schema.Struct({
messageStart: Schema.optional(Schema.Struct({ role: Schema.String })),
contentBlockStart: Schema.optional(
Schema.Struct({
contentBlockIndex: Schema.Number,
start: Schema.optional(
Schema.Struct({
toolUse: Schema.optional(Schema.Struct({ toolUseId: Schema.String, name: Schema.String })),
}),
),
}),
),
contentBlockDelta: Schema.optional(
Schema.Struct({
contentBlockIndex: Schema.Number,
delta: Schema.optional(
Schema.Struct({
text: Schema.optional(Schema.String),
toolUse: Schema.optional(Schema.Struct({ input: Schema.String })),
reasoningContent: Schema.optional(
Schema.Struct({
text: Schema.optional(Schema.String),
signature: Schema.optional(Schema.String),
}),
),
}),
),
}),
),
contentBlockStop: Schema.optional(Schema.Struct({ contentBlockIndex: Schema.Number })),
messageStop: Schema.optional(
Schema.Struct({
stopReason: Schema.String,
additionalModelResponseFields: Schema.optional(Schema.Unknown),
}),
),
metadata: Schema.optional(
Schema.Struct({
usage: Schema.optional(BedrockUsageSchema),
metrics: Schema.optional(Schema.Unknown),
}),
),
internalServerException: Schema.optional(Schema.Struct({ message: Schema.String })),
modelStreamErrorException: Schema.optional(Schema.Struct({ message: Schema.String })),
validationException: Schema.optional(Schema.Struct({ message: Schema.String })),
throttlingException: Schema.optional(Schema.Struct({ message: Schema.String })),
serviceUnavailableException: Schema.optional(Schema.Struct({ message: Schema.String })),
})
type BedrockEvent = Schema.Schema.Type<typeof BedrockEvent>
// =============================================================================
// Request Lowering
// =============================================================================
const lowerToolSpec = (tool: ToolDefinition): BedrockToolSpec => ({
toolSpec: {
name: tool.name,
description: tool.description,
inputSchema: { json: tool.inputSchema },
},
})
const lowerTools = (breakpoints: BedrockCache.Breakpoints, tools: ReadonlyArray<ToolDefinition>): BedrockTool[] => {
const result: BedrockTool[] = []
for (const tool of tools) {
result.push(lowerToolSpec(tool))
const cachePoint = BedrockCache.block(breakpoints, tool.cache)
if (cachePoint) result.push(cachePoint)
}
return result
}
const textWithCache = (
breakpoints: BedrockCache.Breakpoints,
text: string,
cache: CacheHint | undefined,
): Array<BedrockTextBlock | BedrockCache.CachePointBlock> => {
const cachePoint = BedrockCache.block(breakpoints, cache)
return cachePoint ? [{ text }, cachePoint] : [{ text }]
}
const lowerToolChoice = (toolChoice: NonNullable<LLMRequest["toolChoice"]>) =>
ProviderShared.matchToolChoice("Bedrock Converse", toolChoice, {
auto: () => ({ auto: {} }) as const,
none: () => undefined,
required: () => ({ any: {} }) as const,
tool: (name) => ({ tool: { name } }) as const,
})
const bedrockMetadata = (metadata: Record<string, unknown>): ProviderMetadata => ({ bedrock: metadata })
const reasoningSignature = (part: ReasoningPart) => {
const bedrock = part.providerMetadata?.bedrock
return (
part.encrypted ??
(ProviderShared.isRecord(bedrock) && typeof bedrock.signature === "string" ? bedrock.signature : undefined)
)
}
const lowerToolCall = (part: ToolCallPart): BedrockToolUseBlock => ({
toolUse: {
toolUseId: part.id,
name: part.name,
input: part.input,
},
})
const lowerToolResultContent = Effect.fn("BedrockConverse.lowerToolResultContent")(function* (part: ToolResultPart) {
if (part.result.type === "text" || part.result.type === "error")
return [{ text: ProviderShared.toolResultText(part) }]
if (part.result.type === "json") return [{ json: part.result.value }]
const content: Array<Schema.Schema.Type<typeof BedrockToolResultContentItem>> = []
for (const item of part.result.value) {
if (item.type === "text") {
content.push({ text: item.text })
continue
}
const media = yield* BedrockMedia.lower({
type: "media",
mediaType: item.mime,
data: item.uri,
filename: item.name,
})
if (!("image" in media))
return yield* ProviderShared.invalidRequest("Bedrock Converse only supports image media in tool results")
content.push(media)
}
return content
})
const lowerToolResult = Effect.fn("BedrockConverse.lowerToolResult")(function* (part: ToolResultPart) {
return {
toolResult: {
toolUseId: part.id,
content: yield* lowerToolResultContent(part),
status: part.result.type === "error" ? "error" : "success",
},
} satisfies BedrockToolResultBlock
})
const lowerMessages = Effect.fn("BedrockConverse.lowerMessages")(function* (
request: LLMRequest,
breakpoints: BedrockCache.Breakpoints,
) {
const messages: BedrockMessage[] = []
for (const message of request.messages) {
if (message.role === "system") {
const part = yield* ProviderShared.wrappedSystemUpdate("Bedrock Converse", message)
const content = textWithCache(breakpoints, part.text, part.cache)
const previous = messages.at(-1)
if (previous?.role === "user")
messages[messages.length - 1] = { role: "user", content: [...previous.content, ...content] }
else messages.push({ role: "user", content })
continue
}
if (message.role === "user") {
const content: BedrockUserBlock[] = []
for (const part of message.content) {
if (!ProviderShared.supportsContent(part, ["text", "media"]))
return yield* ProviderShared.unsupportedContent("Bedrock Converse", "user", ["text", "media"])
if (part.type === "text") {
content.push(...textWithCache(breakpoints, part.text, part.cache))
continue
}
if (part.type === "media") {
content.push(yield* BedrockMedia.lower(part))
continue
}
}
messages.push({ role: "user", content })
continue
}
if (message.role === "assistant") {
const content: BedrockAssistantBlock[] = []
for (const part of message.content) {
if (!ProviderShared.supportsContent(part, ["text", "reasoning", "tool-call"]))
return yield* ProviderShared.unsupportedContent("Bedrock Converse", "assistant", [
"text",
"reasoning",
"tool-call",
])
if (part.type === "text") {
content.push(...textWithCache(breakpoints, part.text, part.cache))
continue
}
if (part.type === "reasoning") {
content.push({
reasoningContent: {
reasoningText: { text: part.text, signature: reasoningSignature(part) },
},
})
continue
}
if (part.type === "tool-call") {
content.push(lowerToolCall(part))
continue
}
}
messages.push({ role: "assistant", content })
continue
}
const content: BedrockUserBlock[] = []
for (const part of message.content) {
if (!ProviderShared.supportsContent(part, ["tool-result"]))
return yield* ProviderShared.unsupportedContent("Bedrock Converse", "tool", ["tool-result"])
content.push(yield* lowerToolResult(part))
const cachePoint = BedrockCache.block(breakpoints, part.cache)
if (cachePoint) content.push(cachePoint)
}
messages.push({ role: "user", content })
}
return messages
})
// System prompts share the cache-point convention: emit the text block, then
// optionally a positional `cachePoint` marker.
const lowerSystem = (
breakpoints: BedrockCache.Breakpoints,
system: ReadonlyArray<LLMRequest["system"][number]>,
): BedrockSystemBlock[] => system.flatMap((part) => textWithCache(breakpoints, part.text, part.cache))
const fromRequest = Effect.fn("BedrockConverse.fromRequest")(function* (request: LLMRequest) {
const toolChoice = request.toolChoice ? yield* lowerToolChoice(request.toolChoice) : undefined
const generation = request.generation
// Bedrock-Claude shares Anthropic's 4-breakpoint cap. Spend the budget in
// tools → system → messages order to favour the highest-impact prefixes.
const breakpoints = BedrockCache.breakpoints()
const toolConfig =
request.tools.length > 0 && request.toolChoice?.type !== "none"
? { tools: lowerTools(breakpoints, request.tools), toolChoice }
: undefined
const system = request.system.length === 0 ? undefined : lowerSystem(breakpoints, request.system)
const messages = yield* lowerMessages(request, breakpoints)
if (breakpoints.dropped > 0) {
yield* Effect.logWarning(
`Bedrock Converse: dropped ${breakpoints.dropped} cache breakpoint(s); the API allows at most ${BedrockCache.BEDROCK_BREAKPOINT_CAP} per request.`,
)
}
return {
modelId: request.model.id,
messages,
system,
inferenceConfig:
generation?.maxTokens === undefined &&
generation?.temperature === undefined &&
generation?.topP === undefined &&
(generation?.stop === undefined || generation.stop.length === 0)
? undefined
: {
maxTokens: generation?.maxTokens,
temperature: generation?.temperature,
topP: generation?.topP,
stopSequences: generation?.stop,
},
toolConfig,
}
})
// =============================================================================
// Stream Parsing
// =============================================================================
const mapFinishReason = (reason: string): FinishReason => {
if (reason === "end_turn" || reason === "stop_sequence") return "stop"
if (reason === "max_tokens") return "length"
if (reason === "tool_use") return "tool-calls"
if (reason === "content_filtered" || reason === "guardrail_intervened") return "content-filter"
return "unknown"
}
// AWS Bedrock Converse reports `inputTokens` (inclusive total) with
// `cacheReadInputTokens` and `cacheWriteInputTokens` as subsets. Pass
// the total through and derive the non-cached breakdown. Bedrock does
// not break reasoning out of `outputTokens` for any current model.
const mapUsage = (usage: BedrockUsageSchema | undefined): Usage | undefined => {
if (!usage) return undefined
const cacheTotal = (usage.cacheReadInputTokens ?? 0) + (usage.cacheWriteInputTokens ?? 0)
const nonCached = ProviderShared.subtractTokens(usage.inputTokens, cacheTotal)
return new Usage({
inputTokens: usage.inputTokens,
outputTokens: usage.outputTokens,
nonCachedInputTokens: nonCached,
cacheReadInputTokens: usage.cacheReadInputTokens,
cacheWriteInputTokens: usage.cacheWriteInputTokens,
totalTokens: ProviderShared.totalTokens(usage.inputTokens, usage.outputTokens, usage.totalTokens),
providerMetadata: { bedrock: usage },
})
}
interface ParserState {
readonly tools: ToolStream.State<number>
// Bedrock splits the finish into `messageStop` (carries `stopReason`) and
// `metadata` (carries usage). Hold the terminal event in state so `onHalt`
// can emit exactly one finish after both chunks have had a chance to arrive.
readonly pendingFinish: { readonly reason: FinishReason; readonly usage?: Usage } | undefined
readonly hasToolCalls: boolean
readonly lifecycle: Lifecycle.State
readonly reasoningSignatures: Readonly<Record<number, string>>
}
const step = (state: ParserState, event: BedrockEvent) =>
Effect.gen(function* () {
if (event.contentBlockStart?.start?.toolUse) {
const index = event.contentBlockStart.contentBlockIndex
const events: LLMEvent[] = []
const lifecycle = Lifecycle.stepStart(state.lifecycle, events)
return [
{
...state,
lifecycle,
tools: ToolStream.start(state.tools, index, {
id: event.contentBlockStart.start.toolUse.toolUseId,
name: event.contentBlockStart.start.toolUse.name,
}),
},
[
...events,
LLMEvent.toolInputStart({
id: event.contentBlockStart.start.toolUse.toolUseId,
name: event.contentBlockStart.start.toolUse.name,
}),
],
] as const
}
if (event.contentBlockDelta?.delta?.text) {
const events: LLMEvent[] = []
return [
{
...state,
lifecycle: Lifecycle.textDelta(
state.lifecycle,
events,
`text-${event.contentBlockDelta.contentBlockIndex}`,
event.contentBlockDelta.delta.text,
),
},
events,
] as const
}
if (event.contentBlockDelta?.delta?.reasoningContent) {
const index = event.contentBlockDelta.contentBlockIndex
const reasoning = event.contentBlockDelta.delta.reasoningContent
const events: LLMEvent[] = []
return [
{
...state,
lifecycle: reasoning.text
? Lifecycle.reasoningDelta(state.lifecycle, events, `reasoning-${index}`, reasoning.text)
: state.lifecycle,
reasoningSignatures: reasoning.signature
? { ...state.reasoningSignatures, [index]: reasoning.signature }
: state.reasoningSignatures,
},
events,
] as const
}
if (event.contentBlockDelta?.delta?.toolUse) {
const index = event.contentBlockDelta.contentBlockIndex
const result = ToolStream.appendExisting(
ADAPTER,
state.tools,
index,
event.contentBlockDelta.delta.toolUse.input,
"Bedrock Converse tool delta is missing its tool call",
)
if (ToolStream.isError(result)) return yield* result
const events: LLMEvent[] = []
const lifecycle = result.events.length ? Lifecycle.stepStart(state.lifecycle, events) : state.lifecycle
events.push(...result.events)
return [{ ...state, lifecycle, tools: result.tools }, events] as const
}
if (event.contentBlockStop) {
const index = event.contentBlockStop.contentBlockIndex
const result = yield* ToolStream.finish(ADAPTER, state.tools, index)
const events: LLMEvent[] = []
const resultEvents = result.events ?? []
const lifecycle = resultEvents.length
? Lifecycle.stepStart(state.lifecycle, events)
: Lifecycle.reasoningEnd(
Lifecycle.textEnd(state.lifecycle, events, `text-${index}`),
events,
`reasoning-${index}`,
state.reasoningSignatures[index]
? bedrockMetadata({ signature: state.reasoningSignatures[index] })
: undefined,
)
events.push(...resultEvents)
return [
{
...state,
hasToolCalls: resultEvents.some(LLMEvent.is.toolCall) ? true : state.hasToolCalls,
lifecycle,
tools: result.tools,
reasoningSignatures: Object.fromEntries(
Object.entries(state.reasoningSignatures).filter(([key]) => key !== String(index)),
),
},
events,
] as const
}
if (event.messageStop) {
return [
{
...state,
pendingFinish: { reason: mapFinishReason(event.messageStop.stopReason), usage: state.pendingFinish?.usage },
},
[],
] as const
}
if (event.metadata) {
const usage = mapUsage(event.metadata.usage)
return [{ ...state, pendingFinish: { reason: state.pendingFinish?.reason ?? "stop", usage } }, []] as const
}
if (event.internalServerException || event.modelStreamErrorException || event.serviceUnavailableException) {
const message =
event.internalServerException?.message ??
event.modelStreamErrorException?.message ??
event.serviceUnavailableException?.message ??
"Bedrock Converse stream error"
return [state, [LLMEvent.providerError({ message, retryable: true })]] as const
}
if (event.validationException || event.throttlingException) {
const message =
event.validationException?.message ?? event.throttlingException?.message ?? "Bedrock Converse error"
return [
state,
[
LLMEvent.providerError({
message,
classification: event.validationException && isContextOverflow(message) ? "context-overflow" : undefined,
retryable: event.throttlingException !== undefined,
}),
],
] as const
}
return [state, []] as const
})
const framing = BedrockEventStream.framing(ADAPTER)
const onHalt = (state: ParserState): ReadonlyArray<LLMEvent> =>
state.pendingFinish
? (() => {
const events: LLMEvent[] = []
Lifecycle.finish(state.lifecycle, events, {
reason:
state.pendingFinish.reason === "stop" && state.hasToolCalls ? "tool-calls" : state.pendingFinish.reason,
usage: state.pendingFinish.usage,
})
return events
})()
: []
// =============================================================================
// Protocol And Bedrock Route
// =============================================================================
/**
* The Bedrock Converse protocol — request body construction, body schema, and
* the streaming-event state machine.
*/
export const protocol = Protocol.make({
id: ADAPTER,
body: {
schema: BedrockConverseBody,
from: fromRequest,
},
stream: {
event: BedrockEvent,
initial: () => ({
tools: ToolStream.empty<number>(),
pendingFinish: undefined,
hasToolCalls: false,
lifecycle: Lifecycle.initial(),
reasoningSignatures: {},
}),
step,
onHalt,
},
})
export const route = Route.make({
id: ADAPTER,
provider: "bedrock",
protocol,
// Bedrock's URL embeds the region in the route endpoint host and the
// validated modelId in the path. We read the validated body so the URL
// matches the body that gets signed.
endpoint: Endpoint.path<BedrockConverseBody>(
({ body }) => `/model/${encodeURIComponent(body.modelId)}/converse-stream`,
),
auth: BedrockAuth.auth,
framing,
})
export const sigV4Auth = BedrockAuth.sigV4
export * as BedrockConverse from "./bedrock-converse"

87
packages/llm/src/protocols/bedrock-event-stream.ts Normal file
View File

@@ -0,0 +1,87 @@
import { EventStreamCodec } from "@smithy/eventstream-codec"
import { fromUtf8, toUtf8 } from "@smithy/util-utf8"
import { Effect, Stream } from "effect"
import type { Framing } from "../route/framing"
import { ProviderShared } from "./shared"
// Bedrock streams responses using the AWS event stream binary protocol — each
// frame is `[length:4][headers-length:4][prelude-crc:4][headers][payload][crc:4]`.
// We use `@smithy/eventstream-codec` to validate framing and CRCs, then
// reconstruct the JSON wrapping by `:event-type` so the chunk schema can match.
const eventCodec = new EventStreamCodec(toUtf8, fromUtf8)
const utf8 = new TextDecoder()
// Cursor-tracking buffer state. Bytes accumulate in `buffer`; `offset` is the
// read position. Reading by `subarray` is zero-copy. We only allocate a fresh
// buffer when a new network chunk arrives and we need to append.
interface FrameBufferState {
readonly buffer: Uint8Array
readonly offset: number
}
const initialFrameBuffer: FrameBufferState = { buffer: new Uint8Array(0), offset: 0 }
const appendChunk = (state: FrameBufferState, chunk: Uint8Array): FrameBufferState => {
const remaining = state.buffer.length - state.offset
// Compact: drop the consumed prefix and append the new chunk in one alloc.
// This bounds buffer growth to at most one network chunk past the live
// window, regardless of stream length.
const next = new Uint8Array(remaining + chunk.length)
next.set(state.buffer.subarray(state.offset), 0)
next.set(chunk, remaining)
return { buffer: next, offset: 0 }
}
const consumeFrames = (route: string) => (state: FrameBufferState, chunk: Uint8Array) =>
Effect.gen(function* () {
let cursor = appendChunk(state, chunk)
const out: object[] = []
while (cursor.buffer.length - cursor.offset >= 4) {
const view = cursor.buffer.subarray(cursor.offset)
const totalLength = new DataView(view.buffer, view.byteOffset, view.byteLength).getUint32(0, false)
if (view.length < totalLength) break
const decoded = yield* Effect.try({
try: () => eventCodec.decode(view.subarray(0, totalLength)),
catch: (error) =>
ProviderShared.eventError(
route,
`Failed to decode Bedrock Converse event-stream frame: ${
error instanceof Error ? error.message : String(error)
}`,
),
})
cursor = { buffer: cursor.buffer, offset: cursor.offset + totalLength }
if (decoded.headers[":message-type"]?.value !== "event") continue
const eventType = decoded.headers[":event-type"]?.value
if (typeof eventType !== "string") continue
const payload = utf8.decode(decoded.body)
if (!payload) continue
// The AWS event stream pads short payloads with a `p` field. Drop it
// before handing the object to the chunk schema. JSON decode goes
// through the shared Schema-driven codec to satisfy the package rule
// against ad-hoc `JSON.parse` calls.
const parsed = (yield* ProviderShared.parseJson(
route,
payload,
"Failed to parse Bedrock Converse event-stream payload",
)) as Record<string, unknown>
delete parsed.p
out.push({ [eventType]: parsed })
}
return [cursor, out] as const
})
/**
* AWS event-stream framing for Bedrock Converse. Each frame is decoded by
* `@smithy/eventstream-codec` (length + header + payload + CRC) and rewrapped
* under its `:event-type` header so the chunk schema can match the JSON
* payload directly.
*/
export const framing = (route: string): Framing<object> => ({
id: "aws-event-stream",
frame: (bytes) => bytes.pipe(Stream.mapAccumEffect(() => initialFrameBuffer, consumeFrames(route))),
})
export * as BedrockEventStream from "./bedrock-event-stream"

487
packages/llm/src/protocols/gemini.ts Normal file
View File

@@ -0,0 +1,487 @@
import { Effect, Schema } from "effect"
import { Route } from "../route/client"
import { Auth } from "../route/auth"
import { Endpoint } from "../route/endpoint"
import { Framing } from "../route/framing"
import { Protocol } from "../route/protocol"
import {
LLMEvent,
Usage,
type FinishReason,
type LLMRequest,
type MediaPart,
type ProviderMetadata,
type TextPart,
type ToolCallPart,
type ToolDefinition,
type ToolContent,
} from "../schema"
import { JsonObject, optionalArray, ProviderShared } from "./shared"
import { GeminiToolSchema } from "./utils/gemini-tool-schema"
import { Lifecycle } from "./utils/lifecycle"
const ADAPTER = "gemini"
const IMAGE_MIMES = new Set<string>(ProviderShared.IMAGE_MIMES)
export const DEFAULT_BASE_URL = "https://generativelanguage.googleapis.com/v1beta"
// =============================================================================
// Request Body Schema
// =============================================================================
const GeminiTextPart = Schema.Struct({
text: Schema.String,
thought: Schema.optional(Schema.Boolean),
thoughtSignature: Schema.optional(Schema.String),
})
const GeminiInlineDataPart = Schema.Struct({
inlineData: Schema.Struct({
mimeType: Schema.String,
data: Schema.String,
}),
})
const GeminiFunctionCallPart = Schema.Struct({
functionCall: Schema.Struct({
name: Schema.String,
args: Schema.Unknown,
}),
thoughtSignature: Schema.optional(Schema.String),
})
const GeminiFunctionResponsePart = Schema.Struct({
functionResponse: Schema.Struct({
name: Schema.String,
response: Schema.Unknown,
}),
})
const GeminiContentPart = Schema.Union([
GeminiTextPart,
GeminiInlineDataPart,
GeminiFunctionCallPart,
GeminiFunctionResponsePart,
])
const GeminiContent = Schema.Struct({
role: Schema.Literals(["user", "model"]),
parts: Schema.Array(GeminiContentPart),
})
type GeminiContent = Schema.Schema.Type<typeof GeminiContent>
const GeminiSystemInstruction = Schema.Struct({
parts: Schema.Array(Schema.Struct({ text: Schema.String })),
})
const GeminiFunctionDeclaration = Schema.Struct({
name: Schema.String,
description: Schema.String,
parameters: Schema.optional(JsonObject),
})
const GeminiTool = Schema.Struct({
functionDeclarations: Schema.Array(GeminiFunctionDeclaration),
})
const GeminiToolConfig = Schema.Struct({
functionCallingConfig: Schema.Struct({
mode: Schema.Literals(["AUTO", "NONE", "ANY"]),
allowedFunctionNames: optionalArray(Schema.String),
}),
})
const GeminiThinkingConfig = Schema.Struct({
thinkingBudget: Schema.optional(Schema.Number),
includeThoughts: Schema.optional(Schema.Boolean),
})
const GeminiGenerationConfig = Schema.Struct({
maxOutputTokens: Schema.optional(Schema.Number),
temperature: Schema.optional(Schema.Number),
topP: Schema.optional(Schema.Number),
topK: Schema.optional(Schema.Number),
stopSequences: optionalArray(Schema.String),
thinkingConfig: Schema.optional(GeminiThinkingConfig),
})
const GeminiBodyFields = {
contents: Schema.Array(GeminiContent),
systemInstruction: Schema.optional(GeminiSystemInstruction),
tools: optionalArray(GeminiTool),
toolConfig: Schema.optional(GeminiToolConfig),
generationConfig: Schema.optional(GeminiGenerationConfig),
}
const GeminiBody = Schema.Struct(GeminiBodyFields)
export type GeminiBody = Schema.Schema.Type<typeof GeminiBody>
const GeminiUsage = Schema.Struct({
cachedContentTokenCount: Schema.optional(Schema.Number),
thoughtsTokenCount: Schema.optional(Schema.Number),
promptTokenCount: Schema.optional(Schema.Number),
candidatesTokenCount: Schema.optional(Schema.Number),
totalTokenCount: Schema.optional(Schema.Number),
})
type GeminiUsage = Schema.Schema.Type<typeof GeminiUsage>
const GeminiCandidate = Schema.Struct({
content: Schema.optional(GeminiContent),
finishReason: Schema.optional(Schema.String),
})
const GeminiEvent = Schema.Struct({
candidates: optionalArray(GeminiCandidate),
usageMetadata: Schema.optional(GeminiUsage),
})
type GeminiEvent = Schema.Schema.Type<typeof GeminiEvent>
interface ParserState {
readonly finishReason?: string
readonly hasToolCalls: boolean
readonly nextToolCallId: number
readonly usage?: Usage
readonly lifecycle: Lifecycle.State
readonly reasoningSignature?: string
}
// =============================================================================
// Tool Schema Conversion
// =============================================================================
// Tool-schema conversion has two distinct concerns:
//
// 1. Sanitize — fix common authoring mistakes Gemini rejects: integer/number
// enums (must be strings), `required` entries that don't match a property,
// untyped arrays (`items` must be present), and `properties`/`required`
// keys on non-object scalars. Mirrors OpenCode's historical Gemini rules.
//
// 2. Project — lossy mapping from JSON Schema to Gemini's schema dialect:
// drop empty objects, derive `nullable: true` from `type: [..., "null"]`,
// coerce `const` to `[const]` enum, recurse properties/items, propagate
// only an allowlisted set of keys (description, required, format, type,
// properties, items, allOf, anyOf, oneOf, minLength). Anything outside the
// allowlist (e.g. `additionalProperties`, `$ref`) is silently dropped.
//
// Sanitize runs first, then project. The implementation lives in
// `utils/gemini-tool-schema` so this protocol keeps the same shape as the other
// provider protocols.
// =============================================================================
// Request Lowering
// =============================================================================
const lowerTool = (tool: ToolDefinition) => ({
name: tool.name,
description: tool.description,
parameters: GeminiToolSchema.convert(tool.inputSchema),
})
const lowerToolConfig = (toolChoice: NonNullable<LLMRequest["toolChoice"]>) =>
ProviderShared.matchToolChoice("Gemini", toolChoice, {
auto: () => ({ functionCallingConfig: { mode: "AUTO" as const } }),
none: () => ({ functionCallingConfig: { mode: "NONE" as const } }),
required: () => ({ functionCallingConfig: { mode: "ANY" as const } }),
tool: (name) => ({ functionCallingConfig: { mode: "ANY" as const, allowedFunctionNames: [name] } }),
})
const lowerUserPart = Effect.fn("Gemini.lowerUserPart")(function* (part: TextPart | MediaPart) {
if (part.type === "text") return { text: part.text }
const media = yield* ProviderShared.validateMedia("Gemini", part, IMAGE_MIMES)
return { inlineData: { mimeType: media.mime, data: media.base64 } }
})
const googleMetadata = (metadata: Record<string, unknown>): ProviderMetadata => ({ google: metadata })
const thoughtSignature = (providerMetadata: ProviderMetadata | undefined) => {
const google = providerMetadata?.google
return ProviderShared.isRecord(google) && typeof google.thoughtSignature === "string"
? google.thoughtSignature
: undefined
}
const lowerToolCall = (part: ToolCallPart) => ({
functionCall: { name: part.name, args: part.input },
thoughtSignature: thoughtSignature(part.providerMetadata),
})
const lowerMessages = Effect.fn("Gemini.lowerMessages")(function* (request: LLMRequest) {
const contents: GeminiContent[] = []
for (const message of request.messages) {
if (message.role === "system") {
const part = yield* ProviderShared.wrappedSystemUpdate("Gemini", message)
const previous = contents.at(-1)
if (previous?.role === "user")
contents[contents.length - 1] = { role: "user", parts: [...previous.parts, { text: part.text }] }
else contents.push({ role: "user", parts: [{ text: part.text }] })
continue
}
if (message.role === "user") {
const parts: Array<Schema.Schema.Type<typeof GeminiContentPart>> = []
for (const part of message.content) {
if (!ProviderShared.supportsContent(part, ["text", "media"]))
return yield* ProviderShared.unsupportedContent("Gemini", "user", ["text", "media"])
parts.push(yield* lowerUserPart(part))
}
contents.push({ role: "user", parts })
continue
}
if (message.role === "assistant") {
const parts: Array<Schema.Schema.Type<typeof GeminiContentPart>> = []
for (const part of message.content) {
if (!ProviderShared.supportsContent(part, ["text", "reasoning", "tool-call"]))
return yield* ProviderShared.unsupportedContent("Gemini", "assistant", ["text", "reasoning", "tool-call"])
if (part.type === "text") {
parts.push({ text: part.text })
continue
}
if (part.type === "reasoning") {
parts.push({ text: part.text, thought: true, thoughtSignature: thoughtSignature(part.providerMetadata) })
continue
}
if (part.type === "tool-call") {
parts.push(lowerToolCall(part))
continue
}
}
contents.push({ role: "model", parts })
continue
}
const parts: Array<Schema.Schema.Type<typeof GeminiContentPart>> = []
for (const part of message.content) {
if (!ProviderShared.supportsContent(part, ["tool-result"]))
return yield* ProviderShared.unsupportedContent("Gemini", "tool", ["tool-result"])
if (part.result.type !== "content") {
parts.push({
functionResponse: {
name: part.name,
response: {
name: part.name,
content: ProviderShared.toolResultText(part),
},
},
})
continue
}
const content: ReadonlyArray<ToolContent> = part.result.value
const text = content.filter((item) => item.type === "text").map((item) => item.text)
parts.push({
functionResponse: {
name: part.name,
response: {
name: part.name,
content: text.join("\n"),
},
},
})
for (const item of content) {
if (item.type === "text") continue
const media = yield* ProviderShared.validateToolFile("Gemini", item, IMAGE_MIMES)
parts.push({ inlineData: { mimeType: media.mime, data: media.base64 } })
}
}
contents.push({ role: "user", parts })
}
return contents
})
const geminiOptions = (request: LLMRequest) => request.providerOptions?.gemini
const thinkingConfig = (request: LLMRequest) => {
const value = geminiOptions(request)?.thinkingConfig
if (!ProviderShared.isRecord(value)) return undefined
const result = {
thinkingBudget: typeof value.thinkingBudget === "number" ? value.thinkingBudget : undefined,
includeThoughts: typeof value.includeThoughts === "boolean" ? value.includeThoughts : undefined,
}
return Object.values(result).some((item) => item !== undefined) ? result : undefined
}
const fromRequest = Effect.fn("Gemini.fromRequest")(function* (request: LLMRequest) {
const toolsEnabled = request.tools.length > 0 && request.toolChoice?.type !== "none"
const generation = request.generation
const generationConfig = {
maxOutputTokens: generation?.maxTokens,
temperature: generation?.temperature,
topP: generation?.topP,
topK: generation?.topK,
stopSequences: generation?.stop,
thinkingConfig: thinkingConfig(request),
}
return {
contents: yield* lowerMessages(request),
systemInstruction:
request.system.length === 0 ? undefined : { parts: [{ text: ProviderShared.joinText(request.system) }] },
tools: toolsEnabled ? [{ functionDeclarations: request.tools.map(lowerTool) }] : undefined,
toolConfig: toolsEnabled && request.toolChoice ? yield* lowerToolConfig(request.toolChoice) : undefined,
generationConfig: Object.values(generationConfig).some((value) => value !== undefined)
? generationConfig
: undefined,
}
})
// =============================================================================
// Stream Parsing
// =============================================================================
// Gemini reports `promptTokenCount` (inclusive total) with a
// `cachedContentTokenCount` subset. `candidatesTokenCount` is *exclusive*
// of `thoughtsTokenCount` — visible-only, not a total — so we sum the two
// to produce the inclusive `outputTokens` the rest of the contract expects.
const mapUsage = (usage: GeminiUsage | undefined) => {
if (!usage) return undefined
const cached = usage.cachedContentTokenCount
const nonCached = ProviderShared.subtractTokens(usage.promptTokenCount, cached)
// `candidatesTokenCount` is visible-only; sum with thoughts to produce the
// inclusive `outputTokens` the contract expects. Only compute the total
// when the visible component is reported — otherwise we'd fabricate an
// inclusive number from a partial breakdown.
const outputTokens =
usage.candidatesTokenCount !== undefined ? usage.candidatesTokenCount + (usage.thoughtsTokenCount ?? 0) : undefined
return new Usage({
inputTokens: usage.promptTokenCount,
outputTokens,
nonCachedInputTokens: nonCached,
cacheReadInputTokens: cached,
reasoningTokens: usage.thoughtsTokenCount,
totalTokens: ProviderShared.totalTokens(usage.promptTokenCount, outputTokens, usage.totalTokenCount),
providerMetadata: { google: usage },
})
}
const mapFinishReason = (finishReason: string | undefined, hasToolCalls: boolean): FinishReason => {
if (finishReason === "STOP") return hasToolCalls ? "tool-calls" : "stop"
if (finishReason === "MAX_TOKENS") return "length"
if (
finishReason === "IMAGE_SAFETY" ||
finishReason === "RECITATION" ||
finishReason === "SAFETY" ||
finishReason === "BLOCKLIST" ||
finishReason === "PROHIBITED_CONTENT" ||
finishReason === "SPII"
)
return "content-filter"
if (finishReason === "MALFORMED_FUNCTION_CALL") return "error"
return "unknown"
}
const finish = (state: ParserState): ReadonlyArray<LLMEvent> =>
state.finishReason || state.usage
? (() => {
const events: LLMEvent[] = []
const lifecycle = state.reasoningSignature
? Lifecycle.reasoningEnd(
state.lifecycle,
events,
"reasoning-0",
googleMetadata({ thoughtSignature: state.reasoningSignature }),
)
: state.lifecycle
Lifecycle.finish(lifecycle, events, {
reason: mapFinishReason(state.finishReason, state.hasToolCalls),
usage: state.usage,
})
return events
})()
: []
const step = (state: ParserState, event: GeminiEvent) => {
const nextState = {
...state,
usage: event.usageMetadata ? (mapUsage(event.usageMetadata) ?? state.usage) : state.usage,
}
const candidate = event.candidates?.[0]
if (!candidate?.content)
return Effect.succeed([
{ ...nextState, finishReason: candidate?.finishReason ?? nextState.finishReason },
[],
] as const)
const events: LLMEvent[] = []
let hasToolCalls = nextState.hasToolCalls
let lifecycle = nextState.lifecycle
let nextToolCallId = nextState.nextToolCallId
let reasoningSignature = nextState.reasoningSignature
for (const part of candidate.content.parts) {
if ("thoughtSignature" in part && part.thoughtSignature && "thought" in part && part.thought)
reasoningSignature = part.thoughtSignature
if ("text" in part && part.text.length > 0) {
lifecycle = part.thought
? Lifecycle.reasoningDelta(
lifecycle,
events,
"reasoning-0",
part.text,
part.thoughtSignature ? googleMetadata({ thoughtSignature: part.thoughtSignature }) : undefined,
)
: Lifecycle.textDelta(lifecycle, events, "text-0", part.text)
continue
}
if ("functionCall" in part) {
const input = part.functionCall.args
const id = `tool_${nextToolCallId++}`
lifecycle = Lifecycle.stepStart(lifecycle, events)
events.push(
LLMEvent.toolCall({
id,
name: part.functionCall.name,
input,
providerMetadata: part.thoughtSignature
? googleMetadata({ thoughtSignature: part.thoughtSignature })
: undefined,
}),
)
hasToolCalls = true
}
}
return Effect.succeed([
{
...nextState,
hasToolCalls,
lifecycle,
nextToolCallId,
reasoningSignature,
finishReason: candidate.finishReason ?? nextState.finishReason,
},
events,
] as const)
}
// =============================================================================
// Protocol And Gemini Route
// =============================================================================
/**
* The Gemini protocol — request body construction, body schema, and the
* streaming-event state machine. Used by Google AI Studio Gemini and (once
* registered) Vertex Gemini.
*/
export const protocol = Protocol.make({
id: ADAPTER,
body: {
schema: GeminiBody,
from: fromRequest,
},
stream: {
event: Protocol.jsonEvent(GeminiEvent),
initial: () => ({ hasToolCalls: false, nextToolCallId: 0, lifecycle: Lifecycle.initial() }),
step,
onHalt: finish,
},
})
export const route = Route.make({
id: ADAPTER,
provider: "google",
protocol,
// Gemini's path embeds the model id and pins SSE framing at the URL level.
endpoint: Endpoint.path(({ request }) => `/models/${request.model.id}:streamGenerateContent?alt=sse`, {
baseURL: DEFAULT_BASE_URL,
}),
auth: Auth.none,
framing: Framing.sse,
})
export * as Gemini from "./gemini"

6
packages/llm/src/protocols/index.ts Normal file
View File

@@ -0,0 +1,6 @@
export * as AnthropicMessages from "./anthropic-messages"
export * as BedrockConverse from "./bedrock-converse"
export * as Gemini from "./gemini"
export * as OpenAIChat from "./openai-chat"
export * as OpenAICompatibleChat from "./openai-compatible-chat"
export * as OpenAIResponses from "./openai-responses"

493
packages/llm/src/protocols/openai-chat.ts Normal file
View File

@@ -0,0 +1,493 @@
import { Effect, Schema } from "effect"
import { Route } from "../route/client"
import { Auth } from "../route/auth"
import { Endpoint } from "../route/endpoint"
import { HttpTransport } from "../route/transport"
import { Protocol } from "../route/protocol"
import {
LLMEvent,
Usage,
type FinishReason,
type LLMRequest,
type MediaPart,
type ReasoningPart,
type TextPart,
type ToolCallPart,
type ToolDefinition,
type ToolContent,
} from "../schema"
import { isRecord, JsonObject, optionalArray, optionalNull, ProviderShared } from "./shared"
import { OpenAIOptions } from "./utils/openai-options"
import { Lifecycle } from "./utils/lifecycle"
import { ToolStream } from "./utils/tool-stream"
const ADAPTER = "openai-chat"
const IMAGE_MIMES = new Set<string>(ProviderShared.IMAGE_MIMES)
export const DEFAULT_BASE_URL = "https://api.openai.com/v1"
export const PATH = "/chat/completions"
// =============================================================================
// Request Body Schema
// =============================================================================
// The body schema is the provider-native JSON body. `fromRequest` below builds
// this shape from the common `LLMRequest`, then `Route.make` validates and
// JSON-encodes it before transport.
const OpenAIChatFunction = Schema.Struct({
name: Schema.String,
description: Schema.String,
parameters: JsonObject,
})
const OpenAIChatTool = Schema.Struct({
type: Schema.tag("function"),
function: OpenAIChatFunction,
})
type OpenAIChatTool = Schema.Schema.Type<typeof OpenAIChatTool>
const OpenAIChatAssistantToolCall = Schema.Struct({
id: Schema.String,
type: Schema.tag("function"),
function: Schema.Struct({
name: Schema.String,
arguments: Schema.String,
}),
})
type OpenAIChatAssistantToolCall = Schema.Schema.Type<typeof OpenAIChatAssistantToolCall>
const OpenAIChatUserContent = Schema.Union([
Schema.Struct({ type: Schema.Literal("text"), text: Schema.String }),
Schema.Struct({
type: Schema.Literal("image_url"),
image_url: Schema.Struct({ url: Schema.String }),
}),
])
const OpenAIChatMessage = Schema.Union([
Schema.Struct({ role: Schema.Literal("system"), content: Schema.String }),
Schema.Struct({
role: Schema.Literal("user"),
content: Schema.Union([Schema.String, Schema.Array(OpenAIChatUserContent)]),
}),
Schema.Struct({
role: Schema.Literal("assistant"),
content: Schema.NullOr(Schema.String),
tool_calls: optionalArray(OpenAIChatAssistantToolCall),
reasoning_content: Schema.optional(Schema.String),
}),
Schema.Struct({ role: Schema.Literal("tool"), tool_call_id: Schema.String, content: Schema.String }),
]).pipe(Schema.toTaggedUnion("role"))
type OpenAIChatMessage = Schema.Schema.Type<typeof OpenAIChatMessage>
const OpenAIChatToolChoice = Schema.Union([
Schema.Literals(["auto", "none", "required"]),
Schema.Struct({
type: Schema.tag("function"),
function: Schema.Struct({ name: Schema.String }),
}),
])
export const bodyFields = {
model: Schema.String,
messages: Schema.Array(OpenAIChatMessage),
tools: optionalArray(OpenAIChatTool),
tool_choice: Schema.optional(OpenAIChatToolChoice),
stream: Schema.Literal(true),
stream_options: Schema.optional(Schema.Struct({ include_usage: Schema.Boolean })),
store: Schema.optional(Schema.Boolean),
reasoning_effort: Schema.optional(OpenAIOptions.OpenAIReasoningEffort),
max_tokens: Schema.optional(Schema.Number),
temperature: Schema.optional(Schema.Number),
top_p: Schema.optional(Schema.Number),
frequency_penalty: Schema.optional(Schema.Number),
presence_penalty: Schema.optional(Schema.Number),
seed: Schema.optional(Schema.Number),
stop: optionalArray(Schema.String),
}
const OpenAIChatBody = Schema.Struct(bodyFields)
export type OpenAIChatBody = Schema.Schema.Type<typeof OpenAIChatBody>
// =============================================================================
// Streaming Event Schema
// =============================================================================
// The event schema is one decoded SSE `data:` payload. `Framing.sse` splits the
// byte stream into strings, then `Protocol.jsonEvent` decodes each string into
// this provider-native event shape.
const OpenAIChatUsage = Schema.Struct({
prompt_tokens: Schema.optional(Schema.Number),
completion_tokens: Schema.optional(Schema.Number),
total_tokens: Schema.optional(Schema.Number),
prompt_tokens_details: optionalNull(
Schema.Struct({
cached_tokens: Schema.optional(Schema.Number),
}),
),
completion_tokens_details: optionalNull(
Schema.Struct({
reasoning_tokens: Schema.optional(Schema.Number),
}),
),
})
const OpenAIChatToolCallDeltaFunction = Schema.Struct({
name: optionalNull(Schema.String),
arguments: optionalNull(Schema.String),
})
const OpenAIChatToolCallDelta = Schema.Struct({
index: Schema.Number,
id: optionalNull(Schema.String),
function: optionalNull(OpenAIChatToolCallDeltaFunction),
})
type OpenAIChatToolCallDelta = Schema.Schema.Type<typeof OpenAIChatToolCallDelta>
const OpenAIChatDelta = Schema.Struct({
content: optionalNull(Schema.String),
reasoning_content: optionalNull(Schema.String),
tool_calls: optionalNull(Schema.Array(OpenAIChatToolCallDelta)),
})
const OpenAIChatChoice = Schema.Struct({
delta: optionalNull(OpenAIChatDelta),
finish_reason: optionalNull(Schema.String),
})
const OpenAIChatEvent = Schema.Struct({
choices: Schema.Array(OpenAIChatChoice),
usage: optionalNull(OpenAIChatUsage),
})
type OpenAIChatEvent = Schema.Schema.Type<typeof OpenAIChatEvent>
type OpenAIChatRequestMessage = LLMRequest["messages"][number]
interface ParserState {
readonly tools: ToolStream.State<number>
readonly toolCallEvents: ReadonlyArray<LLMEvent>
readonly usage?: Usage
readonly finishReason?: FinishReason
readonly lifecycle: Lifecycle.State
}
const invalid = ProviderShared.invalidRequest
// =============================================================================
// Request Lowering
// =============================================================================
// Lowering is the only place that knows how common LLM messages map onto the
// OpenAI Chat wire format. Keep provider quirks here instead of leaking native
// fields into `LLMRequest`.
const lowerTool = (tool: ToolDefinition): OpenAIChatTool => ({
type: "function",
function: {
name: tool.name,
description: tool.description,
parameters: ProviderShared.openAiToolInputSchema(tool.inputSchema),
},
})
const lowerToolChoice = (toolChoice: NonNullable<LLMRequest["toolChoice"]>) =>
ProviderShared.matchToolChoice("OpenAI Chat", toolChoice, {
auto: () => "auto" as const,
none: () => "none" as const,
required: () => "required" as const,
tool: (name) => ({ type: "function" as const, function: { name } }),
})
const lowerToolCall = (part: ToolCallPart): OpenAIChatAssistantToolCall => ({
id: part.id,
type: "function",
function: {
name: part.name,
arguments: ProviderShared.encodeJson(part.input),
},
})
const lowerMedia = Effect.fn("OpenAIChat.lowerMedia")(function* (part: MediaPart) {
const media = yield* ProviderShared.validateMedia("OpenAI Chat", part, IMAGE_MIMES)
return { type: "image_url" as const, image_url: { url: media.dataUrl } }
})
const openAICompatibleReasoningContent = (native: unknown) =>
isRecord(native) && typeof native.reasoning_content === "string" ? native.reasoning_content : undefined
const lowerUserMessage = Effect.fn("OpenAIChat.lowerUserMessage")(function* (message: OpenAIChatRequestMessage) {
const content: Array<Schema.Schema.Type<typeof OpenAIChatUserContent>> = []
for (const part of message.content) {
if (part.type === "text") {
content.push({ type: "text", text: part.text })
continue
}
if (part.type === "media") {
content.push(yield* lowerMedia(part))
continue
}
return yield* ProviderShared.unsupportedContent("OpenAI Chat", "user", ["text", "media"])
}
if (content.every((part) => part.type === "text"))
return { role: "user" as const, content: content.map((part) => part.text).join("") }
return { role: "user" as const, content }
})
const lowerAssistantMessage = Effect.fn("OpenAIChat.lowerAssistantMessage")(function* (
message: OpenAIChatRequestMessage,
) {
const content: TextPart[] = []
const reasoning: ReasoningPart[] = []
const toolCalls: OpenAIChatAssistantToolCall[] = []
for (const part of message.content) {
if (!ProviderShared.supportsContent(part, ["text", "reasoning", "tool-call"]))
return yield* ProviderShared.unsupportedContent("OpenAI Chat", "assistant", ["text", "reasoning", "tool-call"])
if (part.type === "text") {
content.push(part)
continue
}
if (part.type === "reasoning") {
reasoning.push(part)
continue
}
if (part.type === "tool-call") {
toolCalls.push(lowerToolCall(part))
continue
}
}
return {
role: "assistant" as const,
content: content.length === 0 ? null : ProviderShared.joinText(content),
tool_calls: toolCalls.length === 0 ? undefined : toolCalls,
reasoning_content:
reasoning.length > 0
? reasoning.map((part) => part.text).join("")
: openAICompatibleReasoningContent(message.native?.openaiCompatible),
}
})
const lowerToolMessages = Effect.fn("OpenAIChat.lowerToolMessages")(function* (message: OpenAIChatRequestMessage) {
const messages: OpenAIChatMessage[] = []
const images: Array<Schema.Schema.Type<typeof OpenAIChatUserContent>> = []
for (const part of message.content) {
if (!ProviderShared.supportsContent(part, ["tool-result"]))
return yield* ProviderShared.unsupportedContent("OpenAI Chat", "tool", ["tool-result"])
if (part.result.type !== "content") {
messages.push({ role: "tool", tool_call_id: part.id, content: ProviderShared.toolResultText(part) })
continue
}
const content: ReadonlyArray<ToolContent> = part.result.value
const text = content.filter((item) => item.type === "text").map((item) => item.text)
messages.push({ role: "tool", tool_call_id: part.id, content: text.join("\n") })
const files = content.filter((item) => item.type === "file")
images.push(
...(yield* Effect.forEach(files, (item) =>
lowerMedia({ type: "media", mediaType: item.mime, data: item.uri, filename: item.name }),
)),
)
}
return { messages, images }
})
const lowerMessage = Effect.fn("OpenAIChat.lowerMessage")(function* (message: OpenAIChatRequestMessage) {
if (message.role === "user") return [yield* lowerUserMessage(message)]
if (message.role === "assistant") return [yield* lowerAssistantMessage(message)]
return (yield* lowerToolMessages(message)).messages
})
const lowerMessages = Effect.fn("OpenAIChat.lowerMessages")(function* (request: LLMRequest) {
const system: OpenAIChatMessage[] =
request.system.length === 0 ? [] : [{ role: "system", content: ProviderShared.joinText(request.system) }]
const messages = [...system]
const pendingImages: Array<Schema.Schema.Type<typeof OpenAIChatUserContent>> = []
const flushImages = () => {
if (pendingImages.length === 0) return
messages.push({ role: "user", content: pendingImages.splice(0) })
}
for (const message of request.messages) {
if (message.role === "system") {
const part = yield* ProviderShared.wrappedSystemUpdate("OpenAI Chat", message)
if (pendingImages.length > 0) {
messages.push({ role: "user", content: [...pendingImages.splice(0), { type: "text", text: part.text }] })
continue
}
const previous = messages.at(-1)
if (previous?.role === "user" && typeof previous.content === "string")
messages[messages.length - 1] = { role: "user", content: `${previous.content}\n${part.text}` }
else if (previous?.role === "user" && Array.isArray(previous.content))
messages[messages.length - 1] = {
role: "user",
content: [...previous.content, { type: "text", text: part.text }],
}
else messages.push({ role: "user", content: part.text })
continue
}
if (message.role === "tool") {
const lowered = yield* lowerToolMessages(message)
messages.push(...lowered.messages)
pendingImages.push(...lowered.images)
continue
}
flushImages()
messages.push(...(yield* lowerMessage(message)))
}
flushImages()
return messages
})
const lowerOptions = Effect.fn("OpenAIChat.lowerOptions")(function* (request: LLMRequest) {
const store = OpenAIOptions.store(request)
const reasoningEffort = OpenAIOptions.reasoningEffort(request)
if (reasoningEffort && !OpenAIOptions.isReasoningEffort(reasoningEffort))
return yield* invalid(`OpenAI Chat does not support reasoning effort ${reasoningEffort}`)
return {
...(store !== undefined ? { store } : {}),
...(reasoningEffort ? { reasoning_effort: reasoningEffort } : {}),
}
})
const fromRequest = Effect.fn("OpenAIChat.fromRequest")(function* (request: LLMRequest) {
// `fromRequest` returns the provider body only. Endpoint, auth, framing,
// validation, and HTTP execution are composed by `Route.make`.
const generation = request.generation
return {
model: request.model.id,
messages: yield* lowerMessages(request),
tools: request.tools.length === 0 ? undefined : request.tools.map(lowerTool),
tool_choice: request.toolChoice ? yield* lowerToolChoice(request.toolChoice) : undefined,
stream: true as const,
stream_options: { include_usage: true },
max_tokens: generation?.maxTokens,
temperature: generation?.temperature,
top_p: generation?.topP,
frequency_penalty: generation?.frequencyPenalty,
presence_penalty: generation?.presencePenalty,
seed: generation?.seed,
stop: generation?.stop,
...(yield* lowerOptions(request)),
}
})
// =============================================================================
// Stream Parsing
// =============================================================================
// Streaming parsers are small state machines: every event returns a new state
// plus the common `LLMEvent`s produced by that event. Tool calls are accumulated
// because OpenAI streams JSON arguments across multiple deltas.
const mapFinishReason = (reason: string | null | undefined): FinishReason => {
if (reason === "stop") return "stop"
if (reason === "length") return "length"
if (reason === "content_filter") return "content-filter"
if (reason === "function_call" || reason === "tool_calls") return "tool-calls"
return "unknown"
}
// OpenAI Chat reports `prompt_tokens` (inclusive total) with a
// `cached_tokens` subset, and `completion_tokens` (inclusive total) with
// a `reasoning_tokens` subset. We pass the inclusive totals through and
// derive the non-cached breakdown so the `LLM.Usage` contract is
// satisfied on both sides.
const mapUsage = (usage: OpenAIChatEvent["usage"]): Usage | undefined => {
if (!usage) return undefined
const cached = usage.prompt_tokens_details?.cached_tokens
const reasoning = usage.completion_tokens_details?.reasoning_tokens
const nonCached = ProviderShared.subtractTokens(usage.prompt_tokens, cached)
return new Usage({
inputTokens: usage.prompt_tokens,
outputTokens: usage.completion_tokens,
nonCachedInputTokens: nonCached,
cacheReadInputTokens: cached,
reasoningTokens: reasoning,
totalTokens: ProviderShared.totalTokens(usage.prompt_tokens, usage.completion_tokens, usage.total_tokens),
providerMetadata: { openai: usage },
})
}
const step = (state: ParserState, event: OpenAIChatEvent) =>
Effect.gen(function* () {
const events: LLMEvent[] = []
const usage = mapUsage(event.usage) ?? state.usage
const choice = event.choices[0]
const finishReason = choice?.finish_reason ? mapFinishReason(choice.finish_reason) : state.finishReason
const delta = choice?.delta
const toolDeltas = delta?.tool_calls ?? []
let tools = state.tools
let lifecycle = state.lifecycle
if (delta?.reasoning_content)
lifecycle = Lifecycle.reasoningDelta(lifecycle, events, "reasoning-0", delta.reasoning_content)
if (delta?.content) lifecycle = Lifecycle.textDelta(lifecycle, events, "text-0", delta.content)
for (const tool of toolDeltas) {
const result = ToolStream.appendOrStart(
ADAPTER,
tools,
tool.index,
{ id: tool.id ?? undefined, name: tool.function?.name ?? undefined, text: tool.function?.arguments ?? "" },
"OpenAI Chat tool call delta is missing id or name",
)
if (ToolStream.isError(result)) return yield* result
tools = result.tools
if (result.events.length) lifecycle = Lifecycle.stepStart(lifecycle, events)
events.push(...result.events)
}
// Finalize accumulated tool inputs eagerly when finish_reason arrives so
// JSON parse failures fail the stream at the boundary rather than at halt.
const finished =
finishReason !== undefined && state.finishReason === undefined && Object.keys(tools).length > 0
? yield* ToolStream.finishAll(ADAPTER, tools)
: undefined
return [
{
tools: finished?.tools ?? tools,
toolCallEvents: finished?.events ?? state.toolCallEvents,
usage,
finishReason,
lifecycle,
},
events,
] as const
})
const finishEvents = (state: ParserState): ReadonlyArray<LLMEvent> => {
const events: LLMEvent[] = []
const hasToolCalls = state.toolCallEvents.length > 0
const reason = state.finishReason === "stop" && hasToolCalls ? "tool-calls" : state.finishReason
const lifecycle = state.toolCallEvents.length ? Lifecycle.stepStart(state.lifecycle, events) : state.lifecycle
events.push(...state.toolCallEvents)
if (reason) Lifecycle.finish(lifecycle, events, { reason, usage: state.usage })
return events
}
// =============================================================================
// Protocol And OpenAI Route
// =============================================================================
/**
* The OpenAI Chat protocol — request body construction, body schema, and the
* streaming-event state machine. Reused by every route that speaks OpenAI Chat
* over HTTP+SSE: native OpenAI, DeepSeek, TogetherAI, Cerebras, Baseten,
* Fireworks, DeepInfra, and (once added) Azure OpenAI Chat.
*/
export const protocol = Protocol.make({
id: ADAPTER,
body: {
schema: OpenAIChatBody,
from: fromRequest,
},
stream: {
event: Protocol.jsonEvent(OpenAIChatEvent),
initial: () => ({ tools: ToolStream.empty<number>(), toolCallEvents: [], lifecycle: Lifecycle.initial() }),
step,
onHalt: finishEvents,
},
})
export const httpTransport = HttpTransport.sseJson.with<OpenAIChatBody>()
export const route = Route.make({
id: ADAPTER,
provider: "openai",
protocol,
endpoint: Endpoint.path(PATH, { baseURL: DEFAULT_BASE_URL }),
auth: Auth.none,
transport: httpTransport,
})
export * as OpenAIChat from "./openai-chat"

24
packages/llm/src/protocols/openai-compatible-chat.ts Normal file
View File

@@ -0,0 +1,24 @@
import { Route, type RouteRoutedModelInput } from "../route/client"
import { Endpoint } from "../route/endpoint"
import { Framing } from "../route/framing"
import * as OpenAIChat from "./openai-chat"
const ADAPTER = "openai-compatible-chat"
export type OpenAICompatibleChatModelInput = RouteRoutedModelInput
/**
* Route for non-OpenAI providers that expose an OpenAI Chat-compatible
* `/chat/completions` endpoint. Reuses `OpenAIChat.protocol` end-to-end and
* overrides only the route id so providers can be resolved per-family without
* colliding with native OpenAI. Provider helpers configure the route endpoint
* before model selection.
*/
export const route = Route.make({
id: ADAPTER,
protocol: OpenAIChat.protocol,
endpoint: Endpoint.path("/chat/completions"),
framing: Framing.sse,
})
export * as OpenAICompatibleChat from "./openai-compatible-chat"

1004
packages/llm/src/protocols/openai-responses.ts Normal file
View File

File diff suppressed because it is too large Load Diff

349
packages/llm/src/protocols/shared.ts Normal file
View File

@@ -0,0 +1,349 @@
import { Buffer } from "node:buffer"
import { Effect, JsonSchema, Schema, Stream } from "effect"
import * as Sse from "effect/unstable/encoding/Sse"
import { Headers, HttpClientRequest } from "effect/unstable/http"
import {
InvalidProviderOutputReason,
InvalidRequestReason,
LLMError,
type ContentPart,
type LLMRequest,
type MediaPart,
type ToolFileContent,
type TextPart,
type ToolResultPart,
} from "../schema"
import { isRecord } from "../utils/record"
export { isRecord }
export const Json = Schema.fromJsonString(Schema.Unknown)
export const decodeJson = Schema.decodeUnknownSync(Json)
export const encodeJson = Schema.encodeSync(Json)
export const JsonObject = Schema.Record(Schema.String, Schema.Unknown)
export const optionalArray = <const S extends Schema.Top>(schema: S) => Schema.optional(Schema.Array(schema))
export const optionalNull = <const S extends Schema.Top>(schema: S) => Schema.optional(Schema.NullOr(schema))
/** OpenAI function schemas require one flat object at the top level. */
export const openAiToolInputSchema = (schema: JsonSchema.JsonSchema): JsonSchema.JsonSchema => {
const variants = Array.isArray(schema.anyOf) ? schema.anyOf.filter(isRecord) : []
const flattened =
variants.length === 0
? { ...schema, type: "object" }
: {
...Object.fromEntries(Object.entries(schema).filter(([key]) => key !== "anyOf")),
type: "object",
properties: variants.reduce(
(properties, variant) => ({ ...(isRecord(variant.properties) ? variant.properties : {}), ...properties }),
{},
),
additionalProperties: false,
}
const normalized = removeNullSchemas(flattened)
return isRecord(normalized) ? normalized : { type: "object" }
}
const removeNullSchemas = (value: unknown): unknown => {
if (Array.isArray(value)) return value.map(removeNullSchemas)
if (!isRecord(value)) return value
const fields = Object.fromEntries(
Object.entries(value)
.filter(([key]) => key !== "anyOf")
.map(([key, field]) => [key, removeNullSchemas(field)]),
)
if (!Array.isArray(value.anyOf)) return fields
const variants = value.anyOf.filter((variant) => !isRecord(variant) || variant.type !== "null").map(removeNullSchemas)
if (variants.length === 1 && isRecord(variants[0])) return { ...fields, ...variants[0] }
return { ...fields, anyOf: variants }
}
/**
* Streaming tool-call accumulator. Adapters that build a tool call across
* multiple `tool-input-delta` chunks store the partial JSON input string here
* and finalize it with `parseToolInput` once the call completes.
*/
export interface ToolAccumulator {
readonly id: string
readonly name: string
readonly input: string
}
/**
* `Usage.totalTokens` policy shared by every route. Honors a provider-
* supplied total; otherwise falls back to `inputTokens + outputTokens` only
* when at least one is defined. Returns `undefined` when neither input nor
* output is known so routes don't publish a misleading `0`.
*
* Under the additive `LLM.Usage` contract, `inputTokens` and `outputTokens`
* are the non-cached input and visible output only. The provider-supplied
* `total` is the source of truth when present; the computed fallback
* under-counts cache and reasoning by design and exists mainly so
* Anthropic-style providers (which don't surface a total) still get a
* sensible aggregate on the input + output axes.
*/
export const totalTokens = (
inputTokens: number | undefined,
outputTokens: number | undefined,
total: number | undefined,
) => {
if (total !== undefined) return total
if (inputTokens === undefined && outputTokens === undefined) return undefined
return (inputTokens ?? 0) + (outputTokens ?? 0)
}
/**
* Subtract `subtrahend` from `total`, clamping to zero if the provider
* reports a non-sensical breakdown (e.g. `cached_tokens > prompt_tokens`).
* Used by protocol mappers when deriving a non-overlapping breakdown field
* from a provider's inclusive total — `nonCachedInputTokens` from
* `inputTokens - cacheReadInputTokens - cacheWriteInputTokens`.
*
* If `total` is `undefined`, returns `undefined` (we don't fabricate
* counts). If `subtrahend` is `undefined`, returns `total` unchanged. The
* provider-native breakdown stays available on `Usage.native` for debugging.
*/
export const subtractTokens = (total: number | undefined, subtrahend: number | undefined): number | undefined => {
if (total === undefined) return undefined
if (subtrahend === undefined) return total
return Math.max(0, total - subtrahend)
}
/**
* Sum a list of optional token counts, returning `undefined` only when
* every value is `undefined` (so we don't fabricate a `0`). Used by
* protocol mappers to derive the inclusive `inputTokens` total from a
* provider that natively reports a non-overlapping breakdown
* (e.g. Anthropic, whose `input_tokens` is already non-cached only).
*/
export const sumTokens = (...values: ReadonlyArray<number | undefined>): number | undefined => {
if (values.every((value) => value === undefined)) return undefined
return values.reduce((acc: number, value) => acc + (value ?? 0), 0)
}
export const eventError = (route: string, message: string, raw?: string) =>
new LLMError({
module: "ProviderShared",
method: "stream",
reason: new InvalidProviderOutputReason({ route, message, raw }),
})
export const parseJson = (route: string, input: string, message: string) =>
Effect.try({
try: () => decodeJson(input),
catch: () => eventError(route, message, input),
})
/**
* Join the `text` field of a list of parts with newlines. Used by routes
* that flatten system / message content arrays into a single provider string
* (OpenAI Chat `system` content, OpenAI Responses `system` content, Gemini
* `systemInstruction.parts[].text`).
*/
export const joinText = (parts: ReadonlyArray<{ readonly text: string }>) => parts.map((part) => part.text).join("\n")
const escapeSystemUpdateText = (text: string) =>
text.replaceAll("&", "&amp;").replaceAll("<", "&lt;").replaceAll(">", "&gt;")
/**
* Stable fallback representation for chronological `Message.system(...)`
* updates on routes that do not support that privileged role natively. The
* wrapper remains visibly lower-authority user text, preserves the original
* temporal position, and XML-escapes content so it cannot close the wrapper.
*/
export const wrapSystemUpdate = (parts: ReadonlyArray<{ readonly text: string }>) =>
`<system-update>\n${escapeSystemUpdateText(joinText(parts))}\n</system-update>`
/**
* Chronological system updates deliberately accept text only. Do not insert
* raw retrieved, tool, or web content into privileged updates: keep untrusted
* data in ordinary user/tool messages instead.
*/
export const systemUpdateText = Effect.fn("ProviderShared.systemUpdateText")(function* (
route: string,
message: LLMRequest["messages"][number],
) {
const content: TextPart[] = []
for (const part of message.content) {
if (!supportsContent(part, ["text"])) return yield* unsupportedContent(route, "system", ["text"])
content.push(part)
}
return content
})
/** Lower an unsupported privileged update into visible, in-order user text. */
export const wrappedSystemUpdate = Effect.fn("ProviderShared.wrappedSystemUpdate")(function* (
route: string,
message: LLMRequest["messages"][number],
) {
const content = yield* systemUpdateText(route, message)
return { type: "text" as const, text: wrapSystemUpdate(content), cache: content.at(-1)?.cache }
})
/**
* Parse the streamed JSON input of a tool call. Treats an empty string as
* `"{}"` — providers occasionally finish a tool call without ever emitting
* input deltas (e.g. zero-arg tools). The error message is uniform across
* routes: `Invalid JSON input for <route> tool call <name>`.
*/
export const parseToolInput = (route: string, name: string, raw: string) =>
parseJson(route, raw || "{}", `Invalid JSON input for ${route} tool call ${name}`)
export const IMAGE_MIMES = ["image/png", "image/jpeg", "image/gif", "image/webp"] as const
export const MAX_MEDIA_ENCODED_BYTES = 8 * 1024 * 1024
export const MAX_MEDIA_DECODED_BYTES = 6 * 1024 * 1024
const base64Pattern = /^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/
export interface ValidatedMedia {
readonly mime: string
readonly base64: string
readonly dataUrl: string
readonly bytes: Uint8Array
}
export const validateMedia = Effect.fn("ProviderShared.validateMedia")(function* (
route: string,
part: MediaPart,
supportedMimes: ReadonlySet<string>,
) {
const mime = part.mediaType.toLowerCase()
if (!supportedMimes.has(mime)) return yield* invalidRequest(`${route} does not support media type ${part.mediaType}`)
let base64: string
if (typeof part.data !== "string") {
if (part.data.byteLength > MAX_MEDIA_DECODED_BYTES)
return yield* invalidRequest(`${route} media exceeds the ${MAX_MEDIA_DECODED_BYTES} byte decoded limit`)
base64 = Buffer.from(part.data).toString("base64")
} else if (part.data.startsWith("data:")) {
const match = /^data:([^;,]+);base64,([A-Za-z0-9+/]*={0,2})$/s.exec(part.data)
if (!match) return yield* invalidRequest(`${route} media data URL must contain valid base64`)
if (match[1]!.toLowerCase() !== mime)
return yield* invalidRequest(`${route} media type ${part.mediaType} does not match data URL type ${match[1]}`)
base64 = match[2]!
} else {
base64 = part.data
}
if (Buffer.byteLength(base64, "utf8") > MAX_MEDIA_ENCODED_BYTES)
return yield* invalidRequest(`${route} media exceeds the ${MAX_MEDIA_ENCODED_BYTES} byte encoded limit`)
if (!base64 || base64.length % 4 !== 0 || !base64Pattern.test(base64))
return yield* invalidRequest(`${route} media must contain valid base64`)
const bytes = Buffer.from(base64, "base64")
if (bytes.byteLength > MAX_MEDIA_DECODED_BYTES)
return yield* invalidRequest(`${route} media exceeds the ${MAX_MEDIA_DECODED_BYTES} byte decoded limit`)
if (bytes.toString("base64") !== base64) return yield* invalidRequest(`${route} media must contain canonical base64`)
return { mime, base64, dataUrl: `data:${mime};base64,${base64}`, bytes } satisfies ValidatedMedia
})
export const validateToolFile = (route: string, part: ToolFileContent, supportedMimes: ReadonlySet<string>) =>
validateMedia(route, { type: "media", mediaType: part.mime, data: part.uri, filename: part.name }, supportedMimes)
export const trimBaseUrl = (value: string) => value.replace(/\/+$/, "")
export const toolResultText = (part: ToolResultPart) => {
if (part.result.type === "text" || part.result.type === "error") return String(part.result.value)
if (part.result.type === "content") return encodeJson(part.result.value)
return encodeJson(part.result.value)
}
export const errorText = (error: unknown) => {
if (error instanceof Error) return error.message
if (typeof error === "string") return error
if (typeof error === "number" || typeof error === "boolean" || typeof error === "bigint") return String(error)
if (error === null) return "null"
if (error === undefined) return "undefined"
return "Unknown stream error"
}
/**
* `framing` step for Server-Sent Events. Decodes UTF-8, runs the SSE channel
* decoder, and drops empty / `[DONE]` keep-alive events so the downstream
* `decodeChunk` sees one JSON string per element. The SSE channel emits a
* `Retry` control event on its error channel; we drop it here (we don't
* implement client-driven retries) so the public error channel stays
* `LLMError`.
*/
export const sseFraming = (bytes: Stream.Stream<Uint8Array, LLMError>): Stream.Stream<string, LLMError> =>
bytes.pipe(
Stream.decodeText(),
Stream.pipeThroughChannel(Sse.decode()),
Stream.catchTag("Retry", () => Stream.empty),
Stream.filter((event) => event.data.length > 0 && event.data !== "[DONE]"),
Stream.map((event) => event.data),
)
/**
* Canonical invalid-request constructor. Lift one-line `const invalid =
* (message) => invalidRequest(message)` aliases out of every
* route so the error constructor lives in one place. If we ever extend
* `InvalidRequestReason` with route context or trace metadata, the change
* lands here.
*/
export const invalidRequest = (message: string) =>
new LLMError({
module: "ProviderShared",
method: "request",
reason: new InvalidRequestReason({ message }),
})
export const matchToolChoice = <Auto, None, Required, Tool>(
route: string,
toolChoice: NonNullable<LLMRequest["toolChoice"]>,
cases: {
readonly auto: () => Auto
readonly none: () => None
readonly required: () => Required
readonly tool: (name: string) => Tool
},
) =>
Effect.gen(function* () {
if (toolChoice.type === "auto") return cases.auto()
if (toolChoice.type === "none") return cases.none()
if (toolChoice.type === "required") return cases.required()
if (!toolChoice.name) return yield* invalidRequest(`${route} tool choice requires a tool name`)
return cases.tool(toolChoice.name)
})
type ContentType = ContentPart["type"]
const formatContentTypes = (types: ReadonlyArray<ContentType>) => {
if (types.length <= 1) return types[0] ?? ""
if (types.length === 2) return `${types[0]} and ${types[1]}`
return `${types.slice(0, -1).join(", ")}, and ${types.at(-1)}`
}
export const supportsContent = <const Type extends ContentType>(
part: ContentPart,
types: ReadonlyArray<Type>,
): part is Extract<ContentPart, { readonly type: Type }> => (types as ReadonlyArray<ContentType>).includes(part.type)
export const unsupportedContent = (
route: string,
role: LLMRequest["messages"][number]["role"],
types: ReadonlyArray<ContentType>,
) => invalidRequest(`${route} ${role} messages only support ${formatContentTypes(types)} content for now`)
/**
* Build a `validate` step from a Schema decoder. Replaces the per-route
* lambda body `(payload) => decode(payload).pipe(Effect.mapError((e) =>
* invalid(e.message)))`. Any decode error is translated into
* `LLMError` carrying the original parse-error message.
*/
export const validateWith =
<A, I, E extends { readonly message: string }>(decode: (input: I) => Effect.Effect<A, E>) =>
(payload: I) =>
decode(payload).pipe(Effect.mapError((error) => invalidRequest(error.message)))
/**
* Build an HTTP POST with a JSON body. Sets `content-type: application/json`
* automatically after caller-supplied headers so routes cannot accidentally
* send JSON with a stale content type. The body is passed pre-encoded so
* routes can choose between
* `Schema.encodeSync(payload)` and `ProviderShared.encodeJson(payload)`.
*/
export const jsonPost = (input: { readonly url: string; readonly body: string; readonly headers?: Headers.Input }) =>
HttpClientRequest.post(input.url).pipe(
HttpClientRequest.setHeaders(Headers.set(Headers.fromInput(input.headers), "content-type", "application/json")),
HttpClientRequest.bodyText(input.body, "application/json"),
)
export * as ProviderShared from "./shared"

70
packages/llm/src/protocols/utils/bedrock-auth.ts Normal file
View File

@@ -0,0 +1,70 @@
import { AwsV4Signer } from "aws4fetch"
import { Effect } from "effect"
import { Headers } from "effect/unstable/http"
import { Auth, type AuthInput } from "../../route/auth"
import { ProviderShared } from "../shared"
/**
* AWS credentials for SigV4 signing. Bedrock also supports Bearer API key auth,
* which provider facades configure as route auth instead of SigV4. STS-vended
* credentials should be refreshed by the consumer (rebuild the model) before
* they expire; the route does not refresh.
*/
export interface Credentials {
readonly region: string
readonly accessKeyId: string
readonly secretAccessKey: string
readonly sessionToken?: string
}
const signRequest = (input: {
readonly url: string
readonly body: string
readonly headers: Headers.Headers
readonly credentials: Credentials
}) =>
Effect.tryPromise({
try: async () => {
const signed = await new AwsV4Signer({
url: input.url,
method: "POST",
headers: Object.entries(input.headers),
body: input.body,
region: input.credentials.region,
accessKeyId: input.credentials.accessKeyId,
secretAccessKey: input.credentials.secretAccessKey,
sessionToken: input.credentials.sessionToken,
service: "bedrock",
}).sign()
return Object.fromEntries(signed.headers.entries())
},
catch: (error) =>
ProviderShared.invalidRequest(
`Bedrock Converse SigV4 signing failed: ${error instanceof Error ? error.message : String(error)}`,
),
})
/** Sign the exact JSON bytes with SigV4 using credentials configured on the route. */
export const sigV4 = (credentials: Credentials | undefined) =>
Auth.custom((input: AuthInput) => {
return Effect.gen(function* () {
if (!credentials) {
return yield* ProviderShared.invalidRequest(
"Bedrock Converse requires either route bearer auth or AWS credentials configured on the route",
)
}
const headersForSigning = Headers.set(input.headers, "content-type", "application/json")
const signed = yield* signRequest({
url: input.url,
body: input.body,
headers: headersForSigning,
credentials,
})
return Headers.setAll(headersForSigning, signed)
})
})
/** Bedrock route auth defaults to SigV4 and expects credentials from route configuration. */
export const auth = sigV4(undefined)
export * as BedrockAuth from "./bedrock-auth"

37
packages/llm/src/protocols/utils/bedrock-cache.ts Normal file
View File

@@ -0,0 +1,37 @@
import { Schema } from "effect"
import type { CacheHint } from "../../schema"
import { newBreakpoints, ttlBucket, type Breakpoints } from "./cache"
// Bedrock cache markers are positional: emit a `cachePoint` block immediately
// after the content the caller wants treated as a cacheable prefix. Bedrock
// accepts optional `ttl: "5m" | "1h"` on cachePoint, mirroring Anthropic.
export const CachePointBlock = Schema.Struct({
cachePoint: Schema.Struct({
type: Schema.tag("default"),
ttl: Schema.optional(Schema.Literals(["5m", "1h"])),
}),
})
export type CachePointBlock = Schema.Schema.Type<typeof CachePointBlock>
// Bedrock-Claude enforces the same 4-breakpoint cap as the Anthropic Messages
// API. Callers pass a shared counter through every `block()` call site so the
// budget is respected across `system`, `messages`, and `tools`.
export const BEDROCK_BREAKPOINT_CAP = 4
export type { Breakpoints } from "./cache"
export const breakpoints = () => newBreakpoints(BEDROCK_BREAKPOINT_CAP)
const DEFAULT_5M: CachePointBlock = { cachePoint: { type: "default" } }
const DEFAULT_1H: CachePointBlock = { cachePoint: { type: "default", ttl: "1h" } }
export const block = (breakpoints: Breakpoints, cache: CacheHint | undefined): CachePointBlock | undefined => {
if (cache?.type !== "ephemeral" && cache?.type !== "persistent") return undefined
if (breakpoints.remaining <= 0) {
breakpoints.dropped += 1
return undefined
}
breakpoints.remaining -= 1
return ttlBucket(cache.ttlSeconds) === "1h" ? DEFAULT_1H : DEFAULT_5M
}
export * as BedrockCache from "./bedrock-cache"

90
packages/llm/src/protocols/utils/bedrock-media.ts Normal file
View File

@@ -0,0 +1,90 @@
import { Effect, Schema } from "effect"
import type { MediaPart } from "../../schema"
import { ProviderShared } from "../shared"
// Bedrock Converse accepts image `format` as the file extension and
// `source.bytes` as base64 in the JSON wire format.
export const ImageFormat = Schema.Literals(["png", "jpeg", "gif", "webp"])
export type ImageFormat = Schema.Schema.Type<typeof ImageFormat>
export const ImageBlock = Schema.Struct({
image: Schema.Struct({
format: ImageFormat,
source: Schema.Struct({ bytes: Schema.String }),
}),
})
export type ImageBlock = Schema.Schema.Type<typeof ImageBlock>
// Bedrock document blocks require a user-facing name so the model can refer to
// the uploaded document.
export const DocumentFormat = Schema.Literals(["pdf", "csv", "doc", "docx", "xls", "xlsx", "html", "txt", "md"])
export type DocumentFormat = Schema.Schema.Type<typeof DocumentFormat>
export const DocumentBlock = Schema.Struct({
document: Schema.Struct({
format: DocumentFormat,
name: Schema.String,
source: Schema.Struct({ bytes: Schema.String }),
}),
})
export type DocumentBlock = Schema.Schema.Type<typeof DocumentBlock>
const IMAGE_FORMATS = {
"image/png": "png",
"image/jpeg": "jpeg",
"image/jpg": "jpeg",
"image/gif": "gif",
"image/webp": "webp",
} as const satisfies Record<string, ImageFormat>
const DOCUMENT_FORMATS = {
"application/pdf": "pdf",
"text/csv": "csv",
"application/msword": "doc",
"application/vnd.openxmlformats-officedocument.wordprocessingml.document": "docx",
"application/vnd.ms-excel": "xls",
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet": "xlsx",
"text/html": "html",
"text/plain": "txt",
"text/markdown": "md",
} as const satisfies Record<string, DocumentFormat>
const documentBlock = (part: MediaPart, format: DocumentFormat, bytes: string): DocumentBlock => ({
document: {
format,
name: part.filename ?? `document.${format}`,
source: { bytes },
},
})
// Route by MIME. Known image/document formats lower into a typed block; anything
// else fails with a clear error instead of silently degrading to a malformed
// document block. Image MIME types not in `IMAGE_FORMATS` (e.g. `image/svg+xml`)
// get an image-specific error so the caller knows it's a format-support issue,
// not a kind-detection issue.
export const lower = Effect.fn("BedrockMedia.lower")(function* (part: MediaPart) {
const mime = part.mediaType.toLowerCase()
const imageFormat = IMAGE_FORMATS[mime as keyof typeof IMAGE_FORMATS]
if (imageFormat) {
const media = yield* ProviderShared.validateMedia(
"Bedrock Converse",
part,
new Set<string>(Object.keys(IMAGE_FORMATS)),
)
return { image: { format: imageFormat, source: { bytes: media.base64 } } } satisfies ImageBlock
}
if (mime.startsWith("image/"))
return yield* ProviderShared.invalidRequest(`Bedrock Converse does not support image media type ${part.mediaType}`)
const documentFormat = DOCUMENT_FORMATS[mime as keyof typeof DOCUMENT_FORMATS]
if (documentFormat) {
const media = yield* ProviderShared.validateMedia(
"Bedrock Converse",
part,
new Set<string>(Object.keys(DOCUMENT_FORMATS)),
)
return documentBlock(part, documentFormat, media.base64)
}
return yield* ProviderShared.invalidRequest(`Bedrock Converse does not support media type ${part.mediaType}`)
})
export * as BedrockMedia from "./bedrock-media"

16
packages/llm/src/protocols/utils/cache.ts Normal file
View File

@@ -0,0 +1,16 @@
// Shared helpers for provider cache-marker lowering. Anthropic and Bedrock
// both enforce a 4-breakpoint cap per request and accept the same `5m`/`1h`
// TTL buckets, so the counter and TTL mapping live here.
export interface Breakpoints {
remaining: number
dropped: number
}
export const newBreakpoints = (cap: number): Breakpoints => ({ remaining: cap, dropped: 0 })
// Returns `"1h"` for any `ttlSeconds >= 3600`, otherwise `undefined` (the
// provider default 5m). Anthropic & Bedrock both treat anything shorter than
// an hour as 5m.
export const ttlBucket = (ttlSeconds: number | undefined): "1h" | undefined =>
ttlSeconds !== undefined && ttlSeconds >= 3600 ? "1h" : undefined

101
packages/llm/src/protocols/utils/gemini-tool-schema.ts Normal file
View File

@@ -0,0 +1,101 @@
import { ProviderShared } from "../shared"
// Gemini accepts a JSON Schema-like dialect for tool parameters, but rejects a
// handful of common JSON Schema shapes. Keep this projection isolated so the
// Gemini protocol file still reads like the other protocol modules.
const SCHEMA_INTENT_KEYS = [
"type",
"properties",
"items",
"prefixItems",
"enum",
"const",
"$ref",
"additionalProperties",
"patternProperties",
"required",
"not",
"if",
"then",
"else",
]
const isRecord = ProviderShared.isRecord
const hasCombiner = (schema: unknown) =>
isRecord(schema) && (Array.isArray(schema.anyOf) || Array.isArray(schema.oneOf) || Array.isArray(schema.allOf))
const hasSchemaIntent = (schema: unknown) =>
isRecord(schema) && (hasCombiner(schema) || SCHEMA_INTENT_KEYS.some((key) => key in schema))
const sanitizeNode = (schema: unknown): unknown => {
if (!isRecord(schema)) return Array.isArray(schema) ? schema.map(sanitizeNode) : schema
const result: Record<string, unknown> = Object.fromEntries(
Object.entries(schema).map(([key, value]) => [
key,
key === "enum" && Array.isArray(value) ? value.map(String) : sanitizeNode(value),
]),
)
if (Array.isArray(result.enum) && (result.type === "integer" || result.type === "number")) result.type = "string"
const properties = result.properties
if (result.type === "object" && isRecord(properties) && Array.isArray(result.required)) {
result.required = result.required.filter((field) => typeof field === "string" && field in properties)
}
if (result.type === "array" && !hasCombiner(result)) {
result.items = result.items ?? {}
if (isRecord(result.items) && !hasSchemaIntent(result.items)) result.items = { ...result.items, type: "string" }
}
if (typeof result.type === "string" && result.type !== "object" && !hasCombiner(result)) {
delete result.properties
delete result.required
}
return result
}
const emptyObjectSchema = (schema: Record<string, unknown>) =>
schema.type === "object" &&
(!isRecord(schema.properties) || Object.keys(schema.properties).length === 0) &&
!schema.additionalProperties
const projectNode = (schema: unknown): Record<string, unknown> | undefined => {
if (!isRecord(schema)) return undefined
if (emptyObjectSchema(schema)) return undefined
return Object.fromEntries(
[
["description", schema.description],
["required", schema.required],
["format", schema.format],
["type", Array.isArray(schema.type) ? schema.type.filter((type) => type !== "null")[0] : schema.type],
["nullable", Array.isArray(schema.type) && schema.type.includes("null") ? true : undefined],
["enum", schema.const !== undefined ? [schema.const] : schema.enum],
[
"properties",
isRecord(schema.properties)
? Object.fromEntries(Object.entries(schema.properties).map(([key, value]) => [key, projectNode(value)]))
: undefined,
],
[
"items",
Array.isArray(schema.items)
? schema.items.map(projectNode)
: schema.items === undefined
? undefined
: projectNode(schema.items),
],
["allOf", Array.isArray(schema.allOf) ? schema.allOf.map(projectNode) : undefined],
["anyOf", Array.isArray(schema.anyOf) ? schema.anyOf.map(projectNode) : undefined],
["oneOf", Array.isArray(schema.oneOf) ? schema.oneOf.map(projectNode) : undefined],
["minLength", schema.minLength],
].filter((entry) => entry[1] !== undefined),
)
}
export const convert = (schema: unknown) => projectNode(sanitizeNode(schema))
export * as GeminiToolSchema from "./gemini-tool-schema"

102
packages/llm/src/protocols/utils/lifecycle.ts Normal file
View File

@@ -0,0 +1,102 @@
import { LLMEvent, type FinishReason, type ProviderMetadata, type Usage } from "../../schema"
export interface State {
readonly stepStarted: boolean
readonly text: ReadonlySet<string>
readonly reasoning: ReadonlySet<string>
}
export const initial = (): State => ({ stepStarted: false, text: new Set(), reasoning: new Set() })
export const stepStart = (state: State, events: LLMEvent[]): State => {
if (state.stepStarted) return state
events.push(LLMEvent.stepStart({ index: 0 }))
return { ...state, stepStarted: true }
}
export const textDelta = (state: State, events: LLMEvent[], id: string, text: string): State => {
const stepped = stepStart(state, events)
if (stepped.text.has(id)) {
events.push(LLMEvent.textDelta({ id, text }))
return stepped
}
events.push(LLMEvent.textStart({ id }), LLMEvent.textDelta({ id, text }))
return { ...stepped, text: new Set([...stepped.text, id]) }
}
export const reasoningStart = (
state: State,
events: LLMEvent[],
id: string,
providerMetadata?: ProviderMetadata,
): State => {
if (state.reasoning.has(id)) return state
const stepped = stepStart(state, events)
events.push(LLMEvent.reasoningStart({ id, providerMetadata }))
return { ...stepped, reasoning: new Set([...stepped.reasoning, id]) }
}
export const reasoningDelta = (
state: State,
events: LLMEvent[],
id: string,
text: string,
providerMetadata?: ProviderMetadata,
): State => {
const started = reasoningStart(state, events, id, providerMetadata)
events.push(LLMEvent.reasoningDelta({ id, text }))
return started
}
export const reasoningEnd = (
state: State,
events: LLMEvent[],
id: string,
providerMetadata?: ProviderMetadata,
): State => {
if (!state.reasoning.has(id)) return state
const stepped = stepStart(state, events)
events.push(LLMEvent.reasoningEnd({ id, providerMetadata }))
const reasoning = new Set(stepped.reasoning)
reasoning.delete(id)
return { ...stepped, reasoning }
}
export const textEnd = (state: State, events: LLMEvent[], id: string, providerMetadata?: ProviderMetadata): State => {
if (!state.text.has(id)) return state
const stepped = stepStart(state, events)
events.push(LLMEvent.textEnd({ id, providerMetadata }))
const text = new Set(stepped.text)
text.delete(id)
return { ...stepped, text }
}
const closeOpenBlocks = (state: State, events: LLMEvent[]): State => {
for (const id of state.reasoning) events.push(LLMEvent.reasoningEnd({ id }))
for (const id of state.text) events.push(LLMEvent.textEnd({ id }))
return { ...state, text: new Set(), reasoning: new Set() }
}
export const finish = (
state: State,
events: LLMEvent[],
input: {
readonly reason: FinishReason
readonly usage?: Usage
readonly providerMetadata?: ProviderMetadata
},
): State => {
const stepped = closeOpenBlocks(stepStart(state, events), events)
events.push(
LLMEvent.stepFinish({
index: 0,
reason: input.reason,
usage: input.usage,
providerMetadata: input.providerMetadata,
}),
LLMEvent.finish(input),
)
return { ...stepped, stepStarted: false }
}
export * as Lifecycle from "./lifecycle"

93
packages/llm/src/protocols/utils/openai-options.ts Normal file
View File

@@ -0,0 +1,93 @@
import { Schema } from "effect"
import type { LLMRequest, ReasoningEffort, TextVerbosity as TextVerbosityValue } from "../../schema"
import { ReasoningEfforts, TextVerbosity } from "../../schema"
export const OpenAIReasoningEfforts = ReasoningEfforts.filter(
(effort): effort is Exclude<ReasoningEffort, "max"> => effort !== "max",
)
export type OpenAIReasoningEffort = (typeof OpenAIReasoningEfforts)[number]
// Mirrors OpenAI's `ResponseIncludable` union from the official SDK. Keep this
// in lockstep with `openai-node/src/resources/responses/responses.ts`.
export const OpenAIResponseIncludables = [
"file_search_call.results",
"web_search_call.results",
"web_search_call.action.sources",
"message.input_image.image_url",
"computer_call_output.output.image_url",
"code_interpreter_call.outputs",
"reasoning.encrypted_content",
"message.output_text.logprobs",
] as const
export type OpenAIResponseIncludable = (typeof OpenAIResponseIncludables)[number]
export const OpenAIServiceTiers = ["auto", "default", "flex", "priority"] as const
export type OpenAIServiceTier = (typeof OpenAIServiceTiers)[number]
const REASONING_EFFORTS = new Set<string>(ReasoningEfforts)
const OPENAI_REASONING_EFFORTS = new Set<string>(OpenAIReasoningEfforts)
const TEXT_VERBOSITY = new Set<string>(["low", "medium", "high"])
const INCLUDABLES = new Set<string>(OpenAIResponseIncludables)
const SERVICE_TIERS = new Set<string>(OpenAIServiceTiers)
export const OpenAIReasoningEffort = Schema.Literals(OpenAIReasoningEfforts)
export const OpenAITextVerbosity = TextVerbosity
export const OpenAIResponseIncludable = Schema.Literals(OpenAIResponseIncludables)
export const OpenAIServiceTier = Schema.Literals(OpenAIServiceTiers)
const isAnyReasoningEffort = (effort: unknown): effort is ReasoningEffort =>
typeof effort === "string" && REASONING_EFFORTS.has(effort)
export const isReasoningEffort = (effort: unknown): effort is OpenAIReasoningEffort =>
typeof effort === "string" && OPENAI_REASONING_EFFORTS.has(effort)
const isTextVerbosity = (value: unknown): value is TextVerbosityValue =>
typeof value === "string" && TEXT_VERBOSITY.has(value)
const options = (request: LLMRequest) => request.providerOptions?.openai
export const store = (request: LLMRequest): boolean | undefined => {
const value = options(request)?.store
return typeof value === "boolean" ? value : undefined
}
export const reasoningEffort = (request: LLMRequest): ReasoningEffort | undefined => {
const value = options(request)?.reasoningEffort
return isAnyReasoningEffort(value) ? value : undefined
}
export const reasoningSummary = (request: LLMRequest): "auto" | undefined =>
options(request)?.reasoningSummary === "auto" ? "auto" : undefined
// Resolve the OpenAI Responses `include` field. Filters out unknown
// includable values defensively so a typo in upstream config drops the
// invalid entry instead of poisoning the wire body. An empty array (either
// passed directly or produced by filtering) is treated as "no include" and
// returns undefined so the request body omits the field entirely.
export const include = (request: LLMRequest): ReadonlyArray<OpenAIResponseIncludable> | undefined => {
const value = options(request)?.include
if (!Array.isArray(value)) return undefined
const filtered = value.filter((entry): entry is OpenAIResponseIncludable => INCLUDABLES.has(entry))
return filtered.length > 0 ? filtered : undefined
}
export const promptCacheKey = (request: LLMRequest) => {
const value = options(request)?.promptCacheKey
return typeof value === "string" ? value : undefined
}
export const textVerbosity = (request: LLMRequest) => {
const value = options(request)?.textVerbosity
return isTextVerbosity(value) ? value : undefined
}
export const serviceTier = (request: LLMRequest) => {
const value = options(request)?.serviceTier
return typeof value === "string" && SERVICE_TIERS.has(value) ? (value as OpenAIServiceTier) : undefined
}
export const instructions = (request: LLMRequest) => {
const value = options(request)?.instructions
return typeof value === "string" ? value : undefined
}
export * as OpenAIOptions from "./openai-options"

218
packages/llm/src/protocols/utils/tool-stream.ts Normal file
View File

@@ -0,0 +1,218 @@
import { Effect } from "effect"
import { LLMError, LLMEvent, type ProviderMetadata, type ToolCall } from "../../schema"
import { eventError, parseToolInput, type ToolAccumulator } from "../shared"
type StreamKey = string | number
/**
* One pending streamed tool call. Providers emit the tool identity and JSON
* argument text across separate chunks; `input` is the raw JSON string collected
* so far, not the parsed object.
*/
export interface PendingTool extends ToolAccumulator {
readonly providerExecuted?: boolean
readonly providerMetadata?: ProviderMetadata
}
/**
* Sparse parser state keyed by the provider's stream-local tool identifier.
*
* This key is not the final tool-call id (`call_...`). It is the id/index the
* provider uses while streaming a partial call: OpenAI Chat / Anthropic /
* Bedrock use numeric content indexes, while OpenAI Responses uses string
* `item_id`s. The generic keeps each protocol internally consistent.
*/
export type State<K extends StreamKey> = Partial<Record<K, PendingTool>>
/**
* Result of adding argument text to one pending tool call. It returns both the
* next `tools` state and the updated `tool` because parsers often need the
* current id/name immediately. `events` contains lifecycle and delta events
* produced by the append; metadata-only deltas update identity without output.
*/
export interface AppendOutcome<K extends StreamKey> {
readonly tools: State<K>
readonly tool: PendingTool
readonly events: ReadonlyArray<LLMEvent>
}
/** Create empty accumulator state for one provider stream. */
export const empty = <K extends StreamKey>(): State<K> => ({})
const withTool = <K extends StreamKey>(tools: State<K>, key: K, tool: PendingTool): State<K> => {
return { ...tools, [key]: tool }
}
const withoutTool = <K extends StreamKey>(tools: State<K>, key: K): State<K> => {
const next = { ...tools }
delete next[key]
return next
}
const inputStart = (tool: PendingTool) =>
LLMEvent.toolInputStart({
id: tool.id,
name: tool.name,
providerMetadata: tool.providerMetadata,
})
const inputDelta = (tool: PendingTool, text: string) =>
LLMEvent.toolInputDelta({
id: tool.id,
name: tool.name,
text,
})
const toolCall = (route: string, tool: PendingTool, inputOverride?: string) =>
parseToolInput(route, tool.name, inputOverride ?? tool.input).pipe(
Effect.map(
(input): ToolCall =>
LLMEvent.toolCall({
id: tool.id,
name: tool.name,
input,
providerExecuted: tool.providerExecuted ? true : undefined,
providerMetadata: tool.providerMetadata,
}),
),
)
/** Store the updated tool and produce the optional public delta event. */
const appendTool = <K extends StreamKey>(
tools: State<K>,
key: K,
tool: PendingTool,
text: string,
): AppendOutcome<K> => {
const events: LLMEvent[] = []
if (!tools[key]) events.push(inputStart(tool))
if (text.length > 0) events.push(inputDelta(tool, text))
return {
tools: withTool(tools, key, tool),
tool,
events,
}
}
export const isError = <K extends StreamKey>(result: AppendOutcome<K> | LLMError): result is LLMError =>
result instanceof LLMError
/**
* Register a tool call whose start event arrived before any argument deltas.
* Used by Anthropic `content_block_start`, Bedrock `contentBlockStart`, and
* OpenAI Responses `response.output_item.added`.
*/
export const start = <K extends StreamKey>(
tools: State<K>,
key: K,
tool: Omit<PendingTool, "input"> & { readonly input?: string },
) => withTool(tools, key, { ...tool, input: tool.input ?? "" })
/**
* Append a streamed argument delta, starting the tool if this provider encodes
* identity on the first delta instead of a separate start event. OpenAI Chat has
* this shape: `tool_calls[].index` is the stream key, and `id` / `name` may only
* appear on the first delta for that index.
*/
export const appendOrStart = <K extends StreamKey>(
route: string,
tools: State<K>,
key: K,
delta: { readonly id?: string; readonly name?: string; readonly text: string },
missingToolMessage: string,
): AppendOutcome<K> | LLMError => {
const current = tools[key]
const id = delta.id ?? current?.id
const name = delta.name ?? current?.name
if (!id || !name) return eventError(route, missingToolMessage)
const tool = {
id,
name,
input: `${current?.input ?? ""}${delta.text}`,
providerExecuted: current?.providerExecuted,
providerMetadata: current?.providerMetadata,
}
if (current && delta.text.length === 0 && current.id === id && current.name === name)
return { tools, tool: current, events: [] }
return appendTool(tools, key, tool, delta.text)
}
/**
* Append argument text to a tool that must already have been started. This keeps
* protocols honest when their stream grammar promises a start event before any
* argument delta.
*/
export const appendExisting = <K extends StreamKey>(
route: string,
tools: State<K>,
key: K,
text: string,
missingToolMessage: string,
): AppendOutcome<K> | LLMError => {
const current = tools[key]
if (!current) return eventError(route, missingToolMessage)
if (text.length === 0) return { tools, tool: current, events: [] }
return appendTool(tools, key, { ...current, input: `${current.input}${text}` }, text)
}
/**
* Finalize one pending tool call: parse the accumulated raw JSON, remove it
* from state, and return the optional public `tool-call` event. Missing keys are
* a no-op because some providers emit stop events for non-tool content blocks.
*/
export const finish = <K extends StreamKey>(route: string, tools: State<K>, key: K) =>
Effect.gen(function* () {
const tool = tools[key]
if (!tool) return { tools }
return {
tools: withoutTool(tools, key),
events: [
LLMEvent.toolInputEnd({ id: tool.id, name: tool.name, providerMetadata: tool.providerMetadata }),
yield* toolCall(route, tool),
],
}
})
/**
* Finalize one pending tool call with an authoritative final input string.
* OpenAI Responses can send accumulated deltas and then repeat the completed
* arguments on `response.output_item.done`; the final value wins.
*/
export const finishWithInput = <K extends StreamKey>(route: string, tools: State<K>, key: K, input: string) =>
Effect.gen(function* () {
const tool = tools[key]
if (!tool) return { tools }
return {
tools: withoutTool(tools, key),
events: [
LLMEvent.toolInputEnd({ id: tool.id, name: tool.name, providerMetadata: tool.providerMetadata }),
yield* toolCall(route, tool, input),
],
}
})
/**
* Finalize every pending tool call at once. OpenAI Chat has this shape: it does
* not emit per-tool stop events, so all accumulated calls finish when the choice
* receives a terminal `finish_reason`.
*/
export const finishAll = <K extends StreamKey>(route: string, tools: State<K>) =>
Effect.gen(function* () {
const pending = Object.values<PendingTool | undefined>(tools).filter(
(tool): tool is PendingTool => tool !== undefined,
)
return {
tools: empty<K>(),
events: yield* Effect.forEach(pending, (tool) =>
toolCall(route, tool).pipe(
Effect.map((call) => [
LLMEvent.toolInputEnd({ id: tool.id, name: tool.name, providerMetadata: tool.providerMetadata }),
call,
]),
),
).pipe(Effect.map((events) => events.flat())),
}
})
export * as ToolStream from "./tool-stream"

32
packages/llm/src/provider-error.ts Normal file
View File

@@ -0,0 +1,32 @@
import { Schema } from "effect"
import { LLMError, ProviderErrorEvent } from "./schema"
const patterns = [
/prompt is too long/i,
/input is too long for requested model/i,
/exceeds the context window/i,
/input token count.*exceeds the maximum/i,
/maximum prompt length is \d+/i,
/reduce the length of the messages/i,
/maximum context length is \d+ tokens/i,
/exceeds the limit of \d+/i,
/exceeds the available context size/i,
/greater than the context length/i,
/context window exceeds limit/i,
/exceeded model token limit/i,
/context[_ ]length[_ ]exceeded/i,
/request entity too large/i,
/context length is only \d+ tokens/i,
/input length.*exceeds.*context length/i,
/prompt too long; exceeded (?:max )?context length/i,
/too large for model with \d+ maximum context length/i,
/model_context_window_exceeded/i,
]
export const isContextOverflow = (message: string) =>
patterns.some((pattern) => pattern.test(message)) || /^4(00|13)\s*(status code)?\s*\(no body\)/i.test(message)
export const isContextOverflowFailure = (failure: unknown) =>
failure instanceof LLMError
? failure.reason._tag === "InvalidRequest" && failure.reason.classification === "context-overflow"
: Schema.is(ProviderErrorEvent)(failure) && failure.classification === "context-overflow"

37
packages/llm/src/provider.ts Normal file
View File

@@ -0,0 +1,37 @@
import type { RouteDefaultsInput } from "./route/client"
import type { Model, ModelID, ProviderID } from "./schema"
export type ModelOptions = RouteDefaultsInput
/**
* Advanced structural provider definition helper. Built-in providers should
* prefer explicit `configure(options).model(id)` facades so deployment config is
* chosen before model selection. The optional `apis` map remains for external
* structural providers that expose multiple route selectors behind one provider.
*/
export type ModelFactory<Options extends ModelOptions = ModelOptions> = (
id: string | ModelID,
options?: Options,
) => Model
type AnyModelFactory = (...args: never[]) => Model
export interface Definition<Factory extends AnyModelFactory = ModelFactory> {
readonly id: ProviderID
readonly model: Factory
readonly apis?: Record<string, AnyModelFactory>
}
type DefinitionShape = {
readonly id: ProviderID
readonly model: (...args: never[]) => Model
readonly apis?: Record<string, (...args: never[]) => Model>
}
type NoExtraFields<Input, Shape> = Input & Record<Exclude<keyof Input, keyof Shape>, never>
export const make = <DefinitionType extends DefinitionShape>(
definition: NoExtraFields<DefinitionType, DefinitionShape>,
) => definition
export * as Provider from "./provider"

43
packages/llm/src/providers/amazon-bedrock.ts Normal file
View File

@@ -0,0 +1,43 @@
import type { RouteDefaultsInput } from "../route/client"
import { Auth } from "../route/auth"
import { ProviderID, type ModelID } from "../schema"
import * as BedrockConverse from "../protocols/bedrock-converse"
import type { BedrockCredentials } from "../protocols/bedrock-converse"
export const id = ProviderID.make("amazon-bedrock")
export type Config = RouteDefaultsInput & {
readonly apiKey?: string
readonly headers?: Record<string, string>
readonly credentials?: BedrockCredentials
/** AWS region. Defaults to `us-east-1` when neither this nor `credentials.region` is set. */
readonly region?: string
/** Override the computed `https://bedrock-runtime.<region>.amazonaws.com` URL. */
readonly baseURL?: string
}
export const routes = [BedrockConverse.route]
const bedrockBaseURL = (region: string) => `https://bedrock-runtime.${region}.amazonaws.com`
const configuredRoute = (input: Config) => {
const { apiKey, credentials, region, baseURL, ...rest } = input
const resolvedRegion = region ?? credentials?.region ?? "us-east-1"
return BedrockConverse.route.with({
...rest,
provider: id,
endpoint: { baseURL: baseURL ?? bedrockBaseURL(resolvedRegion) },
auth: apiKey === undefined ? BedrockConverse.sigV4Auth(credentials) : Auth.bearer(apiKey),
})
}
export const configure = (input: Config = {}) => {
const route = configuredRoute(input)
return {
id,
model: (modelID: string | ModelID) => route.model({ id: modelID }),
configure,
}
}
export const provider = configure()
export const model = provider.model

35
packages/llm/src/providers/anthropic.ts Normal file
View File

@@ -0,0 +1,35 @@
import type { RouteDefaultsInput } from "../route/client"
import { Auth } from "../route/auth"
import type { ProviderAuthOption } from "../route/auth-options"
import { ProviderID, type ModelID } from "../schema"
import * as AnthropicMessages from "../protocols/anthropic-messages"
export const id = ProviderID.make("anthropic")
export const routes = [AnthropicMessages.route]
export type Config = RouteDefaultsInput & ProviderAuthOption<"optional"> & { readonly baseURL?: string }
const auth = (options: ProviderAuthOption<"optional">) => {
if ("auth" in options && options.auth) return options.auth
return Auth.optional("apiKey" in options ? options.apiKey : undefined, "apiKey")
.orElse(Auth.config("ANTHROPIC_API_KEY"))
.pipe(Auth.header("x-api-key"))
}
const configuredRoute = (input: Config) => {
const { apiKey: _, auth: _auth, baseURL, ...rest } = input
return AnthropicMessages.route.with({ ...rest, endpoint: { baseURL }, auth: auth(input) })
}
export const configure = (input: Config = {}) => {
const route = configuredRoute(input)
return {
id,
model: (modelID: string | ModelID) => route.model({ id: modelID }),
configure,
}
}
export const provider = configure()
export const model = provider.model

110
packages/llm/src/providers/azure.ts Normal file
View File

@@ -0,0 +1,110 @@
import { Auth } from "../route/auth"
import { type AtLeastOne, type ProviderAuthOption } from "../route/auth-options"
import type { Route as RouteDef, RouteDefaultsInput } from "../route/client"
import { ProviderID, type ModelID } from "../schema"
import * as OpenAIChat from "../protocols/openai-chat"
import * as OpenAIResponses from "../protocols/openai-responses"
import { withOpenAIOptions, type OpenAIProviderOptionsInput } from "./openai-options"
export const id = ProviderID.make("azure")
const routeAuth = Auth.remove("authorization")
// Azure needs the customer's resource URL; supply either `resourceName`
// (helper builds the URL) or `baseURL` directly.
type AzureURL = AtLeastOne<{ readonly resourceName: string; readonly baseURL: string }>
export type ModelOptions = AzureURL &
RouteDefaultsInput &
ProviderAuthOption<"optional"> & {
readonly apiVersion?: string
readonly queryParams?: Record<string, string>
readonly useCompletionUrls?: boolean
readonly providerOptions?: OpenAIProviderOptionsInput
}
export type Config = ModelOptions
const resourceBaseURL = (resourceName: string) => `https://${resourceName.trim()}.openai.azure.com/openai/v1`
const responsesRoute = OpenAIResponses.route.with({
id: "azure-openai-responses",
provider: id,
auth: routeAuth,
endpoint: {
query: { "api-version": "v1" },
},
})
const chatRoute = OpenAIChat.route.with({
id: "azure-openai-chat",
provider: id,
auth: routeAuth,
endpoint: {
query: { "api-version": "v1" },
},
})
export const routes = [responsesRoute, chatRoute]
const defaults = (input: Config) => {
const {
apiKey: _,
apiVersion: _apiVersion,
resourceName: _resourceName,
useCompletionUrls: _useCompletionUrls,
baseURL: _baseURL,
queryParams: _queryParams,
...rest
} = input
if ("auth" in rest) {
const { auth: _, ...withoutAuth } = rest
return withoutAuth
}
return rest
}
const auth = (input: Config) => {
if ("auth" in input && input.auth) return input.auth
return Auth.remove("authorization").andThen(
Auth.optional("apiKey" in input ? input.apiKey : undefined, "apiKey")
.orElse(Auth.config("AZURE_OPENAI_API_KEY"))
.pipe(Auth.header("api-key")),
)
}
const configuredRoute = <Body, Prepared>(route: RouteDef<Body, Prepared>, input: Config) =>
route.with({
auth: auth(input),
endpoint: {
// AtLeastOne guarantees at least one is set; baseURL wins if both are.
baseURL: input.baseURL ?? resourceBaseURL(input.resourceName!),
query: {
...(input.apiVersion ? { "api-version": input.apiVersion } : {}),
...input.queryParams,
},
},
})
export const configure = (input: Config) => {
const configuredResponsesRoute = configuredRoute(responsesRoute, input)
const configuredChatRoute = configuredRoute(chatRoute, input)
const modelDefaults = defaults(input)
const responses = (modelID: string | ModelID) =>
configuredResponsesRoute.with(withOpenAIOptions(modelID, modelDefaults)).model({ id: modelID })
const chat = (modelID: string | ModelID) =>
configuredChatRoute.with(withOpenAIOptions(modelID, modelDefaults)).model({ id: modelID })
return {
id,
model: (modelID: string | ModelID) => (input.useCompletionUrls === true ? chat(modelID) : responses(modelID)),
responses,
chat,
configure,
}
}
export const provider = {
id,
configure,
}

127
packages/llm/src/providers/cloudflare.ts Normal file
View File

@@ -0,0 +1,127 @@
import type { Config, Redacted } from "effect"
import * as OpenAICompatibleChat from "../protocols/openai-compatible-chat"
import { Auth } from "../route/auth"
import { AuthOptions, type AtLeastOne, type ProviderAuthOption } from "../route/auth-options"
import type { RouteDefaultsInput } from "../route/client"
import { ProviderID, type ModelID } from "../schema"
export const aiGatewayID = ProviderID.make("cloudflare-ai-gateway")
export const workersAIID = ProviderID.make("cloudflare-workers-ai")
export const aiGatewayAuthEnvVars = ["CLOUDFLARE_API_TOKEN", "CF_AIG_TOKEN"] as const
export const workersAIAuthEnvVars = ["CLOUDFLARE_API_KEY", "CLOUDFLARE_WORKERS_AI_TOKEN"] as const
type CloudflareSecret = string | Redacted.Redacted | Config.Config<string | Redacted.Redacted>
type GatewayURL = AtLeastOne<{
readonly accountId: string
readonly baseURL: string
}> & {
readonly gatewayId?: string
}
export type AIGatewayOptions = GatewayURL &
RouteDefaultsInput &
ProviderAuthOption<"optional"> & {
/** Cloudflare AI Gateway authentication token. Sent as `cf-aig-authorization`. */
readonly gatewayApiKey?: CloudflareSecret
}
type WorkersAIURL = AtLeastOne<{
readonly accountId: string
readonly baseURL: string
}>
export type WorkersAIOptions = WorkersAIURL & RouteDefaultsInput & ProviderAuthOption<"optional">
export const aiGatewayBaseURL = (input: GatewayURL) => {
if (input.baseURL) return input.baseURL
if (!input.accountId) throw new Error("CloudflareAIGateway.configure requires accountId unless baseURL is supplied")
return `https://gateway.ai.cloudflare.com/v1/${encodeURIComponent(input.accountId)}/${encodeURIComponent(input.gatewayId?.trim() || "default")}/compat`
}
const aiGatewayAuth = (input: AIGatewayOptions) => {
if ("auth" in input && input.auth) return input.auth
const gateway = Auth.optional(input.gatewayApiKey, "gatewayApiKey")
.orElse(Auth.config("CLOUDFLARE_API_TOKEN"))
.orElse(Auth.config("CF_AIG_TOKEN"))
.pipe(Auth.bearerHeader("cf-aig-authorization"))
if (!("apiKey" in input) || input.apiKey === undefined) return gateway
if (input.gatewayApiKey === undefined) return Auth.bearer(input.apiKey)
return Auth.bearerHeader("cf-aig-authorization", input.gatewayApiKey).andThen(Auth.bearer(input.apiKey))
}
export const workersAIBaseURL = (input: WorkersAIURL) => {
if (input.baseURL) return input.baseURL
if (!input.accountId) throw new Error("CloudflareWorkersAI.configure requires accountId unless baseURL is supplied")
return `https://api.cloudflare.com/client/v4/accounts/${encodeURIComponent(input.accountId)}/ai/v1`
}
const workersAIAuth = (input: WorkersAIOptions) => {
return AuthOptions.bearer(input, workersAIAuthEnvVars)
}
export const aiGatewayRoute = OpenAICompatibleChat.route.with({
id: "cloudflare-ai-gateway",
provider: aiGatewayID,
})
export const workersAIRoute = OpenAICompatibleChat.route.with({
id: "cloudflare-workers-ai",
provider: workersAIID,
})
export const routes = [aiGatewayRoute, workersAIRoute]
const aiGatewayDefaults = (options: AIGatewayOptions) => {
const {
accountId: _accountId,
gatewayId: _gatewayId,
apiKey: _apiKey,
gatewayApiKey: _gatewayApiKey,
baseURL: _baseURL,
auth: _auth,
...rest
} = options
return rest
}
const workersAIDefaults = (options: WorkersAIOptions) => {
const { accountId: _accountId, apiKey: _apiKey, auth: _auth, baseURL: _baseURL, ...rest } = options
return rest
}
const configureAIGateway = (options: AIGatewayOptions) => {
const route = aiGatewayRoute.with({
...aiGatewayDefaults(options),
endpoint: { baseURL: aiGatewayBaseURL(options) },
auth: aiGatewayAuth(options),
})
return {
id: aiGatewayID,
model: (modelID: string | ModelID) => route.model({ id: modelID }),
configure: configureAIGateway,
}
}
const configureWorkersAI = (options: WorkersAIOptions) => {
const route = workersAIRoute.with({
...workersAIDefaults(options),
endpoint: { baseURL: workersAIBaseURL(options) },
auth: workersAIAuth(options),
})
return {
id: workersAIID,
model: (modelID: string | ModelID) => route.model({ id: modelID }),
configure: configureWorkersAI,
}
}
export const CloudflareAIGateway = {
id: aiGatewayID,
configure: configureAIGateway,
}
export const CloudflareWorkersAI = {
id: workersAIID,
configure: configureWorkersAI,
}

66
packages/llm/src/providers/github-copilot.ts Normal file
View File

@@ -0,0 +1,66 @@
import { AuthOptions, type ProviderAuthOption } from "../route/auth-options"
import type { RouteDefaultsInput } from "../route/client"
import { ProviderID, type ModelID } from "../schema"
import * as OpenAIChat from "../protocols/openai-chat"
import * as OpenAIResponses from "../protocols/openai-responses"
import { withOpenAIOptions, type OpenAIProviderOptionsInput } from "./openai-options"
export const id = ProviderID.make("github-copilot")
// GitHub Copilot has no canonical public URL — callers (opencode, etc.) must
// supply `baseURL` explicitly.
export type ModelOptions = Omit<RouteDefaultsInput, "providerOptions"> &
ProviderAuthOption<"optional"> & {
readonly baseURL: string
readonly providerOptions?: OpenAIProviderOptionsInput
}
export const shouldUseResponsesApi = (modelID: string | ModelID) => {
const model = String(modelID)
const match = /^gpt-(\d+)/.exec(model)
if (!match) return false
return Number(match[1]) >= 5 && !model.startsWith("gpt-5-mini")
}
export const routes = [OpenAIResponses.route, OpenAIChat.route]
const chatRoute = OpenAIChat.route.with({ provider: id })
const responsesRoute = OpenAIResponses.route.with({ provider: id })
const defaults = (options: ModelOptions) => {
const { apiKey: _, auth: _auth, baseURL: _baseURL, ...rest } = options
return rest
}
const configuredResponsesRoute = (options: ModelOptions) =>
responsesRoute.with({
endpoint: { baseURL: options.baseURL },
auth: AuthOptions.bearer(options, []),
})
const configuredChatRoute = (options: ModelOptions) =>
chatRoute.with({
endpoint: { baseURL: options.baseURL },
auth: AuthOptions.bearer(options, []),
})
export const configure = (options: ModelOptions) => {
const responsesRoute = configuredResponsesRoute(options)
const chatRoute = configuredChatRoute(options)
const responses = (modelID: string | ModelID) =>
responsesRoute.with(withOpenAIOptions(modelID, defaults(options))).model({ id: modelID })
const chat = (modelID: string | ModelID) =>
chatRoute.with(withOpenAIOptions(modelID, defaults(options))).model({ id: modelID })
return {
id,
model: (modelID: string | ModelID) => (shouldUseResponsesApi(modelID) ? responses(modelID) : chat(modelID)),
responses,
chat,
configure,
}
}
export const provider = {
id,
configure,
}

35
packages/llm/src/providers/google.ts Normal file
View File

@@ -0,0 +1,35 @@
import type { RouteDefaultsInput } from "../route/client"
import { Auth } from "../route/auth"
import type { ProviderAuthOption } from "../route/auth-options"
import { ProviderID, type ModelID } from "../schema"
import * as Gemini from "../protocols/gemini"
export const id = ProviderID.make("google")
export const routes = [Gemini.route]
export type Config = RouteDefaultsInput & ProviderAuthOption<"optional"> & { readonly baseURL?: string }
const auth = (options: ProviderAuthOption<"optional">) => {
if ("auth" in options && options.auth) return options.auth
return Auth.optional("apiKey" in options ? options.apiKey : undefined, "apiKey")
.orElse(Auth.config("GOOGLE_GENERATIVE_AI_API_KEY"))
.pipe(Auth.header("x-goog-api-key"))
}
const configuredRoute = (input: Config) => {
const { apiKey: _, auth: _auth, baseURL, ...rest } = input
return Gemini.route.with({ ...rest, endpoint: { baseURL }, auth: auth(input) })
}
export const configure = (input: Config = {}) => {
const route = configuredRoute(input)
return {
id,
model: (modelID: string | ModelID) => route.model({ id: modelID }),
configure,
}
}
export const provider = configure()
export const model = provider.model

11
packages/llm/src/providers/index.ts Normal file
View File

@@ -0,0 +1,11 @@
export * as Anthropic from "./anthropic"
export * as AmazonBedrock from "./amazon-bedrock"
export * as Azure from "./azure"
export * as Cloudflare from "./cloudflare"
export { CloudflareAIGateway, CloudflareWorkersAI } from "./cloudflare"
export * as GitHubCopilot from "./github-copilot"
export * as Google from "./google"
export * as OpenAI from "./openai"
export * as OpenAICompatible from "./openai-compatible"
export * as OpenRouter from "./openrouter"
export * as XAI from "./xai"

20
packages/llm/src/providers/openai-compatible-profile.ts Normal file
View File

@@ -0,0 +1,20 @@
export interface OpenAICompatibleProfile {
readonly provider: string
readonly baseURL: string
}
export const profiles = {
baseten: { provider: "baseten", baseURL: "https://inference.baseten.co/v1" },
cerebras: { provider: "cerebras", baseURL: "https://api.cerebras.ai/v1" },
deepinfra: { provider: "deepinfra", baseURL: "https://api.deepinfra.com/v1/openai" },
deepseek: { provider: "deepseek", baseURL: "https://api.deepseek.com/v1" },
fireworks: { provider: "fireworks", baseURL: "https://api.fireworks.ai/inference/v1" },
groq: { provider: "groq", baseURL: "https://api.groq.com/openai/v1" },
openrouter: { provider: "openrouter", baseURL: "https://openrouter.ai/api/v1" },
togetherai: { provider: "togetherai", baseURL: "https://api.together.xyz/v1" },
xai: { provider: "xai", baseURL: "https://api.x.ai/v1" },
} as const satisfies Record<string, OpenAICompatibleProfile>
export const byProvider: Record<string, OpenAICompatibleProfile> = Object.fromEntries(
Object.values(profiles).map((profile) => [profile.provider, profile]),
)

65
packages/llm/src/providers/openai-compatible.ts Normal file
View File

@@ -0,0 +1,65 @@
import { ProviderID, type ModelID } from "../schema"
import * as OpenAICompatibleChat from "../protocols/openai-compatible-chat"
import type { RouteDefaultsInput } from "../route/client"
import { AuthOptions, type ProviderAuthOption } from "../route/auth-options"
import { profiles, type OpenAICompatibleProfile } from "./openai-compatible-profile"
export const id = ProviderID.make("openai-compatible")
type GenericModelOptions = RouteDefaultsInput &
ProviderAuthOption<"optional"> & {
readonly provider?: string
readonly baseURL: string
}
export type FamilyModelOptions = RouteDefaultsInput &
ProviderAuthOption<"optional"> & {
readonly baseURL?: string
}
export const routes = [OpenAICompatibleChat.route]
export const configure = (input: GenericModelOptions) => {
const provider = input.provider ?? "openai-compatible"
const { provider: _, baseURL, apiKey: _apiKey, auth: _auth, ...rest } = input
const route = OpenAICompatibleChat.route.with({
...rest,
provider,
endpoint: { baseURL },
auth: AuthOptions.bearer(input, []),
})
return {
id: ProviderID.make(provider),
model: (modelID: string | ModelID) => route.model({ id: modelID, provider: ProviderID.make(provider) }),
configure,
}
}
const define = (profile: OpenAICompatibleProfile) => {
const configureProfile = (input: FamilyModelOptions = {}) => {
const facade = configure({
...input,
baseURL: input.baseURL ?? profile.baseURL,
provider: profile.provider,
})
return {
id: ProviderID.make(profile.provider),
model: facade.model,
configure: configureProfile,
}
}
return configureProfile()
}
export const provider = {
id,
configure,
}
export const baseten = define(profiles.baseten)
export const cerebras = define(profiles.cerebras)
export const deepinfra = define(profiles.deepinfra)
export const deepseek = define(profiles.deepseek)
export const fireworks = define(profiles.fireworks)
export const groq = define(profiles.groq)
export const togetherai = define(profiles.togetherai)

83
packages/llm/src/providers/openai-options.ts Normal file
View File

@@ -0,0 +1,83 @@
import type { ProviderOptions, ReasoningEffort, TextVerbosity } from "../schema"
import { mergeProviderOptions } from "../schema"
import type { OpenAIResponseIncludable, OpenAIServiceTier } from "../protocols/utils/openai-options"
export type { OpenAIResponseIncludable, OpenAIServiceTier } from "../protocols/utils/openai-options"
export interface OpenAIOptionsInput {
readonly [key: string]: unknown
readonly store?: boolean
readonly promptCacheKey?: string
readonly reasoningEffort?: ReasoningEffort
readonly reasoningSummary?: "auto"
// OpenAI Responses `include` wire field. Mirrors the official SDK's
// `ResponseIncludable[]` union exactly so AI SDK callers and direct
// native-SDK callers share one shape and no translation is required.
readonly include?: ReadonlyArray<OpenAIResponseIncludable>
readonly textVerbosity?: TextVerbosity
readonly serviceTier?: OpenAIServiceTier
}
export type OpenAIProviderOptionsInput = ProviderOptions & {
readonly openai?: OpenAIOptionsInput
}
const definedEntries = (input: Record<string, unknown>) =>
Object.entries(input).filter((entry) => entry[1] !== undefined)
const openAIProviderOptions = (options: OpenAIOptionsInput | undefined): ProviderOptions | undefined => {
const openai = Object.fromEntries(
definedEntries({
store: options?.store,
promptCacheKey: options?.promptCacheKey,
reasoningEffort: options?.reasoningEffort,
reasoningSummary: options?.reasoningSummary,
include: options?.include,
textVerbosity: options?.textVerbosity,
serviceTier: options?.serviceTier,
}),
)
if (Object.keys(openai).length === 0) return undefined
return { openai }
}
export const gpt5DefaultOptions = (
modelID: string,
options: { readonly textVerbosity?: boolean } = {},
): ProviderOptions | undefined => {
const id = modelID.toLowerCase()
if (!id.includes("gpt-5") || id.includes("gpt-5-chat") || id.includes("gpt-5-pro")) return undefined
return openAIProviderOptions({
reasoningEffort: "medium",
reasoningSummary: "auto",
// GPT-5 reasoning models are configured stateless (`store: false`) by
// `openAIDefaultOptions` below, so the only way a follow-up turn can
// carry reasoning state is via the encrypted reasoning include. Without
// this, callers using the default model facade get reasoning summaries
// they cannot replay statelessly.
include: ["reasoning.encrypted_content"],
textVerbosity:
options.textVerbosity === true && id.includes("gpt-5.") && !id.includes("codex") && !id.includes("-chat")
? "low"
: undefined,
})
}
export const openAIDefaultOptions = (
modelID: string,
options: { readonly textVerbosity?: boolean } = {},
): ProviderOptions | undefined =>
mergeProviderOptions(openAIProviderOptions({ store: false }), gpt5DefaultOptions(modelID, options))
export const withOpenAIOptions = <Options extends { readonly providerOptions?: OpenAIProviderOptionsInput }>(
modelID: string,
options: Options,
defaults: { readonly textVerbosity?: boolean } = {},
): Omit<Options, "providerOptions"> & { readonly providerOptions?: ProviderOptions } => {
return {
...options,
providerOptions: mergeProviderOptions(openAIDefaultOptions(modelID, defaults), options.providerOptions),
}
}
export * as OpenAIProviderOptions from "./openai-options"

63
packages/llm/src/providers/openai.ts Normal file
View File

@@ -0,0 +1,63 @@
import { AuthOptions, type ProviderAuthOption } from "../route/auth-options"
import type { Route, RouteDefaultsInput } from "../route/client"
import { ProviderID, type ModelID } from "../schema"
import * as OpenAIChat from "../protocols/openai-chat"
import * as OpenAIResponses from "../protocols/openai-responses"
import { withOpenAIOptions, type OpenAIProviderOptionsInput } from "./openai-options"
export type { OpenAIOptionsInput, OpenAIResponseIncludable } from "./openai-options"
export const id = ProviderID.make("openai")
export const routes = [OpenAIResponses.route, OpenAIResponses.webSocketRoute, OpenAIChat.route]
// This provider facade wraps the lower-level Responses and Chat model factories
// with OpenAI-specific conveniences: typed options, API-key sugar, env fallback,
// and default option normalization.
export type Config = RouteDefaultsInput &
ProviderAuthOption<"optional"> & {
readonly baseURL?: string
readonly queryParams?: Record<string, string>
readonly providerOptions?: OpenAIProviderOptionsInput
}
const auth = (options: ProviderAuthOption<"optional">) => AuthOptions.bearer(options, "OPENAI_API_KEY")
const defaults = (input: Config) => {
const { apiKey: _, auth: _auth, baseURL: _baseURL, queryParams: _queryParams, ...rest } = input
return rest
}
const configuredRoute = <Body, Prepared>(route: Route<Body, Prepared>, input: Config) =>
route.with({
auth: auth(input),
endpoint: { baseURL: input.baseURL, query: input.queryParams },
})
export const configure = (input: Config = {}) => {
const responsesRoute = configuredRoute(OpenAIResponses.route, input)
const responsesWebSocketRoute = configuredRoute(OpenAIResponses.webSocketRoute, input)
const chatRoute = configuredRoute(OpenAIChat.route, input)
const modelDefaults = defaults(input)
const responses = (id: string | ModelID) =>
responsesRoute.with(withOpenAIOptions(id, modelDefaults, { textVerbosity: true })).model({ id })
const responsesWebSocket = (id: string | ModelID) =>
responsesWebSocketRoute.with(withOpenAIOptions(id, modelDefaults, { textVerbosity: true })).model({ id })
const chat = (id: string | ModelID) => chatRoute.with(withOpenAIOptions(id, modelDefaults)).model({ id })
return {
id,
model: responses,
responses,
responsesWebSocket,
chat,
configure,
}
}
export const provider = configure()
export const model = provider.model
export const responses = provider.responses
export const responsesWebSocket = provider.responsesWebSocket
export const chat = provider.chat

98
packages/llm/src/providers/openrouter.ts Normal file
View File

@@ -0,0 +1,98 @@
import { Effect, Schema } from "effect"
import { Route, type RouteDefaultsInput } from "../route/client"
import { Endpoint } from "../route/endpoint"
import { Framing } from "../route/framing"
import { Protocol } from "../route/protocol"
import { AuthOptions, type ProviderAuthOption } from "../route/auth-options"
import { ProviderID, type ModelID, type ProviderOptions } from "../schema"
import * as OpenAICompatibleProfiles from "./openai-compatible-profile"
import * as OpenAIChat from "../protocols/openai-chat"
import { isRecord } from "../protocols/shared"
export const profile = OpenAICompatibleProfiles.profiles.openrouter
export const id = ProviderID.make(profile.provider)
const ADAPTER = "openrouter"
export interface OpenRouterOptions {
readonly [key: string]: unknown
readonly usage?: boolean | Record<string, unknown>
readonly reasoning?: Record<string, unknown>
readonly promptCacheKey?: string
}
export type OpenRouterProviderOptionsInput = ProviderOptions & {
readonly openrouter?: OpenRouterOptions
}
export type ModelOptions = Omit<RouteDefaultsInput, "providerOptions"> &
ProviderAuthOption<"optional"> & {
readonly baseURL?: string
readonly providerOptions?: OpenRouterProviderOptionsInput
}
const OpenRouterBody = Schema.StructWithRest(Schema.Struct(OpenAIChat.bodyFields), [
Schema.Record(Schema.String, Schema.Any),
])
export type OpenRouterBody = Schema.Schema.Type<typeof OpenRouterBody>
export const protocol = Protocol.make({
id: "openrouter-chat",
body: {
schema: OpenRouterBody,
from: (request) =>
OpenAIChat.protocol.body.from(request).pipe(
Effect.map(
(body) =>
({
...body,
...bodyOptions(request.providerOptions?.openrouter),
}) as OpenRouterBody,
),
),
},
stream: OpenAIChat.protocol.stream,
})
const bodyOptions = (input: unknown) => {
const openrouter = isRecord(input) ? input : {}
return {
...(openrouter.usage === true
? { usage: { include: true } }
: isRecord(openrouter.usage)
? { usage: openrouter.usage }
: {}),
...(isRecord(openrouter.reasoning) ? { reasoning: openrouter.reasoning } : {}),
...(typeof openrouter.promptCacheKey === "string" ? { prompt_cache_key: openrouter.promptCacheKey } : {}),
}
}
export const route = Route.make({
id: ADAPTER,
provider: profile.provider,
protocol,
endpoint: Endpoint.path("/chat/completions", { baseURL: profile.baseURL }),
framing: Framing.sse,
})
export const routes = [route]
const configuredRoute = (input: ModelOptions) => {
const { apiKey: _, auth: _auth, baseURL, ...rest } = input
return route.with({
...rest,
endpoint: { baseURL: baseURL ?? profile.baseURL },
auth: AuthOptions.bearer(input, "OPENROUTER_API_KEY"),
})
}
export const configure = (input: ModelOptions = {}) => {
const route = configuredRoute(input)
return {
id,
model: (modelID: string | ModelID) => route.model({ id: modelID }),
configure,
}
}
export const provider = configure()
export const model = provider.model

56
packages/llm/src/providers/xai.ts Normal file
View File

@@ -0,0 +1,56 @@
import { AuthOptions, type ProviderAuthOption } from "../route/auth-options"
import type { RouteDefaultsInput } from "../route/client"
import { ProviderID, type ModelID } from "../schema"
import * as OpenAICompatibleProfiles from "./openai-compatible-profile"
import * as OpenAICompatibleChat from "../protocols/openai-compatible-chat"
import * as OpenAIResponses from "../protocols/openai-responses"
export const id = ProviderID.make("xai")
export type ModelOptions = RouteDefaultsInput &
ProviderAuthOption<"optional"> & {
readonly baseURL?: string
}
export const routes = [OpenAIResponses.route, OpenAICompatibleChat.route]
const auth = (options: ProviderAuthOption<"optional">) => AuthOptions.bearer(options, "XAI_API_KEY")
const configuredResponsesRoute = (input: ModelOptions) => {
const { apiKey: _, auth: _auth, baseURL, ...rest } = input
return OpenAIResponses.route.with({
...rest,
provider: id,
endpoint: { baseURL: baseURL ?? OpenAICompatibleProfiles.profiles.xai.baseURL },
auth: auth(input),
})
}
const configuredChatRoute = (input: ModelOptions) => {
const { apiKey: _, auth: _auth, baseURL, ...rest } = input
return OpenAICompatibleChat.route.with({
...rest,
provider: id,
endpoint: { baseURL: baseURL ?? OpenAICompatibleProfiles.profiles.xai.baseURL },
auth: auth(input),
})
}
export const configure = (input: ModelOptions = {}) => {
const responsesRoute = configuredResponsesRoute(input)
const chatRoute = configuredChatRoute(input)
const responses = (modelID: string | ModelID) => responsesRoute.model({ id: modelID })
const chat = (modelID: string | ModelID) => chatRoute.model({ id: modelID })
return {
id,
model: responses,
responses,
chat,
configure,
}
}
export const provider = configure()
export const model = provider.model
export const responses = provider.responses
export const chat = provider.chat

57
packages/llm/src/route/auth-options.ts Normal file
View File

@@ -0,0 +1,57 @@
import type { Config, Redacted } from "effect"
import { Auth } from "./auth"
export type ApiKeyMode = "optional" | "required"
export type AuthOverride = {
readonly auth: Auth
readonly apiKey?: never
}
export type OptionalApiKeyAuth = {
readonly apiKey?: string | Redacted.Redacted<string> | Config.Config<string | Redacted.Redacted<string>>
readonly auth?: never
}
export type RequiredApiKeyAuth = {
readonly apiKey: string | Redacted.Redacted<string> | Config.Config<string | Redacted.Redacted<string>>
readonly auth?: never
}
export type ProviderAuthOption<Mode extends ApiKeyMode> =
| AuthOverride
| (Mode extends "optional" ? OptionalApiKeyAuth : RequiredApiKeyAuth)
export type ModelOptions<Base, Mode extends ApiKeyMode> = Omit<Base, "apiKey" | "auth"> & ProviderAuthOption<Mode>
export type ModelArgs<Base, Mode extends ApiKeyMode> = Mode extends "optional"
? readonly [options?: ModelOptions<Base, Mode>]
: readonly [options: ModelOptions<Base, Mode>]
export type ModelFactory<Base, Mode extends ApiKeyMode, Model> = (id: string, ...args: ModelArgs<Base, Mode>) => Model
/**
* Require at least one of the keys in `T`. Use for option shapes where any
* subset of fields is acceptable but at least one must be present (e.g. Azure
* accepts `resourceName` or `baseURL`).
*/
export type AtLeastOne<T> = {
[K in keyof T]: Required<Pick<T, K>> & Partial<Omit<T, K>>
}[keyof T]
/**
* Standard bearer-auth resolution for providers: honor an explicit `auth`
* override, otherwise resolve `apiKey` (option > config var) and apply it as
* a bearer token.
*/
export const bearer = (options: ProviderAuthOption<"optional">, envVar: string | ReadonlyArray<string>): Auth => {
if ("auth" in options && options.auth) return options.auth
return (Array.isArray(envVar) ? envVar : [envVar])
.reduce(
(auth, name) => auth.orElse(Auth.config(name)),
Auth.optional("apiKey" in options ? options.apiKey : undefined, "apiKey"),
)
.bearer()
}
export * as AuthOptions from "./auth-options"

156
packages/llm/src/route/auth.ts Normal file
View File

@@ -0,0 +1,156 @@
import { Config, Effect, Redacted } from "effect"
import { Headers } from "effect/unstable/http"
import { AuthenticationReason, InvalidRequestReason, LLMError, type LLMRequest } from "../schema"
export class MissingCredentialError extends Error {
readonly _tag = "MissingCredentialError"
constructor(readonly source: string) {
super(`Missing auth credential: ${source}`)
}
}
export type CredentialError = MissingCredentialError | Config.ConfigError
export type AuthError = CredentialError | LLMError
type Secret = string | Redacted.Redacted | Config.Config<string | Redacted.Redacted>
export interface AuthInput {
readonly request: LLMRequest
readonly method: "POST" | "GET"
readonly url: string
readonly body: string
readonly headers: Headers.Headers
}
export interface Credential {
readonly load: Effect.Effect<Redacted.Redacted, CredentialError>
readonly orElse: (that: Credential) => Credential
readonly bearer: () => Auth
readonly header: (name: string) => Auth
readonly pipe: <A>(f: (self: Credential) => A) => A
}
export interface Auth {
readonly apply: (input: AuthInput) => Effect.Effect<Headers.Headers, AuthError>
readonly andThen: (that: Auth) => Auth
readonly orElse: (that: Auth) => Auth
readonly pipe: <A>(f: (self: Auth) => A) => A
}
export const isAuth = (input: unknown): input is Auth =>
typeof input === "object" && input !== null && "apply" in input && typeof input.apply === "function"
const credential = (load: Effect.Effect<Redacted.Redacted, CredentialError>): Credential => {
const self: Credential = {
load,
orElse: (that) => credential(load.pipe(Effect.catch(() => that.load))),
bearer: () => fromCredential(self, (secret) => ({ authorization: `Bearer ${secret}` })),
header: (name) => fromCredential(self, (secret) => ({ [name]: secret })),
pipe: (f) => f(self),
}
return self
}
const auth = (apply: Auth["apply"]): Auth => {
const self: Auth = {
apply,
andThen: (that) =>
auth((input) => apply(input).pipe(Effect.flatMap((headers) => that.apply({ ...input, headers })))),
orElse: (that) => auth((input) => apply(input).pipe(Effect.catch(() => that.apply(input)))),
pipe: (f) => f(self),
}
return self
}
const fromCredential = (source: Credential, render: (secret: string) => Headers.Input) =>
auth((input) =>
source.load.pipe(Effect.map((secret) => Headers.setAll(input.headers, render(Redacted.value(secret))))),
)
const secretEffect = (secret: string | Redacted.Redacted, source: string) => {
const redacted = typeof secret === "string" ? Redacted.make(secret) : secret
if (Redacted.value(redacted) === "") return Effect.fail(new MissingCredentialError(source))
return Effect.succeed(redacted)
}
const credentialFromSecret = (secret: Secret, source: string) => {
if (typeof secret === "string" || Redacted.isRedacted(secret)) return credential(secretEffect(secret, source))
return credential(
Effect.gen(function* () {
return yield* secretEffect(yield* secret, source)
}),
)
}
export const value = (secret: string, source = "value") => credentialFromSecret(secret, source)
export const optional = (secret: Secret | undefined, source = "optional value") =>
secret === undefined
? credential(Effect.fail(new MissingCredentialError(source)))
: credentialFromSecret(secret, source)
export const config = (name: string) => credentialFromSecret(Config.redacted(name), name)
export const effect = (load: Effect.Effect<Redacted.Redacted, CredentialError>) => credential(load)
export const none = auth((input) => Effect.succeed(input.headers))
export const headers = (input: Headers.Input) =>
auth((inputAuth) => Effect.succeed(Headers.setAll(inputAuth.headers, input)))
export const remove = (name: string) => auth((input) => Effect.succeed(Headers.remove(input.headers, name)))
export const custom = (apply: (input: AuthInput) => Effect.Effect<Headers.Headers, LLMError>) => auth(apply)
export const passthrough = none
const credentialInput = (source: Secret | Credential) =>
typeof source === "string" || Redacted.isRedacted(source) || Config.isConfig(source)
? credentialFromSecret(source, "value")
: source
export function bearer(source: Secret | Credential): Auth
export function bearer(source: Secret | Credential) {
return credentialInput(source).bearer()
}
export const apiKey = bearer
export function header(name: string): (source: Secret | Credential) => Auth
export function header(name: string, source: Secret | Credential): Auth
export function header(name: string, source?: Secret | Credential) {
if (source === undefined) {
return (next: Secret | Credential) => credentialInput(next).header(name)
}
return credentialInput(source).header(name)
}
export function bearerHeader(name: string): (source: Secret | Credential) => Auth
export function bearerHeader(name: string, source: Secret | Credential): Auth
export function bearerHeader(name: string, source?: Secret | Credential) {
const render = (input: Secret | Credential) =>
fromCredential(credentialInput(input), (secret) => ({ [name]: `Bearer ${secret}` }))
if (source === undefined) return render
return render(source)
}
const toLLMError = (error: AuthError): LLMError => {
if (error instanceof MissingCredentialError || error instanceof Config.ConfigError) {
return new LLMError({
module: "Auth",
method: "apply",
reason:
error instanceof MissingCredentialError
? new AuthenticationReason({ message: error.message, kind: "missing" })
: new InvalidRequestReason({ message: `Failed to resolve auth config: ${error.message}` }),
})
}
return error
}
export const toEffect =
(input: Auth) =>
(authInput: AuthInput): Effect.Effect<Headers.Headers, LLMError> =>
input.apply(authInput).pipe(Effect.mapError(toLLMError))
export * as Auth from "./auth"

434
packages/llm/src/route/client.ts Normal file
View File

@@ -0,0 +1,434 @@
import { Cause, Context, Effect, Layer, Schema, Stream } from "effect"
import * as Option from "effect/Option"
import { Auth, type Auth as AuthDef } from "./auth"
import { Endpoint, type EndpointPatch } from "./endpoint"
import { RequestExecutor } from "./executor"
import type { Framing } from "./framing"
import { HttpTransport } from "./transport"
import type { Transport, TransportRuntime } from "./transport"
import { WebSocketExecutor } from "./transport"
import type { Protocol } from "./protocol"
import { applyCachePolicy } from "../cache-policy"
import * as ProviderShared from "../protocols/shared"
import type { LLMError, LLMEvent, PreparedRequestOf, ProtocolID, ProviderOptions } from "../schema"
import {
GenerationOptions,
HttpOptions,
LLMRequest,
LLMResponse,
Model,
ModelLimits,
LLMError as LLMErrorClass,
PreparedRequest,
ProviderID,
mergeGenerationOptions,
mergeHttpOptions,
mergeProviderOptions,
} from "../schema"
export interface RouteBody<Body> {
/** Schema for the validated provider-native body sent as the JSON request. */
readonly schema: Schema.Codec<Body, unknown>
/** Build the provider-native body from a common `LLMRequest`. */
readonly from: (request: LLMRequest) => Effect.Effect<Body, LLMError>
}
export interface Route<Body, Prepared = unknown> {
readonly id: string
readonly provider?: ProviderID
readonly protocol: ProtocolID
readonly endpoint: Endpoint<Body>
readonly auth: AuthDef
readonly transport: Transport<Body, Prepared, unknown>
readonly defaults: RouteDefaults
readonly body: RouteBody<Body>
readonly with: (patch: RoutePatch<Body, Prepared>) => Route<Body, Prepared>
readonly model: (input: RouteMappedModelInput) => Model
readonly prepareTransport: (body: Body, request: LLMRequest) => Effect.Effect<Prepared, LLMError>
readonly streamPrepared: (
prepared: Prepared,
request: LLMRequest,
runtime: TransportRuntime,
) => Stream.Stream<LLMEvent, LLMError>
}
// Route registries intentionally erase body generics after construction.
// Normal call sites use `OpenAIChat.route`; callers only need body types
// when preparing a request with a protocol-specific type assertion.
// oxlint-disable-next-line typescript-eslint/no-explicit-any
export type AnyRoute = Route<any, any>
export type HttpOptionsInput = HttpOptions.Input
export type RouteModelInput = Omit<Model.Input, "provider" | "route">
export type RouteRoutedModelInput = Omit<Model.Input, "route">
export interface RouteDefaults {
readonly headers?: Record<string, string>
readonly limits?: ModelLimits
readonly generation?: GenerationOptions
readonly providerOptions?: ProviderOptions
readonly http?: HttpOptions
}
export interface RouteDefaultsInput {
readonly headers?: Record<string, string>
readonly limits?: ModelLimits.Input
readonly generation?: GenerationOptions.Input
readonly providerOptions?: ProviderOptions
readonly http?: HttpOptions.Input
}
export interface RoutePatch<Body, Prepared> extends RouteDefaultsInput {
readonly id?: string
readonly provider?: string | ProviderID
readonly auth?: AuthDef
readonly transport?: Transport<Body, Prepared, unknown>
readonly endpoint?: EndpointPatch<Body>
}
type RouteMappedModelInput = RouteModelInput | RouteRoutedModelInput
const makeRouteModel = (route: AnyRoute, mapped: RouteMappedModelInput) => {
const provider = route.provider ?? ("provider" in mapped ? mapped.provider : undefined)
if (!provider) throw new Error(`Route.model(${route.id}) requires a provider`)
if (!endpointBaseURL(route.endpoint))
throw new Error(`Route.model(${route.id}) requires an endpoint baseURL — configure it on the route first`)
return Model.make({
...mapped,
provider,
route,
})
}
const mergeRouteDefaults = (base: RouteDefaults | undefined, patch: RouteDefaultsInput): RouteDefaults => {
const headers = mergeHeaders(base?.headers, patch.headers)
return {
...base,
...patch,
headers,
limits: patch.limits === undefined ? base?.limits : ModelLimits.make(patch.limits),
generation: mergeGenerationOptions(generationOptions(base?.generation), generationOptions(patch.generation)),
providerOptions: mergeProviderOptions(base?.providerOptions, patch.providerOptions),
http: mergeHttpOptions(
base?.http,
httpOptions(patch.http),
headers === undefined ? undefined : new HttpOptions({ headers }),
),
}
}
const endpointBaseURL = <Body>(endpoint: Endpoint<Body>) =>
typeof endpoint.baseURL === "string" ? endpoint.baseURL : undefined
const mergeHeaders = (...items: ReadonlyArray<Record<string, string> | undefined>) => {
const entries = items.flatMap((item) =>
item === undefined ? [] : Object.entries(item).filter((entry): entry is [string, string] => entry[1] !== undefined),
)
if (entries.length === 0) return undefined
return Object.fromEntries(entries)
}
export const generationOptions = (input: GenerationOptions.Input | undefined) =>
input === undefined ? undefined : GenerationOptions.make(input)
export const httpOptions = (input: HttpOptionsInput | undefined) => {
if (input === undefined) return input
return HttpOptions.make(input)
}
export interface Interface {
/**
* Compile a request through protocol body construction, validation, and HTTP
* preparation without sending it. Returns the prepared request including the
* provider-native body.
*
* Pass a `Body` type argument to statically expose the route's body
* shape (e.g. `prepare<OpenAIChatBody>(...)`) — the runtime body is
* identical, so this is a type-level assertion the caller makes about which
* route the request will resolve to.
*/
readonly prepare: <Body = unknown>(request: LLMRequest) => Effect.Effect<PreparedRequestOf<Body>, LLMError>
readonly stream: StreamMethod
readonly generate: GenerateMethod
}
export interface StreamMethod {
(request: LLMRequest): Stream.Stream<LLMEvent, LLMError>
}
export interface GenerateMethod {
(request: LLMRequest): Effect.Effect<LLMResponse, LLMError>
}
export class Service extends Context.Service<Service, Interface>()("@opencode/LLMClient") {}
const resolveRequestOptions = (request: LLMRequest) =>
LLMRequest.update(request, {
generation:
mergeGenerationOptions(request.model.route.defaults.generation, request.generation) ?? new GenerationOptions({}),
providerOptions: mergeProviderOptions(request.model.route.defaults.providerOptions, request.providerOptions),
http: mergeHttpOptions(request.model.route.defaults.http, request.http),
})
export interface MakeInput<Body, Frame, Event, State> {
/** Route id used in diagnostics and prepared request metadata. */
readonly id: string
/** Provider identity for route-owned model construction. */
readonly provider?: string | ProviderID
/** Semantic API contract — owns body construction, body schema, and parsing. */
readonly protocol: Protocol<Body, Frame, Event, State>
/** Where the request is sent. */
readonly endpoint: Endpoint<Body>
/** Per-request transport auth. Provider facades override this via `route.with(...)`. */
readonly auth?: AuthDef
/** Stream framing — bytes -> frames before `protocol.stream.event` decoding. */
readonly framing: Framing<Frame>
/** Static / per-request headers added before `auth` runs. */
readonly headers?: (input: { readonly request: LLMRequest }) => Record<string, string>
/** Route/request defaults used when compiling requests for this route. */
readonly defaults?: RouteDefaultsInput
}
export interface MakeTransportInput<Body, Prepared, Frame, Event, State> {
/** Route id used in diagnostics and prepared request metadata. */
readonly id: string
/** Provider identity for route-owned model construction. */
readonly provider?: string | ProviderID
/** Semantic API contract — owns body construction, body schema, and parsing. */
readonly protocol: Protocol<Body, Frame, Event, State>
/** Where the request is sent. */
readonly endpoint: Endpoint<Body>
/** Per-request transport auth. Provider facades override this via `route.with(...)`. */
readonly auth?: AuthDef
/** Static / per-request headers added before `auth` runs. */
readonly headers?: (input: { readonly request: LLMRequest }) => Record<string, string>
/** Runnable transport route. */
readonly transport: Transport<Body, Prepared, Frame>
/** Route/request defaults used when compiling requests for this route. */
readonly defaults?: RouteDefaultsInput
}
const streamError = (route: string, message: string, cause: Cause.Cause<unknown>) => {
const failed = cause.reasons.find(Cause.isFailReason)?.error
if (failed instanceof LLMErrorClass) return failed
return ProviderShared.eventError(route, message, Cause.pretty(cause))
}
function makeFromTransport<Body, Prepared, Frame, Event, State>(
input: MakeTransportInput<Body, Prepared, Frame, Event, State>,
): Route<Body, Prepared> {
const protocol = input.protocol
const encodeBody = Schema.encodeSync(Schema.fromJsonString(protocol.body.schema))
const decodeEventEffect = Schema.decodeUnknownEffect(protocol.stream.event)
const decodeEvent = (route: string) => (frame: Frame) =>
decodeEventEffect(frame).pipe(
Effect.mapError(() =>
ProviderShared.eventError(
input.id,
`Invalid ${route} stream event`,
typeof frame === "string" ? frame : ProviderShared.encodeJson(frame),
),
),
)
type BuiltRouteInput = Omit<MakeTransportInput<Body, Prepared, Frame, Event, State>, "defaults"> & {
readonly defaults?: RouteDefaults
}
const build = (routeInput: BuiltRouteInput): Route<Body, Prepared> => {
const route: Route<Body, Prepared> = {
id: routeInput.id,
provider: routeInput.provider === undefined ? undefined : ProviderID.make(routeInput.provider),
protocol: protocol.id,
endpoint: routeInput.endpoint,
auth: routeInput.auth ?? Auth.none,
transport: routeInput.transport,
defaults: routeInput.defaults ?? {},
body: protocol.body,
with: (patch: RoutePatch<Body, Prepared>) => {
const { id, provider, auth, transport, endpoint, ...defaults } = patch
return build({
...routeInput,
id: id ?? routeInput.id,
provider: provider ?? routeInput.provider,
auth: auth ?? routeInput.auth,
endpoint: endpoint ? Endpoint.merge(routeInput.endpoint, endpoint) : routeInput.endpoint,
transport: (transport as Transport<Body, Prepared, Frame> | undefined) ?? routeInput.transport,
defaults: mergeRouteDefaults(route.defaults, defaults),
})
},
model: (input) => makeRouteModel(route, input),
prepareTransport: (body, request) =>
routeInput.transport.prepare({
body,
request,
endpoint: routeInput.endpoint,
auth: routeInput.auth ?? Auth.none,
encodeBody,
headers: routeInput.headers,
}),
streamPrepared: (prepared: Prepared, request: LLMRequest, runtime: TransportRuntime) => {
const route = `${request.model.provider}/${request.model.route.id}`
const events = routeInput.transport
.frames(prepared, request, runtime)
.pipe(
Stream.mapEffect(decodeEvent(route)),
protocol.stream.terminal ? Stream.takeUntil(protocol.stream.terminal) : (stream) => stream,
)
return events.pipe(
Stream.mapAccumEffect(
() => protocol.stream.initial(request),
protocol.stream.step,
protocol.stream.onHalt ? { onHalt: protocol.stream.onHalt } : undefined,
),
Stream.catchCause((cause) => Stream.fail(streamError(route, `Failed to read ${route} stream`, cause))),
)
},
} satisfies Route<Body, Prepared>
return route
}
return build({ ...input, defaults: mergeRouteDefaults(undefined, input.defaults ?? {}) })
}
export function make<Body, Prepared, Frame, Event, State>(
input: MakeTransportInput<Body, Prepared, Frame, Event, State>,
): Route<Body, Prepared>
/**
* Build a `Route` by composing the four orthogonal pieces of a deployment:
*
* - `Protocol` — what is the API I'm speaking?
* - `Endpoint` — where do I send the request?
* - `Auth` — how do I authenticate it?
* - `Framing` — how do I cut the response stream into protocol frames?
*
* Plus optional `headers` for cross-cutting deployment concerns (provider
* version pins, per-deployment quirks).
*
* This is the canonical route constructor. If a new route does not fit
* this four-axis model, add a purpose-built constructor rather than widening
* the public surface preemptively.
*/
export function make<Body, Frame, Event, State>(
input: MakeInput<Body, Frame, Event, State>,
): Route<Body, HttpTransport.HttpPrepared<Frame>>
export function make<Body, Prepared, Frame, Event, State>(
input: MakeInput<Body, Frame, Event, State> | MakeTransportInput<Body, Prepared, Frame, Event, State>,
): Route<Body, Prepared> | Route<Body, HttpTransport.HttpPrepared<Frame>> {
if ("transport" in input) return makeFromTransport(input)
const protocol = input.protocol
return makeFromTransport({
id: input.id,
provider: input.provider,
protocol,
endpoint: input.endpoint,
auth: input.auth,
headers: input.headers,
transport: HttpTransport.httpJson({ framing: input.framing }),
defaults: input.defaults,
})
}
// `compile` is the important boundary: it turns a common `LLMRequest` into a
// validated provider body plus transport-private prepared data, but does not
// execute transport.
const compile = Effect.fn("LLM.compile")(function* (request: LLMRequest) {
const resolved = applyCachePolicy(resolveRequestOptions(request))
const route = resolved.model.route
const body = yield* route.body
.from(resolved)
.pipe(Effect.flatMap(ProviderShared.validateWith(Schema.decodeUnknownEffect(route.body.schema))))
const prepared = yield* route.prepareTransport(body, resolved)
return {
request: resolved,
route,
body,
prepared,
}
})
const prepareWith = Effect.fn("LLMClient.prepare")(function* (request: LLMRequest) {
const compiled = yield* compile(request)
return new PreparedRequest({
id: compiled.request.id ?? "request",
route: compiled.route.id,
protocol: compiled.route.protocol,
model: compiled.request.model,
body: compiled.body,
metadata: { transport: compiled.route.transport.id },
})
})
const streamRequestWith = (runtime: TransportRuntime) => (request: LLMRequest) =>
Stream.unwrap(
Effect.gen(function* () {
const compiled = yield* compile(request)
return compiled.route.streamPrepared(compiled.prepared, compiled.request, runtime)
}),
)
const generateWith = (stream: Interface["stream"]) =>
Effect.fn("LLM.generate")(function* (request: LLMRequest) {
return new LLMResponse(
yield* stream(request).pipe(
Stream.runFold(
() => ({ events: [] as LLMEvent[], usage: undefined as LLMResponse["usage"] }),
(acc, event) => {
acc.events.push(event)
if ("usage" in event && event.usage !== undefined) acc.usage = event.usage
return acc
},
),
),
)
})
export const prepare = <Body = unknown>(request: LLMRequest) =>
prepareWith(request) as Effect.Effect<PreparedRequestOf<Body>, LLMError>
export function stream(request: LLMRequest): Stream.Stream<LLMEvent, LLMError> {
return Stream.unwrap(
Effect.gen(function* () {
return (yield* Service).stream(request)
}),
) as Stream.Stream<LLMEvent, LLMError>
}
export function generate(request: LLMRequest): Effect.Effect<LLMResponse, LLMError> {
return Effect.gen(function* () {
return yield* (yield* Service).generate(request)
}) as Effect.Effect<LLMResponse, LLMError>
}
export const streamRequest = (request: LLMRequest) =>
Stream.unwrap(
Effect.gen(function* () {
return (yield* Service).stream(request)
}),
)
export const layer: Layer.Layer<Service, never, RequestExecutor.Service> = Layer.effect(
Service,
Effect.gen(function* () {
const stream = streamRequestWith({
http: yield* RequestExecutor.Service,
webSocket: Option.getOrUndefined(yield* Effect.serviceOption(WebSocketExecutor.Service)),
})
return Service.of({ prepare: prepareWith as Interface["prepare"], stream, generate: generateWith(stream) })
}),
)
export const Route = { make } as const
export const LLMClient = {
Service,
layer,
prepare,
stream,
generate,
} as const

53
packages/llm/src/route/endpoint.ts Normal file
View File

@@ -0,0 +1,53 @@
import type { LLMRequest } from "../schema"
import * as ProviderShared from "../protocols/shared"
export interface EndpointInput<Body> {
readonly request: LLMRequest
readonly body: Body
}
export type EndpointPart<Body> = string | ((input: EndpointInput<Body>) => string)
/**
* Declarative URL construction for one route.
*
* `Endpoint` carries URL construction for one route. Routes with a canonical
* host put `baseURL` here; provider helpers can override it by configuring the
* route before selecting a model.
*
* `path` may be a string or a function of `EndpointInput`, for routes whose
* URL embeds the model id, region, or another body field (e.g. Bedrock,
* Gemini).
*/
export interface Endpoint<Body> {
readonly baseURL?: string
readonly path: EndpointPart<Body>
readonly query?: Record<string, string>
}
export type EndpointPatch<Body> = Partial<Endpoint<Body>>
/** Construct an `Endpoint` from a path string or path function. */
export const path = <Body>(value: EndpointPart<Body>, options: Omit<Endpoint<Body>, "path"> = {}): Endpoint<Body> => ({
...options,
path: value,
})
export const merge = <Body>(base: Endpoint<Body>, patch: EndpointPatch<Body>): Endpoint<Body> => ({
...base,
...patch,
baseURL: patch.baseURL ?? base.baseURL,
path: patch.path ?? base.path,
query: patch.query === undefined ? base.query : { ...base.query, ...patch.query },
})
const renderPart = <Body>(part: EndpointPart<Body>, input: EndpointInput<Body>) =>
typeof part === "function" ? part(input) : part
export const render = <Body>(endpoint: Endpoint<Body>, input: EndpointInput<Body>) => {
const url = new URL(`${ProviderShared.trimBaseUrl(endpoint.baseURL ?? "")}${renderPart(endpoint.path, input)}`)
for (const [key, value] of Object.entries(endpoint.query ?? {})) url.searchParams.set(key, value)
return url
}
export * as Endpoint from "./endpoint"

385
packages/llm/src/route/executor.ts Normal file
View File

@@ -0,0 +1,385 @@
import { Cause, Context, Effect, Layer, Random } from "effect"
import {
FetchHttpClient,
Headers,
HttpClient,
HttpClientError,
HttpClientRequest,
HttpClientResponse,
} from "effect/unstable/http"
import {
AuthenticationReason,
ContentPolicyReason,
HttpContext,
HttpRateLimitDetails,
HttpRequestDetails,
HttpResponseDetails,
InvalidRequestReason,
LLMError,
ProviderInternalReason,
QuotaExceededReason,
RateLimitReason,
TransportReason,
UnknownProviderReason,
} from "../schema"
import { isContextOverflow } from "../provider-error"
export interface Interface {
readonly execute: (
request: HttpClientRequest.HttpClientRequest,
) => Effect.Effect<HttpClientResponse.HttpClientResponse, LLMError>
}
export class Service extends Context.Service<Service, Interface>()("@opencode/LLM/RequestExecutor") {}
const BODY_LIMIT = 16_384
const MAX_RETRIES = 2
const BASE_DELAY_MS = 500
const MAX_DELAY_MS = 10_000
const REDACTED = "<redacted>"
// One source of truth for what counts as a sensitive name across headers,
// URL query keys, and field names embedded inside request/response bodies.
//
// `SENSITIVE_NAME` is used as both a substring matcher (for free-form header
// names like `Authorization` / `X-API-Key`) and as the body-field alternation
// list. `SHORT_QUERY_NAME` covers anchored short keys like `?key=…` / `?sig=…`
// that are too generic to redact substring-style without false positives.
const SENSITIVE_NAME_SOURCE =
"authorization|api[-_]?key|access[-_]?token|refresh[-_]?token|id[-_]?token|token|secret|credential|signature|x-amz-signature"
const SENSITIVE_NAME = new RegExp(SENSITIVE_NAME_SOURCE, "i")
const SHORT_QUERY_NAME = /^(key|sig)$/i
const SENSITIVE_BODY_FIELD = new RegExp(`(?:${SENSITIVE_NAME_SOURCE}|key)`, "i")
const REDACT_JSON_FIELD = new RegExp(`("(?:${SENSITIVE_BODY_FIELD.source})"\\s*:\\s*)"[^"]*"`, "gi")
const REDACT_QUERY_FIELD = new RegExp(`((?:${SENSITIVE_BODY_FIELD.source})=)[^&\\s"]+`, "gi")
const isSensitiveHeaderName = (name: string) => SENSITIVE_NAME.test(name)
const isSensitiveQueryName = (name: string) => isSensitiveHeaderName(name) || SHORT_QUERY_NAME.test(name)
const redactHeaders = (headers: Headers.Headers, redactedNames: ReadonlyArray<string | RegExp>) =>
Object.fromEntries(
Object.entries(Headers.redact(headers, [...redactedNames, SENSITIVE_NAME])).map(([name, value]) => [
name,
String(value),
]),
)
const redactUrl = (value: string) => {
if (!URL.canParse(value)) return REDACTED
const url = new URL(value)
url.searchParams.forEach((_, key) => {
if (isSensitiveQueryName(key)) url.searchParams.set(key, REDACTED)
})
return url.toString()
}
const normalizedHeaders = (headers: Headers.Headers) =>
Object.fromEntries(Object.entries(headers).map(([key, value]) => [key.toLowerCase(), value]))
const requestId = (headers: Record<string, string>) => {
return (
headers["x-request-id"] ??
headers["request-id"] ??
headers["x-amzn-requestid"] ??
headers["x-amz-request-id"] ??
headers["x-goog-request-id"] ??
headers["cf-ray"]
)
}
const retryableStatus = (status: number) => status === 429 || status === 503 || status === 504 || status === 529
const retryAfterMs = (headers: Record<string, string>) => {
const millis = Number(headers["retry-after-ms"])
if (Number.isFinite(millis)) return Math.max(0, millis)
const value = headers["retry-after"]
if (!value) return undefined
const seconds = Number(value)
if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000)
const date = Date.parse(value)
if (!Number.isNaN(date)) return Math.max(0, date - Date.now())
return undefined
}
const addRateLimitValue = (target: Record<string, string>, key: string, value: string) => {
if (key.length > 0) target[key] = value
}
const rateLimitDetails = (headers: Record<string, string>, retryAfter: number | undefined) => {
const limit: Record<string, string> = {}
const remaining: Record<string, string> = {}
const reset: Record<string, string> = {}
Object.entries(headers).forEach(([name, value]) => {
const openaiLimit = /^x-ratelimit-limit-(.+)$/.exec(name)?.[1]
if (openaiLimit) return addRateLimitValue(limit, openaiLimit, value)
const openaiRemaining = /^x-ratelimit-remaining-(.+)$/.exec(name)?.[1]
if (openaiRemaining) return addRateLimitValue(remaining, openaiRemaining, value)
const openaiReset = /^x-ratelimit-reset-(.+)$/.exec(name)?.[1]
if (openaiReset) return addRateLimitValue(reset, openaiReset, value)
const anthropic = /^anthropic-ratelimit-(.+)-(limit|remaining|reset)$/.exec(name)
if (!anthropic) return
if (anthropic[2] === "limit") return addRateLimitValue(limit, anthropic[1], value)
if (anthropic[2] === "remaining") return addRateLimitValue(remaining, anthropic[1], value)
return addRateLimitValue(reset, anthropic[1], value)
})
if (
retryAfter === undefined &&
Object.keys(limit).length === 0 &&
Object.keys(remaining).length === 0 &&
Object.keys(reset).length === 0
)
return undefined
return new HttpRateLimitDetails({
retryAfterMs: retryAfter,
limit: Object.keys(limit).length === 0 ? undefined : limit,
remaining: Object.keys(remaining).length === 0 ? undefined : remaining,
reset: Object.keys(reset).length === 0 ? undefined : reset,
})
}
const requestDetails = (request: HttpClientRequest.HttpClientRequest, redactedNames: ReadonlyArray<string | RegExp>) =>
new HttpRequestDetails({
method: request.method,
url: redactUrl(request.url),
headers: redactHeaders(request.headers, redactedNames),
})
const responseDetails = (
response: HttpClientResponse.HttpClientResponse,
redactedNames: ReadonlyArray<string | RegExp>,
) =>
new HttpResponseDetails({
status: response.status,
headers: redactHeaders(response.headers, redactedNames),
})
const secretValues = (request: HttpClientRequest.HttpClientRequest) => {
const values = new Set<string>()
const add = (value: string) => {
if (value.length < 4) return
values.add(value)
values.add(encodeURIComponent(value))
}
Object.entries(request.headers).forEach(([name, value]) => {
if (!isSensitiveHeaderName(name)) return
add(value)
const bearer = /^Bearer\s+(.+)$/i.exec(value)?.[1]
if (bearer) add(bearer)
})
if (!URL.canParse(request.url)) return values
new URL(request.url).searchParams.forEach((value, key) => {
if (isSensitiveQueryName(key)) add(value)
})
return values
}
// Two passes: structural (redact `"name": "value"` and `name=value` patterns
// for any field name that looks sensitive) plus literal (replace any actual
// secret values we sent in the request, in case the response echoes one back).
const redactBody = (body: string, request: HttpClientRequest.HttpClientRequest) =>
Array.from(secretValues(request)).reduce(
(text, secret) => text.split(secret).join(REDACTED),
body.replace(REDACT_JSON_FIELD, `$1"${REDACTED}"`).replace(REDACT_QUERY_FIELD, `$1${REDACTED}`),
)
const responseBody = (body: string | void, request: HttpClientRequest.HttpClientRequest) => {
if (body === undefined) return {}
const redacted = redactBody(body, request)
if (redacted.length <= BODY_LIMIT) return { body: redacted }
return { body: redacted.slice(0, BODY_LIMIT), bodyTruncated: true }
}
const providerMessage = (status: number, body: { readonly body?: string }) => {
if (body.body && body.body.length <= 500) return `Provider request failed with HTTP ${status}: ${body.body}`
return `Provider request failed with HTTP ${status}`
}
const responseHttp = (input: {
readonly request: HttpClientRequest.HttpClientRequest
readonly response: HttpClientResponse.HttpClientResponse
readonly redactedNames: ReadonlyArray<string | RegExp>
readonly body: ReturnType<typeof responseBody>
readonly requestId?: string | undefined
readonly rateLimit?: HttpRateLimitDetails | undefined
}) =>
new HttpContext({
request: requestDetails(input.request, input.redactedNames),
response: responseDetails(input.response, input.redactedNames),
...input.body,
requestId: input.requestId,
rateLimit: input.rateLimit,
})
const statusReason = (input: {
readonly status: number
readonly message: string
readonly retryAfterMs?: number | undefined
readonly rateLimit?: HttpRateLimitDetails | undefined
readonly http: HttpContext
}) => {
const body = input.http.body ?? ""
if (/content[-_\s]?policy|content_filter|safety/i.test(body)) {
return new ContentPolicyReason({ message: input.message, http: input.http })
}
if (input.status === 401) {
return new AuthenticationReason({ message: input.message, kind: "invalid", http: input.http })
}
if (input.status === 403) {
return new AuthenticationReason({ message: input.message, kind: "insufficient-permissions", http: input.http })
}
if (input.status === 429) {
if (/insufficient[-_\s]?quota|quota[-_\s]?exceeded/i.test(body)) {
return new QuotaExceededReason({ message: input.message, http: input.http })
}
return new RateLimitReason({
message: input.message,
retryAfterMs: input.retryAfterMs,
rateLimit: input.rateLimit,
http: input.http,
})
}
if (
input.status === 400 ||
input.status === 404 ||
input.status === 409 ||
input.status === 413 ||
input.status === 422
) {
return new InvalidRequestReason({
message: input.message,
classification: isContextOverflow(body) ? "context-overflow" : undefined,
http: input.http,
})
}
if (input.status >= 500 || retryableStatus(input.status)) {
return new ProviderInternalReason({
message: input.message,
status: input.status,
retryAfterMs: input.retryAfterMs,
http: input.http,
})
}
return new UnknownProviderReason({ message: input.message, status: input.status, http: input.http })
}
const statusError =
(request: HttpClientRequest.HttpClientRequest, redactedNames: ReadonlyArray<string | RegExp>) =>
(response: HttpClientResponse.HttpClientResponse) =>
Effect.gen(function* () {
if (response.status < 400) return response
const body = yield* response.text.pipe(Effect.catch(() => Effect.void))
const headers = normalizedHeaders(response.headers)
const retryAfter = retryAfterMs(headers)
const rateLimit = rateLimitDetails(headers, retryAfter)
const details = responseBody(body, request)
return yield* new LLMError({
module: "RequestExecutor",
method: "execute",
reason: statusReason({
status: response.status,
message: providerMessage(response.status, details),
retryAfterMs: retryAfter,
rateLimit,
http: responseHttp({
request,
response,
redactedNames,
body: details,
requestId: requestId(headers),
rateLimit,
}),
}),
})
})
const toHttpError = (redactedNames: ReadonlyArray<string | RegExp>) => (error: unknown) => {
const transportError = (input: {
readonly message: string
readonly kind?: string | undefined
readonly request?: HttpClientRequest.HttpClientRequest | undefined
}) =>
new LLMError({
module: "RequestExecutor",
method: "execute",
reason: new TransportReason({
message: input.message,
kind: input.kind,
url: input.request ? redactUrl(input.request.url) : undefined,
http: input.request ? new HttpContext({ request: requestDetails(input.request, redactedNames) }) : undefined,
}),
})
if (Cause.isTimeoutError(error)) {
return transportError({ message: error.message, kind: "Timeout" })
}
if (!HttpClientError.isHttpClientError(error)) {
return transportError({ message: "HTTP transport failed" })
}
const request = "request" in error ? error.request : undefined
if (error.reason._tag === "TransportError") {
return transportError({
message: error.reason.description ?? "HTTP transport failed",
kind: error.reason._tag,
request,
})
}
return transportError({
message: `HTTP transport failed: ${error.reason._tag}`,
kind: error.reason._tag,
request,
})
}
const retryDelay = (error: LLMError, attempt: number) => {
if (error.retryAfterMs !== undefined) return Effect.succeed(Math.min(error.retryAfterMs, MAX_DELAY_MS))
return Random.nextBetween(
Math.min(BASE_DELAY_MS * 2 ** attempt * 0.8, MAX_DELAY_MS),
Math.min(BASE_DELAY_MS * 2 ** attempt * 1.2, MAX_DELAY_MS),
).pipe(Effect.map((delay) => Math.round(delay)))
}
const retryStatusFailures = <A, R>(
effect: Effect.Effect<A, LLMError, R>,
retries = MAX_RETRIES,
attempt = 0,
): Effect.Effect<A, LLMError, R> =>
Effect.catchTag(effect, "LLM.Error", (error): Effect.Effect<A, LLMError, R> => {
if (!error.retryable || retries <= 0) return Effect.fail(error)
return retryDelay(error, attempt).pipe(
Effect.flatMap((delay) => Effect.sleep(delay)),
Effect.flatMap(() => retryStatusFailures(effect, retries - 1, attempt + 1)),
)
})
export const layer: Layer.Layer<Service, never, HttpClient.HttpClient> = Layer.effect(
Service,
Effect.gen(function* () {
const http = yield* HttpClient.HttpClient
const executeOnce = (request: HttpClientRequest.HttpClientRequest) =>
Effect.gen(function* () {
const redactedNames = yield* Headers.CurrentRedactedNames
return yield* http
.execute(request)
.pipe(Effect.mapError(toHttpError(redactedNames)), Effect.flatMap(statusError(request, redactedNames)))
})
return Service.of({
execute: (request) => retryStatusFailures(executeOnce(request)),
})
}),
)
export const defaultLayer = layer.pipe(Layer.provide(FetchHttpClient.layer))
export * as RequestExecutor from "./executor"

27
packages/llm/src/route/framing.ts Normal file
View File

@@ -0,0 +1,27 @@
import type { Stream } from "effect"
import * as ProviderShared from "../protocols/shared"
import type { LLMError } from "../schema"
/**
* Decode a streaming HTTP response body into provider-protocol frames.
*
* `Framing` is the byte-stream-shaped seam between transport and protocol:
*
* - SSE (`Framing.sse`) — UTF-8 decode the body, run the SSE channel decoder,
* drop empty / `[DONE]` keep-alives. Each emitted frame is the JSON `data:`
* payload of one event.
* - AWS event stream — length-prefixed binary frames with CRC checksums.
* Each emitted frame is one parsed binary event record.
*
* The frame type is opaque to this layer; the protocol's `decode` step turns
* a frame into a typed chunk.
*/
export interface Framing<Frame> {
readonly id: string
readonly frame: (bytes: Stream.Stream<Uint8Array, LLMError>) => Stream.Stream<Frame, LLMError>
}
/** Server-Sent Events framing. Used by every JSON-streaming HTTP provider. */
export const sse: Framing<string> = { id: "sse", frame: ProviderShared.sseFraming }
export * as Framing from "./framing"

25
packages/llm/src/route/index.ts Normal file
View File

@@ -0,0 +1,25 @@
export { Route, LLMClient } from "./client"
export type {
Route as RouteShape,
RouteModelInput,
RouteRoutedModelInput,
RouteDefaults,
RouteDefaultsInput,
AnyRoute,
Interface as LLMClientShape,
Service as LLMClientService,
} from "./client"
export * from "./executor"
export { Auth } from "./auth"
export { AuthOptions } from "./auth-options"
export { Endpoint } from "./endpoint"
export { Framing } from "./framing"
export { Protocol } from "./protocol"
export { HttpTransport, WebSocketExecutor, WebSocketTransport } from "./transport"
export * as Transport from "./transport"
export type { Auth as AuthShape, AuthInput, Credential, CredentialError } from "./auth"
export type { ApiKeyMode, AuthOverride, ProviderAuthOption } from "./auth-options"
export type { Endpoint as EndpointFn, EndpointInput } from "./endpoint"
export type { Framing as FramingDef } from "./framing"
export type { Protocol as ProtocolDef } from "./protocol"
export type { Transport as TransportDef, TransportRuntime } from "./transport"

84
packages/llm/src/route/protocol.ts Normal file
View File

@@ -0,0 +1,84 @@
import { Schema, type Effect } from "effect"
import type { LLMError, LLMEvent, LLMRequest, ProtocolID } from "../schema"
/**
* The semantic API contract of one model server family.
*
* A `Protocol` owns the parts of a route that are intrinsic to "what does
* this API look like": how a common `LLMRequest` becomes a provider-native
* body, what schema that body must satisfy before it is JSON-encoded, and
* how the streaming response decodes back into common `LLMEvent`s.
*
* Examples:
*
* - `OpenAIChat.protocol` — chat completions style
* - `OpenAIResponses.protocol` — responses API
* - `AnthropicMessages.protocol` — messages API with content blocks
* - `Gemini.protocol` — generateContent
* - `BedrockConverse.protocol` — Converse with binary event-stream framing
*
* A `Protocol` is **not** a deployment. It does not know which URL, which
* headers, or which auth scheme to use. Those are deployment concerns owned
* by `Route.make(...)` along with the chosen `Endpoint`, `Auth`,
* and `Framing`. This separation is what lets DeepSeek, TogetherAI, Cerebras,
* etc. all reuse `OpenAIChat.protocol` without forking 300 lines per provider.
*
* The four type parameters reflect the pipeline:
*
* - `Body` — provider-native request body candidate. `Route.make(...)`
* validates and JSON-encodes it with `body.schema`.
* - `Frame` — one unit of the framed response stream. SSE: a JSON data
* string. AWS event stream: a parsed binary frame.
* - `Event` — schema-decoded provider event produced from one frame.
* - `State` — accumulator threaded through `stream.step` to translate event
* sequences into `LLMEvent` sequences.
*/
export interface Protocol<Body, Frame, Event, State> {
/** Stable id for the wire protocol implementation. */
readonly id: ProtocolID
/** Request side: schema for the provider-native body and how to build it. */
readonly body: ProtocolBody<Body>
/** Response side: streaming state machine. */
readonly stream: ProtocolStream<Frame, Event, State>
}
export interface ProtocolBody<Body> {
/** Schema for the validated provider-native body sent as the JSON request. */
readonly schema: Schema.Codec<Body, unknown>
/** Build the provider-native body from a common `LLMRequest`. */
readonly from: (request: LLMRequest) => Effect.Effect<Body, LLMError>
}
export interface ProtocolStream<Frame, Event, State> {
/** Schema for one decoded streaming event, decoded from a transport frame. */
readonly event: Schema.Codec<Event, Frame>
/** Initial parser state. Called once per response with the resolved request. */
readonly initial: (request: LLMRequest) => State
/** Translate one event into emitted `LLMEvent`s plus the next state. */
readonly step: (state: State, event: Event) => Effect.Effect<readonly [State, ReadonlyArray<LLMEvent>], LLMError>
/** Optional request-completion signal for transports that do not end naturally. */
readonly terminal?: (event: Event) => boolean
/** Optional flush emitted when the framed stream ends. */
readonly onHalt?: (state: State) => ReadonlyArray<LLMEvent>
}
/**
* Construct a `Protocol` from its body and stream pieces:
*
* - `body.schema` infers the provider-native request body shape.
* - `body.from` ties the common `LLMRequest` to the provider body.
* - `stream.event` infers the decoded streaming event and the wire frame.
* - `stream.initial`, `stream.step`, and `stream.onHalt` infer the parser state.
*
* Provider implementations should usually call `Protocol.make({ ... })`
* without explicit type arguments; the schemas and parser functions are the
* source of truth. The constructor remains as the public seam for future
* cross-cutting concerns such as tracing or instrumentation.
*/
export const make = <Body, Frame, Event, State>(
input: Protocol<Body, Frame, Event, State>,
): Protocol<Body, Frame, Event, State> => input
export const jsonEvent = <const S extends Schema.Top>(schema: S) => Schema.fromJsonString(schema)
export * as Protocol from "./protocol"

108
packages/llm/src/route/transport/http.ts Normal file
View File

@@ -0,0 +1,108 @@
import { Effect, Stream } from "effect"
import { Headers, HttpClientRequest } from "effect/unstable/http"
import { Auth } from "../auth"
import { render as renderEndpoint } from "../endpoint"
import { Framing, type Framing as FramingDef } from "../framing"
import type { Transport, TransportPrepareInput } from "./index"
import * as ProviderShared from "../../protocols/shared"
import { mergeJsonRecords, type LLMRequest } from "../../schema"
export type JsonRequestInput<Body> = TransportPrepareInput<Body>
export interface JsonRequestParts<Body = unknown> {
readonly url: string
readonly jsonBody: Body | Record<string, unknown>
readonly bodyText: string
readonly headers: Headers.Headers
}
export interface HttpPrepared<Frame> {
readonly request: HttpClientRequest.HttpClientRequest
readonly framing: FramingDef<Frame>
}
const applyQuery = (url: string, query: Record<string, string> | undefined) => {
if (!query) return url
const next = new URL(url)
Object.entries(query).forEach(([key, value]) => next.searchParams.set(key, value))
return next.toString()
}
const bodyWithOverlay = <Body>(body: Body, request: LLMRequest, encodeBody: (body: Body) => string) =>
Effect.gen(function* () {
if (request.http?.body === undefined) return { jsonBody: body, bodyText: encodeBody(body) }
if (ProviderShared.isRecord(body)) {
const overlaid = mergeJsonRecords(body, request.http.body) ?? {}
return { jsonBody: overlaid, bodyText: ProviderShared.encodeJson(overlaid) }
}
return yield* ProviderShared.invalidRequest("http.body can only overlay JSON object request bodies")
})
export const jsonRequestParts = <Body>(input: JsonRequestInput<Body>) =>
Effect.gen(function* () {
const url = applyQuery(
renderEndpoint(input.endpoint, { request: input.request, body: input.body }).toString(),
input.request.http?.query,
)
const body = yield* bodyWithOverlay(input.body, input.request, input.encodeBody)
const headers = yield* Auth.toEffect(input.auth)({
request: input.request,
method: "POST",
url,
body: body.bodyText,
headers: Headers.fromInput({
...input.headers?.({ request: input.request }),
...input.request.http?.headers,
}),
})
return { url, jsonBody: body.jsonBody, bodyText: body.bodyText, headers }
})
export interface HttpJsonInput<_Body, Frame> {
readonly framing: FramingDef<Frame>
}
export type HttpJsonPatch<Body, Frame> = Partial<HttpJsonInput<Body, Frame>>
export interface HttpJsonTransport<Body, Frame> extends Transport<Body, HttpPrepared<Frame>, Frame> {
readonly with: (patch: HttpJsonPatch<Body, Frame>) => HttpJsonTransport<Body, Frame>
}
export const httpJson = <Body, Frame>(input: HttpJsonInput<Body, Frame>): HttpJsonTransport<Body, Frame> => ({
id: "http-json",
with: (patch) => httpJson({ ...input, ...patch }),
prepare: (prepareInput) =>
jsonRequestParts({
...prepareInput,
}).pipe(
Effect.map((parts) => ({
request: ProviderShared.jsonPost({ url: parts.url, body: parts.bodyText, headers: parts.headers }),
framing: input.framing,
})),
),
frames: (prepared, request, runtime) =>
Stream.unwrap(
runtime.http
.execute(prepared.request)
.pipe(
Effect.map((response) =>
prepared.framing.frame(
response.stream.pipe(
Stream.mapError((error) =>
ProviderShared.eventError(
`${request.model.provider}/${request.model.route.id}`,
`Failed to read ${request.model.provider}/${request.model.route.id} stream`,
ProviderShared.errorText(error),
),
),
),
),
),
),
),
})
export const sseJson = {
id: "http-json/sse",
with: <Body>() => httpJson<Body, string>({ framing: Framing.sse }),
} as const

33
packages/llm/src/route/transport/index.ts Normal file
View File

@@ -0,0 +1,33 @@
import type { Effect, Stream } from "effect"
import type { Endpoint } from "../endpoint"
import type { Auth } from "../auth"
import type { Interface as RequestExecutorInterface } from "../executor"
import type { Interface as WebSocketExecutorInterface } from "./websocket"
import type { LLMError, LLMRequest } from "../../schema"
export interface TransportRuntime {
readonly http: RequestExecutorInterface
readonly webSocket?: WebSocketExecutorInterface
}
export interface Transport<Body, Prepared, Frame> {
readonly id: string
readonly prepare: (input: TransportPrepareInput<Body>) => Effect.Effect<Prepared, LLMError>
readonly frames: (
prepared: Prepared,
request: LLMRequest,
runtime: TransportRuntime,
) => Stream.Stream<Frame, LLMError>
}
export interface TransportPrepareInput<Body> {
readonly body: Body
readonly request: LLMRequest
readonly endpoint: Endpoint<Body>
readonly auth: Auth
readonly encodeBody: (body: Body) => string
readonly headers?: (input: { readonly request: LLMRequest }) => Record<string, string>
}
export * as HttpTransport from "./http"
export { WebSocketExecutor, WebSocketTransport } from "./websocket"

280
packages/llm/src/route/transport/websocket.ts Normal file
View File

@@ -0,0 +1,280 @@
import { Cause, Context, Effect, Layer, Queue, Stream } from "effect"
import { Headers } from "effect/unstable/http"
import { LLMError, TransportReason } from "../../schema"
import * as HttpTransport from "./http"
import type { Transport } from "./index"
export interface WebSocketRequest {
readonly url: string
readonly headers: Headers.Headers
}
export interface WebSocketConnection {
readonly sendText: (message: string) => Effect.Effect<void, LLMError>
readonly messages: Stream.Stream<string | Uint8Array, LLMError>
readonly close: Effect.Effect<void, never>
}
export interface Interface {
readonly open: (input: WebSocketRequest) => Effect.Effect<WebSocketConnection, LLMError>
}
type WebSocketConstructorWithHeaders = new (
url: string,
options?: { readonly headers?: Headers.Headers },
) => globalThis.WebSocket
export class Service extends Context.Service<Service, Interface>()("@opencode/LLM/WebSocketExecutor") {}
const transportError = (
method: string,
message: string,
input: { readonly url?: string; readonly kind?: string } = {},
) =>
new LLMError({
module: "WebSocketExecutor",
method,
reason: new TransportReason({ message, url: input.url, kind: input.kind }),
})
const eventMessage = (event: Event) => {
if ("message" in event && typeof event.message === "string") return event.message
return event.type
}
const binaryMessage = (data: unknown) => {
if (data instanceof Uint8Array) return data
if (data instanceof ArrayBuffer) return new Uint8Array(data)
if (ArrayBuffer.isView(data)) return new Uint8Array(data.buffer, data.byteOffset, data.byteLength)
return undefined
}
const waitOpen = (ws: globalThis.WebSocket, input: WebSocketRequest) => {
if (ws.readyState === globalThis.WebSocket.OPEN) return Effect.void
if (ws.readyState === globalThis.WebSocket.CLOSING || ws.readyState === globalThis.WebSocket.CLOSED) {
return Effect.fail(
transportError("open", `WebSocket closed before opening (state ${ws.readyState})`, {
url: input.url,
kind: "open",
}),
)
}
return Effect.callback<void, LLMError>((resume, signal) => {
const cleanup = () => {
ws.removeEventListener("open", onOpen)
ws.removeEventListener("error", onError)
ws.removeEventListener("close", onClose)
signal.removeEventListener("abort", onAbort)
}
const onAbort = () => {
cleanup()
if (ws.readyState !== globalThis.WebSocket.CLOSED && ws.readyState !== globalThis.WebSocket.CLOSING)
ws.close(1000)
}
const onOpen = () => {
cleanup()
resume(Effect.void)
}
const onError = (event: Event) => {
cleanup()
resume(
Effect.fail(
transportError("open", `Failed to open WebSocket: ${eventMessage(event)}`, { url: input.url, kind: "open" }),
),
)
}
const onClose = (event: CloseEvent) => {
cleanup()
resume(
Effect.fail(
transportError("open", `WebSocket closed before opening with code ${event.code}`, {
url: input.url,
kind: "open",
}),
),
)
}
ws.addEventListener("open", onOpen, { once: true })
ws.addEventListener("error", onError, { once: true })
ws.addEventListener("close", onClose, { once: true })
signal.addEventListener("abort", onAbort, { once: true })
})
}
const webSocketUrl = (value: string) =>
Effect.try({
try: () => {
const url = new URL(value)
if (url.protocol === "https:") {
url.protocol = "wss:"
return url.toString()
}
if (url.protocol === "http:") {
url.protocol = "ws:"
return url.toString()
}
throw new Error(`Unsupported WebSocket URL protocol ${url.protocol}`)
},
catch: (error) =>
transportError("prepare", error instanceof Error ? error.message : "Invalid WebSocket URL", {
url: value,
kind: "websocket",
}),
})
export const open = (input: WebSocketRequest) =>
Effect.try({
try: () =>
new (globalThis.WebSocket as unknown as WebSocketConstructorWithHeaders)(input.url, { headers: input.headers }),
catch: (error) =>
transportError("open", error instanceof Error ? error.message : "Failed to construct WebSocket", {
url: input.url,
kind: "open",
}),
}).pipe(Effect.flatMap((ws) => fromWebSocket(ws, input)))
export const layer: Layer.Layer<Service> = Layer.succeed(Service, Service.of({ open }))
export const fromWebSocket = (
ws: globalThis.WebSocket,
input: WebSocketRequest,
): Effect.Effect<WebSocketConnection, LLMError> =>
Effect.gen(function* () {
yield* waitOpen(ws, input)
const messages = yield* Queue.bounded<string | Uint8Array, LLMError | Cause.Done<void>>(128)
const onMessage = (event: MessageEvent) => {
if (typeof event.data === "string") return Queue.offerUnsafe(messages, event.data)
const binary = binaryMessage(event.data)
if (binary) return Queue.offerUnsafe(messages, binary)
Queue.failCauseUnsafe(
messages,
Cause.fail(
transportError("message", "Unsupported WebSocket message payload", { url: input.url, kind: "message" }),
),
)
}
const onError = (event: Event) => {
Queue.failCauseUnsafe(
messages,
Cause.fail(
transportError("message", `WebSocket error: ${eventMessage(event)}`, { url: input.url, kind: "message" }),
),
)
}
const onClose = (event: CloseEvent) => {
if (event.code === 1000 || event.code === 1005) return Queue.endUnsafe(messages)
Queue.failCauseUnsafe(
messages,
Cause.fail(
transportError("message", `WebSocket closed with code ${event.code}`, { url: input.url, kind: "close" }),
),
)
}
const cleanup = Effect.sync(() => {
ws.removeEventListener("message", onMessage)
ws.removeEventListener("error", onError)
ws.removeEventListener("close", onClose)
}).pipe(Effect.andThen(Queue.shutdown(messages)))
ws.addEventListener("message", onMessage)
ws.addEventListener("error", onError)
ws.addEventListener("close", onClose)
return {
sendText: (message) =>
Effect.try({
try: () => ws.send(message),
catch: (error) =>
transportError("sendText", error instanceof Error ? error.message : "Failed to send WebSocket message", {
url: input.url,
kind: "write",
}),
}),
messages: Stream.fromQueue(messages),
close: cleanup.pipe(
Effect.andThen(
Effect.sync(() => {
if (ws.readyState === globalThis.WebSocket.CLOSED || ws.readyState === globalThis.WebSocket.CLOSING) return
ws.close(1000)
}),
),
),
}
})
export const messageText = (message: string | Uint8Array, decoder: TextDecoder) =>
typeof message === "string" ? message : decoder.decode(message)
export interface JsonPrepared {
readonly url: string
readonly headers: Headers.Headers
readonly message: string
}
export interface JsonInput<Body, Message> {
readonly toMessage: (body: Body | Record<string, unknown>) => Effect.Effect<Message, LLMError>
readonly encodeMessage: (message: Message) => string
}
export type JsonPatch<Body, Message> = Partial<JsonInput<Body, Message>>
export interface JsonTransport<Body, Message> extends Transport<Body, JsonPrepared, string> {
readonly with: (patch: JsonPatch<Body, Message>) => JsonTransport<Body, Message>
}
export const json = <Body, Message>(input: JsonInput<Body, Message>): JsonTransport<Body, Message> => ({
id: "websocket-json",
with: (patch) => json({ ...input, ...patch }),
prepare: (prepareInput) =>
Effect.gen(function* () {
const parts = yield* HttpTransport.jsonRequestParts({
...prepareInput,
})
return {
url: yield* webSocketUrl(parts.url),
headers: parts.headers,
message: input.encodeMessage(yield* input.toMessage(parts.jsonBody)),
}
}),
frames: (prepared, _request, runtime) => {
const webSocket = runtime.webSocket
if (!webSocket) {
return Stream.fail(
transportError("json", "WebSocket JSON transport requires WebSocketExecutor.Service", {
url: prepared.url,
kind: "websocket",
}),
)
}
const decoder = new TextDecoder()
return Stream.unwrap(
Effect.gen(function* () {
const connection = yield* Effect.acquireRelease(
webSocket.open({ url: prepared.url, headers: prepared.headers }),
(connection) => connection.close,
)
yield* connection.sendText(prepared.message)
return connection.messages.pipe(Stream.map((message) => messageText(message, decoder)))
}),
)
},
})
export const jsonTransport = {
id: "websocket-json",
with: json,
} as const
export const WebSocketExecutor = {
Service,
layer,
open,
fromWebSocket,
messageText,
} as const
export const WebSocketTransport = {
json,
jsonTransport,
} as const

207
packages/llm/src/schema/errors.ts Normal file
View File

@@ -0,0 +1,207 @@
import { Schema } from "effect"
import { ModelID, ProviderID, ProviderMetadata, RouteID } from "./ids"
export const ProviderFailureClassification = Schema.Literal("context-overflow")
export type ProviderFailureClassification = typeof ProviderFailureClassification.Type
export class HttpRequestDetails extends Schema.Class<HttpRequestDetails>("LLM.HttpRequestDetails")({
method: Schema.String,
url: Schema.String,
headers: Schema.Record(Schema.String, Schema.String),
}) {}
export class HttpResponseDetails extends Schema.Class<HttpResponseDetails>("LLM.HttpResponseDetails")({
status: Schema.Number,
headers: Schema.Record(Schema.String, Schema.String),
}) {}
export class HttpRateLimitDetails extends Schema.Class<HttpRateLimitDetails>("LLM.HttpRateLimitDetails")({
retryAfterMs: Schema.optional(Schema.Number),
limit: Schema.optional(Schema.Record(Schema.String, Schema.String)),
remaining: Schema.optional(Schema.Record(Schema.String, Schema.String)),
reset: Schema.optional(Schema.Record(Schema.String, Schema.String)),
}) {}
export class HttpContext extends Schema.Class<HttpContext>("LLM.HttpContext")({
request: HttpRequestDetails,
response: Schema.optional(HttpResponseDetails),
body: Schema.optional(Schema.String),
bodyTruncated: Schema.optional(Schema.Boolean),
requestId: Schema.optional(Schema.String),
rateLimit: Schema.optional(HttpRateLimitDetails),
}) {}
export class InvalidRequestReason extends Schema.Class<InvalidRequestReason>("LLM.Error.InvalidRequest")({
_tag: Schema.tag("InvalidRequest"),
message: Schema.String,
parameter: Schema.optional(Schema.String),
classification: Schema.optional(ProviderFailureClassification),
providerMetadata: Schema.optional(ProviderMetadata),
http: Schema.optional(HttpContext),
}) {
get retryable() {
return false
}
}
export class NoRouteReason extends Schema.Class<NoRouteReason>("LLM.Error.NoRoute")({
_tag: Schema.tag("NoRoute"),
route: RouteID,
provider: ProviderID,
model: ModelID,
}) {
get retryable() {
return false
}
get message() {
return `No LLM route for ${this.provider}/${this.model} using ${this.route}`
}
}
export class AuthenticationReason extends Schema.Class<AuthenticationReason>("LLM.Error.Authentication")({
_tag: Schema.tag("Authentication"),
message: Schema.String,
kind: Schema.Literals(["missing", "invalid", "expired", "insufficient-permissions", "unknown"]),
providerMetadata: Schema.optional(ProviderMetadata),
http: Schema.optional(HttpContext),
}) {
get retryable() {
return false
}
}
export class RateLimitReason extends Schema.Class<RateLimitReason>("LLM.Error.RateLimit")({
_tag: Schema.tag("RateLimit"),
message: Schema.String,
retryAfterMs: Schema.optional(Schema.Number),
rateLimit: Schema.optional(HttpRateLimitDetails),
providerMetadata: Schema.optional(ProviderMetadata),
http: Schema.optional(HttpContext),
}) {
get retryable() {
return true
}
}
export class QuotaExceededReason extends Schema.Class<QuotaExceededReason>("LLM.Error.QuotaExceeded")({
_tag: Schema.tag("QuotaExceeded"),
message: Schema.String,
providerMetadata: Schema.optional(ProviderMetadata),
http: Schema.optional(HttpContext),
}) {
get retryable() {
return false
}
}
export class ContentPolicyReason extends Schema.Class<ContentPolicyReason>("LLM.Error.ContentPolicy")({
_tag: Schema.tag("ContentPolicy"),
message: Schema.String,
providerMetadata: Schema.optional(ProviderMetadata),
http: Schema.optional(HttpContext),
}) {
get retryable() {
return false
}
}
export class ProviderInternalReason extends Schema.Class<ProviderInternalReason>("LLM.Error.ProviderInternal")({
_tag: Schema.tag("ProviderInternal"),
message: Schema.String,
status: Schema.Number,
retryAfterMs: Schema.optional(Schema.Number),
providerMetadata: Schema.optional(ProviderMetadata),
http: Schema.optional(HttpContext),
}) {
get retryable() {
return true
}
}
export class TransportReason extends Schema.Class<TransportReason>("LLM.Error.Transport")({
_tag: Schema.tag("Transport"),
message: Schema.String,
kind: Schema.optional(Schema.String),
url: Schema.optional(Schema.String),
http: Schema.optional(HttpContext),
}) {
get retryable() {
return false
}
}
export class InvalidProviderOutputReason extends Schema.Class<InvalidProviderOutputReason>(
"LLM.Error.InvalidProviderOutput",
)({
_tag: Schema.tag("InvalidProviderOutput"),
message: Schema.String,
route: Schema.optional(Schema.String),
raw: Schema.optional(Schema.String),
providerMetadata: Schema.optional(ProviderMetadata),
}) {
get retryable() {
return false
}
}
export class UnknownProviderReason extends Schema.Class<UnknownProviderReason>("LLM.Error.UnknownProvider")({
_tag: Schema.tag("UnknownProvider"),
message: Schema.String,
status: Schema.optional(Schema.Number),
providerMetadata: Schema.optional(ProviderMetadata),
http: Schema.optional(HttpContext),
}) {
get retryable() {
return false
}
}
export const LLMErrorReason = Schema.Union([
InvalidRequestReason,
NoRouteReason,
AuthenticationReason,
RateLimitReason,
QuotaExceededReason,
ContentPolicyReason,
ProviderInternalReason,
TransportReason,
InvalidProviderOutputReason,
UnknownProviderReason,
]).pipe(Schema.toTaggedUnion("_tag"))
export type LLMErrorReason = Schema.Schema.Type<typeof LLMErrorReason>
export class LLMError extends Schema.TaggedErrorClass<LLMError>()("LLM.Error", {
module: Schema.String,
method: Schema.String,
reason: LLMErrorReason,
}) {
override readonly cause = this.reason
get retryable() {
return this.reason.retryable
}
get retryAfterMs() {
return "retryAfterMs" in this.reason ? this.reason.retryAfterMs : undefined
}
override get message() {
return `${this.module}.${this.method}: ${this.reason.message}`
}
}
/**
* Failure type for tool execute handlers. Handlers must map their internal
* errors to this shape; the runtime catches `ToolFailure`s and surfaces them
* as `tool-error` events plus a `tool-result` of `type: "error"` so the model
* can self-correct.
*
* Anything thrown or yielded by a handler that is not a `ToolFailure` is
* treated as a defect and fails the stream.
*/
export class ToolFailure extends Schema.TaggedErrorClass<ToolFailure>()("LLM.ToolFailure", {
message: Schema.String,
error: Schema.optional(Schema.Defect),
metadata: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
}) {}

372
packages/llm/src/schema/events.ts Normal file
View File

@@ -0,0 +1,372 @@
import { Schema } from "effect"
import { ContentBlockID, FinishReason, ProtocolID, ProviderMetadata, RouteID, ToolCallID } from "./ids"
import { ModelSchema } from "./options"
import { ToolOutput, ToolResultValue } from "./messages"
import { ProviderFailureClassification } from "./errors"
/**
* Token usage reported by an LLM provider.
*
* **Inclusive totals** (match AI SDK / OpenAI / LangChain convention — a
* reader from any of those ecosystems sees the number they expect):
*
* - `inputTokens` — total prompt tokens, *including* cached reads/writes.
* - `outputTokens` — total output tokens, *including* reasoning.
* - `totalTokens` — provider-supplied total, or `inputTokens + outputTokens`.
*
* **Non-overlapping breakdown** (every field is independently meaningful;
* consumers never have to subtract):
*
* - `nonCachedInputTokens` — the "fresh" portion of the prompt.
* - `cacheReadInputTokens` — input tokens served from cache.
* - `cacheWriteInputTokens` — input tokens written to cache.
* - `reasoningTokens` — subset of `outputTokens` spent on hidden reasoning.
*
* **Invariant**: `nonCachedInputTokens + cacheReadInputTokens +
* cacheWriteInputTokens = inputTokens`, and `reasoningTokens ≤ outputTokens`.
* Each protocol mapper computes whichever side it doesn't get natively,
* with `Math.max(0, …)` clamping for defense against provider bugs. Because
* every breakdown field is stored independently, downstream consumers can
* read whatever they need (cost-by-category, context-pressure, AI-SDK-style
* inclusive total) without ever subtracting — eliminating the underflow
* class of bug where a clamped difference would silently store the wrong
* value.
*
* **Semantics by provider**:
*
* - OpenAI Chat / Responses / Gemini / Bedrock: provider reports inclusive
* `inputTokens` and an inclusive `outputTokens`; mapper subtracts to
* derive the breakdown.
* - Anthropic: provider reports the breakdown natively (`input_tokens` is
* non-cached only); mapper sums to derive the inclusive `inputTokens`.
* Anthropic does *not* break extended-thinking out of `output_tokens`, so
* `reasoningTokens` is `undefined` and `outputTokens` carries the
* combined total — a documented limitation of the Anthropic API.
*
* `providerMetadata` always carries the provider's raw usage payload —
* keyed by provider name (`{ openai: ... }`, `{ anthropic: ... }`, etc.)
* — for fields we don't normalize and for billing-level audit trails.
* Matches the same escape-hatch field on `LLMEvent`.
*/
export class Usage extends Schema.Class<Usage>("LLM.Usage")({
inputTokens: Schema.optional(Schema.Number),
outputTokens: Schema.optional(Schema.Number),
nonCachedInputTokens: Schema.optional(Schema.Number),
cacheReadInputTokens: Schema.optional(Schema.Number),
cacheWriteInputTokens: Schema.optional(Schema.Number),
reasoningTokens: Schema.optional(Schema.Number),
totalTokens: Schema.optional(Schema.Number),
providerMetadata: Schema.optional(ProviderMetadata),
}) {
/**
* Visible output tokens — `outputTokens` minus `reasoningTokens`, clamped
* to zero. The one place subtraction happens in this contract; the clamp
* means a provider reporting `reasoningTokens > outputTokens` produces a
* harmless zero rather than a negative that crashes downstream schemas.
*/
get visibleOutputTokens() {
return Math.max(0, (this.outputTokens ?? 0) - (this.reasoningTokens ?? 0))
}
static from(input: UsageInput) {
return input instanceof Usage ? input : new Usage(input)
}
}
export type UsageInput = Usage | ConstructorParameters<typeof Usage>[0]
export const StepStart = Schema.Struct({
type: Schema.tag("step-start"),
index: Schema.Number,
}).annotate({ identifier: "LLM.Event.StepStart" })
export type StepStart = Schema.Schema.Type<typeof StepStart>
export const TextStart = Schema.Struct({
type: Schema.tag("text-start"),
id: ContentBlockID,
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.TextStart" })
export type TextStart = Schema.Schema.Type<typeof TextStart>
export const TextDelta = Schema.Struct({
type: Schema.tag("text-delta"),
id: ContentBlockID,
text: Schema.String,
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.TextDelta" })
export type TextDelta = Schema.Schema.Type<typeof TextDelta>
export const TextEnd = Schema.Struct({
type: Schema.tag("text-end"),
id: ContentBlockID,
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.TextEnd" })
export type TextEnd = Schema.Schema.Type<typeof TextEnd>
export const ReasoningStart = Schema.Struct({
type: Schema.tag("reasoning-start"),
id: ContentBlockID,
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.ReasoningStart" })
export type ReasoningStart = Schema.Schema.Type<typeof ReasoningStart>
export const ReasoningDelta = Schema.Struct({
type: Schema.tag("reasoning-delta"),
id: ContentBlockID,
text: Schema.String,
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.ReasoningDelta" })
export type ReasoningDelta = Schema.Schema.Type<typeof ReasoningDelta>
export const ReasoningEnd = Schema.Struct({
type: Schema.tag("reasoning-end"),
id: ContentBlockID,
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.ReasoningEnd" })
export type ReasoningEnd = Schema.Schema.Type<typeof ReasoningEnd>
export const ToolInputStart = Schema.Struct({
type: Schema.tag("tool-input-start"),
id: ToolCallID,
name: Schema.String,
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.ToolInputStart" })
export type ToolInputStart = Schema.Schema.Type<typeof ToolInputStart>
export const ToolInputDelta = Schema.Struct({
type: Schema.tag("tool-input-delta"),
id: ToolCallID,
name: Schema.String,
text: Schema.String,
}).annotate({ identifier: "LLM.Event.ToolInputDelta" })
export type ToolInputDelta = Schema.Schema.Type<typeof ToolInputDelta>
export const ToolInputEnd = Schema.Struct({
type: Schema.tag("tool-input-end"),
id: ToolCallID,
name: Schema.String,
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.ToolInputEnd" })
export type ToolInputEnd = Schema.Schema.Type<typeof ToolInputEnd>
export const ToolCall = Schema.Struct({
type: Schema.tag("tool-call"),
id: ToolCallID,
name: Schema.String,
input: Schema.Unknown,
providerExecuted: Schema.optional(Schema.Boolean),
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.ToolCall" })
export type ToolCall = Schema.Schema.Type<typeof ToolCall>
export const ToolResult = Schema.Struct({
type: Schema.tag("tool-result"),
id: ToolCallID,
name: Schema.String,
result: ToolResultValue,
output: Schema.optional(ToolOutput),
providerExecuted: Schema.optional(Schema.Boolean),
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.ToolResult" })
export type ToolResult = Schema.Schema.Type<typeof ToolResult>
export const ToolError = Schema.Struct({
type: Schema.tag("tool-error"),
id: ToolCallID,
name: Schema.String,
message: Schema.String,
error: Schema.optional(Schema.Defect),
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.ToolError" })
export type ToolError = Schema.Schema.Type<typeof ToolError>
export const StepFinish = Schema.Struct({
type: Schema.tag("step-finish"),
index: Schema.Number,
reason: FinishReason,
usage: Schema.optional(Usage),
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.StepFinish" })
export type StepFinish = Schema.Schema.Type<typeof StepFinish>
export const Finish = Schema.Struct({
type: Schema.tag("finish"),
reason: FinishReason,
usage: Schema.optional(Usage),
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.Finish" })
export type Finish = Schema.Schema.Type<typeof Finish>
export const ProviderErrorEvent = Schema.Struct({
type: Schema.tag("provider-error"),
message: Schema.String,
classification: Schema.optional(ProviderFailureClassification),
retryable: Schema.optional(Schema.Boolean),
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.ProviderError" })
export type ProviderErrorEvent = Schema.Schema.Type<typeof ProviderErrorEvent>
const llmEventTagged = Schema.Union([
StepStart,
TextStart,
TextDelta,
TextEnd,
ReasoningStart,
ReasoningDelta,
ReasoningEnd,
ToolInputStart,
ToolInputDelta,
ToolInputEnd,
ToolCall,
ToolResult,
ToolError,
StepFinish,
Finish,
ProviderErrorEvent,
]).pipe(Schema.toTaggedUnion("type"))
type WithID<Event extends { readonly id: unknown }, ID> = Omit<Event, "type" | "id"> & { readonly id: ID | string }
type WithUsage<Event extends { readonly usage?: Usage }> = Omit<Event, "type" | "usage"> & {
readonly usage?: UsageInput
}
const contentBlockID = (value: ContentBlockID | string) => ContentBlockID.make(value)
const toolCallID = (value: ToolCallID | string) => ToolCallID.make(value)
/**
* camelCase aliases for `LLMEvent.guards` (provided by `Schema.toTaggedUnion`).
* Lets consumers write `events.filter(LLMEvent.is.toolCall)` instead of
* `events.filter(LLMEvent.guards["tool-call"])`.
*/
export const LLMEvent = Object.assign(llmEventTagged, {
stepStart: StepStart.make,
textStart: (input: WithID<TextStart, ContentBlockID>) => TextStart.make({ ...input, id: contentBlockID(input.id) }),
textDelta: (input: WithID<TextDelta, ContentBlockID>) => TextDelta.make({ ...input, id: contentBlockID(input.id) }),
textEnd: (input: WithID<TextEnd, ContentBlockID>) => TextEnd.make({ ...input, id: contentBlockID(input.id) }),
reasoningStart: (input: WithID<ReasoningStart, ContentBlockID>) =>
ReasoningStart.make({ ...input, id: contentBlockID(input.id) }),
reasoningDelta: (input: WithID<ReasoningDelta, ContentBlockID>) =>
ReasoningDelta.make({ ...input, id: contentBlockID(input.id) }),
reasoningEnd: (input: WithID<ReasoningEnd, ContentBlockID>) =>
ReasoningEnd.make({ ...input, id: contentBlockID(input.id) }),
toolInputStart: (input: WithID<ToolInputStart, ToolCallID>) =>
ToolInputStart.make({ ...input, id: toolCallID(input.id) }),
toolInputDelta: (input: WithID<ToolInputDelta, ToolCallID>) =>
ToolInputDelta.make({ ...input, id: toolCallID(input.id) }),
toolInputEnd: (input: WithID<ToolInputEnd, ToolCallID>) => ToolInputEnd.make({ ...input, id: toolCallID(input.id) }),
toolCall: (input: WithID<ToolCall, ToolCallID>) => ToolCall.make({ ...input, id: toolCallID(input.id) }),
toolResult: (input: WithID<ToolResult, ToolCallID>) =>
ToolResult.make({
...input,
id: toolCallID(input.id),
output: input.output === undefined ? undefined : ToolOutput.make(input.output.structured, input.output.content),
}),
toolError: (input: WithID<ToolError, ToolCallID>) => ToolError.make({ ...input, id: toolCallID(input.id) }),
stepFinish: (input: WithUsage<StepFinish>) =>
StepFinish.make({
...input,
usage: input.usage === undefined ? undefined : Usage.from(input.usage),
}),
finish: (input: WithUsage<Finish>) =>
Finish.make({
...input,
usage: input.usage === undefined ? undefined : Usage.from(input.usage),
}),
providerError: ProviderErrorEvent.make,
is: {
stepStart: llmEventTagged.guards["step-start"],
textStart: llmEventTagged.guards["text-start"],
textDelta: llmEventTagged.guards["text-delta"],
textEnd: llmEventTagged.guards["text-end"],
reasoningStart: llmEventTagged.guards["reasoning-start"],
reasoningDelta: llmEventTagged.guards["reasoning-delta"],
reasoningEnd: llmEventTagged.guards["reasoning-end"],
toolInputStart: llmEventTagged.guards["tool-input-start"],
toolInputDelta: llmEventTagged.guards["tool-input-delta"],
toolInputEnd: llmEventTagged.guards["tool-input-end"],
toolCall: llmEventTagged.guards["tool-call"],
toolResult: llmEventTagged.guards["tool-result"],
toolError: llmEventTagged.guards["tool-error"],
stepFinish: llmEventTagged.guards["step-finish"],
finish: llmEventTagged.guards.finish,
providerError: llmEventTagged.guards["provider-error"],
},
})
export type LLMEvent = Schema.Schema.Type<typeof llmEventTagged>
export class PreparedRequest extends Schema.Class<PreparedRequest>("LLM.PreparedRequest")({
id: Schema.String,
route: RouteID,
protocol: ProtocolID,
model: ModelSchema,
body: Schema.Unknown,
metadata: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
}) {}
/**
* A `PreparedRequest` whose `body` is typed as `Body`. Use with the generic
* on `LLMClient.prepare<Body>(...)` when the caller knows which route their
* request will resolve to and wants its native shape statically exposed
* (debug UIs, request previews, plan rendering).
*
* The runtime body is identical — the route still emits `body: unknown` — so
* this is a type-level assertion the caller makes about what they expect to
* find. The prepare runtime does not validate the assertion.
*/
export type PreparedRequestOf<Body> = Omit<PreparedRequest, "body"> & {
readonly body: Body
}
const responseText = (events: ReadonlyArray<LLMEvent>) =>
events
.filter(LLMEvent.is.textDelta)
.map((event) => event.text)
.join("")
const responseReasoning = (events: ReadonlyArray<LLMEvent>) =>
events
.filter(LLMEvent.is.reasoningDelta)
.map((event) => event.text)
.join("")
const responseUsage = (events: ReadonlyArray<LLMEvent>) =>
events.reduce<Usage | undefined>(
(usage, event) => ("usage" in event && event.usage !== undefined ? event.usage : usage),
undefined,
)
export class LLMResponse extends Schema.Class<LLMResponse>("LLM.Response")({
events: Schema.Array(LLMEvent),
usage: Schema.optional(Usage),
}) {
/** Concatenated assistant text assembled from streamed `text-delta` events. */
get text() {
return responseText(this.events)
}
/** Concatenated reasoning text assembled from streamed `reasoning-delta` events. */
get reasoning() {
return responseReasoning(this.events)
}
/** Completed tool calls emitted by the provider. */
get toolCalls() {
return this.events.filter(LLMEvent.is.toolCall)
}
}
export namespace LLMResponse {
export type Output = LLMResponse | { readonly events: ReadonlyArray<LLMEvent>; readonly usage?: Usage }
/** Concatenate assistant text from a response or collected event list. */
export const text = (response: Output) => responseText(response.events)
/** Return response usage, falling back to the latest usage-bearing event. */
export const usage = (response: Output) => response.usage ?? responseUsage(response.events)
/** Return completed tool calls from a response or collected event list. */
export const toolCalls = (response: Output) => response.events.filter(LLMEvent.is.toolCall)
/** Concatenate reasoning text from a response or collected event list. */
export const reasoning = (response: Output) => responseReasoning(response.events)
}

43
packages/llm/src/schema/ids.ts Normal file
View File

@@ -0,0 +1,43 @@
import { Schema } from "effect"
/** Stable string identifier for a protocol implementation. */
export const ProtocolID = Schema.String
export type ProtocolID = Schema.Schema.Type<typeof ProtocolID>
/** Stable string identifier for the runnable route. */
export const RouteID = Schema.String
export type RouteID = Schema.Schema.Type<typeof RouteID>
export const ModelID = Schema.String.pipe(Schema.brand("LLM.ModelID"))
export type ModelID = typeof ModelID.Type
export const ProviderID = Schema.String.pipe(Schema.brand("LLM.ProviderID"))
export type ProviderID = typeof ProviderID.Type
export const ResponseID = Schema.String
export type ResponseID = Schema.Schema.Type<typeof ResponseID>
export const ContentBlockID = Schema.String
export type ContentBlockID = Schema.Schema.Type<typeof ContentBlockID>
export const ToolCallID = Schema.String
export type ToolCallID = Schema.Schema.Type<typeof ToolCallID>
export const ReasoningEfforts = ["none", "minimal", "low", "medium", "high", "xhigh", "max"] as const
export const ReasoningEffort = Schema.Literals(ReasoningEfforts)
export type ReasoningEffort = Schema.Schema.Type<typeof ReasoningEffort>
export const TextVerbosity = Schema.Literals(["low", "medium", "high"])
export type TextVerbosity = Schema.Schema.Type<typeof TextVerbosity>
export const MessageRole = Schema.Literals(["system", "user", "assistant", "tool"])
export type MessageRole = Schema.Schema.Type<typeof MessageRole>
export const FinishReason = Schema.Literals(["stop", "length", "tool-calls", "content-filter", "error", "unknown"])
export type FinishReason = Schema.Schema.Type<typeof FinishReason>
export const JsonSchema = Schema.Record(Schema.String, Schema.Unknown)
export type JsonSchema = Schema.Schema.Type<typeof JsonSchema>
export const ProviderMetadata = Schema.Record(Schema.String, Schema.Record(Schema.String, Schema.Unknown))
export type ProviderMetadata = Schema.Schema.Type<typeof ProviderMetadata>

5
packages/llm/src/schema/index.ts Normal file
View File

@@ -0,0 +1,5 @@
export * from "./ids"
export * from "./options"
export * from "./messages"
export * from "./events"
export * from "./errors"

327
packages/llm/src/schema/messages.ts Normal file
View File

@@ -0,0 +1,327 @@
import { Schema } from "effect"
import { JsonSchema, MessageRole, ProviderMetadata } from "./ids"
import { CacheHint, CachePolicy, GenerationOptions, HttpOptions, ModelSchema, ProviderOptions } from "./options"
import { isRecord } from "../utils/record"
const systemPartSchema = Schema.Struct({
type: Schema.Literal("text"),
text: Schema.String,
cache: Schema.optional(CacheHint),
metadata: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
}).annotate({ identifier: "LLM.SystemPart" })
export type SystemPart = Schema.Schema.Type<typeof systemPartSchema>
const makeSystemPart = (text: string): SystemPart => ({ type: "text", text })
export const SystemPart = Object.assign(systemPartSchema, {
make: makeSystemPart,
content: (input?: string | SystemPart | ReadonlyArray<SystemPart>) => {
if (input === undefined) return []
return typeof input === "string" ? [makeSystemPart(input)] : Array.isArray(input) ? [...input] : [input]
},
})
export const TextPart = Schema.Struct({
type: Schema.Literal("text"),
text: Schema.String,
cache: Schema.optional(CacheHint),
metadata: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Content.Text" })
export type TextPart = Schema.Schema.Type<typeof TextPart>
export const MediaPart = Schema.Struct({
type: Schema.Literal("media"),
mediaType: Schema.String,
data: Schema.Union([Schema.String, Schema.Uint8Array]),
filename: Schema.optional(Schema.String),
metadata: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
}).annotate({ identifier: "LLM.Content.Media" })
export type MediaPart = Schema.Schema.Type<typeof MediaPart>
export const ToolTextContent = Schema.Struct({
type: Schema.Literal("text"),
text: Schema.String,
}).annotate({ identifier: "Tool.TextContent" })
export type ToolTextContent = typeof ToolTextContent.Type
export const ToolFileContent = Schema.Struct({
type: Schema.Literal("file"),
uri: Schema.String,
mime: Schema.String,
name: Schema.optional(Schema.String),
}).annotate({ identifier: "Tool.FileContent" })
export type ToolFileContent = typeof ToolFileContent.Type
/** Ordered, provider-independent content shown to models and UIs after a tool succeeds. */
export const ToolContent = Schema.Union([ToolTextContent, ToolFileContent]).pipe(Schema.toTaggedUnion("type"))
export type ToolContent = Schema.Schema.Type<typeof ToolContent>
const isToolResultValue = (value: unknown): value is ToolResultValue =>
isRecord(value) &&
(value.type === "text" || value.type === "json" || value.type === "error" || value.type === "content") &&
"value" in value
export const ToolResultValue = Object.assign(
Schema.Union([
Schema.Struct({
type: Schema.Literal("json"),
value: Schema.Unknown,
}),
Schema.Struct({
type: Schema.Literal("text"),
value: Schema.Unknown,
}),
Schema.Struct({
type: Schema.Literal("error"),
value: Schema.Unknown,
}),
Schema.Struct({
type: Schema.Literal("content"),
value: Schema.Array(ToolContent),
}),
]).annotate({ identifier: "LLM.ToolResult" }),
{
is: isToolResultValue,
make: (value: unknown, type: ToolResultValue["type"] = "json"): ToolResultValue => {
if (isToolResultValue(value)) return value
if (type === "content") return { type, value: Array.isArray(value) ? value : [] }
return { type, value }
},
},
)
export type ToolResultValue = Schema.Schema.Type<typeof ToolResultValue>
export interface ToolOutput {
readonly structured: unknown
readonly content: ReadonlyArray<ToolContent>
}
export const ToolOutput = Object.assign(
Schema.Struct({
structured: Schema.Unknown,
content: Schema.Array(ToolContent),
}).annotate({ identifier: "LLM.ToolOutput" }),
{
make: (structured: unknown, content: ReadonlyArray<ToolContent> = []): ToolOutput => ({ structured, content }),
fromResultValue: (result: ToolResultValue): ToolOutput | undefined => {
switch (result.type) {
case "json":
return { structured: result.value, content: [] }
case "text":
return { structured: {}, content: [{ type: "text", text: toolResultText(result.value) }] }
case "content":
return { structured: {}, content: result.value }
case "error":
return undefined
}
},
toResultValue: (output: ToolOutput): ToolResultValue => {
if (output.content.length === 0) return { type: "json", value: output.structured }
if (output.content.length === 1 && output.content[0]?.type === "text")
return { type: "text", value: output.content[0].text }
return { type: "content", value: output.content }
},
},
)
const toolResultText = (value: unknown) => {
if (typeof value === "string") return value
try {
return JSON.stringify(value) ?? String(value)
} catch {
return String(value)
}
}
export const ToolCallPart = Object.assign(
Schema.Struct({
type: Schema.Literal("tool-call"),
id: Schema.String,
name: Schema.String,
input: Schema.Unknown,
providerExecuted: Schema.optional(Schema.Boolean),
metadata: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Content.ToolCall" }),
{
make: (input: Omit<ToolCallPart, "type">): ToolCallPart => ({ type: "tool-call", ...input }),
},
)
export type ToolCallPart = Schema.Schema.Type<typeof ToolCallPart>
export const ToolResultPart = Object.assign(
Schema.Struct({
type: Schema.Literal("tool-result"),
id: Schema.String,
name: Schema.String,
result: ToolResultValue,
providerExecuted: Schema.optional(Schema.Boolean),
cache: Schema.optional(CacheHint),
metadata: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Content.ToolResult" }),
{
make: (
input: Omit<ToolResultPart, "type" | "result"> & {
readonly result: unknown
readonly resultType?: ToolResultValue["type"]
},
): ToolResultPart => ({
type: "tool-result",
id: input.id,
name: input.name,
result: ToolResultValue.make(input.result, input.resultType),
providerExecuted: input.providerExecuted,
cache: input.cache,
metadata: input.metadata,
providerMetadata: input.providerMetadata,
}),
},
)
export type ToolResultPart = Schema.Schema.Type<typeof ToolResultPart>
export const ReasoningPart = Schema.Struct({
type: Schema.Literal("reasoning"),
text: Schema.String,
encrypted: Schema.optional(Schema.String),
metadata: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Content.Reasoning" })
export type ReasoningPart = Schema.Schema.Type<typeof ReasoningPart>
export const ContentPart = Schema.Union([TextPart, MediaPart, ToolCallPart, ToolResultPart, ReasoningPart]).pipe(
Schema.toTaggedUnion("type"),
)
export type ContentPart = Schema.Schema.Type<typeof ContentPart>
export class Message extends Schema.Class<Message>("LLM.Message")({
id: Schema.optional(Schema.String),
role: MessageRole,
content: Schema.Array(ContentPart),
metadata: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
native: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
}) {}
export namespace Message {
export type ContentInput = string | ContentPart | ReadonlyArray<ContentPart>
export type SystemContentInput = string | TextPart | ReadonlyArray<TextPart>
export type Input = Omit<ConstructorParameters<typeof Message>[0], "content"> & {
readonly content: ContentInput
}
export const text = (value: string): ContentPart => ({ type: "text", text: value })
export const content = (input: ContentInput) =>
typeof input === "string" ? [text(input)] : Array.isArray(input) ? [...input] : [input]
export const make = (input: Message | Input) => {
if (input instanceof Message) return input
return new Message({ ...input, content: content(input.content) })
}
export const user = (content: ContentInput) => make({ role: "user", content })
export const assistant = (content: ContentInput) => make({ role: "assistant", content })
/**
* Add an operator-authored instruction at this chronological point in the
* conversation. This is distinct from the initial `LLMRequest.system`
* prompt. Keep raw retrieved, tool, and web content out of privileged system
* updates; pass that untrusted content through ordinary user/tool channels.
*/
export const system = (content: SystemContentInput) => make({ role: "system", content })
export const tool = (result: ToolResultPart | Parameters<typeof ToolResultPart.make>[0]) =>
make({ role: "tool", content: ["type" in result ? result : ToolResultPart.make(result)] })
}
export class ToolDefinition extends Schema.Class<ToolDefinition>("LLM.ToolDefinition")({
name: Schema.String,
description: Schema.String,
inputSchema: JsonSchema,
outputSchema: Schema.optional(JsonSchema),
cache: Schema.optional(CacheHint),
metadata: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
native: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
}) {}
export namespace ToolDefinition {
export type Input = ToolDefinition | ConstructorParameters<typeof ToolDefinition>[0]
/** Normalize tool definition input into the canonical `ToolDefinition` class. */
export const make = (input: Input) => (input instanceof ToolDefinition ? input : new ToolDefinition(input))
}
export class ToolChoice extends Schema.Class<ToolChoice>("LLM.ToolChoice")({
type: Schema.Literals(["auto", "none", "required", "tool"]),
name: Schema.optional(Schema.String),
}) {}
export namespace ToolChoice {
export type Mode = Exclude<ToolChoice["type"], "tool">
export type Input = ToolChoice | ConstructorParameters<typeof ToolChoice>[0] | ToolDefinition | string
const isMode = (value: string): value is Mode => value === "auto" || value === "none" || value === "required"
/** Select a specific named tool. */
export const named = (value: string) => new ToolChoice({ type: "tool", name: value })
/** Normalize ergonomic tool-choice inputs into the canonical `ToolChoice` class. */
export const make = (input: Input) => {
if (input instanceof ToolChoice) return input
if (input instanceof ToolDefinition) return named(input.name)
if (typeof input === "string") return isMode(input) ? new ToolChoice({ type: input }) : named(input)
return new ToolChoice(input)
}
}
export const ResponseFormat = Schema.Union([
Schema.Struct({ type: Schema.Literal("text") }),
Schema.Struct({ type: Schema.Literal("json"), schema: JsonSchema }),
Schema.Struct({ type: Schema.Literal("tool"), tool: ToolDefinition }),
]).pipe(Schema.toTaggedUnion("type"))
export type ResponseFormat = Schema.Schema.Type<typeof ResponseFormat>
export class LLMRequest extends Schema.Class<LLMRequest>("LLM.Request")({
id: Schema.optional(Schema.String),
model: ModelSchema,
system: Schema.Array(SystemPart),
messages: Schema.Array(Message),
tools: Schema.Array(ToolDefinition),
toolChoice: Schema.optional(ToolChoice),
generation: Schema.optional(GenerationOptions),
providerOptions: Schema.optional(ProviderOptions),
http: Schema.optional(HttpOptions),
responseFormat: Schema.optional(ResponseFormat),
cache: Schema.optional(CachePolicy),
metadata: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
}) {}
export namespace LLMRequest {
export type Input = ConstructorParameters<typeof LLMRequest>[0]
export const input = (request: LLMRequest): Input => ({
id: request.id,
model: request.model,
system: request.system,
messages: request.messages,
tools: request.tools,
toolChoice: request.toolChoice,
generation: request.generation,
providerOptions: request.providerOptions,
http: request.http,
responseFormat: request.responseFormat,
cache: request.cache,
metadata: request.metadata,
})
export const update = (request: LLMRequest, patch: Partial<Input>) => {
if (Object.keys(patch).length === 0) return request
return new LLMRequest({
...input(request),
...patch,
model: patch.model ?? request.model,
})
}
}

221
packages/llm/src/schema/options.ts Normal file
View File

@@ -0,0 +1,221 @@
import { Schema } from "effect"
import { JsonSchema, ModelID, ProviderID } from "./ids"
import type { AnyRoute } from "../route/client"
import { isRecord } from "../utils/record"
export const mergeJsonRecords = (
...items: ReadonlyArray<Record<string, unknown> | undefined>
): Record<string, unknown> | undefined => {
const defined = items.filter((item): item is Record<string, unknown> => item !== undefined)
if (defined.length === 0) return undefined
if (defined.length === 1 && Object.values(defined[0]).every((value) => value !== undefined)) return defined[0]
const result: Record<string, unknown> = {}
for (const item of defined) {
for (const [key, value] of Object.entries(item)) {
if (value === undefined) continue
result[key] = isRecord(result[key]) && isRecord(value) ? mergeJsonRecords(result[key], value) : value
}
}
return Object.keys(result).length === 0 ? undefined : result
}
const mergeStringRecords = (
...items: ReadonlyArray<Record<string, string> | undefined>
): Record<string, string> | undefined => {
const defined = items.filter((item): item is Record<string, string> => item !== undefined)
if (defined.length === 0) return undefined
if (defined.length === 1) return defined[0]
const result = Object.fromEntries(
defined.flatMap((item) =>
Object.entries(item).filter((entry): entry is [string, string] => entry[1] !== undefined),
),
)
return Object.keys(result).length === 0 ? undefined : result
}
export const ProviderOptions = Schema.Record(Schema.String, Schema.Record(Schema.String, Schema.Unknown))
export type ProviderOptions = Schema.Schema.Type<typeof ProviderOptions>
export const mergeProviderOptions = (
...items: ReadonlyArray<ProviderOptions | undefined>
): ProviderOptions | undefined => {
const result: Record<string, Record<string, unknown>> = {}
for (const item of items) {
if (!item) continue
for (const [provider, options] of Object.entries(item)) {
const merged = mergeJsonRecords(result[provider], options)
if (merged) result[provider] = merged
}
}
return Object.keys(result).length === 0 ? undefined : result
}
export class HttpOptions extends Schema.Class<HttpOptions>("LLM.HttpOptions")({
body: Schema.optional(JsonSchema),
headers: Schema.optional(Schema.Record(Schema.String, Schema.String)),
query: Schema.optional(Schema.Record(Schema.String, Schema.String)),
}) {}
export namespace HttpOptions {
export type Input = HttpOptions | ConstructorParameters<typeof HttpOptions>[0]
/** Normalize HTTP option input into the canonical `HttpOptions` class. */
export const make = (input: Input) => (input instanceof HttpOptions ? input : new HttpOptions(input))
}
export const mergeHttpOptions = (...items: ReadonlyArray<HttpOptions | undefined>): HttpOptions | undefined => {
const body = mergeJsonRecords(...items.map((item) => item?.body))
const headers = mergeStringRecords(...items.map((item) => item?.headers))
const query = mergeStringRecords(...items.map((item) => item?.query))
if (!body && !headers && !query) return undefined
return new HttpOptions({ body, headers, query })
}
export class GenerationOptions extends Schema.Class<GenerationOptions>("LLM.GenerationOptions")({
maxTokens: Schema.optional(Schema.Number),
temperature: Schema.optional(Schema.Number),
topP: Schema.optional(Schema.Number),
topK: Schema.optional(Schema.Number),
frequencyPenalty: Schema.optional(Schema.Number),
presencePenalty: Schema.optional(Schema.Number),
seed: Schema.optional(Schema.Number),
stop: Schema.optional(Schema.Array(Schema.String)),
}) {}
export namespace GenerationOptions {
export type Input = GenerationOptions | ConstructorParameters<typeof GenerationOptions>[0]
/** Normalize generation option input into the canonical `GenerationOptions` class. */
export const make = (input: Input = {}) => (input instanceof GenerationOptions ? input : new GenerationOptions(input))
}
export type GenerationOptionsFields = {
readonly maxTokens?: number
readonly temperature?: number
readonly topP?: number
readonly topK?: number
readonly frequencyPenalty?: number
readonly presencePenalty?: number
readonly seed?: number
readonly stop?: ReadonlyArray<string>
}
export type GenerationOptionsInput = GenerationOptions | GenerationOptionsFields
const latestGeneration = <Key extends keyof GenerationOptionsFields>(
items: ReadonlyArray<GenerationOptionsInput | undefined>,
key: Key,
) => items.findLast((item) => item?.[key] !== undefined)?.[key]
export const mergeGenerationOptions = (...items: ReadonlyArray<GenerationOptionsInput | undefined>) => {
const result = new GenerationOptions({
maxTokens: latestGeneration(items, "maxTokens"),
temperature: latestGeneration(items, "temperature"),
topP: latestGeneration(items, "topP"),
topK: latestGeneration(items, "topK"),
frequencyPenalty: latestGeneration(items, "frequencyPenalty"),
presencePenalty: latestGeneration(items, "presencePenalty"),
seed: latestGeneration(items, "seed"),
stop: latestGeneration(items, "stop"),
})
return Object.values(result).some((value) => value !== undefined) ? result : undefined
}
export class ModelLimits extends Schema.Class<ModelLimits>("LLM.ModelLimits")({
context: Schema.optional(Schema.Number),
output: Schema.optional(Schema.Number),
}) {}
export namespace ModelLimits {
export type Input = ModelLimits | ConstructorParameters<typeof ModelLimits>[0]
/** Normalize model limit input into the canonical `ModelLimits` class. */
export const make = (input: Input | undefined) =>
input instanceof ModelLimits ? input : new ModelLimits(input ?? {})
}
export class Model {
readonly id: ModelID
readonly provider: ProviderID
readonly route: AnyRoute
constructor(input: Model.ConstructorInput) {
this.id = input.id
this.provider = input.provider
this.route = input.route
}
static make(input: Model.Input) {
return new Model({
id: ModelID.make(input.id),
provider: ProviderID.make(input.provider),
route: input.route,
})
}
static input(model: Model): Model.ConstructorInput {
return {
id: model.id,
provider: model.provider,
route: model.route,
}
}
static update(model: Model, patch: Partial<Model.Input>) {
if (Object.keys(patch).length === 0) return model
return Model.make({
...Model.input(model),
...patch,
})
}
}
export namespace Model {
export type ConstructorInput = {
readonly id: ModelID
readonly provider: ProviderID
readonly route: AnyRoute
}
export type Input = Omit<ConstructorInput, "id" | "provider"> & {
readonly id: string | ModelID
readonly provider: string | ProviderID
}
}
export type ModelInput = Model.Input
export const ModelSchema = Schema.declare((value): value is Model => value instanceof Model, { expected: "LLM.Model" })
export class CacheHint extends Schema.Class<CacheHint>("LLM.CacheHint")({
type: Schema.Literals(["ephemeral", "persistent"]),
ttlSeconds: Schema.optional(Schema.Number),
}) {}
// Auto-placement policy for prompt caching. The protocol-neutral lowering step
// reads this and injects `CacheHint`s at the configured boundaries; the
// per-protocol body builders then translate those hints into wire markers as
// usual. `"auto"` is the recommended default for agent loops — it places one
// breakpoint at the last tool definition, one at the last system part, and one
// at the latest user message. The combination of provider invalidation
// hierarchy (tools → system → messages) and Anthropic/Bedrock's 20-block
// lookback means three trailing breakpoints reliably cover the static prefix.
//
// Pass `"none"` to opt out entirely (the legacy behavior). Pass the granular
// object form to override individual choices.
export const CachePolicyObject = Schema.Struct({
tools: Schema.optional(Schema.Boolean),
system: Schema.optional(Schema.Boolean),
messages: Schema.optional(
Schema.Union([
Schema.Literal("latest-user-message"),
Schema.Literal("latest-assistant"),
Schema.Struct({ tail: Schema.Number }),
]),
),
ttlSeconds: Schema.optional(Schema.Number),
})
export type CachePolicyObject = Schema.Schema.Type<typeof CachePolicyObject>
export const CachePolicy = Schema.Union([Schema.Literal("auto"), Schema.Literal("none"), CachePolicyObject])
export type CachePolicy = Schema.Schema.Type<typeof CachePolicy>

78
packages/llm/src/tool-runtime.ts Normal file
View File

@@ -0,0 +1,78 @@
import { Effect } from "effect"
import {
LLMEvent,
type ToolCallPart,
ToolFailure,
ToolOutput,
ToolResultValue,
type ToolOutput as ToolOutputType,
type ToolResultValue as ToolResultValueType,
} from "./schema"
import { type AnyTool, type Tools } from "./tool"
export interface ToolSettlement {
readonly result: ToolResultValueType
readonly output?: ToolOutputType
}
export interface DispatchResult extends ToolSettlement {
readonly events: ReadonlyArray<LLMEvent>
}
/** Execute one canonical tool call without owning provider IO or continuation. */
export const dispatch = (tools: Tools, call: ToolCallPart): Effect.Effect<DispatchResult> => {
const tool = tools[call.name]
if (!tool) return Effect.succeed(result(call, { type: "error", value: `Unknown tool: ${call.name}` }))
if (!tool.execute)
return Effect.succeed(result(call, { type: "error", value: `Tool has no execute handler: ${call.name}` }))
return decodeAndExecute(tool, call).pipe(
Effect.map((value) => result(call, value)),
Effect.catchTag("LLM.ToolFailure", (failure) =>
Effect.succeed(result(call, { type: "error", value: failure.message }, failure.error)),
),
)
}
const decodeAndExecute = (tool: AnyTool, call: ToolCallPart): Effect.Effect<ToolSettlement, ToolFailure> =>
tool._decode(call.input).pipe(
Effect.mapError((error) => new ToolFailure({ message: `Invalid tool input: ${error.message}` })),
Effect.flatMap((decoded) =>
tool.execute!(decoded, { id: call.id, name: call.name }).pipe(
Effect.flatMap((value) =>
tool._encode(value).pipe(
Effect.mapError(
(error) =>
new ToolFailure({
message: `Tool returned an invalid value for its success schema: ${error.message}`,
}),
),
),
),
Effect.map((encoded) => {
if (tool._legacyResult && ToolResultValue.is(encoded))
return { result: encoded, output: ToolOutput.fromResultValue(encoded) }
const output = tool._project(decoded, call.id, encoded)
const result = ToolOutput.toResultValue(output)
return result.type === "error" ? { result } : { result, output }
}),
),
),
)
const result = (call: ToolCallPart, value: ToolResultValueType | ToolSettlement, error?: unknown): DispatchResult => {
const settlement = ToolResultValue.is(value) ? { result: value } : value
return {
result: settlement.result,
output: settlement.output,
events:
settlement.result.type === "error"
? [
LLMEvent.toolError({ id: call.id, name: call.name, message: String(settlement.result.value), error }),
LLMEvent.toolResult({ id: call.id, name: call.name, result: settlement.result }),
]
: [LLMEvent.toolResult({ id: call.id, name: call.name, result: settlement.result, output: settlement.output })],
}
}
export const ToolRuntime = { dispatch } as const

253
packages/llm/src/tool.ts Normal file
View File

@@ -0,0 +1,253 @@
import { Effect, JsonSchema, Schema } from "effect"
import type {
ToolCallPart,
ToolContent,
ToolDefinition as ToolDefinitionClass,
ToolOutput as ToolOutputType,
} from "./schema"
import { ToolDefinition, ToolFailure, ToolOutput } from "./schema"
/**
* Schema constraint for tool parameters / success values: no decoding or
* encoding services are allowed. Tools should be self-contained — anything
* beyond pure data conversion belongs in the handler closure.
*/
export type ToolSchema<T> = Schema.Codec<T, any, never, never>
export interface ToolExecuteContext {
readonly id: ToolCallPart["id"]
readonly name: ToolCallPart["name"]
}
export type ToolExecute<Parameters extends ToolSchema<any>, Success extends ToolSchema<any>> = (
params: Schema.Schema.Type<Parameters>,
context?: ToolExecuteContext,
) => Effect.Effect<Schema.Schema.Type<Success>, ToolFailure>
export interface ToolModelOutputInput<Parameters, Output> {
readonly callID: ToolCallPart["id"]
readonly parameters: Parameters
readonly output: Output
}
export type ToolToModelOutput<Parameters extends ToolSchema<any>, Success extends ToolSchema<any>> = (
input: ToolModelOutputInput<Schema.Schema.Type<Parameters>, Success["Encoded"]>,
) => ReadonlyArray<ToolContent>
/**
* A type-safe LLM tool. Each tool bundles its own description, parameter
* Schema and success Schema. The execute handler is optional: omit it when you
* only want to expose a tool schema to the model and handle tool calls outside
* this package.
*
* Errors must be expressed as `ToolFailure`. Unmapped errors and defects fail
* the stream.
*
* Internally each tool also carries memoized codecs and a precomputed
* `ToolDefinition` so callers do not rebuild them per invocation.
*/
export interface Tool<Parameters extends ToolSchema<any>, Success extends ToolSchema<any>> {
readonly description: string
readonly parameters: Parameters
readonly success: Success
readonly execute?: ToolExecute<Parameters, Success>
readonly toModelOutput?: ToolToModelOutput<Parameters, Success>
readonly toStructuredOutput?: (output: Success["Encoded"]) => unknown
/** @internal */
readonly _decode: (input: unknown) => Effect.Effect<Schema.Schema.Type<Parameters>, Schema.SchemaError>
/** @internal */
readonly _encode: (value: Schema.Schema.Type<Success>) => Effect.Effect<unknown, Schema.SchemaError>
/** @internal */
readonly _project: (
parameters: Schema.Schema.Type<Parameters>,
callID: ToolCallPart["id"],
output: unknown,
) => ToolOutputType
/** @internal */
readonly _legacyResult: boolean
/** @internal */
readonly _definition: ToolDefinitionClass
}
export type AnyTool = Tool<any, any>
export type ExecutableTool<Parameters extends ToolSchema<any>, Success extends ToolSchema<any>> = Tool<
Parameters,
Success
> & {
readonly execute: ToolExecute<Parameters, Success>
}
export type AnyExecutableTool = ExecutableTool<any, any>
export type ExecutableTools = Record<string, AnyExecutableTool>
type TypedToolConfig = {
readonly description: string
readonly parameters: ToolSchema<any>
readonly success: ToolSchema<any>
readonly execute?: ToolExecute<ToolSchema<any>, ToolSchema<any>>
readonly toModelOutput?: ToolToModelOutput<ToolSchema<any>, ToolSchema<any>>
readonly toStructuredOutput?: (output: unknown) => unknown
}
type DynamicToolConfig = {
readonly description: string
readonly jsonSchema: JsonSchema.JsonSchema
readonly outputSchema?: JsonSchema.JsonSchema
readonly execute?: (params: unknown, context?: ToolExecuteContext) => Effect.Effect<unknown, ToolFailure>
readonly toModelOutput?: (input: ToolModelOutputInput<unknown, unknown>) => ReadonlyArray<ToolContent>
readonly toStructuredOutput?: (output: unknown) => unknown
}
/**
* Constructs a tool. Two input modes:
*
* 1. **Typed** — pass Effect `parameters` and `success` Schemas; inputs and
* outputs are statically typed and decoded/encoded automatically.
*
* ```ts
* Tool.make({
* description: "Get current weather",
* parameters: Schema.Struct({ city: Schema.String }),
* success: Schema.Struct({ temperature: Schema.Number }),
* execute: ({ city }) => Effect.succeed({ temperature: 22 }),
* })
* ```
*
* 2. **Dynamic** — pass raw JSON Schema as `jsonSchema`. Use this when the
* schema comes from an external source (MCP server, plugin manifest,
* dynamic config) and is not known at compile time. Inputs are typed as
* `unknown`; the handler is responsible for any validation it needs.
*
* ```ts
* Tool.make({
* description: "Look something up",
* jsonSchema: { type: "object", properties: { ... } },
* execute: (params) => Effect.succeed(...),
* })
* ```
*
* In both modes the produced tool flows through `toDefinitions(...)`
* identically.
*/
export function make<Parameters extends ToolSchema<any>, Success extends ToolSchema<any>>(config: {
readonly description: string
readonly parameters: Parameters
readonly success: Success
readonly execute: ToolExecute<Parameters, Success>
readonly toModelOutput?: ToolToModelOutput<Parameters, Success>
readonly toStructuredOutput?: (output: Success["Encoded"]) => unknown
}): ExecutableTool<Parameters, Success>
export function make<Parameters extends ToolSchema<any>, Success extends ToolSchema<any>>(config: {
readonly description: string
readonly parameters: Parameters
readonly success: Success
readonly execute?: undefined
readonly toModelOutput?: ToolToModelOutput<Parameters, Success>
readonly toStructuredOutput?: (output: Success["Encoded"]) => unknown
}): Tool<Parameters, Success>
export function make(config: {
readonly description: string
readonly jsonSchema: JsonSchema.JsonSchema
readonly outputSchema?: JsonSchema.JsonSchema
readonly execute: (params: unknown, context?: ToolExecuteContext) => Effect.Effect<unknown, ToolFailure>
readonly toModelOutput?: (input: ToolModelOutputInput<unknown, unknown>) => ReadonlyArray<ToolContent>
readonly toStructuredOutput?: (output: unknown) => unknown
}): AnyExecutableTool
export function make(config: {
readonly description: string
readonly jsonSchema: JsonSchema.JsonSchema
readonly outputSchema?: JsonSchema.JsonSchema
readonly execute?: undefined
readonly toModelOutput?: (input: ToolModelOutputInput<unknown, unknown>) => ReadonlyArray<ToolContent>
readonly toStructuredOutput?: (output: unknown) => unknown
}): AnyTool
export function make(config: TypedToolConfig | DynamicToolConfig): AnyTool {
if ("jsonSchema" in config) {
return {
description: config.description,
parameters: Schema.Unknown as ToolSchema<unknown>,
success: Schema.Unknown as ToolSchema<unknown>,
execute: config.execute,
toModelOutput: config.toModelOutput,
toStructuredOutput: config.toStructuredOutput,
_decode: Effect.succeed,
_encode: Effect.succeed,
_project: (parameters, callID, output) =>
project(config.toModelOutput, config.toStructuredOutput, parameters, callID, output),
_legacyResult: config.toModelOutput === undefined && config.toStructuredOutput === undefined,
_definition: new ToolDefinition({
name: "",
description: config.description,
inputSchema: config.jsonSchema,
outputSchema: config.outputSchema,
}),
}
}
return {
description: config.description,
parameters: config.parameters,
success: config.success,
execute: config.execute,
toModelOutput: config.toModelOutput,
toStructuredOutput: config.toStructuredOutput,
_decode: Schema.decodeUnknownEffect(config.parameters),
_encode: Schema.encodeEffect(config.success),
_project: (parameters, callID, output) =>
project(config.toModelOutput, config.toStructuredOutput, parameters, callID, output),
_legacyResult: false,
_definition: new ToolDefinition({
name: "",
description: config.description,
inputSchema: toJsonSchema(config.parameters),
outputSchema: toJsonSchema(config.success),
}),
}
}
/**
* A record of named tools. The record key becomes the tool name on the wire.
*/
export type Tools = Record<string, AnyTool>
/**
* Convert a tools record into the `ToolDefinition[]` shape that
* `LLMRequest.tools` expects.
*
* Tool names come from the record keys, so the per-tool cached
* `_definition` is rebuilt with the correct name here. The JSON Schema body
* is reused.
*/
export const toDefinitions = (tools: Tools): ReadonlyArray<ToolDefinitionClass> =>
Object.entries(tools).map(
([name, item]) =>
new ToolDefinition({
name,
description: item._definition.description,
inputSchema: item._definition.inputSchema,
outputSchema: item._definition.outputSchema,
}),
)
const toJsonSchema = (schema: Schema.Top): JsonSchema.JsonSchema => {
const document = Schema.toJsonSchemaDocument(schema)
if (Object.keys(document.definitions).length === 0) return document.schema
return { ...document.schema, $defs: document.definitions }
}
const project = (
toModelOutput: ((input: ToolModelOutputInput<any, any>) => ReadonlyArray<ToolContent>) | undefined,
toStructuredOutput: ((output: unknown) => unknown) | undefined,
parameters: unknown,
callID: ToolCallPart["id"],
output: unknown,
): ToolOutputType =>
ToolOutput.make(
toStructuredOutput?.(output) ?? output,
toModelOutput?.({ callID, parameters, output }) ??
(typeof output === "string" ? [{ type: "text", text: output }] : []),
)
export { ToolFailure }
export * as Tool from "./tool"

3
packages/llm/src/utils/record.ts Normal file
View File

@@ -0,0 +1,3 @@
/** Plain-record narrowing. Excludes arrays so JSON object checks don't accept tuples as key/value bags. */
export const isRecord = (value: unknown): value is Record<string, unknown> =>
typeof value === "object" && value !== null && !Array.isArray(value)

10
packages/llm/sst-env.d.ts vendored Normal file
View File

@@ -0,0 +1,10 @@
/* This file is auto-generated by SST. Do not edit. */
/* tslint:disable */
/* eslint-disable */
/* deno-fmt-ignore-file */
/* biome-ignore-all lint: auto-generated */
/// <reference path="../../sst-env.d.ts" />
import "sst"
export {}

164
packages/llm/test/adapter.test.ts Normal file
View File

@@ -0,0 +1,164 @@
import { describe, expect } from "bun:test"
import { Effect, Schema, Stream } from "effect"
import { LLM } from "../src"
import { Route, Endpoint, LLMClient, Protocol, type FramingDef } from "../src/route"
import { Model } from "../src/schema"
import { testEffect } from "./lib/effect"
import { dynamicResponse } from "./lib/http"
const updateModel = (model: Model, patch: Partial<Model.Input>) => Model.update(model, patch)
const Json = Schema.fromJsonString(Schema.Unknown)
const encodeJson = Schema.encodeSync(Json)
type FakeBody = {
readonly body: string
}
const FakeEvent = Schema.Union([
Schema.Struct({ type: Schema.Literal("text"), text: Schema.String }),
Schema.Struct({ type: Schema.Literal("finish"), reason: Schema.Literal("stop") }),
])
type FakeEvent = Schema.Schema.Type<typeof FakeEvent>
const decodeFakeEvents = Schema.decodeUnknownEffect(Schema.fromJsonString(Schema.Array(FakeEvent)))
const fakeFraming: FramingDef<FakeEvent> = {
id: "fake-json-array",
frame: (bytes) =>
Stream.fromEffect(
bytes.pipe(
Stream.decodeText(),
Stream.runFold(
() => "",
(text, event) => text + event,
),
Effect.flatMap(decodeFakeEvents),
Effect.orDie,
),
).pipe(Stream.flatMap(Stream.fromIterable)),
}
const raiseEvent = (event: FakeEvent): import("../src/schema").LLMEvent =>
event.type === "finish"
? { type: "finish", reason: event.reason }
: { type: "text-delta", id: "text-0", text: event.text }
const fakeProtocol = Protocol.make<FakeBody, FakeEvent, FakeEvent, void>({
id: "fake",
body: {
schema: Schema.Struct({
body: Schema.String,
}),
from: (request) =>
Effect.succeed({
body: [
...request.messages
.flatMap((message) => message.content)
.filter((part) => part.type === "text")
.map((part) => part.text),
...request.tools.map((tool) => `tool:${tool.name}:${tool.description}`),
].join("\n"),
}),
},
stream: {
event: FakeEvent,
initial: () => undefined,
step: (state, event) => Effect.succeed([state, [raiseEvent(event)]] as const),
},
})
const fake = Route.make({
id: "fake",
protocol: fakeProtocol,
endpoint: Endpoint.path("/chat"),
framing: fakeFraming,
})
const configuredFake = fake.with({ endpoint: { baseURL: "https://fake.local" } })
const gemini = Route.make({
id: "gemini-fake",
protocol: fakeProtocol,
endpoint: Endpoint.path("/chat"),
framing: fakeFraming,
})
const configuredGemini = gemini.with({ endpoint: { baseURL: "https://fake.local" } })
const request = LLM.request({
id: "req_1",
model: Model.make({
id: "fake-model",
provider: "fake-provider",
route: configuredFake,
}),
prompt: "hello",
})
const echoLayer = dynamicResponse(({ text, respond }) =>
Effect.succeed(
respond(
encodeJson([
{ type: "text", text: `echo:${text}` },
{ type: "finish", reason: "stop" },
]),
),
),
)
const it = testEffect(echoLayer)
describe("llm route", () => {
it.effect("stream and generate use the route pipeline", () =>
Effect.gen(function* () {
const llm = yield* LLMClient.Service
const events = Array.from(yield* llm.stream(request).pipe(Stream.runCollect))
const response = yield* llm.generate(request)
expect(events.map((event) => event.type)).toEqual(["text-delta", "finish"])
expect(response.events.map((event) => event.type)).toEqual(["text-delta", "finish"])
}),
)
it.effect("selects routes by model route value", () =>
Effect.gen(function* () {
const llm = yield* LLMClient.Service
const prepared = yield* llm.prepare(
LLM.updateRequest(request, { model: updateModel(request.model, { route: configuredGemini }) }),
)
expect(prepared.route).toBe("gemini-fake")
}),
)
it.effect("builds models from configured routes", () =>
Effect.gen(function* () {
const configured = fake.with({ provider: "fake-provider", endpoint: { baseURL: "https://fake.local" } })
expect(configured.model({ id: "fake-model" })).toMatchObject({
provider: "fake-provider",
})
}),
)
it.effect("does not register duplicate route ids globally", () =>
Effect.gen(function* () {
const duplicate = Route.make({
id: "fake",
protocol: Protocol.make({
...fakeProtocol,
body: {
...fakeProtocol.body,
from: () => Effect.succeed({ body: "late-default" }),
},
}),
endpoint: Endpoint.path("/chat", { baseURL: "https://fake.local" }),
framing: fakeFraming,
})
const prepared = yield* (yield* LLMClient.Service).prepare(
LLM.updateRequest(request, { model: updateModel(request.model, { route: duplicate }) }),
)
expect(prepared.body).toEqual({ body: "late-default" })
}),
)
})

168
packages/llm/test/auth-options.types.ts Normal file
View File

@@ -0,0 +1,168 @@
import { Config } from "effect"
import type { Auth } from "../src/route/auth"
import type { ModelFactory } from "../src/route/auth-options"
import { Auth as RuntimeAuth } from "../src/route/auth"
import * as OpenAIChat from "../src/protocols/openai-chat"
import * as AmazonBedrock from "../src/providers/amazon-bedrock"
import * as Anthropic from "../src/providers/anthropic"
import * as Azure from "../src/providers/azure"
import * as Cloudflare from "../src/providers/cloudflare"
import * as GitHubCopilot from "../src/providers/github-copilot"
import * as Google from "../src/providers/google"
import * as OpenAI from "../src/providers/openai"
import * as OpenAICompatible from "../src/providers/openai-compatible"
import * as OpenRouter from "../src/providers/openrouter"
import * as XAI from "../src/providers/xai"
type BaseOptions = {
readonly baseURL?: string
readonly headers?: Record<string, string>
}
type Model = {
readonly id: string
}
declare const auth: Auth
declare const optionalAuthModel: ModelFactory<BaseOptions, "optional", Model>
declare const requiredAuthModel: ModelFactory<BaseOptions, "required", Model>
const configApiKey = Config.redacted("OPENAI_API_KEY")
OpenAIChat.route.model({ id: "gpt-4.1-mini" })
// @ts-expect-error route model selection does not configure endpoints.
OpenAIChat.route.model({ id: "gpt-4.1-mini", baseURL: "https://gateway.example.com/v1" })
// @ts-expect-error route model selection does not configure query params.
OpenAIChat.route.model({ id: "gpt-4.1-mini", queryParams: { debug: "1" } })
// @ts-expect-error route model selection does not configure auth.
OpenAIChat.route.model({ id: "gpt-4.1-mini", auth })
// @ts-expect-error route model selection does not configure api keys.
OpenAIChat.route.model({ id: "gpt-4.1-mini", apiKey: "sk-test" })
optionalAuthModel("gpt-4.1-mini")
optionalAuthModel("gpt-4.1-mini", {})
optionalAuthModel("gpt-4.1-mini", { apiKey: "sk-test" })
optionalAuthModel("gpt-4.1-mini", { apiKey: configApiKey })
optionalAuthModel("gpt-4.1-mini", { auth })
optionalAuthModel("gpt-4.1-mini", { auth, baseURL: "https://gateway.example.com/v1" })
optionalAuthModel("gpt-4.1-mini", { apiKey: "sk-test", headers: { "x-source": "test" } })
// @ts-expect-error auth is an override, so apiKey cannot be supplied with it.
optionalAuthModel("gpt-4.1-mini", { apiKey: "sk-test", auth })
requiredAuthModel("custom-model", { apiKey: "key" })
requiredAuthModel("custom-model", { apiKey: configApiKey })
requiredAuthModel("custom-model", { auth })
requiredAuthModel("custom-model", { auth, headers: { "x-tenant-id": "tenant" } })
// @ts-expect-error providers without config fallback need apiKey or auth.
requiredAuthModel("custom-model")
// @ts-expect-error providers without config fallback need apiKey or auth.
requiredAuthModel("custom-model", {})
// @ts-expect-error auth is an override, so apiKey cannot be supplied with it.
requiredAuthModel("custom-model", { apiKey: "key", auth })
OpenAI.responses("gpt-4.1-mini")
OpenAI.configure({}).responses("gpt-4.1-mini")
OpenAI.configure({ apiKey: "sk-test" }).responses("gpt-4.1-mini")
OpenAI.configure({ apiKey: configApiKey }).responses("gpt-4.1-mini")
OpenAI.configure({ auth: RuntimeAuth.bearer("oauth-token") }).responses("gpt-4.1-mini")
OpenAI.configure({
auth: RuntimeAuth.headers({ authorization: "Bearer gateway" }),
baseURL: "https://gateway.example.com/v1",
}).responses("gpt-4.1-mini")
OpenAI.configure({
generation: { maxTokens: 100 },
providerOptions: { openai: { store: false } },
}).responses("gpt-4.1-mini")
// @ts-expect-error OpenAI model selectors only accept model ids.
OpenAI.configure({ apiKey: "sk-test" }).responses("gpt-4.1-mini", {})
// @ts-expect-error apiKey only accepts string, Redacted<string>, or Config<string | Redacted<string>>.
OpenAI.configure({ apiKey: 123 })
// @ts-expect-error provider helpers reject unknown top-level options.
OpenAI.configure({ bogus: true })
// @ts-expect-error common generation options remain typed.
OpenAI.configure({ generation: { maxTokens: "many" } })
// @ts-expect-error provider-native options remain typed.
OpenAI.configure({ providerOptions: { openai: { store: "false" } } })
// @ts-expect-error auth is an override, so OpenAI rejects apiKey with auth.
OpenAI.configure({ apiKey: "sk-test", auth: RuntimeAuth.bearer("oauth-token") })
OpenAI.chat("gpt-4.1-mini")
OpenAI.configure({ apiKey: "sk-test" }).chat("gpt-4.1-mini")
OpenAI.configure({ apiKey: configApiKey }).chat("gpt-4.1-mini")
OpenAI.configure({ auth: RuntimeAuth.bearer("oauth-token") }).chat("gpt-4.1-mini")
// @ts-expect-error OpenAI chat selectors only accept model ids.
OpenAI.configure({ apiKey: "sk-test" }).chat("gpt-4.1-mini", {})
// @ts-expect-error auth is an override, so OpenAI Chat rejects apiKey with auth.
OpenAI.configure({ apiKey: "sk-test", auth: RuntimeAuth.bearer("oauth-token") })
// @ts-expect-error Azure requires at least one of `resourceName` or `baseURL`.
Azure.configure()
Azure.configure({ apiKey: "azure-key", resourceName: "resource" }).responses("deployment")
Azure.configure({ apiKey: configApiKey, resourceName: "resource" }).responses("deployment")
Azure.configure({ auth: RuntimeAuth.header("api-key", "azure-key"), resourceName: "resource" }).responses("deployment")
// @ts-expect-error Azure model selectors only accept deployment ids.
Azure.configure({ apiKey: "azure-key", resourceName: "resource" }).responses("deployment", {})
// @ts-expect-error auth is an override, so Azure rejects apiKey with auth.
Azure.configure({ resourceName: "resource", apiKey: "azure-key", auth: RuntimeAuth.header("api-key", "override") })
Azure.configure({ apiKey: "azure-key", resourceName: "resource" }).chat("deployment")
Azure.configure({ apiKey: configApiKey, resourceName: "resource" }).chat("deployment")
Azure.configure({ auth: RuntimeAuth.header("api-key", "azure-key"), resourceName: "resource" }).chat("deployment")
// @ts-expect-error Azure chat model selectors only accept deployment ids.
Azure.configure({ apiKey: "azure-key", resourceName: "resource" }).chat("deployment", {})
// @ts-expect-error auth is an override, so Azure Chat rejects apiKey with auth.
Azure.configure({ resourceName: "resource", apiKey: "azure-key", auth: RuntimeAuth.header("api-key", "override") })
Anthropic.configure({ apiKey: "anthropic-key" }).model("claude-haiku")
// @ts-expect-error Anthropic model selectors only accept model ids.
Anthropic.configure({ apiKey: "anthropic-key" }).model("claude-haiku", {})
Google.configure({ apiKey: "google-key" }).model("gemini-2.5-flash")
// @ts-expect-error Google model selectors only accept model ids.
Google.configure({ apiKey: "google-key" }).model("gemini-2.5-flash", {})
AmazonBedrock.configure({ apiKey: "bedrock-key" }).model("anthropic.claude")
// @ts-expect-error Bedrock model selectors only accept model ids.
AmazonBedrock.configure({ apiKey: "bedrock-key" }).model("anthropic.claude", {})
OpenRouter.configure({ apiKey: "openrouter-key" }).model("openai/gpt-4o-mini")
// @ts-expect-error OpenRouter model selectors only accept model ids.
OpenRouter.configure({ apiKey: "openrouter-key" }).model("openai/gpt-4o-mini", {})
XAI.configure({ apiKey: "xai-key" }).responses("grok-4")
XAI.configure({ apiKey: "xai-key" }).chat("grok-4")
// @ts-expect-error xAI Responses selectors only accept model ids.
XAI.configure({ apiKey: "xai-key" }).responses("grok-4", {})
// @ts-expect-error xAI Chat selectors only accept model ids.
XAI.configure({ apiKey: "xai-key" }).chat("grok-4", {})
OpenAICompatible.deepseek.configure({ apiKey: "deepseek-key" }).model("deepseek-chat")
// @ts-expect-error OpenAI-compatible family selectors only accept model ids.
OpenAICompatible.deepseek.configure({ apiKey: "deepseek-key" }).model("deepseek-chat", {})
Cloudflare.CloudflareWorkersAI.configure({ accountId: "account", apiKey: "cf-key" }).model("@cf/meta/llama")
// @ts-expect-error Cloudflare Workers AI model selectors only accept model ids.
Cloudflare.CloudflareWorkersAI.configure({ accountId: "account", apiKey: "cf-key" }).model("@cf/meta/llama", {})
GitHubCopilot.configure({ baseURL: "https://copilot.test", apiKey: "copilot-key" }).model("gpt-4.1")
// @ts-expect-error GitHub Copilot model selectors only accept model ids.
GitHubCopilot.configure({ baseURL: "https://copilot.test", apiKey: "copilot-key" }).model("gpt-4.1", {})

103
packages/llm/test/auth.test.ts Normal file
View File

@@ -0,0 +1,103 @@
import { describe, expect } from "bun:test"
import { ConfigProvider, Effect } from "effect"
import { Headers } from "effect/unstable/http"
import { LLM } from "../src"
import { Auth } from "../src/route/auth"
import * as OpenAIChat from "../src/protocols/openai-chat"
import { Model } from "../src/schema"
import { it } from "./lib/effect"
const request = LLM.request({
id: "req_auth",
model: Model.make({ id: "fake-model", provider: "fake", route: OpenAIChat.route }),
prompt: "hello",
})
const input = {
request,
method: "POST" as const,
url: "https://example.test/v1/chat",
body: "{}",
headers: Headers.fromInput({ "x-existing": "yes" }),
}
const withEnv = (env: Record<string, string>) => Effect.provide(ConfigProvider.layer(ConfigProvider.fromEnv({ env })))
describe("Auth", () => {
it.effect("renders a config credential as bearer auth", () =>
Effect.gen(function* () {
const headers = yield* Auth.config("OPENAI_API_KEY")
.bearer()
.apply(input)
.pipe(withEnv({ OPENAI_API_KEY: "sk-test" }))
expect(headers.authorization).toBe("Bearer sk-test")
expect(headers["x-existing"]).toBe("yes")
}),
)
it.effect("falls back between credential sources before rendering", () =>
Effect.gen(function* () {
const headers = yield* Auth.config("PRIMARY_KEY")
.orElse(Auth.value("fallback-key"))
.pipe(Auth.header("x-api-key"))
.apply(input)
.pipe(withEnv({}))
expect(headers["x-api-key"]).toBe("fallback-key")
expect(headers["x-existing"]).toBe("yes")
}),
)
it.effect("composes header auth in sequence", () =>
Effect.gen(function* () {
const headers = yield* Auth.headers({ "x-tenant-id": "tenant-1" })
.andThen(Auth.bearer("gateway-token"))
.apply(input)
expect(headers["x-tenant-id"]).toBe("tenant-1")
expect(headers.authorization).toBe("Bearer gateway-token")
expect(headers["x-existing"]).toBe("yes")
}),
)
it.effect("renders a direct secret as a custom header", () =>
Effect.gen(function* () {
const headers = yield* Auth.header("api-key", "direct-key").apply(input)
expect(headers["api-key"]).toBe("direct-key")
expect(headers["x-existing"]).toBe("yes")
}),
)
it.effect("renders bearer auth into a custom header", () =>
Effect.gen(function* () {
const headers = yield* Auth.bearerHeader("cf-aig-authorization", "gateway-token").apply(input)
expect(headers["cf-aig-authorization"]).toBe("Bearer gateway-token")
expect(headers["x-existing"]).toBe("yes")
}),
)
it.effect("falls back between full auth values", () =>
Effect.gen(function* () {
const headers = yield* Auth.config("OPENAI_API_KEY")
.bearer()
.orElse(Auth.headers({ authorization: "Bearer supplied" }))
.apply(input)
.pipe(withEnv({}))
expect(headers.authorization).toBe("Bearer supplied")
expect(headers["x-existing"]).toBe("yes")
}),
)
it.effect("can intentionally leave auth untouched", () =>
Effect.gen(function* () {
const headers = yield* Auth.none.apply(input)
expect(headers.authorization).toBeUndefined()
expect(headers["x-existing"]).toBe("yes")
}),
)
})

262
packages/llm/test/cache-policy.test.ts Normal file
View File

@@ -0,0 +1,262 @@
import { describe, expect, test } from "bun:test"
import { Effect } from "effect"
import { CacheHint, LLM, Message } from "../src"
import { Auth, LLMClient } from "../src/route"
import { AmazonBedrock } from "../src/providers"
import * as AnthropicMessages from "../src/protocols/anthropic-messages"
import * as Gemini from "../src/protocols/gemini"
import * as OpenAIChat from "../src/protocols/openai-chat"
import { applyCachePolicy } from "../src/cache-policy"
import { it } from "./lib/effect"
const anthropicModel = AnthropicMessages.route
.with({ endpoint: { baseURL: "https://api.anthropic.test/v1/" }, auth: Auth.header("x-api-key", "test") })
.model({ id: "claude-sonnet-4-5" })
const bedrockModel = AmazonBedrock.configure({
credentials: { region: "us-east-1", accessKeyId: "fixture", secretAccessKey: "fixture" },
}).model("anthropic.claude-3-5-sonnet-20241022-v2:0")
const openaiModel = OpenAIChat.route
.with({ endpoint: { baseURL: "https://api.openai.test/v1/" }, auth: Auth.bearer("test") })
.model({ id: "gpt-4o-mini" })
const geminiModel = Gemini.route
.with({
endpoint: { baseURL: "https://generativelanguage.test/v1beta/" },
auth: Auth.header("x-goog-api-key", "test"),
})
.model({ id: "gemini-2.5-flash" })
describe("applyCachePolicy", () => {
it.effect("undefined cache resolves to 'auto' (the recommended default)", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare(
LLM.request({
model: anthropicModel,
system: "You are concise.",
prompt: "hi",
}),
)
// No explicit cache field → auto policy fires → last system part + latest
// user message both get cache_control markers.
expect(prepared.body).toMatchObject({
system: [{ type: "text", text: "You are concise.", cache_control: { type: "ephemeral" } }],
messages: [{ role: "user", content: [{ type: "text", text: "hi", cache_control: { type: "ephemeral" } }] }],
})
}),
)
it.effect("'auto' marks the last tool, last system part, and latest user message on Anthropic", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare(
LLM.request({
model: anthropicModel,
system: "Sys A",
tools: [{ name: "t1", description: "t1", inputSchema: { type: "object", properties: {} } }],
messages: [
Message.user("first user"),
Message.assistant("assistant reply"),
Message.user("latest user message"),
],
cache: "auto",
}),
)
expect(prepared.body).toMatchObject({
tools: [{ name: "t1", cache_control: { type: "ephemeral" } }],
system: [{ type: "text", text: "Sys A", cache_control: { type: "ephemeral" } }],
messages: [
{ role: "user", content: [{ type: "text", text: "first user" }] },
{ role: "assistant", content: [{ type: "text", text: "assistant reply" }] },
{
role: "user",
content: [{ type: "text", text: "latest user message", cache_control: { type: "ephemeral" } }],
},
],
})
}),
)
it.effect("'auto' is a no-op on OpenAI (implicit caching protocol)", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare(
LLM.request({
model: openaiModel,
system: "Sys",
prompt: "hi",
cache: "auto",
}),
)
const body = prepared.body as { messages: Array<{ content: unknown }> }
// OpenAI doesn't accept cache_control on messages — policy must skip.
const flat = JSON.stringify(body)
expect(flat).not.toContain("cache_control")
expect(flat).not.toContain("cachePoint")
}),
)
it.effect("'auto' is a no-op on Gemini (out-of-band caching protocol)", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare(
LLM.request({
model: geminiModel,
system: "Sys",
prompt: "hi",
cache: "auto",
}),
)
const flat = JSON.stringify(prepared.body)
expect(flat).not.toContain("cache_control")
expect(flat).not.toContain("cachePoint")
}),
)
it.effect("'auto' on Bedrock emits cachePoint markers in the right places", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare(
LLM.request({
model: bedrockModel,
system: "Sys",
tools: [{ name: "t1", description: "t1", inputSchema: { type: "object", properties: {} } }],
messages: [Message.user("first user"), Message.assistant("reply"), Message.user("latest user")],
cache: "auto",
}),
)
expect(prepared.body).toMatchObject({
toolConfig: {
tools: [{ toolSpec: { name: "t1" } }, { cachePoint: { type: "default" } }],
},
system: [{ text: "Sys" }, { cachePoint: { type: "default" } }],
messages: [
{ role: "user", content: [{ text: "first user" }] },
{ role: "assistant", content: [{ text: "reply" }] },
{ role: "user", content: [{ text: "latest user" }, { cachePoint: { type: "default" } }] },
],
})
}),
)
it.effect("'none' disables auto placement even when manual hints exist", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare(
LLM.request({
model: anthropicModel,
system: "Sys",
tools: [{ name: "t1", description: "t1", inputSchema: { type: "object", properties: {} } }],
prompt: "hi",
cache: "none",
}),
)
expect(prepared.body).toMatchObject({
tools: [{ name: "t1", cache_control: undefined }],
system: [{ type: "text", text: "Sys", cache_control: undefined }],
})
}),
)
it.effect("granular object form: tools-only marks just tools", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare(
LLM.request({
model: anthropicModel,
system: "Sys",
tools: [{ name: "t1", description: "t1", inputSchema: { type: "object", properties: {} } }],
prompt: "hi",
cache: { tools: true },
}),
)
expect(prepared.body).toMatchObject({
tools: [{ name: "t1", cache_control: { type: "ephemeral" } }],
system: [{ type: "text", text: "Sys", cache_control: undefined }],
})
}),
)
it.effect("auto policy preserves manual CacheHints on other parts", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare(
LLM.request({
model: anthropicModel,
system: [
{ type: "text", text: "first system", cache: new CacheHint({ type: "ephemeral", ttlSeconds: 3600 }) },
{ type: "text", text: "last system" },
],
prompt: "hi",
cache: "auto",
}),
)
const body = prepared.body as { system: Array<{ text: string; cache_control?: unknown }> }
expect(body.system[0]?.cache_control).toEqual({ type: "ephemeral", ttl: "1h" })
expect(body.system[1]?.cache_control).toEqual({ type: "ephemeral" })
}),
)
it.effect("ttlSeconds in the policy flows through to wire markers", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare(
LLM.request({
model: anthropicModel,
system: "Sys",
prompt: "hi",
cache: { system: true, ttlSeconds: 3600 },
}),
)
expect(prepared.body).toMatchObject({
system: [{ type: "text", text: "Sys", cache_control: { type: "ephemeral", ttl: "1h" } }],
})
}),
)
it.effect("messages: { tail: 2 } marks the last 2 message boundaries", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare(
LLM.request({
model: anthropicModel,
messages: [Message.user("u1"), Message.assistant("a1"), Message.user("u2"), Message.assistant("a2")],
cache: { messages: { tail: 2 } },
}),
)
const body = prepared.body as { messages: Array<{ content: Array<{ cache_control?: unknown }> }> }
expect(body.messages[0]?.content[0]?.cache_control).toBeUndefined()
expect(body.messages[1]?.content[0]?.cache_control).toBeUndefined()
expect(body.messages[2]?.content[0]?.cache_control).toEqual({ type: "ephemeral" })
expect(body.messages[3]?.content[0]?.cache_control).toEqual({ type: "ephemeral" })
}),
)
it.effect("'latest-assistant' marks the last assistant message", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare(
LLM.request({
model: anthropicModel,
messages: [Message.user("u1"), Message.assistant("a1"), Message.user("u2")],
cache: { messages: "latest-assistant" },
}),
)
const body = prepared.body as { messages: Array<{ content: Array<{ cache_control?: unknown }> }> }
expect(body.messages[0]?.content[0]?.cache_control).toBeUndefined()
expect(body.messages[1]?.content[0]?.cache_control).toEqual({ type: "ephemeral" })
expect(body.messages[2]?.content[0]?.cache_control).toBeUndefined()
}),
)
test("returns the same request reference when policy is a no-op (pure function)", () => {
const request = LLM.request({
model: anthropicModel,
prompt: "hi",
cache: "none",
})
expect(applyCachePolicy(request)).toBe(request)
})
})

104
packages/llm/test/continuation-scenarios.ts Normal file
View File

@@ -0,0 +1,104 @@
import { LLM, Message, ToolCallPart, ToolDefinition, ToolResultPart, type ContentPart, type Model } from "../src"
export const basicContinuation = ["system", "user-text", "assistant-text", "user-follow-up"] as const
export const toolContinuation = ["tool-call", "tool-result"] as const
export const reasoningContinuation = ["assistant-reasoning", "encrypted-reasoning"] as const
export const mediaContinuation = ["user-image"] as const
export const maximalContinuation = [
...basicContinuation,
...toolContinuation,
...reasoningContinuation,
...mediaContinuation,
] as const
export type ContinuationFeature = (typeof maximalContinuation)[number]
export const nativeOpenAIResponsesContinuation = [
...basicContinuation,
...toolContinuation,
"encrypted-reasoning",
...mediaContinuation,
] as const satisfies ReadonlyArray<ContinuationFeature>
export const nativeAnthropicMessagesContinuation = [
...basicContinuation,
...toolContinuation,
"assistant-reasoning",
...mediaContinuation,
] as const satisfies ReadonlyArray<ContinuationFeature>
export const continuationTool = ToolDefinition.make({
name: "get_weather",
description: "Get current weather for a city.",
inputSchema: {
type: "object",
properties: { city: { type: "string" } },
required: ["city"],
additionalProperties: false,
},
})
export function continuationRequest(input: {
readonly id: string
readonly model: Model
readonly features: ReadonlyArray<ContinuationFeature>
readonly image?: string
}) {
const features = new Set(input.features)
const messages = []
const firstUser: ContentPart[] = []
const firstAssistant: ContentPart[] = []
if (features.has("user-text")) firstUser.push({ type: "text", text: "What is shown here?" })
if (features.has("user-image"))
firstUser.push({ type: "media", mediaType: "image/png", data: input.image ?? "AAECAw==" })
if (firstUser.length > 0) messages.push(Message.user(firstUser))
if (features.has("assistant-reasoning"))
firstAssistant.push({
type: "reasoning",
text: "I inspected the previous turn.",
providerMetadata: { anthropic: { signature: "sig_continuation_1" } },
})
if (features.has("encrypted-reasoning"))
firstAssistant.push({
type: "reasoning",
text: "I inspected the previous turn.",
providerMetadata: {
openai: {
itemId: "rs_continuation_1",
reasoningEncryptedContent: "encrypted-continuation-state",
},
},
})
if (features.has("assistant-text")) firstAssistant.push({ type: "text", text: "It shows a small test image." })
if (firstAssistant.length > 0) messages.push(Message.assistant(firstAssistant))
if (features.has("tool-call")) {
messages.push(Message.user("Check the weather in Paris before continuing."))
messages.push(
Message.assistant([ToolCallPart.make({ id: "call_weather_1", name: "get_weather", input: { city: "Paris" } })]),
)
}
if (features.has("tool-result")) {
messages.push(
Message.tool(ToolResultPart.make({ id: "call_weather_1", name: "get_weather", result: { temperature: 22 } })),
)
if (features.has("assistant-text")) messages.push(Message.assistant("Paris is 22 degrees."))
}
if (features.has("user-follow-up"))
messages.push(Message.user("Continue from this conversation in one short sentence."))
return LLM.request({
id: input.id,
model: input.model,
system: features.has("system") ? "You are concise. Continue from the provided history." : undefined,
messages,
tools: features.has("tool-call") ? [continuationTool] : [],
cache: "none",
providerOptions: features.has("encrypted-reasoning")
? { openai: { store: false, include: ["reasoning.encrypted_content"], reasoningSummary: "auto" } }
: undefined,
generation: { maxTokens: 80, temperature: 0 },
})
}

58
packages/llm/test/endpoint.test.ts Normal file
View File

@@ -0,0 +1,58 @@
import { describe, expect, test } from "bun:test"
import { LLM } from "../src"
import * as OpenAIChat from "../src/protocols/openai-chat"
import { Endpoint } from "../src/route"
import { Model } from "../src/schema"
const request = () =>
LLM.request({
model: Model.make({
id: "model-1",
provider: "test",
route: OpenAIChat.route,
}),
prompt: "hello",
})
describe("Endpoint", () => {
test("appends a static path to the model's baseURL", () => {
const url = Endpoint.render(Endpoint.path("/chat", { baseURL: "https://api.example.test/v1/" }), {
request: request(),
body: {},
})
expect(url.toString()).toBe("https://api.example.test/v1/chat")
})
test("endpoint query params are appended to the rendered URL", () => {
const url = Endpoint.render(
Endpoint.path("/chat?alt=sse", {
baseURL: "https://custom.example.test/root/",
query: { "api-version": "2026-01-01", alt: "json" },
}),
{
request: request(),
body: {},
},
)
expect(url.toString()).toBe("https://custom.example.test/root/chat?alt=json&api-version=2026-01-01")
})
test("path may be a function of the validated body", () => {
const url = Endpoint.render(
Endpoint.path<{ readonly modelId: string }>(
({ body }) => `/model/${encodeURIComponent(body.modelId)}/converse-stream`,
{ baseURL: "https://bedrock-runtime.us-east-1.amazonaws.com" },
),
{
request: request(),
body: { modelId: "us.amazon.nova-micro-v1:0" },
},
)
expect(url.toString()).toBe(
"https://bedrock-runtime.us-east-1.amazonaws.com/model/us.amazon.nova-micro-v1%3A0/converse-stream",
)
})
})

458
packages/llm/test/executor.test.ts Normal file
View File

@@ -0,0 +1,458 @@
import { describe, expect } from "bun:test"
import { Effect, Fiber, Layer, Random, Ref } from "effect"
import * as TestClock from "effect/testing/TestClock"
import { Headers, HttpClient, HttpClientRequest, HttpClientResponse } from "effect/unstable/http"
import { LLM, LLMError } from "../src"
import { LLMClient, RequestExecutor } from "../src/route"
import * as OpenAIChat from "../src/protocols/openai-chat"
import { dynamicResponse } from "./lib/http"
import { deltaChunk } from "./lib/openai-chunks"
import { sseRaw } from "./lib/sse"
import { it } from "./lib/effect"
const request = HttpClientRequest.post("https://provider.test/v1/chat?api_key=secret&key=secret&debug=1").pipe(
HttpClientRequest.setHeaders(Headers.fromInput({ authorization: "Bearer secret", "x-safe": "visible" })),
)
const secretRequest = HttpClientRequest.post("https://provider.test/v1/chat?api_key=query-secret-123&debug=1").pipe(
HttpClientRequest.setHeaders(Headers.fromInput({ authorization: "Bearer header-secret-456" })),
)
const responsesLayer = (responses: ReadonlyArray<Response>) =>
RequestExecutor.layer.pipe(
Layer.provide(
Layer.unwrap(
Effect.gen(function* () {
const cursor = yield* Ref.make(0)
return Layer.succeed(
HttpClient.HttpClient,
HttpClient.make((request) =>
Effect.gen(function* () {
const index = yield* Ref.getAndUpdate(cursor, (value) => value + 1)
return HttpClientResponse.fromWeb(request, responses[index] ?? responses[responses.length - 1])
}),
),
)
}),
),
),
)
const countedResponsesLayer = (attempts: Ref.Ref<number>, responses: ReadonlyArray<Response>) =>
RequestExecutor.layer.pipe(
Layer.provide(
Layer.unwrap(
Effect.gen(function* () {
const cursor = yield* Ref.make(0)
return Layer.succeed(
HttpClient.HttpClient,
HttpClient.make((request) =>
Effect.gen(function* () {
yield* Ref.update(attempts, (value) => value + 1)
const index = yield* Ref.getAndUpdate(cursor, (value) => value + 1)
return HttpClientResponse.fromWeb(request, responses[index] ?? responses[responses.length - 1])
}),
),
)
}),
),
),
)
const randomMidpoint = {
nextDoubleUnsafe: () => 0.5,
nextIntUnsafe: () => 0,
}
const expectLLMError = (error: unknown) => {
expect(error).toBeInstanceOf(LLMError)
if (!(error instanceof LLMError)) throw new Error("expected LLMError")
return error
}
const errorHttp = (error: LLMError) => ("http" in error.reason ? error.reason.http : undefined)
describe("RequestExecutor", () => {
it.effect("classifies context overflow responses", () =>
Effect.gen(function* () {
const executor = yield* RequestExecutor.Service
const error = yield* executor.execute(request).pipe(Effect.flip)
expectLLMError(error)
expect(error.reason).toMatchObject({ _tag: "InvalidRequest", classification: "context-overflow" })
}).pipe(
Effect.provide(
responsesLayer([
new Response('{"error":{"code":"context_length_exceeded","message":"prompt too long"}}', {
status: 400,
}),
]),
),
),
)
it.effect("does not classify generic HTTP 413 payload errors as context overflow", () =>
Effect.gen(function* () {
const executor = yield* RequestExecutor.Service
const error = yield* executor.execute(request).pipe(Effect.flip)
expectLLMError(error)
expect(error.reason).toMatchObject({ _tag: "InvalidRequest" })
expect("classification" in error.reason ? error.reason.classification : undefined).toBeUndefined()
}).pipe(Effect.provide(responsesLayer([new Response("request too large", { status: 413 })]))),
)
it.effect("does not classify ordinary invalid requests as context overflow", () =>
Effect.gen(function* () {
const executor = yield* RequestExecutor.Service
const error = yield* executor.execute(request).pipe(Effect.flip)
expectLLMError(error)
expect(error.reason).toMatchObject({ _tag: "InvalidRequest" })
expect("classification" in error.reason ? error.reason.classification : undefined).toBeUndefined()
}).pipe(Effect.provide(responsesLayer([new Response("invalid parameter", { status: 400 })]))),
)
it.effect("returns redacted diagnostics for retryable rate limits", () =>
Effect.gen(function* () {
const executor = yield* RequestExecutor.Service
const error = yield* executor.execute(request).pipe(Effect.flip)
expectLLMError(error)
expect(error).toMatchObject({
retryable: true,
retryAfterMs: 0,
reason: {
_tag: "RateLimit",
rateLimit: { retryAfterMs: 0 },
http: {
requestId: "req_123",
request: {
method: "POST",
url: "https://provider.test/v1/chat?api_key=%3Credacted%3E&key=%3Credacted%3E&debug=1",
headers: { authorization: "<redacted>", "x-safe": "visible" },
},
response: {
status: 429,
headers: {
"retry-after-ms": "0",
"x-request-id": "req_123",
"x-api-key": "<redacted>",
},
},
},
},
})
expect(errorHttp(error)?.body).toBe("rate limited")
}).pipe(
Effect.provide(
responsesLayer(
Array.from(
{ length: 3 },
() =>
new Response("rate limited", {
status: 429,
headers: { "retry-after-ms": "0", "x-request-id": "req_123", "x-api-key": "secret" },
}),
),
),
),
),
)
it.effect("honors current redacted header names in diagnostics", () =>
Effect.gen(function* () {
const executor = yield* RequestExecutor.Service
const error = yield* executor.execute(request).pipe(Effect.flip)
expectLLMError(error)
expect(errorHttp(error)?.request.headers["x-safe"]).toBe("<redacted>")
expect(errorHttp(error)?.response?.headers["x-safe"]).toBe("<redacted>")
}).pipe(
Effect.provide(responsesLayer([new Response("bad", { status: 400, headers: { "x-safe": "response-secret" } })])),
Effect.provideService(Headers.CurrentRedactedNames, ["x-safe"]),
),
)
it.effect("extracts OpenAI-style rate-limit diagnostics", () =>
Effect.gen(function* () {
const executor = yield* RequestExecutor.Service
const error = yield* executor.execute(request).pipe(Effect.flip)
expectLLMError(error)
expect(error.reason).toMatchObject({ _tag: "RateLimit" })
expect(error.reason._tag === "RateLimit" ? error.reason.rateLimit : undefined).toEqual({
retryAfterMs: 0,
limit: { requests: "500", tokens: "30000" },
remaining: { requests: "499", tokens: "29900" },
reset: { requests: "1s", tokens: "10s" },
})
}).pipe(
Effect.provide(
responsesLayer(
Array.from(
{ length: 3 },
() =>
new Response("rate limited", {
status: 429,
headers: {
"retry-after-ms": "0",
"x-ratelimit-limit-requests": "500",
"x-ratelimit-limit-tokens": "30000",
"x-ratelimit-remaining-requests": "499",
"x-ratelimit-remaining-tokens": "29900",
"x-ratelimit-reset-requests": "1s",
"x-ratelimit-reset-tokens": "10s",
},
}),
),
),
),
),
)
it.effect("extracts Anthropic-style rate-limit diagnostics", () =>
Effect.gen(function* () {
const executor = yield* RequestExecutor.Service
const error = yield* executor.execute(request).pipe(Effect.flip)
expectLLMError(error)
expect(error.reason).toMatchObject({ _tag: "ProviderInternal" })
expect(errorHttp(error)?.rateLimit).toEqual({
retryAfterMs: 0,
limit: { requests: "100", "input-tokens": "10000" },
remaining: { requests: "12", "input-tokens": "9000" },
reset: { requests: "2026-05-06T12:00:00Z", "input-tokens": "2026-05-06T12:00:10Z" },
})
}).pipe(
Effect.provide(
responsesLayer(
Array.from(
{ length: 3 },
() =>
new Response("overloaded", {
status: 529,
headers: {
"retry-after-ms": "0",
"anthropic-ratelimit-requests-limit": "100",
"anthropic-ratelimit-requests-remaining": "12",
"anthropic-ratelimit-requests-reset": "2026-05-06T12:00:00Z",
"anthropic-ratelimit-input-tokens-limit": "10000",
"anthropic-ratelimit-input-tokens-remaining": "9000",
"anthropic-ratelimit-input-tokens-reset": "2026-05-06T12:00:10Z",
},
}),
),
),
),
),
)
it.effect("retries retryable status responses before returning the stream", () =>
Effect.gen(function* () {
const executor = yield* RequestExecutor.Service
const response = yield* executor.execute(request)
expect(response.status).toBe(200)
expect(yield* response.text).toBe("ok")
}).pipe(
Effect.provide(
responsesLayer([
new Response("busy", { status: 503, headers: { "retry-after-ms": "0" } }),
new Response("ok", { status: 200 }),
]),
),
),
)
it.effect("marks 504 and 529 status responses retryable", () =>
Effect.gen(function* () {
const failWith = (status: number) =>
Effect.gen(function* () {
const executor = yield* RequestExecutor.Service
const error = yield* executor.execute(request).pipe(Effect.flip)
expectLLMError(error)
expect(error.reason).toMatchObject({ _tag: "ProviderInternal", status })
expect(error.retryable).toBe(true)
}).pipe(
Effect.provide(
responsesLayer(
Array.from(
{ length: 3 },
() =>
new Response("retry", {
status,
headers: { "retry-after-ms": "0" },
}),
),
),
),
)
yield* failWith(504)
yield* failWith(529)
}),
)
it.effect("does not retry non-retryable status responses and truncates large bodies", () =>
Effect.gen(function* () {
const executor = yield* RequestExecutor.Service
const error = yield* executor.execute(request).pipe(Effect.flip)
expectLLMError(error)
expect(error.reason).toMatchObject({ _tag: "Authentication" })
expect(error.retryable).toBe(false)
expect(errorHttp(error)?.bodyTruncated).toBe(true)
expect(errorHttp(error)?.body).toHaveLength(16_384)
}).pipe(
Effect.provide(
responsesLayer([
new Response("x".repeat(20_000), { status: 401 }),
new Response("should not retry", { status: 200 }),
]),
),
),
)
it.effect("redacts common secret fields in response bodies", () =>
Effect.gen(function* () {
const executor = yield* RequestExecutor.Service
const error = yield* executor.execute(request).pipe(Effect.flip)
expectLLMError(error)
expect(errorHttp(error)?.body).toContain('"key":"<redacted>"')
expect(errorHttp(error)?.body).toContain("api_key=<redacted>")
expect(errorHttp(error)?.body).not.toContain("body-secret")
expect(errorHttp(error)?.body).not.toContain("query-secret")
}).pipe(
Effect.provide(
responsesLayer([
new Response('{"error":{"message":"bad","key":"body-secret","detail":"api_key=query-secret"}}', {
status: 400,
}),
]),
),
),
)
it.effect("redacts echoed request secret values in response bodies", () =>
Effect.gen(function* () {
const executor = yield* RequestExecutor.Service
const error = yield* executor.execute(secretRequest).pipe(Effect.flip)
expectLLMError(error)
expect(errorHttp(error)?.body).toContain("provider echoed <redacted>")
expect(errorHttp(error)?.body).toContain("authorization <redacted>")
expect(errorHttp(error)?.body).not.toContain("query-secret-123")
expect(errorHttp(error)?.body).not.toContain("header-secret-456")
}).pipe(
Effect.provide(
responsesLayer([
new Response("provider echoed query-secret-123 and authorization header-secret-456", { status: 400 }),
]),
),
),
)
it.effect("honors Retry-After delta seconds before retrying", () =>
Effect.gen(function* () {
const attempts = yield* Ref.make(0)
return yield* Effect.gen(function* () {
const executor = yield* RequestExecutor.Service
const fiber = yield* executor.execute(request).pipe(Effect.forkChild)
yield* Effect.yieldNow
expect(yield* Ref.get(attempts)).toBe(1)
yield* TestClock.adjust(1_999)
yield* Effect.yieldNow
expect(yield* Ref.get(attempts)).toBe(1)
yield* TestClock.adjust(1)
const response = yield* Fiber.join(fiber)
expect(response.status).toBe(200)
expect(yield* Ref.get(attempts)).toBe(2)
}).pipe(
Effect.provide(
countedResponsesLayer(attempts, [
new Response("busy", { status: 503, headers: { "retry-after": "2" } }),
new Response("ok", { status: 200 }),
]),
),
)
}),
)
it.effect("uses exponential jittered delay when retry-after is absent", () =>
Effect.gen(function* () {
const attempts = yield* Ref.make(0)
return yield* Effect.gen(function* () {
const executor = yield* RequestExecutor.Service
const fiber = yield* executor.execute(request).pipe(Effect.flip, Effect.forkChild)
yield* Effect.yieldNow
expect(yield* Ref.get(attempts)).toBe(1)
yield* TestClock.adjust(499)
yield* Effect.yieldNow
expect(yield* Ref.get(attempts)).toBe(1)
yield* TestClock.adjust(1)
yield* Effect.yieldNow
expect(yield* Ref.get(attempts)).toBe(2)
yield* TestClock.adjust(999)
yield* Effect.yieldNow
expect(yield* Ref.get(attempts)).toBe(2)
yield* TestClock.adjust(1)
const error = yield* Fiber.join(fiber)
expectLLMError(error)
expect(error.reason).toMatchObject({ _tag: "ProviderInternal" })
expect(yield* Ref.get(attempts)).toBe(3)
}).pipe(
Effect.provide(
countedResponsesLayer(attempts, [
new Response("busy", { status: 503 }),
new Response("still busy", { status: 503 }),
new Response("done retrying", { status: 503 }),
]),
),
)
}).pipe(Effect.provideService(Random.Random, randomMidpoint)),
)
it.effect("does not retry after a successful response reaches stream parsing", () =>
Effect.gen(function* () {
const attempts = yield* Ref.make(0)
const model = OpenAIChat.route
.with({ endpoint: { baseURL: "https://api.openai.test/v1" } })
.model({ id: "gpt-4o-mini" })
const error = yield* LLMClient.generate(LLM.request({ model, prompt: "Say hello." })).pipe(
Effect.provide(
dynamicResponse((input) =>
Ref.update(attempts, (value) => value + 1).pipe(
Effect.as(
input.respond(
sseRaw(
`data: ${JSON.stringify(deltaChunk({ role: "assistant", content: "Hello" }))}`,
"data: not-json",
),
{ headers: { "content-type": "text/event-stream" } },
),
),
),
),
),
Effect.flip,
)
expectLLMError(error)
expect(error.reason).toMatchObject({ _tag: "InvalidProviderOutput" })
expect(yield* Ref.get(attempts)).toBe(1)
}),
)
})

62
packages/llm/test/exports.test.ts Normal file
View File

@@ -0,0 +1,62 @@
import { describe, expect, test } from "bun:test"
import { LLM, LLMClient, Provider } from "@opencode-ai/llm"
import { Route, Protocol } from "@opencode-ai/llm/route"
import { Provider as ProviderSubpath } from "@opencode-ai/llm/provider"
import {
CloudflareAIGateway,
CloudflareWorkersAI,
OpenAI,
OpenAICompatible,
OpenRouter,
XAI,
} from "@opencode-ai/llm/providers"
import * as GitHubCopilot from "@opencode-ai/llm/providers/github-copilot"
import { OpenAIChat, OpenAICompatibleChat, OpenAIResponses } from "@opencode-ai/llm/protocols"
import * as AnthropicMessages from "@opencode-ai/llm/protocols/anthropic-messages"
describe("public exports", () => {
test("root exposes app-facing runtime APIs", () => {
expect(LLM.request).toBeFunction()
expect(LLMClient.Service).toBeFunction()
expect(LLMClient.layer).toBeDefined()
expect(Provider.make).toBeFunction()
expect(ProviderSubpath.make).toBe(Provider.make)
})
test("route barrel exposes route-authoring APIs", () => {
expect(Route.make).toBeFunction()
expect(Protocol.make).toBeFunction()
})
test("provider barrels expose user-facing facades", () => {
expect(OpenAI.model).toBeFunction()
expect(OpenAI.provider.model).toBe(OpenAI.model)
expect(OpenAI.provider.responses).toBe(OpenAI.responses)
expect(OpenAI.provider.responsesWebSocket).toBe(OpenAI.responsesWebSocket)
expect(OpenAI.configure({ apiKey: "fixture" }).responses).toBeFunction()
expect(OpenAICompatible.deepseek.model).toBeFunction()
expect(CloudflareAIGateway.configure).toBeFunction()
expect(CloudflareAIGateway.configure({ accountId: "fixture", gatewayApiKey: "fixture" }).model).toBeFunction()
expect(CloudflareWorkersAI.configure).toBeFunction()
expect(CloudflareWorkersAI.configure({ accountId: "fixture", apiKey: "fixture" }).model).toBeFunction()
expect(OpenRouter.model).toBeFunction()
expect(OpenRouter.provider.model).toBe(OpenRouter.model)
expect(XAI.model).toBeFunction()
expect(XAI.provider.model).toBe(XAI.model)
expect(XAI.provider.responses).toBe(XAI.responses)
expect(XAI.provider.chat).toBe(XAI.chat)
expect(XAI.configure({ apiKey: "fixture" }).responses("grok-4.3").route.id).toBe("openai-responses")
expect(XAI.configure({ apiKey: "fixture" }).chat("grok-4.3").route.id).toBe("openai-compatible-chat")
expect(
GitHubCopilot.configure({ baseURL: "https://api.githubcopilot.test", apiKey: "fixture" }).model,
).toBeFunction()
})
test("protocol barrels expose supported low-level routes", () => {
expect(OpenAIChat.route.id).toBe("openai-chat")
expect(OpenAICompatibleChat.route.id).toBe("openai-compatible-chat")
expect(OpenAIResponses.route.id).toBe("openai-responses")
expect(OpenAIResponses.webSocketRoute.id).toBe("openai-responses-websocket")
expect(AnthropicMessages.route.id).toBe("anthropic-messages")
})
})

BIN
packages/llm/test/fixtures/media/restroom.png vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 14 KiB

48
packages/llm/test/fixtures/recordings/anthropic-messages-cache/writes-then-reads-cache-control-on-identical-second-call.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

29
packages/llm/test/fixtures/recordings/anthropic-messages/accepts-malformed-assistant-tool-order-with-default-patch.json vendored Normal file
View File

@@ -0,0 +1,29 @@
{
"version": 1,
"metadata": {
"name": "anthropic-messages/accepts-malformed-assistant-tool-order-with-default-patch",
"recordedAt": "2026-05-05T20:09:16.245Z",
"tags": ["prefix:anthropic-messages", "provider:anthropic", "protocol:anthropic-messages", "tool"]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.anthropic.com/v1/messages",
"headers": {
"anthropic-version": "2023-06-01",
"content-type": "application/json"
},
"body": "{\"model\":\"claude-haiku-4-5-20251001\",\"messages\":[{\"role\":\"assistant\",\"content\":[{\"type\":\"text\",\"text\":\"I will check the weather.\"}]},{\"role\":\"assistant\",\"content\":[{\"type\":\"tool_use\",\"id\":\"call_1\",\"name\":\"get_weather\",\"input\":{\"city\":\"Paris\"}}]},{\"role\":\"user\",\"content\":[{\"type\":\"tool_result\",\"tool_use_id\":\"call_1\",\"content\":\"{\\\"temperature\\\":\\\"72F\\\"}\"}]},{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"Use that result to answer briefly.\",\"cache_control\":{\"type\":\"ephemeral\"}}]}],\"tools\":[{\"name\":\"get_weather\",\"description\":\"Get weather\",\"input_schema\":{\"type\":\"object\",\"properties\":{}}}],\"stream\":true,\"max_tokens\":4096}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream; charset=utf-8"
},
"body": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-haiku-4-5-20251001\",\"id\":\"msg_01SikJVFaMR1XLMtavUhvuog\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"stop_reason\":null,\"stop_sequence\":null,\"stop_details\":null,\"usage\":{\"input_tokens\":638,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":1,\"service_tier\":\"standard\",\"inference_geo\":\"not_available\"}} }\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"text\",\"text\":\"\"} }\n\nevent: ping\ndata: {\"type\": \"ping\"}\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"The\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" weather in Paris is currently 72°F.\"} }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0}\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\",\"stop_sequence\":null,\"stop_details\":null},\"usage\":{\"input_tokens\":638,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"output_tokens\":14} }\n\nevent: message_stop\ndata: {\"type\":\"message_stop\" }\n\n"
}
}
]
}

43
packages/llm/test/fixtures/recordings/anthropic-messages/anthropic-opus-4-7-image-tool-result.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

56
packages/llm/test/fixtures/recordings/anthropic-messages/claude-opus-4-7-drives-a-tool-loop.json vendored Normal file
View File

@@ -0,0 +1,56 @@
{
"version": 1,
"metadata": {
"name": "anthropic-messages/claude-opus-4-7-drives-a-tool-loop",
"recordedAt": "2026-05-03T19:59:44.186Z",
"tags": [
"prefix:anthropic-messages",
"provider:anthropic",
"protocol:anthropic-messages",
"tool",
"tool-loop",
"golden",
"flagship"
]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.anthropic.com/v1/messages",
"headers": {
"anthropic-version": "2023-06-01",
"content-type": "application/json"
},
"body": "{\"model\":\"claude-opus-4-7\",\"system\":[{\"type\":\"text\",\"text\":\"Use the get_weather tool, then answer in one short sentence.\"}],\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"What is the weather in Paris?\"}]}],\"tools\":[{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"input_schema\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false}}],\"stream\":true,\"max_tokens\":80}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream; charset=utf-8"
},
"body": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-opus-4-7\",\"id\":\"msg_01DgAEgLgB1ZhavZon4qGE1t\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"stop_reason\":null,\"stop_sequence\":null,\"stop_details\":null,\"usage\":{\"input_tokens\":798,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":0,\"service_tier\":\"standard\",\"inference_geo\":\"global\"}} }\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"tool_use\",\"id\":\"toolu_01M8nJQQMxqpv1VaPYuJKT4j\",\"name\":\"get_weather\",\"input\":{},\"caller\":{\"type\":\"direct\"}} }\n\nevent: ping\ndata: {\"type\": \"ping\"}\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"{\\\"city\\\": \"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"\\\"Pa\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"ris\\\"}\"} }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0 }\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"tool_use\",\"stop_sequence\":null,\"stop_details\":null},\"usage\":{\"input_tokens\":798,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"output_tokens\":66} }\n\nevent: message_stop\ndata: {\"type\":\"message_stop\" }\n\n"
}
},
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.anthropic.com/v1/messages",
"headers": {
"anthropic-version": "2023-06-01",
"content-type": "application/json"
},
"body": "{\"model\":\"claude-opus-4-7\",\"system\":[{\"type\":\"text\",\"text\":\"Use the get_weather tool, then answer in one short sentence.\"}],\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"What is the weather in Paris?\"}]},{\"role\":\"assistant\",\"content\":[{\"type\":\"tool_use\",\"id\":\"toolu_01M8nJQQMxqpv1VaPYuJKT4j\",\"name\":\"get_weather\",\"input\":{\"city\":\"Paris\"}}]},{\"role\":\"user\",\"content\":[{\"type\":\"tool_result\",\"tool_use_id\":\"toolu_01M8nJQQMxqpv1VaPYuJKT4j\",\"content\":\"{\\\"temperature\\\":22,\\\"condition\\\":\\\"sunny\\\"}\"}]}],\"tools\":[{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"input_schema\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false}}],\"stream\":true,\"max_tokens\":80}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream; charset=utf-8"
},
"body": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-opus-4-7\",\"id\":\"msg_011KJqj32QjkrUAiBFxhmEoG\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"stop_reason\":null,\"stop_sequence\":null,\"stop_details\":null,\"usage\":{\"input_tokens\":895,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":5,\"service_tier\":\"standard\",\"inference_geo\":\"global\"}} }\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"text\",\"text\":\"\"} }\n\nevent: ping\ndata: {\"type\": \"ping\"}\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"Paris is curr\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"ently sunny at 22°C.\"} }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0 }\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\",\"stop_sequence\":null,\"stop_details\":null},\"usage\":{\"input_tokens\":895,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"output_tokens\":19}}\n\nevent: message_stop\ndata: {\"type\":\"message_stop\"}\n\n"
}
}
]
}

29
packages/llm/test/fixtures/recordings/anthropic-messages/rejects-malformed-assistant-tool-order-without-patch.json vendored Normal file
View File

@@ -0,0 +1,29 @@
{
"version": 1,
"metadata": {
"name": "anthropic-messages/rejects-malformed-assistant-tool-order-without-patch",
"recordedAt": "2026-05-05T20:08:42.597Z",
"tags": ["prefix:anthropic-messages", "provider:anthropic", "protocol:anthropic-messages", "tool", "sad-path"]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.anthropic.com/v1/messages",
"headers": {
"anthropic-version": "2023-06-01",
"content-type": "application/json"
},
"body": "{\"model\":\"claude-haiku-4-5-20251001\",\"messages\":[{\"role\":\"assistant\",\"content\":[{\"type\":\"tool_use\",\"id\":\"call_1\",\"name\":\"get_weather\",\"input\":{\"city\":\"Paris\"}},{\"type\":\"text\",\"text\":\"I will check the weather.\"}]},{\"role\":\"user\",\"content\":[{\"type\":\"tool_result\",\"tool_use_id\":\"call_1\",\"content\":\"{\\\"temperature\\\":\\\"72F\\\"}\"}]},{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"Use that result to answer briefly.\"}]}],\"tools\":[{\"name\":\"get_weather\",\"description\":\"Get weather\",\"input_schema\":{\"type\":\"object\",\"properties\":{}}}],\"stream\":true,\"max_tokens\":4096}"
},
"response": {
"status": 400,
"headers": {
"content-type": "application/json"
},
"body": "{\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"message\":\"messages.1: `tool_use` ids were found without `tool_result` blocks immediately after: call_1. Each `tool_use` block must have a corresponding `tool_result` block in the next message.\"},\"request_id\":\"req_011Cak2XdJgnzxKCY2BC2Beh\"}"
}
}
]
}

29
packages/llm/test/fixtures/recordings/anthropic-messages/streams-text.json vendored Normal file
View File

@@ -0,0 +1,29 @@
{
"version": 1,
"metadata": {
"name": "anthropic-messages/streams-text",
"recordedAt": "2026-04-28T21:18:45.535Z",
"tags": ["prefix:anthropic-messages", "provider:anthropic", "protocol:anthropic-messages"]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.anthropic.com/v1/messages",
"headers": {
"anthropic-version": "2023-06-01",
"content-type": "application/json"
},
"body": "{\"model\":\"claude-haiku-4-5-20251001\",\"system\":[{\"type\":\"text\",\"text\":\"You are concise.\"}],\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"Reply with exactly: Hello!\"}]}],\"stream\":true,\"max_tokens\":20,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream; charset=utf-8"
},
"body": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-haiku-4-5-20251001\",\"id\":\"msg_01UodR8c3ezAK8rAfi8HAs8g\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"stop_reason\":null,\"stop_sequence\":null,\"stop_details\":null,\"usage\":{\"input_tokens\":18,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":2,\"service_tier\":\"standard\",\"inference_geo\":\"not_available\"}} }\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"text\",\"text\":\"\"} }\n\nevent: ping\ndata: {\"type\": \"ping\"}\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"Hello!\"} }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0 }\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\",\"stop_sequence\":null,\"stop_details\":null},\"usage\":{\"input_tokens\":18,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"output_tokens\":5} }\n\nevent: message_stop\ndata: {\"type\":\"message_stop\" }\n\n"
}
}
]
}

29
packages/llm/test/fixtures/recordings/anthropic-messages/streams-tool-call.json vendored Normal file
View File

@@ -0,0 +1,29 @@
{
"version": 1,
"metadata": {
"name": "anthropic-messages/streams-tool-call",
"recordedAt": "2026-04-28T21:18:46.878Z",
"tags": ["prefix:anthropic-messages", "provider:anthropic", "protocol:anthropic-messages", "tool"]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.anthropic.com/v1/messages",
"headers": {
"anthropic-version": "2023-06-01",
"content-type": "application/json"
},
"body": "{\"model\":\"claude-haiku-4-5-20251001\",\"system\":[{\"type\":\"text\",\"text\":\"Call tools exactly as requested.\"}],\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"Call get_weather with city exactly Paris.\"}]}],\"tools\":[{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"input_schema\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false}}],\"tool_choice\":{\"type\":\"tool\",\"name\":\"get_weather\"},\"stream\":true,\"max_tokens\":80,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream; charset=utf-8"
},
"body": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-haiku-4-5-20251001\",\"id\":\"msg_01RYgU7NUPMK4B9v8S7gVpCS\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"stop_reason\":null,\"stop_sequence\":null,\"stop_details\":null,\"usage\":{\"input_tokens\":677,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":16,\"service_tier\":\"standard\",\"inference_geo\":\"not_available\"}} }\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"tool_use\",\"id\":\"toolu_012rmAruviySvUXSjgCPWVRu\",\"name\":\"get_weather\",\"input\":{},\"caller\":{\"type\":\"direct\"}} }\n\nevent: ping\ndata: {\"type\": \"ping\"}\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"{\\\"city\\\":\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\" \\\"Paris\\\"}\"} }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0 }\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"tool_use\",\"stop_sequence\":null,\"stop_details\":null},\"usage\":{\"input_tokens\":677,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"output_tokens\":33} }\n\nevent: message_stop\ndata: {\"type\":\"message_stop\" }\n\n"
}
}
]
}

55
packages/llm/test/fixtures/recordings/bedrock-converse/drives-a-tool-loop.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

29
packages/llm/test/fixtures/recordings/bedrock-converse/streams-a-tool-call.json vendored Normal file
View File

@@ -0,0 +1,29 @@
{
"version": 1,
"metadata": {
"name": "bedrock-converse/streams-a-tool-call",
"recordedAt": "2026-04-28T21:18:46.929Z",
"tags": ["prefix:bedrock-converse", "provider:amazon-bedrock", "protocol:bedrock-converse", "tool"]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://bedrock-runtime.us-east-1.amazonaws.com/model/us.amazon.nova-micro-v1%3A0/converse-stream",
"headers": {
"content-type": "application/json"
},
"body": "{\"modelId\":\"us.amazon.nova-micro-v1:0\",\"messages\":[{\"role\":\"user\",\"content\":[{\"text\":\"Call get_weather with city exactly Paris.\"}]}],\"system\":[{\"text\":\"Call tools exactly as requested.\"}],\"inferenceConfig\":{\"maxTokens\":80,\"temperature\":0},\"toolConfig\":{\"tools\":[{\"toolSpec\":{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"inputSchema\":{\"json\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false}}}}],\"toolChoice\":{\"tool\":{\"name\":\"get_weather\"}}}}"
},
"response": {
"status": 200,
"headers": {
"content-type": "application/vnd.amazon.eventstream"
},
"body": "AAAAuQAAAFL9kIXUCzpldmVudC10eXBlBwAMbWVzc2FnZVN0YXJ0DTpjb250ZW50LXR5cGUHABBhcHBsaWNhdGlvbi9qc29uDTptZXNzYWdlLXR5cGUHAAVldmVudHsicCI6ImFiY2RlZmdoaWprbG1ub3BxcnN0dXZ3eHl6QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVowMTIzNDU2NyIsInJvbGUiOiJhc3Npc3RhbnQifWf51EkAAAEMAAAAV56BJZoLOmV2ZW50LXR5cGUHABFjb250ZW50QmxvY2tTdGFydA06Y29udGVudC10eXBlBwAQYXBwbGljYXRpb24vanNvbg06bWVzc2FnZS10eXBlBwAFZXZlbnR7ImNvbnRlbnRCbG9ja0luZGV4IjowLCJwIjoiYWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXpBQkNERUZHSElKS0xNTk9QUVJTVFUiLCJzdGFydCI6eyJ0b29sVXNlIjp7Im5hbWUiOiJnZXRfd2VhdGhlciIsInRvb2xVc2VJZCI6InRvb2x1c2VfNmExcFB2bmM5OUdMS08zS0drVUEyTiJ9fX2LR7PFAAAA4gAAAFfCOY+BCzpldmVudC10eXBlBwARY29udGVudEJsb2NrRGVsdGENOmNvbnRlbnQtdHlwZQcAEGFwcGxpY2F0aW9uL2pzb24NOm1lc3NhZ2UtdHlwZQcABWV2ZW50eyJjb250ZW50QmxvY2tJbmRleCI6MCwiZGVsdGEiOnsidG9vbFVzZSI6eyJpbnB1dCI6IntcImNpdHlcIjpcIlBhcmlzXCJ9In19LCJwIjoiYWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXpBQkNERUZHSElKS0xNTiJ9RkW+2gAAAIcAAABW5OxHKgs6ZXZlbnQtdHlwZQcAEGNvbnRlbnRCbG9ja1N0b3ANOmNvbnRlbnQtdHlwZQcAEGFwcGxpY2F0aW9uL2pzb24NOm1lc3NhZ2UtdHlwZQcABWV2ZW50eyJjb250ZW50QmxvY2tJbmRleCI6MCwicCI6ImFiYyJ9y6nrtwAAAK4AAABRtlmf/As6ZXZlbnQtdHlwZQcAC21lc3NhZ2VTdG9wDTpjb250ZW50LXR5cGUHABBhcHBsaWNhdGlvbi9qc29uDTptZXNzYWdlLXR5cGUHAAVldmVudHsicCI6ImFiY2RlZmdoaWprbG1ub3BxcnN0dXZ3eHl6QUJDREVGR0hJSktMTU5PUFFSUyIsInN0b3BSZWFzb24iOiJ0b29sX3VzZSJ9MTlQawAAAOIAAABOplInQQs6ZXZlbnQtdHlwZQcACG1ldGFkYXRhDTpjb250ZW50LXR5cGUHABBhcHBsaWNhdGlvbi9qc29uDTptZXNzYWdlLXR5cGUHAAVldmVudHsibWV0cmljcyI6eyJsYXRlbmN5TXMiOjM1NX0sInAiOiJhYmNkZWZnaGlqayIsInVzYWdlIjp7ImlucHV0VG9rZW5zIjo0MTksIm91dHB1dFRva2VucyI6MTYsInNlcnZlclRvb2xVc2FnZSI6e30sInRvdGFsVG9rZW5zIjo0MzV9fU1tVJc=",
"bodyEncoding": "base64"
}
}
]
}

29
packages/llm/test/fixtures/recordings/bedrock-converse/streams-text.json vendored Normal file
View File

@@ -0,0 +1,29 @@
{
"version": 1,
"metadata": {
"name": "bedrock-converse/streams-text",
"recordedAt": "2026-04-28T21:18:46.553Z",
"tags": ["prefix:bedrock-converse", "provider:amazon-bedrock", "protocol:bedrock-converse"]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://bedrock-runtime.us-east-1.amazonaws.com/model/us.amazon.nova-micro-v1%3A0/converse-stream",
"headers": {
"content-type": "application/json"
},
"body": "{\"modelId\":\"us.amazon.nova-micro-v1:0\",\"messages\":[{\"role\":\"user\",\"content\":[{\"text\":\"Say hello.\"}]}],\"system\":[{\"text\":\"Reply with the single word 'Hello'.\"}],\"inferenceConfig\":{\"maxTokens\":16,\"temperature\":0}}"
},
"response": {
"status": 200,
"headers": {
"content-type": "application/vnd.amazon.eventstream"
},
"body": "AAAAmQAAAFI8UarQCzpldmVudC10eXBlBwAMbWVzc2FnZVN0YXJ0DTpjb250ZW50LXR5cGUHABBhcHBsaWNhdGlvbi9qc29uDTptZXNzYWdlLXR5cGUHAAVldmVudHsicCI6ImFiY2RlZmdoaWprbG1ub3BxcnN0dXZ3eHl6QUIiLCJyb2xlIjoiYXNzaXN0YW50In3SL1jNAAAAvQAAAFd4etebCzpldmVudC10eXBlBwARY29udGVudEJsb2NrRGVsdGENOmNvbnRlbnQtdHlwZQcAEGFwcGxpY2F0aW9uL2pzb24NOm1lc3NhZ2UtdHlwZQcABWV2ZW50eyJjb250ZW50QmxvY2tJbmRleCI6MCwiZGVsdGEiOnsidGV4dCI6IkhlbGxvIn0sInAiOiJhYmNkZWZnaGlqa2xtbm9wcXJzdHV2d3h5ekFCQ0RFIn2B0NR6AAAAxgAAAFf2eAZFCzpldmVudC10eXBlBwARY29udGVudEJsb2NrRGVsdGENOmNvbnRlbnQtdHlwZQcAEGFwcGxpY2F0aW9uL2pzb24NOm1lc3NhZ2UtdHlwZQcABWV2ZW50eyJjb250ZW50QmxvY2tJbmRleCI6MCwiZGVsdGEiOnsidGV4dCI6IiJ9LCJwIjoiYWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXpBQkNERUZHSElKS0xNTk9QUVJTIn3XaHMvAAAAhwAAAFbk7EcqCzpldmVudC10eXBlBwAQY29udGVudEJsb2NrU3RvcA06Y29udGVudC10eXBlBwAQYXBwbGljYXRpb24vanNvbg06bWVzc2FnZS10eXBlBwAFZXZlbnR7ImNvbnRlbnRCbG9ja0luZGV4IjowLCJwIjoiYWJjIn3Lqeu3AAAAjwAAAFFK+JlICzpldmVudC10eXBlBwALbWVzc2FnZVN0b3ANOmNvbnRlbnQtdHlwZQcAEGFwcGxpY2F0aW9uL2pzb24NOm1lc3NhZ2UtdHlwZQcABWV2ZW50eyJwIjoiYWJjZGVmZ2hpamtsbW4iLCJzdG9wUmVhc29uIjoiZW5kX3R1cm4ifZ+RQqEAAAECAAAATkXaMzsLOmV2ZW50LXR5cGUHAAhtZXRhZGF0YQ06Y29udGVudC10eXBlBwAQYXBwbGljYXRpb24vanNvbg06bWVzc2FnZS10eXBlBwAFZXZlbnR7Im1ldHJpY3MiOnsibGF0ZW5jeU1zIjozMDZ9LCJwIjoiYWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXpBQkNERUZHSElKS0xNTk9QUVJTVCIsInVzYWdlIjp7ImlucHV0VG9rZW5zIjoxMiwib3V0cHV0VG9rZW5zIjoyLCJzZXJ2ZXJUb29sVXNhZ2UiOnt9LCJ0b3RhbFRva2VucyI6MTR9fSnnkUk=",
"bodyEncoding": "base64"
}
}
]
}

32
packages/llm/test/fixtures/recordings/cloudflare-ai-gateway/cloudflare-ai-gateway-workers-ai-gpt-oss-20b-tools-tool-call.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

32
packages/llm/test/fixtures/recordings/cloudflare-ai-gateway/cloudflare-ai-gateway-workers-ai-llama-3-1-8b-text.json vendored Normal file
View File

@@ -0,0 +1,32 @@
{
"version": 1,
"metadata": {
"name": "cloudflare-ai-gateway/cloudflare-ai-gateway-workers-ai-llama-3-1-8b-text",
"recordedAt": "2026-05-08T15:55:48.952Z",
"provider": "cloudflare-ai-gateway",
"route": "cloudflare-ai-gateway",
"transport": "http",
"model": "workers-ai/@cf/meta/llama-3.1-8b-instruct",
"tags": ["prefix:cloudflare-ai-gateway", "provider:cloudflare-ai-gateway", "text", "golden"]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://gateway.ai.cloudflare.com/v1/{account}/{gateway}/compat/chat/completions",
"headers": {
"content-type": "application/json"
},
"body": "{\"model\":\"workers-ai/@cf/meta/llama-3.1-8b-instruct\",\"messages\":[{\"role\":\"system\",\"content\":\"You are concise.\"},{\"role\":\"user\",\"content\":\"Reply exactly with: Hello!\"}],\"stream\":true,\"stream_options\":{\"include_usage\":true},\"max_tokens\":40,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream"
},
"body": "data: {\"id\":\"id-1778255748911\",\"created\":1778255748,\"model\":\"@cf/meta/llama-3.1-8b-instruct\",\"object\":\"chat.completion.chunk\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Hello\"}}]}\n\ndata: {\"id\":\"id-1778255748911\",\"created\":1778255748,\"model\":\"@cf/meta/llama-3.1-8b-instruct\",\"object\":\"chat.completion.chunk\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"!\"}}]}\n\ndata: {\"id\":\"id-1778255748911\",\"object\":\"chat.completion.chunk\",\"created\":1778255748,\"model\":\"@cf/meta/llama-3.1-8b-instruct\",\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":45,\"completion_tokens\":2,\"total_tokens\":47}}\n\ndata: {\"id\":\"id-1778255748911\",\"object\":\"chat.completion.chunk\",\"created\":1778255748,\"model\":\"@cf/meta/llama-3.1-8b-instruct\",\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":0,\"completion_tokens\":0,\"total_tokens\":0,\"prompt_tokens_details\":{\"cached_tokens\":0}}}\n\ndata: [DONE]\n\n"
}
}
]
}

32
packages/llm/test/fixtures/recordings/cloudflare-workers-ai/cloudflare-workers-ai-gpt-oss-20b-tools-tool-call.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

32
packages/llm/test/fixtures/recordings/cloudflare-workers-ai/cloudflare-workers-ai-llama-3-1-8b-text.json vendored Normal file
View File

@@ -0,0 +1,32 @@
{
"version": 1,
"metadata": {
"name": "cloudflare-workers-ai/cloudflare-workers-ai-llama-3-1-8b-text",
"recordedAt": "2026-05-08T15:56:18.284Z",
"provider": "cloudflare-workers-ai",
"route": "cloudflare-workers-ai",
"transport": "http",
"model": "@cf/meta/llama-3.1-8b-instruct",
"tags": ["prefix:cloudflare-workers-ai", "provider:cloudflare-workers-ai", "text", "golden"]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.cloudflare.com/client/v4/accounts/{account}/ai/v1/chat/completions",
"headers": {
"content-type": "application/json"
},
"body": "{\"model\":\"@cf/meta/llama-3.1-8b-instruct\",\"messages\":[{\"role\":\"system\",\"content\":\"You are concise.\"},{\"role\":\"user\",\"content\":\"Reply exactly with: Hello!\"}],\"stream\":true,\"stream_options\":{\"include_usage\":true},\"max_tokens\":40,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream"
},
"body": "data: {\"id\":\"id-1778255778230\",\"created\":1778255778,\"model\":\"@cf/meta/llama-3.1-8b-instruct\",\"object\":\"chat.completion.chunk\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Hello\"}}]}\n\ndata: {\"id\":\"id-1778255778230\",\"created\":1778255778,\"model\":\"@cf/meta/llama-3.1-8b-instruct\",\"object\":\"chat.completion.chunk\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"!\"}}]}\n\ndata: {\"id\":\"id-1778255778230\",\"object\":\"chat.completion.chunk\",\"created\":1778255778,\"model\":\"@cf/meta/llama-3.1-8b-instruct\",\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":45,\"completion_tokens\":2,\"total_tokens\":47}}\n\ndata: {\"id\":\"id-1778255778230\",\"object\":\"chat.completion.chunk\",\"created\":1778255778,\"model\":\"@cf/meta/llama-3.1-8b-instruct\",\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":0,\"completion_tokens\":0,\"total_tokens\":0,\"prompt_tokens_details\":{\"cached_tokens\":0}}}\n\ndata: [DONE]\n\n"
}
}
]
}

46
packages/llm/test/fixtures/recordings/gemini-cache/reports-cachedcontenttokencount-on-identical-second-call.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

32
packages/llm/test/fixtures/recordings/gemini/gemini-2-5-flash-image.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

28
packages/llm/test/fixtures/recordings/gemini/streams-text.json vendored Normal file
View File

@@ -0,0 +1,28 @@
{
"version": 1,
"metadata": {
"name": "gemini/streams-text",
"recordedAt": "2026-04-28T21:18:47.483Z",
"tags": ["prefix:gemini", "provider:google", "protocol:gemini"]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:streamGenerateContent?alt=sse",
"headers": {
"content-type": "application/json"
},
"body": "{\"contents\":[{\"role\":\"user\",\"parts\":[{\"text\":\"Reply with exactly: Hello!\"}]}],\"systemInstruction\":{\"parts\":[{\"text\":\"You are concise.\"}]},\"generationConfig\":{\"maxOutputTokens\":80,\"temperature\":0}}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream"
},
"body": "data: {\"candidates\": [{\"content\": {\"parts\": [{\"text\": \"Hello!\"}],\"role\": \"model\"},\"finishReason\": \"STOP\",\"index\": 0}],\"usageMetadata\": {\"promptTokenCount\": 11,\"candidatesTokenCount\": 2,\"totalTokenCount\": 29,\"promptTokensDetails\": [{\"modality\": \"TEXT\",\"tokenCount\": 11}],\"thoughtsTokenCount\": 16},\"modelVersion\": \"gemini-2.5-flash\",\"responseId\": \"NyTxaczMAZ-b_uMP6u--iQg\"}\r\n\r\n"
}
}
]
}

28
packages/llm/test/fixtures/recordings/gemini/streams-tool-call.json vendored Normal file
View File

@@ -0,0 +1,28 @@
{
"version": 1,
"metadata": {
"name": "gemini/streams-tool-call",
"recordedAt": "2026-04-28T21:18:48.285Z",
"tags": ["prefix:gemini", "provider:google", "protocol:gemini", "tool"]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:streamGenerateContent?alt=sse",
"headers": {
"content-type": "application/json"
},
"body": "{\"contents\":[{\"role\":\"user\",\"parts\":[{\"text\":\"Call get_weather with city exactly Paris.\"}]}],\"systemInstruction\":{\"parts\":[{\"text\":\"Call tools exactly as requested.\"}]},\"tools\":[{\"functionDeclarations\":[{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"parameters\":{\"required\":[\"city\"],\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}}}}]}],\"toolConfig\":{\"functionCallingConfig\":{\"mode\":\"ANY\",\"allowedFunctionNames\":[\"get_weather\"]}},\"generationConfig\":{\"maxOutputTokens\":80,\"temperature\":0}}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream"
},
"body": "data: {\"candidates\": [{\"content\": {\"parts\": [{\"functionCall\": {\"name\": \"get_weather\",\"args\": {\"city\": \"Paris\"}},\"thoughtSignature\": \"CiQBDDnWx5RcSsS1UMbykQ5HWlrMu6wrxXGUhmZ0uRKLaMhDZaEKXwEMOdbHVoJAlfbOQyKB378pDZ/gkjWr3HP+dWw1us1kMG22g4G3oJvuTq/SrWS+7KYtSlvOxCKhW2l/2/TczpyGyGmANmsusDcxF1SKOYA5/8Hg0nI24MAlT3+91V/MCoUBAQw51seClFLy3E71v2H44F1kpmjgz8FeTRZofrjbaazfrT+w8Yxgdr3UgGagLMY4OadZemQTWckq9IAqRum78hrBg6NGtQvn15SbtfTNqI4PcxX/+qPo4/g4/ZT5kVORDhVqO8BVP/RA5GQ3ce3sRK8hSkvQlXSoXIPpHh6x7hBezIGXzw==\"}],\"role\": \"model\"},\"finishReason\": \"STOP\",\"index\": 0,\"finishMessage\": \"Model generated function call(s).\"}],\"usageMetadata\": {\"promptTokenCount\": 55,\"candidatesTokenCount\": 15,\"totalTokenCount\": 115,\"promptTokensDetails\": [{\"modality\": \"TEXT\",\"tokenCount\": 55}],\"thoughtsTokenCount\": 45},\"modelVersion\": \"gemini-2.5-flash\",\"responseId\": \"NyTxaYuTJ_OW_uMPgIPKgAg\"}\r\n\r\n"
}
}
]
}

28
packages/llm/test/fixtures/recordings/openai-chat/continues-after-tool-result.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

46
packages/llm/test/fixtures/recordings/openai-chat/drives-a-tool-loop-end-to-end.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

28
packages/llm/test/fixtures/recordings/openai-chat/streams-text.json vendored Normal file
View File

@@ -0,0 +1,28 @@
{
"version": 1,
"metadata": {
"name": "openai-chat/streams-text",
"recordedAt": "2026-05-06T01:33:30.542Z",
"tags": ["prefix:openai-chat", "provider:openai", "protocol:openai-chat"]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.openai.com/v1/chat/completions",
"headers": {
"content-type": "application/json"
},
"body": "{\"model\":\"gpt-4o-mini\",\"messages\":[{\"role\":\"system\",\"content\":\"You are concise.\"},{\"role\":\"user\",\"content\":\"Say hello in one short sentence.\"}],\"stream\":true,\"stream_options\":{\"include_usage\":true},\"max_tokens\":20,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream; charset=utf-8"
},
"body": "data: {\"id\":\"chatcmpl-DcLQgbFetadY4JFl0fHK0g7OYsCOL\",\"object\":\"chat.completion.chunk\",\"created\":1778031210,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_57133166c6\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\",\"refusal\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"g9SWm2h6J\"}\n\ndata: {\"id\":\"chatcmpl-DcLQgbFetadY4JFl0fHK0g7OYsCOL\",\"object\":\"chat.completion.chunk\",\"created\":1778031210,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_57133166c6\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Hello\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"lVzwlh\"}\n\ndata: {\"id\":\"chatcmpl-DcLQgbFetadY4JFl0fHK0g7OYsCOL\",\"object\":\"chat.completion.chunk\",\"created\":1778031210,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_57133166c6\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"!\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"onzhziaLGv\"}\n\ndata: {\"id\":\"chatcmpl-DcLQgbFetadY4JFl0fHK0g7OYsCOL\",\"object\":\"chat.completion.chunk\",\"created\":1778031210,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_57133166c6\",\"choices\":[{\"index\":0,\"delta\":{},\"logprobs\":null,\"finish_reason\":\"stop\"}],\"usage\":null,\"obfuscation\":\"LzUj1\"}\n\ndata: {\"id\":\"chatcmpl-DcLQgbFetadY4JFl0fHK0g7OYsCOL\",\"object\":\"chat.completion.chunk\",\"created\":1778031210,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_57133166c6\",\"choices\":[],\"usage\":{\"prompt_tokens\":22,\"completion_tokens\":2,\"total_tokens\":24,\"prompt_tokens_details\":{\"cached_tokens\":0,\"audio_tokens\":0},\"completion_tokens_details\":{\"reasoning_tokens\":0,\"audio_tokens\":0,\"accepted_prediction_tokens\":0,\"rejected_prediction_tokens\":0}},\"obfuscation\":\"emMuPcvvOkI\"}\n\ndata: [DONE]\n\n"
}
}
]
}

28
packages/llm/test/fixtures/recordings/openai-chat/streams-tool-call.json vendored Normal file
View File

@@ -0,0 +1,28 @@
{
"version": 1,
"metadata": {
"name": "openai-chat/streams-tool-call",
"recordedAt": "2026-05-06T01:33:31.127Z",
"tags": ["prefix:openai-chat", "provider:openai", "protocol:openai-chat", "tool"]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.openai.com/v1/chat/completions",
"headers": {
"content-type": "application/json"
},
"body": "{\"model\":\"gpt-4o-mini\",\"messages\":[{\"role\":\"system\",\"content\":\"Call tools exactly as requested.\"},{\"role\":\"user\",\"content\":\"Call get_weather with city exactly Paris.\"}],\"tools\":[{\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false}}}],\"tool_choice\":{\"type\":\"function\",\"function\":{\"name\":\"get_weather\"}},\"stream\":true,\"stream_options\":{\"include_usage\":true},\"max_tokens\":80,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream; charset=utf-8"
},
"body": "data: {\"id\":\"chatcmpl-DcLQgGuIIwnMHqZMRCOwZMLir5SkK\",\"object\":\"chat.completion.chunk\",\"created\":1778031210,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_d0a1738203\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":null,\"tool_calls\":[{\"index\":0,\"id\":\"call_5wBV98AvGPwOyC6a2HtKh85w\",\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"arguments\":\"\"}}],\"refusal\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"hrw8\"}\n\ndata: {\"id\":\"chatcmpl-DcLQgGuIIwnMHqZMRCOwZMLir5SkK\",\"object\":\"chat.completion.chunk\",\"created\":1778031210,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_d0a1738203\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"{\\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"MzOlaTohF20Sbb\"}\n\ndata: {\"id\":\"chatcmpl-DcLQgGuIIwnMHqZMRCOwZMLir5SkK\",\"object\":\"chat.completion.chunk\",\"created\":1778031210,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_d0a1738203\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"city\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"QuYBQ5vYEUVxR\"}\n\ndata: {\"id\":\"chatcmpl-DcLQgGuIIwnMHqZMRCOwZMLir5SkK\",\"object\":\"chat.completion.chunk\",\"created\":1778031210,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_d0a1738203\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\":\\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"spyXlsV2hl6l\"}\n\ndata: {\"id\":\"chatcmpl-DcLQgGuIIwnMHqZMRCOwZMLir5SkK\",\"object\":\"chat.completion.chunk\",\"created\":1778031210,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_d0a1738203\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"Paris\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"Db1cjFKa6YAI\"}\n\ndata: {\"id\":\"chatcmpl-DcLQgGuIIwnMHqZMRCOwZMLir5SkK\",\"object\":\"chat.completion.chunk\",\"created\":1778031210,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_d0a1738203\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\"}\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"oPu35nrhXcjTL5\"}\n\ndata: {\"id\":\"chatcmpl-DcLQgGuIIwnMHqZMRCOwZMLir5SkK\",\"object\":\"chat.completion.chunk\",\"created\":1778031210,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_d0a1738203\",\"choices\":[{\"index\":0,\"delta\":{},\"logprobs\":null,\"finish_reason\":\"stop\"}],\"usage\":null,\"obfuscation\":\"63TVy\"}\n\ndata: {\"id\":\"chatcmpl-DcLQgGuIIwnMHqZMRCOwZMLir5SkK\",\"object\":\"chat.completion.chunk\",\"created\":1778031210,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_d0a1738203\",\"choices\":[],\"usage\":{\"prompt_tokens\":67,\"completion_tokens\":5,\"total_tokens\":72,\"prompt_tokens_details\":{\"cached_tokens\":0,\"audio_tokens\":0},\"completion_tokens_details\":{\"reasoning_tokens\":0,\"audio_tokens\":0,\"accepted_prediction_tokens\":0,\"rejected_prediction_tokens\":0}},\"obfuscation\":\"NxJjur40z4H\"}\n\ndata: [DONE]\n\n"
}
}
]
}

28
packages/llm/test/fixtures/recordings/openai-compatible-chat/deepseek-streams-text.json vendored Normal file
View File

@@ -0,0 +1,28 @@
{
"version": 1,
"metadata": {
"name": "openai-compatible-chat/deepseek-streams-text",
"recordedAt": "2026-04-28T21:18:49.498Z",
"tags": ["prefix:openai-compatible-chat", "protocol:openai-compatible-chat", "provider:deepseek"]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.deepseek.com/v1/chat/completions",
"headers": {
"content-type": "application/json"
},
"body": "{\"model\":\"deepseek-chat\",\"messages\":[{\"role\":\"system\",\"content\":\"You are concise.\"},{\"role\":\"user\",\"content\":\"Reply with exactly: Hello!\"}],\"stream\":true,\"stream_options\":{\"include_usage\":true},\"max_tokens\":20,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream; charset=utf-8"
},
"body": "data: {\"id\":\"0c811926-1e0c-4160-baf8-6e71247c8ad7\",\"object\":\"chat.completion.chunk\",\"created\":1777411128,\"model\":\"deepseek-v4-flash\",\"system_fingerprint\":\"fp_058df29938_prod0820_fp8_kvcache_20260402\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\"},\"logprobs\":null,\"finish_reason\":null}]}\n\ndata: {\"id\":\"0c811926-1e0c-4160-baf8-6e71247c8ad7\",\"object\":\"chat.completion.chunk\",\"created\":1777411128,\"model\":\"deepseek-v4-flash\",\"system_fingerprint\":\"fp_058df29938_prod0820_fp8_kvcache_20260402\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Hello\"},\"logprobs\":null,\"finish_reason\":null}]}\n\ndata: {\"id\":\"0c811926-1e0c-4160-baf8-6e71247c8ad7\",\"object\":\"chat.completion.chunk\",\"created\":1777411128,\"model\":\"deepseek-v4-flash\",\"system_fingerprint\":\"fp_058df29938_prod0820_fp8_kvcache_20260402\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"!\"},\"logprobs\":null,\"finish_reason\":null}]}\n\ndata: {\"id\":\"0c811926-1e0c-4160-baf8-6e71247c8ad7\",\"object\":\"chat.completion.chunk\",\"created\":1777411128,\"model\":\"deepseek-v4-flash\",\"system_fingerprint\":\"fp_058df29938_prod0820_fp8_kvcache_20260402\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"\"},\"logprobs\":null,\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":14,\"completion_tokens\":2,\"total_tokens\":16,\"prompt_tokens_details\":{\"cached_tokens\":0},\"prompt_cache_hit_tokens\":0,\"prompt_cache_miss_tokens\":14}}\n\ndata: [DONE]\n\n"
}
}
]
}

53
packages/llm/test/fixtures/recordings/openai-compatible-chat/groq-llama-3-3-70b-drives-a-tool-loop.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

28
packages/llm/test/fixtures/recordings/openai-compatible-chat/groq-streams-text.json vendored Normal file
View File

@@ -0,0 +1,28 @@
{
"version": 1,
"metadata": {
"name": "openai-compatible-chat/groq-streams-text",
"recordedAt": "2026-05-06T01:35:05.532Z",
"tags": ["prefix:openai-compatible-chat", "protocol:openai-compatible-chat", "provider:groq"]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.groq.com/openai/v1/chat/completions",
"headers": {
"content-type": "application/json"
},
"body": "{\"model\":\"llama-3.3-70b-versatile\",\"messages\":[{\"role\":\"system\",\"content\":\"You are concise.\"},{\"role\":\"user\",\"content\":\"Reply with exactly: Hello!\"}],\"stream\":true,\"stream_options\":{\"include_usage\":true},\"max_tokens\":20,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream"
},
"body": "data: {\"id\":\"chatcmpl-dd5aae9f-7032-44a7-aca8-01027903b4c9\",\"object\":\"chat.completion.chunk\",\"created\":1778031305,\"model\":\"llama-3.3-70b-versatile\",\"system_fingerprint\":\"fp_d42c28f9ce\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\"},\"logprobs\":null,\"finish_reason\":null}],\"x_groq\":{\"id\":\"req_01kqxes8r3fmja0yhxvt665m6h\",\"seed\":687314058}}\n\ndata: {\"id\":\"chatcmpl-dd5aae9f-7032-44a7-aca8-01027903b4c9\",\"object\":\"chat.completion.chunk\",\"created\":1778031305,\"model\":\"llama-3.3-70b-versatile\",\"system_fingerprint\":\"fp_d42c28f9ce\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Hello\"},\"logprobs\":null,\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-dd5aae9f-7032-44a7-aca8-01027903b4c9\",\"object\":\"chat.completion.chunk\",\"created\":1778031305,\"model\":\"llama-3.3-70b-versatile\",\"system_fingerprint\":\"fp_d42c28f9ce\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"!\"},\"logprobs\":null,\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-dd5aae9f-7032-44a7-aca8-01027903b4c9\",\"object\":\"chat.completion.chunk\",\"created\":1778031305,\"model\":\"llama-3.3-70b-versatile\",\"system_fingerprint\":\"fp_d42c28f9ce\",\"choices\":[{\"index\":0,\"delta\":{},\"logprobs\":null,\"finish_reason\":\"stop\"}],\"x_groq\":{\"id\":\"req_01kqxes8r3fmja0yhxvt665m6h\",\"usage\":{\"queue_time\":0.0381395,\"prompt_tokens\":45,\"prompt_time\":0.003985297,\"completion_tokens\":3,\"completion_time\":0.014171875,\"total_tokens\":48,\"total_time\":0.018157172}},\"usage\":{\"queue_time\":0.0381395,\"prompt_tokens\":45,\"prompt_time\":0.003985297,\"completion_tokens\":3,\"completion_time\":0.014171875,\"total_tokens\":48,\"total_time\":0.018157172}}\n\ndata: {\"id\":\"chatcmpl-dd5aae9f-7032-44a7-aca8-01027903b4c9\",\"object\":\"chat.completion.chunk\",\"created\":1778031305,\"model\":\"llama-3.3-70b-versatile\",\"system_fingerprint\":\"fp_d42c28f9ce\",\"choices\":[],\"usage\":{\"queue_time\":0.0381395,\"prompt_tokens\":45,\"prompt_time\":0.003985297,\"completion_tokens\":3,\"completion_time\":0.014171875,\"total_tokens\":48,\"total_time\":0.018157172},\"service_tier\":\"on_demand\"}\n\ndata: [DONE]\n\n"
}
}
]
}

28
packages/llm/test/fixtures/recordings/openai-compatible-chat/groq-streams-tool-call.json vendored Normal file
View File

@@ -0,0 +1,28 @@
{
"version": 1,
"metadata": {
"name": "openai-compatible-chat/groq-streams-tool-call",
"recordedAt": "2026-05-06T01:35:05.706Z",
"tags": ["prefix:openai-compatible-chat", "protocol:openai-compatible-chat", "provider:groq", "tool"]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.groq.com/openai/v1/chat/completions",
"headers": {
"content-type": "application/json"
},
"body": "{\"model\":\"llama-3.3-70b-versatile\",\"messages\":[{\"role\":\"system\",\"content\":\"Call tools exactly as requested.\"},{\"role\":\"user\",\"content\":\"Call get_weather with city exactly Paris.\"}],\"tools\":[{\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false}}}],\"tool_choice\":{\"type\":\"function\",\"function\":{\"name\":\"get_weather\"}},\"stream\":true,\"stream_options\":{\"include_usage\":true},\"max_tokens\":80,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream"
},
"body": "data: {\"id\":\"chatcmpl-05380361-f8e4-444a-ae80-296b4d1d46f7\",\"object\":\"chat.completion.chunk\",\"created\":1778031305,\"model\":\"llama-3.3-70b-versatile\",\"system_fingerprint\":\"fp_0761e44d7b\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":null},\"logprobs\":null,\"finish_reason\":null}],\"x_groq\":{\"id\":\"req_01kqxes8v4fm7baf4smt42f0qn\",\"seed\":1846647562}}\n\ndata: {\"id\":\"chatcmpl-05380361-f8e4-444a-ae80-296b4d1d46f7\",\"object\":\"chat.completion.chunk\",\"created\":1778031305,\"model\":\"llama-3.3-70b-versatile\",\"system_fingerprint\":\"fp_0761e44d7b\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"id\":\"mcf2d8nn1\",\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"arguments\":\"{\\\"city\\\":\\\"Paris\\\"}\"},\"index\":0}]},\"logprobs\":null,\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-05380361-f8e4-444a-ae80-296b4d1d46f7\",\"object\":\"chat.completion.chunk\",\"created\":1778031305,\"model\":\"llama-3.3-70b-versatile\",\"system_fingerprint\":\"fp_0761e44d7b\",\"choices\":[{\"index\":0,\"delta\":{},\"logprobs\":null,\"finish_reason\":\"tool_calls\"}],\"x_groq\":{\"id\":\"req_01kqxes8v4fm7baf4smt42f0qn\",\"usage\":{\"queue_time\":0.07684935,\"prompt_tokens\":249,\"prompt_time\":0.014815006,\"completion_tokens\":10,\"completion_time\":0.036435756,\"total_tokens\":259,\"total_time\":0.051250762}},\"usage\":{\"queue_time\":0.07684935,\"prompt_tokens\":249,\"prompt_time\":0.014815006,\"completion_tokens\":10,\"completion_time\":0.036435756,\"total_tokens\":259,\"total_time\":0.051250762}}\n\ndata: {\"id\":\"chatcmpl-05380361-f8e4-444a-ae80-296b4d1d46f7\",\"object\":\"chat.completion.chunk\",\"created\":1778031305,\"model\":\"llama-3.3-70b-versatile\",\"system_fingerprint\":\"fp_0761e44d7b\",\"choices\":[],\"usage\":{\"queue_time\":0.07684935,\"prompt_tokens\":249,\"prompt_time\":0.014815006,\"completion_tokens\":10,\"completion_time\":0.036435756,\"total_tokens\":259,\"total_time\":0.051250762},\"service_tier\":\"on_demand\"}\n\ndata: [DONE]\n\n"
}
}
]
}

54
packages/llm/test/fixtures/recordings/openai-compatible-chat/openrouter-claude-opus-4-7-drives-a-tool-loop.json vendored Normal file
View File

@@ -0,0 +1,54 @@
{
"version": 1,
"metadata": {
"name": "openai-compatible-chat/openrouter-claude-opus-4-7-drives-a-tool-loop",
"recordedAt": "2026-05-06T01:35:14.282Z",
"tags": [
"prefix:openai-compatible-chat",
"protocol:openai-compatible-chat",
"provider:openrouter",
"tool",
"tool-loop",
"golden",
"flagship"
]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://openrouter.ai/api/v1/chat/completions",
"headers": {
"content-type": "application/json"
},
"body": "{\"model\":\"anthropic/claude-opus-4.7\",\"messages\":[{\"role\":\"system\",\"content\":\"Use the get_weather tool exactly once, then answer in one short sentence.\"},{\"role\":\"user\",\"content\":\"What is the weather in Paris?\"}],\"tools\":[{\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false}}}],\"stream\":true,\"stream_options\":{\"include_usage\":true},\"max_tokens\":80,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream"
},
"body": ": OPENROUTER PROCESSING\n\n: OPENROUTER PROCESSING\n\n: OPENROUTER PROCESSING\n\ndata: {\"id\":\"gen-1778031311-S3NlfYGRwAnOoPoNrThK\",\"object\":\"chat.completion.chunk\",\"created\":1778031311,\"model\":\"anthropic/claude-4.7-opus-20260416\",\"provider\":\"Amazon Bedrock\",\"choices\":[{\"index\":0,\"delta\":{\"content\":null,\"role\":\"assistant\",\"tool_calls\":[{\"index\":0,\"id\":\"toolu_bdrk_01AVRkzbigpMbNJ3zjnuQ6ZE\",\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"arguments\":\"\"}}]},\"finish_reason\":null,\"native_finish_reason\":null}]}\n\ndata: {\"id\":\"gen-1778031311-S3NlfYGRwAnOoPoNrThK\",\"object\":\"chat.completion.chunk\",\"created\":1778031311,\"model\":\"anthropic/claude-4.7-opus-20260416\",\"provider\":\"Amazon Bedrock\",\"choices\":[{\"index\":0,\"delta\":{\"content\":null,\"role\":\"assistant\",\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\"}}]},\"finish_reason\":null,\"native_finish_reason\":null}]}\n\ndata: {\"id\":\"gen-1778031311-S3NlfYGRwAnOoPoNrThK\",\"object\":\"chat.completion.chunk\",\"created\":1778031311,\"model\":\"anthropic/claude-4.7-opus-20260416\",\"provider\":\"Amazon Bedrock\",\"choices\":[{\"index\":0,\"delta\":{\"content\":null,\"role\":\"assistant\",\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\"}}]},\"finish_reason\":null,\"native_finish_reason\":null}]}\n\ndata: {\"id\":\"gen-1778031311-S3NlfYGRwAnOoPoNrThK\",\"object\":\"chat.completion.chunk\",\"created\":1778031311,\"model\":\"anthropic/claude-4.7-opus-20260416\",\"provider\":\"Amazon Bedrock\",\"choices\":[{\"index\":0,\"delta\":{\"content\":null,\"role\":\"assistant\",\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"{\\\"city\\\": \\\"P\"}}]},\"finish_reason\":null,\"native_finish_reason\":null}]}\n\ndata: {\"id\":\"gen-1778031311-S3NlfYGRwAnOoPoNrThK\",\"object\":\"chat.completion.chunk\",\"created\":1778031311,\"model\":\"anthropic/claude-4.7-opus-20260416\",\"provider\":\"Amazon Bedrock\",\"choices\":[{\"index\":0,\"delta\":{\"content\":null,\"role\":\"assistant\",\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"ari\"}}]},\"finish_reason\":null,\"native_finish_reason\":null}]}\n\ndata: {\"id\":\"gen-1778031311-S3NlfYGRwAnOoPoNrThK\",\"object\":\"chat.completion.chunk\",\"created\":1778031311,\"model\":\"anthropic/claude-4.7-opus-20260416\",\"provider\":\"Amazon Bedrock\",\"choices\":[{\"index\":0,\"delta\":{\"content\":null,\"role\":\"assistant\",\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"s\\\"}\"}}]},\"finish_reason\":null,\"native_finish_reason\":null}]}\n\ndata: {\"id\":\"gen-1778031311-S3NlfYGRwAnOoPoNrThK\",\"object\":\"chat.completion.chunk\",\"created\":1778031311,\"model\":\"anthropic/claude-4.7-opus-20260416\",\"provider\":\"Amazon Bedrock\",\"service_tier\":\"standard\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"\",\"role\":\"assistant\"},\"finish_reason\":\"tool_calls\",\"native_finish_reason\":\"tool_use\"}]}\n\ndata: {\"id\":\"gen-1778031311-S3NlfYGRwAnOoPoNrThK\",\"object\":\"chat.completion.chunk\",\"created\":1778031311,\"model\":\"anthropic/claude-4.7-opus-20260416\",\"provider\":\"Amazon Bedrock\",\"service_tier\":\"standard\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"\",\"role\":\"assistant\"},\"finish_reason\":\"tool_calls\",\"native_finish_reason\":\"tool_use\"}],\"usage\":{\"prompt_tokens\":802,\"completion_tokens\":66,\"total_tokens\":868,\"cost\":0.00566,\"is_byok\":false,\"prompt_tokens_details\":{\"cached_tokens\":0,\"cache_write_tokens\":0,\"audio_tokens\":0,\"video_tokens\":0},\"cost_details\":{\"upstream_inference_cost\":0.00566,\"upstream_inference_prompt_cost\":0.00401,\"upstream_inference_completions_cost\":0.00165},\"completion_tokens_details\":{\"reasoning_tokens\":0,\"image_tokens\":0,\"audio_tokens\":0}}}\n\ndata: [DONE]\n\n"
}
},
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://openrouter.ai/api/v1/chat/completions",
"headers": {
"content-type": "application/json"
},
"body": "{\"model\":\"anthropic/claude-opus-4.7\",\"messages\":[{\"role\":\"system\",\"content\":\"Use the get_weather tool exactly once, then answer in one short sentence.\"},{\"role\":\"user\",\"content\":\"What is the weather in Paris?\"},{\"role\":\"assistant\",\"content\":null,\"tool_calls\":[{\"id\":\"toolu_bdrk_01AVRkzbigpMbNJ3zjnuQ6ZE\",\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"arguments\":\"{\\\"city\\\":\\\"Paris\\\"}\"}}]},{\"role\":\"tool\",\"tool_call_id\":\"toolu_bdrk_01AVRkzbigpMbNJ3zjnuQ6ZE\",\"content\":\"{\\\"temperature\\\":22,\\\"condition\\\":\\\"sunny\\\"}\"}],\"tools\":[{\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false}}}],\"stream\":true,\"stream_options\":{\"include_usage\":true},\"max_tokens\":80,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream"
},
"body": ": OPENROUTER PROCESSING\n\ndata: {\"id\":\"gen-1778031313-XM4XZGmFyt6jg3GZ772w\",\"object\":\"chat.completion.chunk\",\"created\":1778031313,\"model\":\"anthropic/claude-4.7-opus-20260416\",\"provider\":\"Amazon Bedrock\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"It\",\"role\":\"assistant\"},\"finish_reason\":null,\"native_finish_reason\":null}]}\n\ndata: {\"id\":\"gen-1778031313-XM4XZGmFyt6jg3GZ772w\",\"object\":\"chat.completion.chunk\",\"created\":1778031313,\"model\":\"anthropic/claude-4.7-opus-20260416\",\"provider\":\"Amazon Bedrock\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"'s sunny and\",\"role\":\"assistant\"},\"finish_reason\":null,\"native_finish_reason\":null}]}\n\ndata: {\"id\":\"gen-1778031313-XM4XZGmFyt6jg3GZ772w\",\"object\":\"chat.completion.chunk\",\"created\":1778031313,\"model\":\"anthropic/claude-4.7-opus-20260416\",\"provider\":\"Amazon Bedrock\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" 22°C in\",\"role\":\"assistant\"},\"finish_reason\":null,\"native_finish_reason\":null}]}\n\ndata: {\"id\":\"gen-1778031313-XM4XZGmFyt6jg3GZ772w\",\"object\":\"chat.completion.chunk\",\"created\":1778031313,\"model\":\"anthropic/claude-4.7-opus-20260416\",\"provider\":\"Amazon Bedrock\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" Paris.\",\"role\":\"assistant\"},\"finish_reason\":null,\"native_finish_reason\":null}]}\n\ndata: {\"id\":\"gen-1778031313-XM4XZGmFyt6jg3GZ772w\",\"object\":\"chat.completion.chunk\",\"created\":1778031313,\"model\":\"anthropic/claude-4.7-opus-20260416\",\"provider\":\"Amazon Bedrock\",\"service_tier\":\"standard\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"\",\"role\":\"assistant\"},\"finish_reason\":\"stop\",\"native_finish_reason\":\"end_turn\"}]}\n\ndata: {\"id\":\"gen-1778031313-XM4XZGmFyt6jg3GZ772w\",\"object\":\"chat.completion.chunk\",\"created\":1778031313,\"model\":\"anthropic/claude-4.7-opus-20260416\",\"provider\":\"Amazon Bedrock\",\"service_tier\":\"standard\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"\",\"role\":\"assistant\"},\"finish_reason\":\"stop\",\"native_finish_reason\":\"end_turn\"}],\"usage\":{\"prompt_tokens\":899,\"completion_tokens\":19,\"total_tokens\":918,\"cost\":0.00497,\"is_byok\":false,\"prompt_tokens_details\":{\"cached_tokens\":0,\"cache_write_tokens\":0,\"audio_tokens\":0,\"video_tokens\":0},\"cost_details\":{\"upstream_inference_cost\":0.00497,\"upstream_inference_prompt_cost\":0.004495,\"upstream_inference_completions_cost\":0.000475},\"completion_tokens_details\":{\"reasoning_tokens\":0,\"image_tokens\":0,\"audio_tokens\":0}}}\n\ndata: [DONE]\n\n"
}
}
]
}

53
packages/llm/test/fixtures/recordings/openai-compatible-chat/openrouter-gpt-4o-mini-drives-a-tool-loop.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

Some files were not shown because too many files have changed in this diff Show More