feat(aircoding): AirCoding V2 baseline — deterministic multi-agent architecture
Forked from OpenCode v1.17.4 with multi-agent system: - 5 agents: aircoding, scheduler, worker, architect, reviewer - Deterministic DAG scheduling engine (coordinator_tick) - Tool whitelists as hard enforcement - AirCoding validation plugin - System prompt injection for routing - V1 requirements: C4 docs, ADR, AGENTS.md, debug-log.md - Design documents in docs/
This commit is contained in:
airlongdian
145
specs/storage/effect-sqlite-package.md
Normal file
145
specs/storage/effect-sqlite-package.md
Normal file
@@ -0,0 +1,145 @@
|
||||
# Effect Drizzle SQLite Package
|
||||
|
||||
## Goal
|
||||
|
||||
Create a small workspace package that vendors the Drizzle `effect-sqlite` adapter shape for our repo. This is not an opencode storage abstraction. It is a local package that ports the Drizzle Effect SQLite implementation so we can use it before/independently of upstream release timing.
|
||||
|
||||
`packages/opencode` will use it internally, but the package itself should be generic: Drizzle + Effect + SQLite. No opencode paths, migrations, tables, transaction hooks, post-commit behavior, or domain language should live in this package.
|
||||
|
||||
## Package Shape
|
||||
|
||||
Add a package similar in style to `packages/http-recorder`:
|
||||
|
||||
- `packages/effect-drizzle-sqlite/package.json`
|
||||
- `packages/effect-drizzle-sqlite/src/index.ts`
|
||||
- `packages/effect-drizzle-sqlite/src/effect-sqlite/*`
|
||||
- `packages/effect-drizzle-sqlite/src/sqlite-core/effect/*`
|
||||
- `packages/effect-drizzle-sqlite/test/sqlite.test.ts`
|
||||
|
||||
Package name:
|
||||
|
||||
- `@opencode-ai/effect-drizzle-sqlite`
|
||||
|
||||
Initial exports:
|
||||
|
||||
```ts
|
||||
export { EffectLogger } from "drizzle-orm/effect-core"
|
||||
export * from "./effect-sqlite/driver"
|
||||
export * from "./effect-sqlite/session"
|
||||
export { migrate } from "./effect-sqlite/migrator"
|
||||
export * as EffectDrizzleSqlite from "."
|
||||
```
|
||||
|
||||
The package should follow Drizzle's adapter naming and semantics as closely as possible. Think of it as a vendored `drizzle-orm/effect-sqlite` package surface, not as a new storage service API.
|
||||
|
||||
## Upstream References
|
||||
|
||||
Use these as implementation references instead of inventing a custom API:
|
||||
|
||||
- Drizzle Effect Postgres current RC:
|
||||
- `/Users/kit/code/open-source/drizzle-orm-rc4-pr/drizzle-orm/src/effect-core/query-effect.ts`
|
||||
- `/Users/kit/code/open-source/drizzle-orm-rc4-pr/integration-tests/tests/pg/effect-sql.test.ts`
|
||||
- SQLite Effect branch/reference:
|
||||
- `/Users/kit/code/open-source/drizzle-orm-beta16/drizzle-orm/src/up-migrations/effect-sqlite.ts`
|
||||
- `/Users/kit/code/open-source/drizzle-orm-beta16/integration-tests/tests/sqlite/effect-sql.test.ts`
|
||||
- `/Users/kit/code/open-source/drizzle-orm-beta16/drizzle-orm/type-tests/sqlite/effect.ts`
|
||||
- Effect SQLite client source of truth:
|
||||
- `/Users/kit/code/open-source/effect-smol/packages/sql/sqlite-bun/src/SqliteClient.ts`
|
||||
- `/Users/kit/code/open-source/effect-smol/packages/sql/sqlite-node/test/Client.test.ts`
|
||||
- `/Users/kit/code/open-source/effect-smol/packages/sql/sqlite-node/test/SqliteMigrator.test.ts`
|
||||
|
||||
Important API patterns from those references:
|
||||
|
||||
- Drizzle queries are Effect-yieldable: `yield* db.select().from(table)`.
|
||||
- Transactions are Effect values: `yield* db.transaction((tx) => Effect.gen(...), { behavior: "immediate" })`.
|
||||
- SQLite clients come from Effect layers such as `SqliteClient.layer({ filename })`.
|
||||
- Migrations can run through Effect SQL/SQLite migrator mechanisms or Drizzle's `effect-sqlite/migrator` when available.
|
||||
|
||||
## Public Surface
|
||||
|
||||
Do not invent an `Interface<TDatabase>` abstraction unless the Drizzle port already has one. The public surface should mirror Drizzle's Effect adapters:
|
||||
|
||||
```ts
|
||||
const db = yield * EffectDrizzleSqlite.make({ relations }).pipe(Effect.provide(EffectDrizzleSqlite.DefaultServices))
|
||||
|
||||
yield * db.select().from(users)
|
||||
yield *
|
||||
db.transaction(
|
||||
(tx) =>
|
||||
Effect.gen(function* () {
|
||||
yield* tx.insert(users).values({ name: "Ada" })
|
||||
}),
|
||||
{ behavior: "immediate" },
|
||||
)
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `make` / `makeWithDefaults` should match the Drizzle Effect SQLite branch as much as possible.
|
||||
- `DefaultServices` should provide Drizzle's default logger/cache services, same as Effect Postgres.
|
||||
- The package should depend on Effect SQL SQLite clients (`@effect/sql-sqlite-bun` and/or node) the same way the Drizzle branch does.
|
||||
- Opencode-specific path/channel selection stays in `packages/opencode`.
|
||||
|
||||
## Opencode Adoption Notes
|
||||
|
||||
These are not package requirements, but they matter for the later opencode adoption PR.
|
||||
|
||||
The current `packages/opencode/src/storage/db.ts` has two non-obvious semantics that the opencode wrapper must preserve when it consumes this adapter:
|
||||
|
||||
- Nested `Database.use` inside `Database.transaction` sees the current transaction, not the root client.
|
||||
- `Database.effect` queues post-commit side effects while inside a transaction, and runs immediately outside a transaction.
|
||||
|
||||
The opencode wrapper can implement that using Effect context instead of `LocalContext`:
|
||||
|
||||
- A private transaction context holding `{ tx, afterCommit }`.
|
||||
- `withDb`/`db` methods read the current transaction context if present, otherwise use the root db.
|
||||
- `transaction` installs a transaction context around the effect.
|
||||
- Nested transactions can either reuse the existing tx initially, matching current behavior, or later use explicit savepoints if needed.
|
||||
|
||||
Do not remove this behavior while moving opencode to Effect SQLite. `SyncEvent.run` depends on transaction composability and `behavior: "immediate"` for sequencing correctness.
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
1. Add `@opencode-ai/effect-drizzle-sqlite` with a minimal in-memory/file SQLite test schema.
|
||||
2. Port the Drizzle Effect SQLite adapter from the SQLite branch into the package, preserving upstream names and API shape.
|
||||
3. Test adapter-level guarantees:
|
||||
- query builders are yieldable Effect values,
|
||||
- `transaction(..., { behavior: "immediate" })` commits successful writes,
|
||||
- failed transaction rolls back,
|
||||
- migrations run once and in order,
|
||||
- close finalizer closes the underlying SQLite database.
|
||||
4. Add `@opencode-ai/effect-drizzle-sqlite` as a dependency of `packages/opencode`.
|
||||
5. Port `packages/opencode/src/storage/db.ts` to be a thin compatibility wrapper over the adapter plus opencode-specific transaction/post-commit context.
|
||||
6. Keep existing call sites working first:
|
||||
- `Database.Client()`
|
||||
- `Database.use(...)`
|
||||
- `Database.transaction(...)`
|
||||
- `Database.effect(...)`
|
||||
7. After compatibility is stable, migrate call sites from callback-style `Database.use` to yielding Effect Drizzle queries directly.
|
||||
8. Only then build domain stores like session/message/project stores on top of opencode's storage wrapper.
|
||||
|
||||
## Why This Is Cleaner Than Starting With SessionStorage
|
||||
|
||||
`SessionStorage` is a useful domain seam, but it does not answer the core adapter problem: how to make Drizzle SQLite Effect-native in this repo.
|
||||
|
||||
An Effect Drizzle SQLite package lets us vendor the adapter once. Then opencode can build its own storage wrapper on top, and `SessionStorage`, `MessageStorage`, event store, and projector writes can all share the same transaction and migration model.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Which client should the first package target: `@effect/sql-sqlite-bun`, `@effect/sql-sqlite-node`, or both behind separate layers?
|
||||
- How much source should we copy from the Drizzle branch versus import from catalog `drizzle-orm` internals?
|
||||
- What is the update path once Drizzle upstream ships `effect-sqlite`?
|
||||
- Should `afterCommit` stay opencode-specific until event publishing moves? Default answer: yes.
|
||||
- Should the compatibility wrapper preserve synchronous return types temporarily, or should the migration intentionally force Effect call sites?
|
||||
- Do CLI/admin raw SQL and sqlite shell stay in `packages/opencode`, or does the storage package expose backend capabilities for them?
|
||||
|
||||
## Recommended First PR
|
||||
|
||||
Make the first PR package-only and intentionally boring:
|
||||
|
||||
- Add `packages/effect-drizzle-sqlite`.
|
||||
- Use a tiny test schema, not opencode domain tables.
|
||||
- Prove Effect Drizzle SQLite queries, transactions, and migrations.
|
||||
- Do not migrate `packages/opencode` yet except possibly adding the dependency if needed for typechecking.
|
||||
|
||||
That gives us a focused place to validate the Effect SQLite approach before disturbing opencode's current database runtime.
|
||||
239
specs/storage/remove-opencode-db.md
Normal file
239
specs/storage/remove-opencode-db.md
Normal file
@@ -0,0 +1,239 @@
|
||||
# Remove `packages/opencode/src/storage/db.ts`
|
||||
|
||||
## Goal
|
||||
|
||||
Remove all production usages of the legacy `packages/opencode/src/storage/db.ts` module.
|
||||
|
||||
This means eliminating imports from `@/storage/db` or `./storage/db`, including:
|
||||
|
||||
- `Database.use(...)`
|
||||
- `Database.transaction(...)`
|
||||
- `Database.effect(...)`
|
||||
- `Database.Client()`
|
||||
- `Database.getPath()`
|
||||
- `Database.TxOrDb` / `Database.Transaction`
|
||||
- drizzle helpers re-exported from `@/storage/db`, such as `eq`
|
||||
|
||||
This does not mean removing SQLite or Drizzle everywhere in one step. The smaller target is deleting the opencode legacy wrapper by moving call sites onto deeper modules or onto the core/effect database adapter directly.
|
||||
|
||||
## Current Inventory
|
||||
|
||||
Production imports from `packages/opencode/src/storage/db.ts` are concentrated in 22 source files:
|
||||
|
||||
- `packages/opencode/src/account/repo.ts`
|
||||
- `packages/opencode/src/cli/cmd/db.ts`
|
||||
- `packages/opencode/src/cli/cmd/import.ts`
|
||||
- `packages/opencode/src/cli/cmd/stats.ts`
|
||||
- `packages/opencode/src/control-plane/workspace.ts`
|
||||
- `packages/opencode/src/index.ts`
|
||||
- `packages/opencode/src/node.ts`
|
||||
- `packages/opencode/src/permission/index.ts`
|
||||
- `packages/opencode/src/project/project.ts`
|
||||
- `packages/opencode/src/server/projectors.ts`
|
||||
- `packages/opencode/src/server/routes/instance/httpapi/handlers/sync.ts`
|
||||
- `packages/opencode/src/server/shared/fence.ts`
|
||||
- `packages/opencode/src/session/message-v2.ts`
|
||||
- `packages/opencode/src/session/projectors.ts`
|
||||
- `packages/opencode/src/session/prompt.ts`
|
||||
- `packages/opencode/src/session/session.ts`
|
||||
- `packages/opencode/src/session/todo.ts`
|
||||
- `packages/opencode/src/share/share-next.ts`
|
||||
- `packages/opencode/src/storage/db.ts`
|
||||
- `packages/opencode/src/sync/index.ts`
|
||||
- `packages/opencode/src/worktree/index.ts`
|
||||
|
||||
There are 65 direct API/type references in those files. The references fall into the groups below.
|
||||
|
||||
## Group 1: Database Runtime And Startup
|
||||
|
||||
Status: Completed. Startup, the public node export, and database CLI tooling no longer import the legacy opencode database wrapper; `packages/opencode/src/storage/db.ts` has been deleted.
|
||||
|
||||
Files:
|
||||
|
||||
- `packages/opencode/src/storage/db.ts`
|
||||
- `packages/opencode/src/index.ts`
|
||||
- `packages/opencode/src/node.ts`
|
||||
- `packages/opencode/src/cli/cmd/db.ts`
|
||||
|
||||
Current usage:
|
||||
|
||||
- `storage/db.ts` opens the singleton database, applies pragmas, exposes callback-style access, holds ambient transaction context, and queues post-commit effects.
|
||||
- `index.ts` no longer performs the removed JSON-to-SQLite migration during startup.
|
||||
- `node.ts` publicly re-exports `Database` from the legacy module.
|
||||
- `cli/cmd/db.ts` uses `Database.getPath()` to print the path, open a readonly Bun SQLite handle, run `sqlite3`, and vacuum.
|
||||
|
||||
Why this group comes first:
|
||||
|
||||
- These call sites define the seam currently used by every other group.
|
||||
- Deleting `storage/db.ts` requires an explicit replacement for database path, client acquisition, migration startup, and close/finalization.
|
||||
|
||||
Target shape:
|
||||
|
||||
- Move database path and client startup behind the core/effect database module rather than the opencode wrapper.
|
||||
- Replace `Database.Client()` with an Effect-provided database service or a narrow startup-only adapter.
|
||||
- Replace the public `node.ts` re-export with either no export or a stable non-legacy database capability.
|
||||
- Keep `cli/cmd/db.ts` as an admin/raw SQLite tool, but make it ask the replacement database path provider instead of importing `@/storage/db`.
|
||||
|
||||
## Group 2: Sync Event Transaction Boundary
|
||||
|
||||
Status: Completed. `SyncEvent` and the opencode projector boundary were removed; session/message event projection now lives in core EventV2/projector infrastructure.
|
||||
|
||||
Files:
|
||||
|
||||
- `packages/opencode/src/sync/index.ts`
|
||||
- `packages/opencode/src/session/projectors.ts`
|
||||
- `packages/opencode/src/server/projectors.ts`
|
||||
|
||||
Current usage:
|
||||
|
||||
- `SyncEvent.run` uses `Database.transaction(..., { behavior: "immediate" })` to allocate event sequence numbers safely.
|
||||
- `SyncEvent.process` wraps projector execution, event sequence writes, event log writes, and post-commit publishing in `Database.transaction(...)`.
|
||||
- `Database.effect(...)` queues publish side effects until after the transaction commits.
|
||||
- Projector functions accept `Database.TxOrDb` so they can write through either a root client or the active transaction.
|
||||
|
||||
Why this group is critical:
|
||||
|
||||
- It depends on the most non-obvious legacy behavior: nested `Database.use` inside a transaction must see the active transaction, and `Database.effect` must not publish until commit.
|
||||
- It is the central seam for session, message, permission, workspace, and server projection writes.
|
||||
|
||||
Target shape:
|
||||
|
||||
- Replace `Database.TxOrDb` with an explicit projector transaction type from the replacement database adapter.
|
||||
- Move transaction context and after-commit behavior into an Effect-native sync event implementation.
|
||||
- Preserve immediate transaction behavior for sequence allocation.
|
||||
- Convert projector registration to accept the new transaction interface before converting every projector body.
|
||||
|
||||
Suggested first step:
|
||||
|
||||
- Create a narrow internal module for sync projection execution, then migrate `SyncEvent.project(...)` and projector type signatures to that module. Keep the implementation backed by the new database adapter until all projector users are moved.
|
||||
|
||||
## Group 3: Domain Repositories Already Behind Services
|
||||
|
||||
Status: Completed. These services no longer import the legacy opencode database wrapper.
|
||||
|
||||
Files:
|
||||
|
||||
- `packages/opencode/src/account/repo.ts`
|
||||
- `packages/opencode/src/project/project.ts`
|
||||
- `packages/opencode/src/control-plane/workspace.ts`
|
||||
- `packages/opencode/src/share/share-next.ts`
|
||||
|
||||
Current usage:
|
||||
|
||||
- These modules already expose Effect services or Effect functions, but internally wrap `Database.use` with local `db(...)` helpers or `Effect.try`.
|
||||
- `account/repo.ts` uses both `Database.use` and `Database.transaction` through a repository interface.
|
||||
- `project/project.ts` has the largest mixed usage: Effect service methods use a local `db(...)` helper, while legacy top-level functions still call `Database.use` directly.
|
||||
- `control-plane/workspace.ts` and `share/share-next.ts` have local Effect wrappers around `Database.use`.
|
||||
|
||||
Why this group is tractable:
|
||||
|
||||
- The public interfaces are already deeper than the database calls.
|
||||
- Most callers should not need to know whether these modules use Drizzle, files, or core services internally.
|
||||
|
||||
Target shape:
|
||||
|
||||
- Inject the replacement database service into each Effect layer and yield Effect Drizzle queries directly.
|
||||
- Replace local callback wrappers with direct Effect queries.
|
||||
- Move remaining synchronous top-level helpers either behind the existing service interface or onto core modules.
|
||||
|
||||
Suggested order:
|
||||
|
||||
- Start with `account/repo.ts`; it has a clear repository interface and few call sites.
|
||||
- Then migrate `share/share-next.ts` and `control-plane/workspace.ts` local wrappers.
|
||||
- Leave `project/project.ts` for last in this group because it mixes project resolution, VCS, global bus emission, migration, and legacy top-level helpers.
|
||||
|
||||
## Group 4: Session And Message Read Models
|
||||
|
||||
Status: Completed. Session/message reads and projector writes have moved off the legacy opencode database wrapper.
|
||||
|
||||
Files:
|
||||
|
||||
- `packages/opencode/src/session/session.ts`
|
||||
- `packages/opencode/src/session/message-v2.ts`
|
||||
- `packages/opencode/src/session/prompt.ts`
|
||||
- `packages/opencode/src/session/todo.ts`
|
||||
- `packages/opencode/src/session/projectors.ts`
|
||||
|
||||
Current usage:
|
||||
|
||||
- `session/session.ts` uses `Database.use` for session reads, list queries, children, part lookup, and global list helpers.
|
||||
- `session/message-v2.ts` uses `Database.use` to page messages, hydrate parts, fetch one message, and fetch parts.
|
||||
- `session/prompt.ts` imports `eq` from `@/storage/db` and reads current prompt-related session/message rows directly.
|
||||
- `session/todo.ts` uses `Database.transaction` for todo replacement and `Database.use` for list reads.
|
||||
- `session/projectors.ts` uses `TxOrDb` for session/message usage projection helpers.
|
||||
|
||||
Why this group should be split:
|
||||
|
||||
- Reads can move independently from projector writes.
|
||||
- Message hydration is used by model prompt construction and session APIs, so changing it without a stable read module would spread query details across callers.
|
||||
- Projector writes are tied to Group 2's transaction type.
|
||||
|
||||
Target shape:
|
||||
|
||||
- Create or use a session/message read module with Effect-native methods for `get`, `list`, `page`, `parts`, and prompt assembly reads.
|
||||
- Move todo persistence either into a session todo repository or into the sync event projection path.
|
||||
- Convert `session/projectors.ts` only after Group 2 defines the replacement projector transaction type.
|
||||
|
||||
Suggested order:
|
||||
|
||||
- Migrate `session/message-v2.ts` reads first because the module already centralizes message pagination and hydration.
|
||||
- Migrate `session/session.ts` read helpers next.
|
||||
- Migrate `session/prompt.ts` after message/session reads exist, and import drizzle operators from `drizzle-orm` if any direct SQL remains temporarily.
|
||||
- Migrate `session/todo.ts` writes with the sync transaction work or move them behind a repository.
|
||||
|
||||
## Group 5: Legacy CLI And One-Off Admin Reads
|
||||
|
||||
Status: Completed. Remaining one-off CLI/admin reads and writes now use core database services or domain services instead of the legacy opencode database wrapper.
|
||||
|
||||
Files:
|
||||
|
||||
- `packages/opencode/src/cli/cmd/import.ts`
|
||||
- `packages/opencode/src/cli/cmd/stats.ts`
|
||||
- `packages/opencode/src/server/shared/fence.ts`
|
||||
- `packages/opencode/src/server/routes/instance/httpapi/handlers/sync.ts`
|
||||
- `packages/opencode/src/worktree/index.ts`
|
||||
- `packages/opencode/src/permission/index.ts`
|
||||
|
||||
Current usage:
|
||||
|
||||
- `cli/cmd/import.ts` writes imported sessions/messages/parts directly with `Database.use`.
|
||||
- `cli/cmd/stats.ts` reads all sessions directly.
|
||||
- `server/shared/fence.ts` queries sessions for fence context.
|
||||
- `handlers/sync.ts` reads event rows for HTTP sync endpoints.
|
||||
- `worktree/index.ts` looks up a project row for worktree behavior.
|
||||
- `permission/index.ts` reads permission rows directly.
|
||||
|
||||
Why this group is mostly cleanup:
|
||||
|
||||
- Most usages are small and can either call an existing domain service or be given a narrow query function.
|
||||
- They are not defining shared transaction semantics.
|
||||
|
||||
Target shape:
|
||||
|
||||
- Replace direct database reads with existing services where possible.
|
||||
- For admin/import commands, prefer dedicated import/stat modules rather than direct database access from command handlers.
|
||||
- For HTTP sync reads, move the event log query behind the sync event module.
|
||||
- For permission and worktree reads, call the permission/project services if available; otherwise add narrow repository methods.
|
||||
|
||||
## Recommended Migration Sequence
|
||||
|
||||
All migration groups are complete or superseded. `packages/opencode/src/storage/db.ts` has been deleted.
|
||||
|
||||
## Superseded: Data Migrations
|
||||
|
||||
Status: Superseded. No opencode data-migration group remains.
|
||||
|
||||
The previous opencode `data-migration.ts` service only backfilled session usage from message rows. That work is now covered by core database migration `packages/core/src/database/migration/20260510033149_session_usage.ts`, so there is no separate opencode data-migration group.
|
||||
|
||||
## Invariants To Preserve
|
||||
|
||||
- Nested reads inside a transaction must use the active transaction, not the root client.
|
||||
- `SyncEvent.run` sequence allocation must keep immediate transaction behavior.
|
||||
- Post-commit publish effects must not run before the transaction commits.
|
||||
- Existing schema ownership remains in `packages/core/src/**/*.sql.ts`; do not move table definitions back into `packages/opencode`.
|
||||
|
||||
## Verification Commands
|
||||
|
||||
- `rg "@/storage/db|./storage/db|Database\.(use|transaction|effect|Client|getPath)|\bTxOrDb\b|\bTransaction\b" packages/opencode/src`
|
||||
- `bun typecheck` from `packages/opencode`
|
||||
- Relevant package tests from `packages/opencode`, not the repo root
|
||||
Reference in New Issue
Block a user