refactor: extract @agents/kg + complete lib/ migration to SDK

.claude/plans/agents-cli/lib-migration/PLAN.md

@@ -0,0 +1,153 @@
+# CLI lib/ Migration Plan
+
+## Goal
+
+Move the 32 remaining TypeScript files from `packages/cli/src/lib/` into their proper packages (`@agents/sdk`, `@agents/kg`) or explicitly mark them as CLI-only. When complete, `cli/src/lib/` retains at most 5 files that are genuinely CLI-specific.
+
+## Current State
+
+| Location | Files | Description |
+|----------|-------|-------------|
+| `packages/cli/src/lib/` | 32 `.ts` files | Mixed concerns: KG, catalog, skill ops, agents, search, plugins |
+| `packages/cli/src/lib/component/` | 3 files | DI wiring: `factory.ts`, `index.ts`, `skill-ops-impl.ts` |
+| `packages/sdk/src/` | ~76 files | SDK context, catalog, providers, UI layers |
+| `packages/core/src/` | 11 files | Low-level primitives: hash, git, runtime, types |
+
+**Total lines to migrate:** ~11,588 (lib/*.ts) plus updates to ~12 command files that import from lib/.
+
+## Target State
+
+After all 6 phases:
+
+```text
+packages/cli/src/lib/
+  agents.ts                    # CLI-specific agent config (implements SDK AgentResolver)
+  component/
+    factory.ts                 # Wires SDK providers with CLI config
+    index.ts                   # Barrel re-exports
+    skill-ops-impl.ts          # DI adapter (imports from SDK, trivial)
+```
+
+Everything else lives in `@agents/sdk` or `@agents/kg`.
+
+## Test Baselines
+
+| Package | Pass | Fail | Notes |
+|---------|------|------|-------|
+| `@agents/core` | 10 | 0 | Stable foundation |
+| `@agents/sdk` | 382 | 0 | Growing as modules move in |
+| `@agents/cli` | 1684 | 10 | 10 known failures (pre-existing) |
+| `@agents/kg` | 0 | 0 | New package -- tests created in Phase 1 |
+
+**Rule:** No phase may increase the fail count in any package. New code must ship with tests.
+
+## Dependency Graph
+
+```text
+Phase 1: KG Extraction ─────────────────────────────────┐
+                                                         │
+Phase 2: Content Parsing → SDK ──────────────────────────┤
+                              │                          │
+                              ├── Phase 3: Catalog ──────┤
+                              │                          │
+Phase 4: Agent Abstraction ───┤                          │
+                              │                          │
+                              ├── Phase 5: Skill Ops ────┤
+                                                         │
+                              Phase 6: Cleanup ──────────┘
+```
+
+- **Phase 1** is independent -- no blockers, no downstream until Phase 6.
+- **Phase 2** is independent -- unlocks Phases 3 and 5.
+- **Phase 3** depends on Phase 2 (manifest.ts and lockfile.ts must be in SDK).
+- **Phase 4** is independent -- but must complete before Phase 5.
+- **Phase 5** depends on Phases 2 + 4 (content parsing + agent abstraction).
+- **Phase 6** depends on all previous phases.
+
+**Parallelism:** Phases 1, 2, and 4 can run concurrently.
+
+## Phase Summary
+
+| Phase | Name | Files Moved | Target Package | Lines |
+|-------|------|-------------|----------------|-------|
+| 1 | KG Extraction | 6 | `@agents/kg` (new) | ~1,369 |
+| 2 | Content Parsing | 4 | `@agents/sdk` | ~1,507 |
+| 3 | Catalog Pipeline | 7 | `@agents/sdk` | ~3,719 |
+| 4 | Agent Abstraction | 0 (create interface) | `@agents/sdk` | ~50 new |
+| 5 | Skill Operations | 11 | `@agents/sdk` | ~2,924 |
+| 6 | Search & Cleanup | 3 + verify | `@agents/sdk` | ~1,781 |
+
+**Total:** 31 files moved/dissolved + 1 file stays (`agents.ts`). Component directory (3 files) stays.
+
+## Global Acceptance Criteria
+
+1. All existing tests pass (core 10/0, SDK 382+/0, CLI 1684/10).
+2. New code has at least one test per exported function.
+3. No circular dependencies between packages.
+4. `bun run packages/cli/src/bin/agents.ts --help` works.
+5. CLI commands that exercised moved modules still pass their integration tests.
+6. `cli/src/lib/` contains exactly 4 files (agents.ts + 3 component/ files).
+7. `packages/kg/` exists with package.json, exports, and passing tests.
+8. No `import ... from '../lib/...'` in any SDK or KG package.
+
+## Error Code Mapping
+
+Modules use `CliError` codes. After migration, SDK modules should use `SdkError` (from `@agents/sdk/util/errors`) instead. Map:
+
+| CLI Code | SDK Code | Module |
+|----------|----------|--------|
+| `E_UNKNOWN_AGENT` | `E_AGENT_UNKNOWN` | agents.ts (stays CLI) |
+| `E_LOCKFILE_PARSE` | `E_STORAGE_BACKEND` | lockfile.ts |
+| `E_MANIFEST_PARSE` | `E_PARSE_FAILED` | manifest.ts |
+| `E_CATALOG_IO` | `E_STORAGE_BACKEND` | catalog*.ts |
+| `E_KG_*` | `E_KG_*` (new) | graph*.ts, embedder.ts |
+| `E_SEARCH_*` | `E_SEARCH_*` | search.ts, meilisearch.ts |
+
+## Test Migration Policy
+
+For each module moved from CLI to SDK/KG:
+1. If a corresponding test exists in `packages/cli/test/`, move it to the destination package's test directory.
+2. Update test imports to use the new package path.
+3. If the test has CLI-specific fixtures or setup, keep it in CLI and update imports to point to the new package.
+
+**Test files per phase:**
+- **Phase 1:** `chunker.test.ts`, `graph.test.ts` --> `packages/kg/test/`
+- **Phase 2:** `manifest.test.ts`, `lockfile.test.ts` --> `packages/sdk/test/`
+- **Phase 3:** `catalog-*.test.ts` (7 files) --> `packages/sdk/test/catalog/`
+- **Phase 5:** `skill-*.test.ts` (7 files) --> `packages/sdk/test/providers/local/`
+- **Phase 6:** `external-skills*.test.ts` --> `packages/sdk/test/`
+
+## Non-Goals
+
+- **No behavioral changes.** This is a pure structural refactor. Functions keep the same signatures and semantics.
+- **No new features.** Each phase produces identical output as before.
+- **No Python changes.** The `chunker.py` and `embedder.py` files stay in cli/src/lib/ (they are Python, not TypeScript).
+- **No command restructuring.** CLI commands keep their current file layout; only their imports change.
+- **No package publishing.** All packages remain `private: true` workspace packages.
+- **No dependency version bumps.** Dependencies move between package.json files at their current versions.
+
+## Phase Details
+
+See individual phase files:
+
+- [Phase 1: KG Extraction](./phase/1-kg-extraction.md)
+- [Phase 2: Content Parsing](./phase/2-content-parsing.md)
+- [Phase 3: Catalog Pipeline](./phase/3-catalog-pipeline.md)
+- [Phase 4: Agent Abstraction](./phase/4-agent-abstraction.md)
+- [Phase 5: Skill Operations](./phase/5-skill-operations.md)
+- [Phase 6: Search & Cleanup](./phase/6-search-and-cleanup.md)
+
+## Migration Strategy per File
+
+For each file:
+
+1. **Copy** to target location (do not delete source yet).
+2. **Update imports** in the copy to use package-relative paths.
+3. **Replace error classes** (`CliError` -> `SdkError` for SDK modules).
+4. **Re-export from source** -- change the original file to `export * from '@agents/<pkg>/...'`.
+5. **Run tests** -- all existing tests must pass via the re-export shim.
+6. **Update consumers** -- change command files to import from the new package.
+7. **Delete shim** -- remove the re-export file from cli/src/lib/.
+8. **Run tests again** -- confirm nothing breaks.
+
+This two-step approach (shim then delete) prevents big-bang breakage. If step 6 causes failures, the shim still works as a fallback.

.claude/plans/agents-cli/lib-migration/phase/1-kg-extraction.md

@@ -0,0 +1,237 @@
+# Phase 1: Extract @agents/kg (Knowledge Graph)
+
+## Goal
+
+Create a new `packages/kg/` workspace package containing all knowledge-graph modules: graph construction, schema validation, file locking, embedding, search indexing, and markdown chunking.
+
+## Non-Goals
+
+- Moving Python files (`chunker.py`, `embedder.py`) -- they stay in `cli/src/lib/`.
+- Changing chunking algorithms or embedding behavior.
+- Adding new KG features.
+- Wiring KG into the SDK catalog system (that is a future concern beyond this migration).
+
+## Prerequisites
+
+- None. This phase has no blockers -- KG modules are self-contained and only consumed by CLI commands and the graph-viewer server.
+
+## Files
+
+### Create (new package scaffold)
+
+| File | Description |
+|------|-------------|
+| `packages/kg/package.json` | Package manifest with KG-specific deps |
+| `packages/kg/tsconfig.json` | TypeScript config extending root |
+| `packages/kg/src/index.ts` | Barrel re-exports |
+| `packages/kg/src/graph.ts` | Graph construction (from `cli/src/lib/graph.ts`, 202 lines) |
+| `packages/kg/src/graph-schema.ts` | JSON Schema validation (from `cli/src/lib/graph-schema.ts`, 121 lines) |
+| `packages/kg/src/graph-lock.ts` | File-based graph locking (from `cli/src/lib/graph-lock.ts`, 158 lines) |
+| `packages/kg/src/embedder.ts` | Ollama embedding client (from `cli/src/lib/embedder.ts`, 59 lines) |
+| `packages/kg/src/meilisearch.ts` | Meilisearch client wrapper (from `cli/src/lib/meilisearch.ts`, 478 lines) |
+| `packages/kg/src/chunker.ts` | Markdown chunking + frontmatter (from `cli/src/lib/chunker.ts`, 364 lines) |
+| `packages/kg/test/chunker.test.ts` | Unit tests for chunker |
+| `packages/kg/test/graph.test.ts` | Unit tests for graph construction |
+| `packages/kg/test/embedder.test.ts` | Unit tests for embedder |
+
+### Modify (update imports)
+
+| File | Change |
+|------|--------|
+| `packages/cli/src/commands/kg.ts` | Change `from '../lib/chunker'` to `from '@agents/kg/chunker'`, etc. |
+| `packages/cli/src/server/graph-viewer/routes/*.ts` | Change `from '../../lib/graph'` to `from '@agents/kg/graph'`, etc. |
+| `packages/cli/package.json` | Add `"@agents/kg": "workspace:*"` dependency |
+| `root package.json` | Add `packages/kg` to workspaces array |
+
+### Delete (after consumers updated)
+
+| File | Lines |
+|------|-------|
+| `packages/cli/src/lib/graph.ts` | 202 |
+| `packages/cli/src/lib/graph-schema.ts` | 121 |
+| `packages/cli/src/lib/graph-lock.ts` | 158 |
+| `packages/cli/src/lib/embedder.ts` | 59 |
+| `packages/cli/src/lib/meilisearch.ts` | 478 |
+| `packages/cli/src/lib/chunker.ts` | 364 |
+
+## Steps
+
+- [ ] **1.1** Create `packages/kg/` directory structure.
+- [ ] **1.2** Write `packages/kg/package.json`:
+
+  ```json
+  {
+    "name": "@agents/kg",
+    "version": "0.1.0",
+    "private": true,
+    "type": "module",
+    "exports": {
+      ".": "./src/index.ts",
+      "./*": "./src/*.ts"
+    },
+    "dependencies": {
+      "@agents/core": "workspace:*",
+      "ajv": "^8.18.0",
+      "ajv-formats": "^3.0.1",
+      "graphology": "^0.26.0",
+      "graphology-layout": "^0.6.1",
+      "graphology-layout-forceatlas2": "^0.10.1",
+      "js-yaml": "^4.1.0",
+      "meilisearch": "^0.56.0",
+      "ollama": "^0.5.0"
+    }
+  }
+  ```
+
+- [ ] **1.3** Write `packages/kg/tsconfig.json` extending root tsconfig.
+- [ ] **1.4** Copy the 6 source files from `cli/src/lib/` to `packages/kg/src/`.
+- [ ] **1.5** Update imports in each copied file:
+  - `chunker.ts`: `@agents/core/runtime` stays (cross-package import is fine).
+  - `meilisearch.ts`: `@agents/core/types` stays.
+  - `graph.ts`, `graph-schema.ts`, `graph-lock.ts`: No external package imports to change (only node built-ins and direct deps).
+  - `embedder.ts`: Only imports `ollama` -- no changes.
+- [ ] **1.6** Replace `CliError` with a KG-specific error type in meilisearch.ts:
+
+  ```typescript
+  // Before
+  import { CliError } from '@agents/core/types'
+  // After
+  import { CliError } from '@agents/core/types'  // Keep CliError for now -- it's from core, not CLI
+  ```
+
+  Note: `CliError` is actually defined in `@agents/core/types`, not in the CLI package. It can be used anywhere. No change needed in Phase 1; error code rename happens in Phase 6 cleanup.
+- [ ] **1.7** Write `packages/kg/src/index.ts` barrel:
+
+  ```typescript
+  export * from './chunker'
+  export * from './embedder'
+  export * from './graph'
+  export * from './graph-lock'
+  export * from './graph-schema'
+  export * from './meilisearch'
+  ```
+
+- [ ] **1.8** Add `"@agents/kg": "workspace:*"` to `packages/cli/package.json` dependencies.
+- [ ] **1.9** Add `"packages/kg"` to root `package.json` workspaces.
+- [ ] **1.10** Run `bun install` from root to link the new workspace package.
+- [ ] **1.11** Update `packages/cli/src/commands/kg.ts` imports:
+
+  ```typescript
+  // Before
+  import { chunkMarkdown, parseFrontmatter } from '../lib/chunker'
+  import { embed } from '../lib/embedder'
+  import { createClient, searchKeyword } from '../lib/meilisearch'
+  // After
+  import { chunkMarkdown, parseFrontmatter } from '@agents/kg/chunker'
+  import { embed } from '@agents/kg/embedder'
+  import { createClient, searchKeyword } from '@agents/kg/meilisearch'
+  ```
+
+- [ ] **1.12** Update `packages/cli/src/server/graph-viewer/routes/*.ts` imports similarly.
+- [ ] **1.13** Update any other CLI files that import from the 6 moved modules (search with `grep`).
+- [ ] **1.14** Write tests for `packages/kg/test/`:
+  - `chunker.test.ts`: Test `parseFrontmatter`, `chunkMarkdown` with sample markdown.
+  - `graph.test.ts`: Test graph construction from mock data.
+  - `embedder.test.ts`: Test embed function signature (mock Ollama).
+- [ ] **1.15** Run full test suite:
+  - `bun test --cwd packages/kg` -- all new tests pass.
+  - `bun test --cwd packages/cli` -- 1684 pass / 10 fail (unchanged).
+- [ ] **1.16** Delete the 6 source files from `cli/src/lib/`.
+- [ ] **1.17** Run CLI test suite again to confirm no regressions.
+
+## Acceptance Criteria
+
+1. `packages/kg/` exists with `package.json`, `tsconfig.json`, `src/`, and `test/`.
+2. `bun test --cwd packages/kg` passes with at least 6 test cases.
+3. `bun test --cwd packages/cli` maintains 1684 pass / 10 fail.
+4. No file in `packages/cli/src/lib/` imports from `packages/kg/src/` via relative path.
+5. `packages/cli/src/lib/` no longer contains: `graph.ts`, `graph-schema.ts`, `graph-lock.ts`, `embedder.ts`, `meilisearch.ts`, `chunker.ts`.
+6. `bun run packages/cli/src/bin/agents.ts kg --help` still works.
+
+## Failure Criteria
+
+- **Stop if:** Moving `chunker.ts` breaks `manifest.ts` (Phase 2 dependency). In that case, leave a re-export shim in `cli/src/lib/chunker.ts`:
+
+  ```typescript
+  export { parseFrontmatter, chunkMarkdown, type Chunk, type ParsedMarkdown } from '@agents/kg/chunker'
+  ```
+
+  This shim allows Phase 1 to complete independently. Phase 2 will update `manifest.ts` to import from `@agents/kg/chunker` or `@agents/sdk/context/frontmatter` directly.
+
+- **Stop if:** Graph-viewer server fails to start after import changes. Debug the import path resolution in Bun's module system before proceeding.
+
+## Fallback Logic
+
+1. If any moved module fails to resolve via `@agents/kg/*`, check that `packages/kg/package.json` exports map is correct and that `bun install` was re-run.
+2. If Ollama types cause issues in `embedder.ts`, add `ollama` to `@agents/kg` devDependencies and use dynamic import.
+3. If test count drops, diff the test output against the baseline and identify which tests regressed -- do not proceed to Phase 2 until resolved.
+
+## Test Files
+
+Move corresponding test files from `packages/cli/test/` to `packages/kg/test/`:
+- `chunker.test.ts` --> `packages/kg/test/chunker.test.ts`
+- `graph.test.ts` --> `packages/kg/test/graph.test.ts`
+
+Update imports in moved tests to use `@agents/kg/*` paths. If tests have CLI-specific fixtures, keep them in CLI and update imports to point to `@agents/kg`.
+
+## Dependency Notes
+
+- `chunker.ts` is imported by `manifest.ts` (for `parseFrontmatter`). After Phase 1 moves chunker to `@agents/kg`, `manifest.ts` temporarily imports from `@agents/kg/chunker`. Phase 2 resolves this by switching manifest to use `@agents/sdk/context/frontmatter.parseFrontmatter` instead, removing the cross-package KG dependency from the content-parsing layer.
+- `meilisearch.ts` is imported by `skill-search-api.ts` (Phase 5). After Phase 1, skill-search-api imports from `@agents/kg/meilisearch`. Phase 6 may move search to SDK, at which point the import path updates again.
+- Graph modules are only consumed by `commands/kg.ts` and `server/graph-viewer/` -- no transitive dependency concerns.
+
+## Examples
+
+### Before (cli/src/commands/kg.ts)
+
+```typescript
+import { chunkMarkdown, parseFrontmatter } from '../lib/chunker'
+import { embed } from '../lib/embedder'
+import { buildGraph } from '../lib/graph'
+```
+
+### After (cli/src/commands/kg.ts)
+
+```typescript
+import { chunkMarkdown, parseFrontmatter } from '@agents/kg/chunker'
+import { embed } from '@agents/kg/embedder'
+import { buildGraph } from '@agents/kg/graph'
+```
+
+### Before (cli/src/lib/manifest.ts)
+
+```typescript
+import { parseFrontmatter } from './chunker'
+```
+
+### After Phase 1 (temporary -- resolved in Phase 2)
+
+```typescript
+import { parseFrontmatter } from '@agents/kg/chunker'
+```
+
+### packages/kg/package.json (new)
+
+```json
+{
+  "name": "@agents/kg",
+  "version": "0.1.0",
+  "private": true,
+  "type": "module",
+  "exports": {
+    ".": "./src/index.ts",
+    "./*": "./src/*.ts"
+  },
+  "dependencies": {
+    "@agents/core": "workspace:*",
+    "ajv": "^8.18.0",
+    "ajv-formats": "^3.0.1",
+    "graphology": "^0.26.0",
+    "graphology-layout": "^0.6.1",
+    "graphology-layout-forceatlas2": "^0.10.1",
+    "js-yaml": "^4.1.0",
+    "meilisearch": "^0.56.0",
+    "ollama": "^0.5.0"
+  }
+}
+```