# VeriThink Privacy & Data-Retention Guide

This document describes what the VeriThink engine's **self-learning** subsystem persists
to disk, **where** it stores it, what each record actually contains, and **how to control,
disable, or clear** it. It is written for privacy-sensitive deployments (e.g. work with
proprietary code, regulated data, or confidential prompts).

All statements below are grounded in the current source. Key files:

- `src/ReasonCore/Config/VeriThinkConfig.cs` — all config keys and their environment variables.
- `src/ReasonCore/Experience/ExperienceRecord.cs`, `JsonExperienceStore.cs`, `SqliteExperienceStore.cs`, `PostgresExperienceStore.cs`, `ExperienceRetentionService.cs`.
- `src/ReasonCore/Learning/JsonProfileStore.cs`, `JsonModelQualityStore.cs`, `IModelQualityStore.cs`, `KeywordMemoryIndex.cs`, `IMemoryIndex.cs`.
- `src/ReasonCore/Telemetry/JsonUsageLedger.cs`, `IUsageLedger.cs`.
- `src/ReasonCore/Llm/SecretRedactor.cs`, `src/ReasonCore/Text/TextEncodingHelper.cs`.
- `src/ReasonCore/Composition/StoreFactory.cs` (store/index selection).
- `src/ReasonService/Program.cs` (`/healthz` `paths` block, `/v1/feedback`, `/v1/verify` write locations).

---

## 1. Overview

VeriThink "learns" from every reasoning run so that future plans improve. That learning is
backed by several on-disk stores, all rooted (by default) under a `.verithink/` directory
relative to the service's working directory:

| Store | Purpose | Contains request/answer content? |
|---|---|---|
| Experience records | Full transcript of each run (request, answer, plan graph) | **Yes — full text** |
| Feedback notes | Per-task outcome + free-text note | **Yes — free-text note** |
| Verify reports | Verification results per task | Partial (verdict + scores) |
| Usage ledger | Cost/token telemetry per run | **Yes — 117-char request preview** |
| Repo profiles | Learned category weights per repo | No (category names + numbers only) |
| Model quality telemetry | Per-model outcome quality (EWMA) | No (model/domain/score only) |
| Memory index | Retrieval of similar past runs | Depends on mode (see §2.7) |

The stores that can hold **sensitive request/answer content** are the experience records,
the feedback notes, and the usage-ledger preview. The profile and model-quality stores hold
only derived numeric signals and category identifiers.

---

## 2. What is stored (per store, with fields)

### 2.1 Experience records — the transcript store

Written after every completed run by `ReasonPipeline` (`src/ReasonCore/Fsm/ReasonPipeline.cs`,
~line 326) via `IExperienceStore.SaveAsync`. Record shape (`ExperienceRecord.cs`):

| Field | What it holds |
|---|---|
| `TaskId` | Run identifier (also the filename in the JSON store). |
| `Domain` | `coding` / `research` / etc. |
| `GoalTitles[]` | Short titles of the derived goals. |
| `CategoriesUsed[]` | Requirement category IDs touched by the plan. |
| `Coverage` | Numeric coverage score. |
| `DurationMs` | Wall-clock duration. |
| `CreatedAt` | ISO-8601 UTC timestamp. |
| `GraphJson` | **The full plan graph**: node titles/categories/meta, synth summary, plan-step titles + details + `addresses`, open questions, per-state trace (state, ms, model, tokens), adequacy checks. |
| `SessionId` | Session/branch key (e.g. `aithink/<branch>`). |
| `TenantId` | Tenant key, if supplied. |
| `RequestText` | **The full original request text.** |
| `AnswerMarkdown` | **The full generated answer/plan markdown.** |
| `Depth`, `SymFsmLevel`, `RequestMode` | Reasoning knobs used. |
| `SessionTurn` | Turn index within a session. |
| `RecallTags[]` | Derived tags used for memory recall. |

**Sensitivity flags:**

- `RequestText` and `AnswerMarkdown` store the **complete** request and answer. In the default
  **JSON store**, both are passed through `SecretRedactor.Redact(...)` on save
  (`JsonExperienceStore.SaveAsync`, ~lines 34–35), which masks bearer tokens, `api_key=...`,
  `sk-...` keys, and `*_KEY/_TOKEN/_SECRET=...` env-style secrets (`SecretRedactor.cs`). It does
  **not** truncate the text and does **not** remove non-secret sensitive content (proprietary
  code, PII, business logic, etc.).
- `GraphJson` is **not** run through the secret redactor. Plan-step details and node titles derived
  from the request can therefore contain paraphrased request content in cleartext.
- **Important — redaction is JSON-store-only.** `SecretRedactor` is referenced only in
  `JsonExperienceStore.cs` (confirmed by searching `src/`). The **SQLite** and **Postgres**
  experience stores (`SqliteExperienceStore.cs`, `PostgresExperienceStore.cs`) persist
  `request_text` / `answer_markdown` **exactly as passed in** — i.e. secret-redaction is **not**
  applied when `VERITHINK_STORE=sqlite` or `postgres`.

An `index.json` (task → domain/goals/categories) is maintained alongside the JSON records.

### 2.2 Feedback notes

Written by `POST /v1/feedback` (`src/ReasonService/Program.cs`, ~line 1368) to
`<ExperienceDir>/feedback/<taskId>.json`. Payload stored: `{ taskId, outcome, note }`.
The `note` is **free-text supplied by the caller** and is **not** redacted. Treat it as
potentially sensitive.

### 2.3 Verify reports

Written by `POST /v1/verify` (`Program.cs`, ~line 1101) to
`<ExperienceDir>/verify/<taskId>.json`. Contains the verification report (per-item statuses,
detail strings, `doneScore`). Detail strings can echo fragments of the artifact/criteria being
verified.

### 2.4 Usage ledger

Appended per run by `JsonUsageLedger` to `<ExperienceDir>/usage-log.jsonl` (JSON-lines).
Entry shape (`IUsageLedger.cs` → `UsageLogEntry`):
`taskId, sessionId, provider, modelCheap, modelStrong, llmCalls, tokensIn, tokensOut,
estimatedCostUsd, durationMs, createdAt, depth, symFsmLevel, requestPreview, resultStatus,
coverage, adequacyScore, traceSummary`.

**Sensitivity flag:** `requestPreview` is the **first 117 characters** of the raw request
(`ReasonPipeline.TryLogUsageAsync`, ~line 381: `request.Request[..117] + "..."`). It is **not**
passed through `SecretRedactor`, so the opening of a prompt is stored verbatim.

### 2.5 Repo profiles

`JsonProfileStore` (`JsonProfileStore.cs`) writes `<ProfileDir>/<repoKey>.json`.
Fields: `categoryWeights` (category ID → weight `0..2`), `mandatoryOverrides`
(category ID → bool), `weightDecayLastAppliedUtc`. **No request or answer text** — only
requirement-category identifiers and numeric weights derived from feedback outcomes.

### 2.6 Model quality telemetry

`JsonModelQualityStore` (`JsonModelQualityStore.cs`) writes a single
`<ModelQualityDir>/model-quality.json`. It is a list of `ModelQualityEntry`
(`IModelQualityStore.cs`): `model, domain, quality (EWMA 0..1), samples, lastSeenUtc`.
**No request/answer content** — only model name, domain, an aggregate score, and a count.

### 2.7 Memory index

Selected by `StoreFactory.CreateMemoryIndex` from `VERITHINK_MEMORY_MODE`:

- `keyword` (**default**): `KeywordMemoryIndex` keeps **no separate persistence** — it reads
  live from the experience store on each search. So it adds no new files; its privacy profile is
  that of the experience store.
- `vector` + `VERITHINK_STORE=json`: `HashVectorMemoryIndex` — in-memory only, no new persistence.
- `vector` + `VERITHINK_STORE=sqlite`/`postgres`: `SqliteVectorMemoryIndex` /
  `PostgresVectorMemoryIndex` persist vector data into the same SQLite/Postgres database.

### 2.8 Derived learning artifacts

Under `<ExperienceDir>`: `learning-report.md`, `promotion-queue.json`, and (when auto-apply is in
preview) `auto-apply-preview.json`. These summarize aggregated outcomes/categories, not raw prompts.

---

## 3. Where it lives (data-location summary)

Default locations assume the JSON store and are **relative to the service working directory**.
You can confirm the resolved paths at runtime via `GET /healthz` → `paths`
(`experienceDir`, `profileDir`, `repoPath`, `repoKey`, and existence flags —
`Program.cs` ~line 539).

| Store | Default file / dir | Env var | What's stored | How to disable / clear |
|---|---|---|---|---|
| Experience records | `./.verithink/experience/<taskId>.json` + `index.json` | `VERITHINK_EXPERIENCE_DIR` | Full request, full answer, plan graph, metadata (§2.1) | `VERITHINK_LEARNING_ENABLED=0` or `VERITHINK_EXPERIENCE_WRITE=0` stop writing (see §4); also retention days, relocate/clear the dir. Delete files to clear. |
| Feedback notes | `./.verithink/experience/feedback/<taskId>.json` | (under `VERITHINK_EXPERIENCE_DIR`) | `{ taskId, outcome, note }`; free-text note | `VERITHINK_FEEDBACK_WRITE=0` stops the note write; don't call `/v1/feedback`; pruned by retention; delete files. |
| Verify reports | `./.verithink/experience/verify/<taskId>.json` | (under `VERITHINK_EXPERIENCE_DIR`) | Verification report / scores | `VERITHINK_VERIFY_WRITE=0` stops the report write; don't call `/v1/verify`; pruned by retention; delete files. |
| Usage ledger | `./.verithink/experience/usage-log.jsonl` | (under `VERITHINK_EXPERIENCE_DIR`) | Cost/token telemetry + 117-char request preview | `VERITHINK_USAGE_LOG=0` stops writing; relocate dir or delete the file. |
| Repo profiles | `./.verithink/profiles/<repoKey>.json` | `VERITHINK_PROFILE_DIR` | Category weights + mandatory overrides (no text) | `VERITHINK_LEARNING_ENABLED=0` stops profile mutations; relocate dir or delete files. |
| Model quality | `./.verithink/model-quality/model-quality.json` | `VERITHINK_MODEL_QUALITY_DIR` | Per-(model,domain) EWMA quality + sample count (no text) | `VERITHINK_MODEL_QUALITY_WRITE=0` (or `VERITHINK_LEARNING_ENABLED=0`) stops writing; `VERITHINK_ROUTER_QUALITY_TELEMETRY=0` stops it *influencing routing*; relocate dir or delete file. |
| Memory index | none (keyword) / SQLite/Postgres DB (vector) | `VERITHINK_MEMORY_MODE`, `VERITHINK_STORE`, `VERITHINK_DB_URL` | Retrieval signal over experience (§2.7) | Keyword mode persists nothing extra; for vector, clear the DB. |
| Learning artifacts | `./.verithink/experience/learning-report.md`, `promotion-queue.json` | (under `VERITHINK_EXPERIENCE_DIR`) | Aggregated outcomes/categories | Protected from retention pruning; delete manually. |

**Alternate backends** (`VERITHINK_STORE`, `StoreFactory.cs`):
- `json` (default) — files as above; **request/answer secret-redacted**.
- `sqlite` — experience in `<ExperienceDir>/experience.db` (or the path in `VERITHINK_DB_URL`,
  `sqlite:` prefix stripped). **Request/answer stored unredacted.**
- `postgres` — experience in the database at `VERITHINK_DB_URL` / `VERITHINK_PG_CONN`.
  **Request/answer stored unredacted.**

---

## 4. Controlling / disabling learning

VeriThink exposes a **master consent switch plus per-store write gates** so a privacy-sensitive
deployment can turn persistence off entirely — no directory tricks required. All gates default to
**true** (learning on), so existing deployments are unchanged.

**Consent + write gates (the primary control):**

| Env var | Config | Default | When `false`, stops writing… |
|---|---|---|---|
| `VERITHINK_LEARNING_ENABLED` | `LearningEnabled` | `true` | **Master consent.** Disables ALL self-learning persistence at once: experience records, memory index, model-quality telemetry, evolution strategies, and all profile auto-updates (budget-exhausted signal, feedback-driven learning, auto-apply). |
| `VERITHINK_EXPERIENCE_WRITE` | `ExperienceWriteEnabled` | `true` | Experience transcript records + memory index + auto-apply. Effective gate is `PersistExperience = LearningEnabled && ExperienceWriteEnabled`. |
| `VERITHINK_MODEL_QUALITY_WRITE` | `ModelQualityWriteEnabled` | `true` | Per-`(model,domain)` quality telemetry (`model-quality.json`). Effective gate is `PersistModelQuality = LearningEnabled && ModelQualityWriteEnabled`. |
| `VERITHINK_USAGE_LOG` | `UsageLogEnabled` | `true` | The usage/cost ledger (`usage-log.jsonl`), including its 117-char request preview. Independent of the master (it is billing telemetry, not learning). |
| `VERITHINK_FEEDBACK_WRITE` | `FeedbackWriteEnabled` | `true` | The `POST /v1/feedback` free-text note file. (Feedback-driven **profile learning** is separately gated by `LearningEnabled`.) |
| `VERITHINK_VERIFY_WRITE` | `VerifyWriteEnabled` | `true` | The `POST /v1/verify` report file. |
| `VERITHINK_VERIFY_FEEDBACK` | `VerifyFeedback` | `true` | Profile weight/mandatory updates from `/v1/verify` outcomes. Requires `LearningEnabled=true` (master consent) — with learning off, verify never mutates profiles. |
| `VERITHINK_LEARNING_JOIN_FEEDBACK` | `LearningJoinFeedback` | `true` | Join lesson failRate to feedback taskIds only (`0` = legacy uses vs failures formula). |
| `VERITHINK_LEARNING_LESSON_MIN_N` | `LearningLessonMinN` | `5` | Min feedback-backed tasks per category before a lesson line appears. |
| `VERITHINK_LEARNING_LESSON_RATE` | `LearningLessonRate` | `0.30` | Min failure rate for lessons when join is on. |
| `VERITHINK_MEMORY_RECENT` | `MemoryRecent` | `200` | Recent experience window for keyword/hash recall (`50` = legacy). |
| `VERITHINK_MEMORY_OUTCOME_WEIGHT` | `MemoryOutcomeWeight` | `true` | Adjust recall score from `feedback/{taskId}.json` at search time (`0` = off). |
| `VERITHINK_COG_BOOST_ON_SUCCESS` | `CogBoostOnSuccess` | `true` | Defer cog usefulness boost until `/v1/verify` or `/v1/feedback` success (`0` = boost in StepLearn). |

Setting `VERITHINK_LEARNING_ENABLED=0` is the one-switch "learn nothing" mode: the engine still
reasons and returns full answers, but writes no experience transcripts, no telemetry, and makes no
profile mutations. Combine with `VERITHINK_USAGE_LOG=0`, `VERITHINK_FEEDBACK_WRITE=0`, and
`VERITHINK_VERIFY_WRITE=0` for a fully non-persisting run. All gate checks are **fail-open** in the
sense that they never throw — a disabled gate simply skips the write.

**Verify the effective gates at runtime:** `GET /healthz` returns a `learning` sub-block with the
effective booleans (`enabled`, `experienceWrite`, `modelQualityWrite`, `usageLog`, `feedbackWrite`,
`verifyWrite`, `persistExperience`, `persistModelQuality`) — safe booleans only, no secrets.
`GET /v1/capabilities` exposes `selfLearningEnabled` for UI/consumers.

**Retention (a secondary control):**

- `VERITHINK_EXPERIENCE_RETENTION_DAYS` (default `0` = keep forever). When `> 0`,
  `ExperienceRetentionService.Prune` deletes experience/feedback/verify files older than N days
  on startup. `index.json`, `learning-report.md`, and `promotion-queue.json` are **protected**
  and never pruned. This prunes the usage ledger's *directory* but the `usage-log.jsonl` file
  itself is a single appended file (not matched by the per-file `*.json` prune), so treat it
  separately.

**Relocation / isolation (works for every store):**

- `VERITHINK_EXPERIENCE_DIR`, `VERITHINK_PROFILE_DIR`, `VERITHINK_MODEL_QUALITY_DIR` — point each
  store at a path you control (e.g. an encrypted volume, a per-tenant directory, or a scratch
  location you wipe between runs).

**Model-quality telemetry:**

- `VERITHINK_ROUTER_QUALITY_TELEMETRY` (default `true`) controls whether the auto-router
  **consumes** learned quality (`ModelRouter.cs`); it does **not** control recording. To stop
  **writing** `model-quality.json`, set `VERITHINK_MODEL_QUALITY_WRITE=0` (or the master
  `VERITHINK_LEARNING_ENABLED=0`): `ReasonPipeline.TryRecordModelQuality` skips the `Record` call
  when `PersistModelQuality` is false. (Its contents are non-sensitive: model name, domain, score,
  count.)

**Auto-apply of lessons:**

- `VERITHINK_AUTO_APPLY_LESSONS` (default `false`) and `VERITHINK_AUTO_APPLY_PREVIEW`
  (default `true`) control whether learned lessons are auto-promoted into profiles vs. written to
  a preview file. Leaving auto-apply off keeps learned signal from silently changing behavior.

**Memory retrieval:**

- `VERITHINK_MEMORY_MODE=keyword` (default) persists nothing beyond the experience store.
  Switching to `vector` with a SQLite/Postgres backend introduces a separate persisted index.

**Off-switches now exist (see the gate table above):** `VERITHINK_LEARNING_ENABLED=0` disables all
learning persistence at once, and the per-store flags (`VERITHINK_EXPERIENCE_WRITE`,
`VERITHINK_MODEL_QUALITY_WRITE`, `VERITHINK_USAGE_LOG`, `VERITHINK_FEEDBACK_WRITE`,
`VERITHINK_VERIFY_WRITE`) turn off individual stores. Directory relocation, short retention, and a
wipeable store backend remain available as additional layers.

---

## 5. Redaction & retention notes

- **Secret redaction** (`SecretRedactor.cs`) masks: `Authorization: Bearer <x>`,
  `api_key=/api-key: <x>` (≥6 chars), `sk-<x>` OpenAI-style keys, and
  `<NAME>_KEY|_TOKEN|_SECRET=<x>` env-style assignments. It is a **best-effort secret masker,
  not a PII/redaction engine** — it will not remove proprietary source, personal data, or
  business content.
- Redaction is applied to `RequestText` and `AnswerMarkdown` **only in the JSON experience
  store**. It is **not** applied to `GraphJson`, to the usage-ledger `requestPreview`, to
  feedback notes, or to the SQLite/Postgres experience stores.
- Text is Unicode-normalized on save/load (`TextEncodingHelper`), and `SanitizeForStorage`
  attempts mojibake recovery but does **not** truncate — the full request text is stored.
- Retention pruning is **startup-time** and file-mtime based; it does not run continuously.
- Corrupt records are skipped/quarantined by the read paths; they are not auto-deleted unless
  retention pruning removes them by age.

---

## 6. Recommendations for privacy-sensitive deployments

1. **Enable retention:** set `VERITHINK_EXPERIENCE_RETENTION_DAYS` to the shortest window that
   still gives useful learning (e.g. `7`–`30`). Remember `usage-log.jsonl` is a single file and is
   effectively not age-pruned — rotate or delete it on your own schedule.
2. **Relocate the data root:** point `VERITHINK_EXPERIENCE_DIR`, `VERITHINK_PROFILE_DIR`, and
   `VERITHINK_MODEL_QUALITY_DIR` at an access-controlled, preferably encrypted, volume. Use
   per-tenant/per-repo directories if you serve multiple parties.
3. **Prefer the JSON store for its redaction** if you must persist request/answer content — the
   SQLite/Postgres stores currently persist those fields **unredacted**. If you use SQLite/Postgres,
   secure the database and assume prompts/answers are stored in cleartext.
4. **Treat `GraphJson`, `usage-log.jsonl` previews, and feedback notes as unredacted.** Avoid
   pasting live secrets or regulated data into requests; the secret masker is not a guarantee.
5. **Keep auto-apply off** (`VERITHINK_AUTO_APPLY_LESSONS=0`, the default) so learned signal does
   not silently alter planning.
6. **For maximum isolation / ephemeral runs:** point the data directories at a scratch location and
   wipe it between sessions, or run against a database you truncate. For an in-process
   "learning disabled" mode, set `VERITHINK_LEARNING_ENABLED=0` (see §4); directory/DB lifecycle
   then only matters for previously written data.
7. **Verify at runtime:** hit `GET /healthz` and read the `paths` block to confirm exactly where
   data is being written before processing sensitive material.
8. **Clearing existing data:** delete the contents of the experience, profile, and model-quality
   directories (or drop the SQLite/Postgres tables) to reset all learned state; keep the directories
   themselves — the stores recreate them on next write.