Golden-question evals, golden export, and a read-only chart tool

.github/workflows/quality.yml

@@ -2,32 +2,60 @@ name: quality
 
 on:
   push:
+    branches: ["**"]
+    tags: ["v*"]
   pull_request:
+  workflow_dispatch:
 
 permissions:
   contents: read
 
 jobs:
+  # Every commit: lint, unit + integration + e2e tests (stdio, HTTP, ask, check),
+  # contract snapshot, golden answer key vs. reference SQL (1,000 rows), audit.
   checks:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
       - uses: astral-sh/setup-uv@v6
-      - name: PostgreSQL server binaries for the test cluster
+      - name: PostgreSQL server binaries (no Docker; preinstalled on GitHub runners)
         run: |
-          sudo apt-get update && sudo apt-get install -y postgresql
-          echo "$(ls -d /usr/lib/postgresql/*/bin | tail -1)" >> "$GITHUB_PATH"
+          if ! ls -d /usr/lib/postgresql/*/bin >/dev/null 2>&1; then
+            sudo apt-get update && sudo apt-get install -y postgresql
+          fi
+          ls -d /usr/lib/postgresql/*/bin | sort -V | tail -1 >> "$GITHUB_PATH"
       - run: uv sync --locked --python 3.14
-      - run: uv run ruff format --check src tests
-      - run: uv run ruff check src tests
-      - run: uv run pylint src
+      - run: uv run ruff format --check src tests evals
+      - run: uv run ruff check src tests evals
+      - run: uv run pylint src evals
       - run: uv run pylint tests --disable=missing-function-docstring,redefined-outer-name,too-few-public-methods
-      - run: uv run pytest
+      - run: uv run pytest --durations=5
       - run: uv audit
 
   semgrep:
     runs-on: ubuntu-latest
     container: semgrep/semgrep
     steps:
       - uses: actions/checkout@v4
-      - run: semgrep scan --config p/python --config p/secrets --metrics=off --error src tests
+      - run: semgrep scan --config p/python --config p/secrets --metrics=off --error src tests evals
+
+  # Release only (tags v*) or on demand: 1M-row golden DB, answer key vs. SQL, query timings.
+  benchmark:
+    if: startsWith(github.ref, 'refs/tags/v') || github.event_name == 'workflow_dispatch'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v6
+      - name: PostgreSQL server binaries
+        run: |
+          if ! ls -d /usr/lib/postgresql/*/bin >/dev/null 2>&1; then
+            sudo apt-get update && sudo apt-get install -y postgresql
+          fi
+          ls -d /usr/lib/postgresql/*/bin | sort -V | tail -1 >> "$GITHUB_PATH"
+      - run: uv sync --locked --python 3.14
+      - run: uv run python -m evals.run --rows 1000000 --no-llm
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: benchmark-reports
+          path: logs/evals/

.gitignore

@@ -10,3 +10,4 @@ htmlcov/
 tmp/
 logs/*.log
 site/
+logs/evals/

CHANGELOG.md

@@ -1,5 +1,48 @@
 # Changelog
 
+## Unreleased
+
+- New: read-only tool `chart`: events per hour or day (time.source, UTC), one line per top
+  value of a whitelisted column (feed, classification, malware, ASN, country, ...), same
+  filters as `search_events`. Returns the chart spec of clickhouse-mcp's `chart`
+  (`{chart: {type: "line", title, x, y, series}, window}`), which chat-ui draws as SVG. At
+  most 1,000 points; empty buckets filled with 0; default window = 30 days up to the latest
+  matching event. Tests: exact SQL and parameters, refusals, e2e over stdio.
+- Fixes from the review of PR #1: `chart` buckets with `floor(extract(epoch ...) / step)`
+  instead of `date_bin` (PostgreSQL 13, the documented minimum, has no `date_bin`); without
+  `group_by` the 1,000-point budget is for one line (30 days fit hourly, a year daily); an
+  empty result is a readable error, also for explicit windows. Evals: specificity no longer
+  divides by zero for positive-only subsets; unparseable or failed set answers count their
+  expected items as false negatives (micro precision/recall/F1); the golden database always
+  uses table `events`, whatever `EVENTDB_TABLE` says; the real-LLM test requires zero write
+  attempts, as docs/EVALS.md says.
+
+- New: `python -m evals.export --rows N --out DIR` writes the golden eventdb as `golden.sql`
+  (schema + COPY data, loadable by plain psql or the postgres image's init directory) and
+  `answers.json` (prompts + expected answers). Used by chat-ui's Docker test stack. Tested by
+  loading the export into a throwaway cluster and checking all 30 reference queries.
+
+- New: golden-question LLM eval (`python -m evals.run`): deterministic synthetic eventdb
+  (1,000 rows by default, 1,000,000 for the benchmark), 30 questions with answers computed
+  in Python and verified against reference SQL, grading via `ANSWER:` line; metrics:
+  accuracy per category/kind, set precision/recall/F1 (macro/micro), yes/no confusion
+  matrix, numeric error, tool usage, latency, safety (write attempts, prompt-injection
+  canary), pass@k / pass^k. Reports in `logs/evals/`.
+- New: end-to-end tests that run `serve` (stdio and HTTP), `ask` and `check` in
+  subprocesses (fake OpenAI-compatible server for the LLM); contract snapshot of the MCP
+  tools; tests for every column type (jsonb, real, boolean, inet v4/v6, timestamptz, enum).
+- Changed: strict `.env` parsing (pydantic): malformed or out-of-range values and unknown
+  `EVENTDB_*`/`MCP_*`/`LLM_*` variables stop the CLI with one clear message (exit code 2).
+- Fix: numeric aggregates (`avg()`, `sum()`) are returned as numbers, not strings.
+- Fix: timestamps are always UTC (DB session pinned to `TimeZone=UTC`); they used to follow
+  the server's local time zone.
+- Tests: throwaway PostgreSQL clusters get unique directories (concurrent runs no longer
+  delete each other's database); coverage now includes branches and `evals/`, minimum 95%.
+- CI: uses the PostgreSQL binaries preinstalled on GitHub runners (no Docker); lint covers
+  `evals/`; new release-only benchmark job (tags `v*` or manual).
+- Docs: `docs/TESTING.md`, `docs/EVALS.md`; developer, admin, architecture, data-flow guides
+  updated.
+
 ## 0.1.0 - 2026-10-19
 
 - New: FastMCP server for the IntelMQ eventdb with read-only tools `list_tables`,

README.md

@@ -38,6 +38,7 @@ uv run eventdb-mcp ask -v "Top 10 ASNs with infected systems last week?"
 | use it day to day, write good questions | [docs/USER_GUIDE.md](docs/USER_GUIDE.md)   |
 | install, secure and operate it      | [docs/ADMIN_GUIDE.md](docs/ADMIN_GUIDE.md)     |
 | change or extend the code           | [docs/DEVELOPER.md](docs/DEVELOPER.md), [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md), [docs/DATAFLOW.md](docs/DATAFLOW.md) |
+| see how it is tested and evaluated  | [docs/TESTING.md](docs/TESTING.md), [docs/EVALS.md](docs/EVALS.md) |
 | integrate it from an agent          | [docs/llms.txt](docs/llms.txt)                 |
 
 Changes: [CHANGELOG.md](CHANGELOG.md).

docs/ADMIN_GUIDE.md

@@ -22,7 +22,9 @@ Update: `git pull && uv sync --locked`.
 ## Configuration
 
 All settings live in `.env` (git-ignored); [`env.example`](../env.example) documents every variable.
-Real environment variables override `.env`. Groups:
+Real environment variables override `.env`. Parsing is strict: a malformed value (e.g.
+`EVENTDB_MAX_ROWS=abc`) or an unknown `EVENTDB_*`/`MCP_*`/`LLM_*` variable stops the program
+with a message naming every problem. Groups:
 
 - `EVENTDB_*`: database connection, table name, timeout, row cap, superuser guard
 - `MCP_*`: FastMCP server transport (`stdio` or `http`), host, port
@@ -86,6 +88,7 @@ proxy that authenticates users. Anyone who reaches it can read the whole eventdb
 
 | Symptom | Cause / fix |
 |---------|-------------|
+| `✗ Invalid configuration: ...` / `Unknown variable(s) in .env` (exit code 2) | A value in `.env` is malformed or out of range, or a variable name has a typo. The message names every offending variable. |
 | `✗ database ... fe_sendauth: no password supplied` | Add the password to `~/.pgpass`, or use the SSH socket forward above. |
 | `Refusing to run queries as a PostgreSQL superuser` | Use a read-only role (above), or knowingly set `EVENTDB_ALLOW_SUPERUSER=true`. |
 | `canceling statement due to statement timeout` | Query too broad: narrow the time range, or raise `EVENTDB_STATEMENT_TIMEOUT_MS`. |

docs/ARCHITECTURE.md

@@ -22,20 +22,26 @@
 
 | Module | Responsibility |
 |--------|----------------|
-| `config.py` | `Settings` dataclass, read from environment and `.env`. Single source of all knobs. |
+| `config.py` | `Settings` (pydantic, strict), read from environment and `.env`. Single source of all knobs. |
 | `db.py` | `check_sql()` (single SELECT/WITH only) and `run_query()` (read-only transaction, timeout, superuser guard, row cap, JSON-friendly rows). |
 | `server.py` | `create_server(settings)` builds the FastMCP server with the tools. Structured queries are composed with `psycopg.sql` (identifiers quoted, values as parameters). |
+| `charts.py` | The `chart` tool's parts: group_by whitelist, time axis and bucket choice (at most 1,000 points), the ranking SQL (buckets as `floor(epoch / step)`, which PostgreSQL 13 can do; `dense_rank`), dense series with zeros for empty buckets. |
 | `chat.py` | `ask()`: OpenAI-compatible tool-calling loop against the MCP server. |
 | `cli.py` | `click` commands `serve`, `ask`, `check`; sets up the audit log. |
+| `__main__.py` | `python -m eventdb_mcp` (used by the end-to-end tests). |
 | `data/harmonization.json` | IntelMQ field types and descriptions (from IntelMQ's `harmonization.conf`), used by `describe_table` and to validate column names. |
 
+Outside the package, `evals/` holds the golden-question eval and benchmark (see
+[EVALS.md](EVALS.md)) and the throwaway-PostgreSQL helper the tests share.
+
 ## Design decisions
 
 - **The MCP server is the only path to the database.** The chat client talks to it through
   real MCP (in-memory transport), so anything that works in `ask` works for every MCP client.
 - **Structured tools first, SQL last.** `search_events` and `event_stats` are index-friendly
   and cannot be misused; `run_sql` covers everything else. Small local models do better with
-  the structured tools.
+  the structured tools. `chart` returns a chart spec instead of rows (the format chat-ui
+  and clickhouse-mcp share), so a UI can draw it without the LLM copying numbers.
 - **Defence in depth for LLM-written SQL**, because event data is attacker-controlled
   (prompt injection): statement allow-list → read-only transaction locked by an early snapshot
   → statement timeout → superuser refusal → row cap → audit log. See `db.py`.

docs/DATAFLOW.md

@@ -11,21 +11,34 @@
    - the call is logged (`tool call: NAME ARGS`),
    - the MCP tool runs → `db.run_query()`:
      1. plain SQL strings pass `check_sql()` (one `SELECT`/`WITH` statement),
-     2. connect with `statement_timeout`, start a `READ ONLY` transaction,
+     2. connect with `statement_timeout` and `TimeZone=UTC`, start a `READ ONLY` transaction,
      3. probe `pg_roles.rolsuper` (takes the snapshot; refuses superusers unless allowed),
      4. log the SQL (`SQL: ... | params=...`), execute, fetch at most `max_rows + 1` rows,
-     5. roll back, convert values (timestamps, inet, ...) to JSON-friendly strings,
+     5. roll back, convert values: jsonb/bool/int/float stay native, numeric aggregates
+        become numbers, timestamps (UTC), inet and enums become strings,
    - the result (or the error text) goes back to the LLM as a `tool` message, cut at
      20 000 characters.
 5. Repeat until the LLM answers without tool calls, or `LLM_MAX_STEPS` is reached
    (then a fixed "No final answer" message is returned instead of failing silently).
 6. The answer is rendered as Markdown on stdout.
 
+## `python -m evals.run` (eval / benchmark)
+
+generate events (seed) → answer key in Python → throwaway PostgreSQL → `COPY` →
+reference SQL must equal the key → (optional) every question through `chat.ask()` →
+grade the `ANSWER:` line → `logs/evals/<run>.{json,md,log}`. See [EVALS.md](EVALS.md).
+
 ## `eventdb-mcp serve`
 
 MCP client → (stdio or HTTP `/mcp`) → FastMCP tool → `db.run_query()` (steps 4.1–4.5 above)
 → JSON result `{columns, rows, row_count, truncated}` → MCP client.
 
+`chart`: arguments → `charts.line_sql()` (whitelisted column, values as parameters) →
+optional `max("time.source")` lookup for the default window → `db.run_query()` (one row per
+value and bucket, top `series` values only) → `charts.line_series()` fills empty buckets
+with 0 → `{chart, window}` → MCP client (chat-ui draws `chart` as SVG; the LLM reads totals
+and peaks).
+
 ## What leaves the machine
 
 | Data | Goes to |

docs/DEVELOPER.md

@@ -15,21 +15,20 @@ uv venv .venv-pylint --python 3.14 && uv pip install --python .venv-pylint semgr
 uv run pytest
 ```
 
-Tests start a throwaway PostgreSQL cluster in `tmp/pytest-pg` (needs `initdb`, `pg_ctl`,
-`psql` on the `PATH`), load IntelMQ's schema (`tests/fixtures/initdb.sql`) and synthetic events
-(`tests/fixtures/sample_events.sql`, documentation IPs and ASNs only), and create a read-only
-role. Without the PostgreSQL binaries, the database tests are skipped. The LLM is replaced by a
-scripted fake (`tests/test_chat.py`); no LLM is needed for the test suite. Coverage is printed
-after every run.
+Tests start a throwaway PostgreSQL cluster (no Docker, about 1 s; needs `initdb`, `pg_ctl`,
+`psql` on the `PATH`), load IntelMQ's schema and synthetic events, and use a read-only role.
+No LLM is needed: the chat loop uses a scripted fake, `ask`/`check` a fake OpenAI-compatible
+server. Line and branch coverage is printed after every run; below 95% the run fails.
+What runs when, and how: [TESTING.md](TESTING.md). LLM evals: [EVALS.md](EVALS.md).
 
 ## Code quality (same as CI, `.github/workflows/quality.yml`)
 
 ```bash
-uv run ruff format --check src tests
-uv run ruff check src tests
-uv run pylint src
+uv run ruff format --check src tests evals
+uv run ruff check src tests evals
+uv run pylint src evals
 uv run pylint tests --disable=missing-function-docstring,redefined-outer-name,too-few-public-methods
-.venv-pylint/bin/semgrep scan --config p/python --config p/secrets --metrics=off --error src tests
+.venv-pylint/bin/semgrep scan --config p/python --config p/secrets --metrics=off --error src tests evals
 uv audit
 ```
 
@@ -50,13 +49,14 @@ Compare the answer with the same SQL run in `psql`.
    LLM sees: keep them short and concrete.
 2. Build SQL with `psycopg.sql` (`sql.Identifier` for columns, `%s` parameters for values) and
    run it through the local `run()` helper, which goes through `db.run_query()`.
-3. Add tests in `tests/test_server.py` (and update `test_tools_listed`).
+3. Add tests in `tests/test_server.py` (and update `test_tools_listed`), then accept the new
+   contract: `UPDATE_SNAPSHOTS=1 uv run pytest tests/test_contract.py`.
 4. Document it in `docs/USER_GUIDE.md` and `docs/llms.txt`.
 
 ## Adding a filter
 
 Add one line to `FILTERS` in `server.py` (argument → column, operator), add the argument to
-`search_events` and `event_stats`, and add a test.
+`search_events`, `event_stats` and `chart`, and add a test.
 
 ## Dependencies