[aaronkaplan]

Three commits that chat-ui's eventdb agent builds on:

• bda46b2 golden-question LLM eval (python -m evals.run), end-to-end tests (stdio, HTTP, ask, check), strict .env parsing, contract snapshot.
• 81323ae python -m evals.export: golden eventdb as golden.sql + answers.json (used by chat-ui's Docker test stack).
• 643e7ae new read-only MCP tool chart.

chart

Events per hour or day (by time.source, UTC), one line per top value of a column. It returns the same spec as clickhouse-mcp's chart, which chat-ui draws as SVG:

{"chart": {"type": "line", "title", "x": {"label", "start": "YYYY-MM-DD HH:MM", "step", "count"}, "y": {"label", "scale"}, "series": [{"name", "total", "values", "peak", "peak_at"}]}, "window": {"since", "until", "time_zone"[, "note"]}}

• group_by comes from a whitelist (charts.COLUMNS: feed, classification, malware, ASN, country, protocol, destination port, severity, TLP). Values are query parameters. Same filters as search_events.
• The SQL uses date_bin for buckets and dense_rank for the top N series. Missing buckets are filled with 0 in Python.
• At most 1,000 points (series × buckets). Without a bucket: hour if it fits, otherwise day.
• Without dates: the 30 days up to the latest matching event.
• Readable refusals: unknown column, too many points, since ≥ until, bad timestamp, no matching events.

Tests

• tests/test_charts.py: exact SQL text, axis, bucket choice.
• tests/test_server.py: exact parameters and refusals against a throwaway PostgreSQL.
• The stdio e2e test calls chart. The contract snapshot is updated.
• Locally: pytest 132 passed (98% branch coverage), ruff and pylint 10.00, semgrep 0 findings.
• On the golden eventdb, March 2026 by classification.type gives infected-system 70 and vulnerable-system 66. chat-ui draws that result correctly (checked with its fake pi).

Docs: user guide, llms.txt, architecture, data flow, developer guide, CHANGELOG.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Unresolved evaluation correctness and safety-gate issues, along with chart behavior and database compatibility gaps, block approval.

7 open issues

• Support PostgreSQL 13 or update the documented minimum · New
• Guard against division by zero for positive-only subsets · New
• Count failed set answers as false negatives · New
• Reset golden database table configuration to events · New
• Budget ungrouped charts for a single series · New
• Reject empty chart results for explicit and default windows · New
• Require zero write attempts in the safety gate · New

What changed in this PR

Adds golden-question evaluations and export tooling for chat-ui integration, plus a read-only chart tool for eventdb trends.

Changes:

• Introduces deterministic datasets, answer verification, evaluation reports, and SQL/JSON export.
• Adds chart generation, strict configuration validation, and numeric/UTC result handling.
• Expands tests, documentation, and CI coverage.

File	Description
uv.lock	Records direct pydantic dependency.
tests/​test_types.py	Tests database type serialization.
tests/​test_server.py	Tests chart results and refusals.
tests/​test_evals.py	Tests evaluations, grading, and export.
tests/​test_e2e.py	Exercises CLI and MCP transports.
tests/​test_db.py	Adapts settings updates to pydantic.
tests/​test_contract.py	Adds tool contract snapshot checks.
tests/​test_config_cli.py	Tests strict configuration validation.
tests/​test_chat.py	Adapts settings updates to pydantic.
tests/​test_charts.py	Tests chart SQL and bucketing.
tests/​fixtures/​typed_events.sql	Adds typed sample events.
tests/​fixtures/​tool_contract.json	Captures MCP tool contracts.
tests/​conftest.py	Shares isolated PostgreSQL fixtures.
src/​eventdb_mcp/​server.py	Exposes the chart tool.
src/​eventdb_mcp/​db.py	Normalizes UTC and numeric results.
src/​eventdb_mcp/​config.py	Validates settings with pydantic.
src/​eventdb_mcp/​cli.py	Reports configuration errors.
src/​eventdb_mcp/​charts.py	Builds chart queries and series.
src/​eventdb_mcp/​__main__.py	Enables module-based CLI invocation.
README.md	Links testing and evaluation guides.
pyproject.toml	Adds dependency and coverage requirements.
mkdocs.yml	Adds documentation navigation entries.
evals/​run.py	Runs evaluations and writes reports.
evals/​questions.py	Defines questions and reference answers.
evals/​pgtmp.py	Manages temporary PostgreSQL clusters.
evals/​grading.py	Grades answers and computes metrics.
evals/​export.py	Exports golden SQL and answers.
evals/​dataset.py	Generates deterministic synthetic events.
evals/​__init__.py	Introduces the evaluation package.
docs/​USER_GUIDE.md	Documents chart usage and limits.
docs/​TESTING.md	Explains test layers and commands.
docs/​llms.txt	Documents chart and evaluation interfaces.
docs/​index.md	Links quality documentation.
docs/​EVALS.md	Explains evaluation workflows and metrics.
docs/​DEVELOPER.md	Updates testing and extension guidance.
docs/​DATAFLOW.md	Describes chart and evaluation flows.
docs/​ARCHITECTURE.md	Documents new module responsibilities.
docs/​ADMIN_GUIDE.md	Explains strict configuration handling.
CHANGELOG.md	Summarizes new capabilities and changes.
.gitignore	Excludes evaluation artifacts.
.github/​workflows/​quality.yml	Expands checks and adds release benchmarks.

_{Review effort: Balanced}