Skip to content

fix(logs): 保留明细过期后仍保留用量汇总 - #477

Merged
caigee-cmd merged 1 commit into
caigee-cmd:mainfrom
NaimioChan:fix/log-retention-rollups
Nov 8, 2026
Merged

caigee-cmd merged 1 commit into
caigee-cmd:mainfrom
NaimioChan:fix/log-retention-rollups

Conversation

@NaimioChan

Copy link
Copy Markdown
Contributor

问题

#462 修掉了子表孤儿行,但没有修「保留期一到,用量历史整体消失」。这两件事的根因不同,前者是级联没生效,后者是保留逻辑本身会删掉唯一的数据副本。

request_logs 在这个项目里是双重角色:既是逐请求明细,也是所有聚合查询读取的表——用量页 /api/logs/stats、费用面板 /api/usage/cost、流量图都 GROUP BY request_logs。而保留任务只有一条按龄 DELETE:

// internal/logs/recorder.go(改动前)
keep := time.Duration(RetentionDays()) * 24 * time.Hour
r.store.PurgeRequestLogs(ctx, keep, MaxLogRows())

所以某一天滑出 CLI2API_LOG_RETENTION_DAYS 时,那天的明细和那天的汇总在同一条语句里一起消失,schema 里也没有任何汇总表可以重建。调大保留期只是把悬崖推远;CLI2API_LOG_MAX_ROWS 又会按 ORDER BY created_at ASC 从最老的开始删,繁忙部署在窗口内就会丢数据。

实测(线上库副本)

检查项 结果
request_logs 行数 19032
三张子表行数合计 49766
子表占用 / 库文件 24.7 MB 中约九成
request_logs 所有文本列合计 2.6 MB
可恢复的最早记录 仅限备份快照覆盖的窗口

这个比例决定了改法:明细是体积,汇总不是。删明细能回收空间,删汇总只是丢信息。

改动

保留拆成两段,不再有「整行按龄删除」这一步。

第一段 ExpireRequestLogDetail(新增):超过 CLI2API_LOG_RETENTION_DAYS 的行,保留全部聚合列,只清明细:

  • 删除 request_attempts、request_usage_details、request_stream_diagnostics 三张子表里对应的行
  • 清空 error_code、error_message、message_roles、empty_message_indexes、requested_reasoning、resolved_reasoning、routing、finished_at
  • 保留 created_at、status、stream、requested_model、mapped_model、account_id、api_key_id、provider、prompt_tokens、completion_tokens、cache_read_tokens、cache_write_tokens、credits、latency_ms、ttfb_ms、error_kind、attempt_count

error_kind 特意留下:错误分布图读它,删了会让那天的错误分类凭空消失。credits 留下还顺带解决了原先依赖 request_usage_details 回退取积分的问题——回退行被删之后,汇总数字仍然来自主表。

第二段 PurgeRequestLogs:超过 CLI2API_LOG_TOTALS_RETENTION_DAYS(默认 730 天)才整行删除。

配套三处:

  1. CLI2API_LOG_MAX_ROWS 改为只统计 detail_expired = 0 的行。只剩汇总的行不计入,否则行数上限会继续按最老优先删,把刚保住的汇总又删掉。
  2. exportWindowHours() 跟随 TotalsRetentionDays() 而不是明细保留期,否则数据库里还有历史,费用面板仍然只给 7 天。
  3. RequestLog 增加 detail_expired 字段,GET /api/logs/requests 与 /api/logs/requests/<id> 都会带出,控制台可以区分「本来没记录 attempt」和「明细已被保留策略清理」。

迁移 026_request_log_detail_expiry.sql 只加一列和一个索引,不重写既有行,schema_migrations 的 checksum 表已同步。

测试

internal/store/request_log_detail_expiry_test.go:

  • TestExpireRequestLogDetailKeepsAggregatesExact — 造一条 30 天前、一条 1 小时前的日志,各带 attempt / usage detail / stream diagnostic。执行过期后断言:子表只剩在窗口内那条的明细;过期行被标记 detail_expired 且明细列已清空、error_kind 仍在;SummarizeRequestLogs、SummarizeModelCosts、SummarizeRequestUsage 的 requests / tokens / credits / 延迟 / 模型排行 / 错误分布逐项不变。这条是本次改动的核心断言。
  • TestExpireRequestLogDetailIsIdempotent — 连续执行两次,第二次影响 0 行,不会重复清理。
  • TestPurgeRequestLogsCapIgnoresTotalsOnlyRows — 3 条只剩汇总 + 2 条带明细,maxRows=2 时删除 0 行;再用 30 天窗口才删掉那 3 条。

另有 TestTotalsRetentionDaysDefaultsAndOverride(含「汇总窗口不得短于明细窗口」的回退)与改写后的 TestExportWindowHoursFollowsTotalsRetention。

docker run golang:1.25 go test ./internal/... 全绿。

兼容性

  • 默认行为对普通部署更安全(7 天明细 + 730 天汇总),但磁盘占用会比原先持续增长得慢而非更快:明细仍按 7 天清,只是行壳留下。
  • CLI2API_LOG_RETENTION_DAYS 语义不变(仍管明细)。新增 CLI2API_LOG_TOTALS_RETENTION_DAYS 可选。
  • 已存在的历史孤儿行不会自动重建,需要部署方自行从备份回填——这部分是 fix(store): make request-log purge drop child rows for real #462 review 里已经确认「先不动」的。

本 PR 由 AI 辅助生成。

request_logs is both the per-request detail log and the table every aggregate
reads: the Usage page, the Cost panel and the traffic chart all GROUP BY it.
The retention job deleted rows with a single age-based DELETE, so when a day
fell out of CLI2API_LOG_RETENTION_DAYS its detail and its totals disappeared in
the same statement, and nothing in the schema could reconstruct the totals.
Raising the window only moved the cliff; the row cap then deleted the oldest
rows first, so a busy deployment still lost history well inside the window.

Retention now runs in two stages:

- ExpireRequestLogDetail keeps the row and clears only what the detail view
  reads: the three child tables, plus error_code/error_message, message_roles,
  empty_message_indexes, requested/resolved_reasoning, routing and finished_at.
  Every aggregate column survives, so the numbers stay exact rather than
  approximated, and because the child rows were ~90% of the file's bytes the
  file still stops growing.
- PurgeRequestLogs deletes the row once it is older than
  CLI2API_LOG_TOTALS_RETENTION_DAYS (default 730).

The detail cap now counts only rows that still carry detail, so totals-only
rows cannot be evicted by it. The Cost panel window follows the totals window,
otherwise history that is still in the database stays invisible.

A live deployment made this concrete: 19032 logs, of which the child tables
held 49766 rows across 24.7 MB while every text column of request_logs summed
to 2.6 MB.

Agent: hermes
Verify: docker golang:1.25 go test ./internal/... all ok
@caigee-cmd
caigee-cmd merged commit 3e821be into caigee-cmd:main Nov 8, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants