Repository navigation
fix(logs): 保留明细过期后仍保留用量汇总 - #477
Merged
Merged
Conversation
request_logs is both the per-request detail log and the table every aggregate reads: the Usage page, the Cost panel and the traffic chart all GROUP BY it. The retention job deleted rows with a single age-based DELETE, so when a day fell out of CLI2API_LOG_RETENTION_DAYS its detail and its totals disappeared in the same statement, and nothing in the schema could reconstruct the totals. Raising the window only moved the cliff; the row cap then deleted the oldest rows first, so a busy deployment still lost history well inside the window. Retention now runs in two stages: - ExpireRequestLogDetail keeps the row and clears only what the detail view reads: the three child tables, plus error_code/error_message, message_roles, empty_message_indexes, requested/resolved_reasoning, routing and finished_at. Every aggregate column survives, so the numbers stay exact rather than approximated, and because the child rows were ~90% of the file's bytes the file still stops growing. - PurgeRequestLogs deletes the row once it is older than CLI2API_LOG_TOTALS_RETENTION_DAYS (default 730). The detail cap now counts only rows that still carry detail, so totals-only rows cannot be evicted by it. The Cost panel window follows the totals window, otherwise history that is still in the database stays invisible. A live deployment made this concrete: 19032 logs, of which the child tables held 49766 rows across 24.7 MB while every text column of request_logs summed to 2.6 MB. Agent: hermes Verify: docker golang:1.25 go test ./internal/... all ok
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题
#462 修掉了子表孤儿行,但没有修「保留期一到,用量历史整体消失」。这两件事的根因不同,前者是级联没生效,后者是保留逻辑本身会删掉唯一的数据副本。
request_logs在这个项目里是双重角色:既是逐请求明细,也是所有聚合查询读取的表——用量页/api/logs/stats、费用面板/api/usage/cost、流量图都GROUP BY request_logs。而保留任务只有一条按龄DELETE:所以某一天滑出
CLI2API_LOG_RETENTION_DAYS时,那天的明细和那天的汇总在同一条语句里一起消失,schema 里也没有任何汇总表可以重建。调大保留期只是把悬崖推远;CLI2API_LOG_MAX_ROWS又会按ORDER BY created_at ASC从最老的开始删,繁忙部署在窗口内就会丢数据。实测(线上库副本)
request_logs行数request_logs所有文本列合计这个比例决定了改法:明细是体积,汇总不是。删明细能回收空间,删汇总只是丢信息。
改动
保留拆成两段,不再有「整行按龄删除」这一步。
第一段
ExpireRequestLogDetail(新增):超过CLI2API_LOG_RETENTION_DAYS的行,保留全部聚合列,只清明细:request_attempts、request_usage_details、request_stream_diagnostics三张子表里对应的行error_code、error_message、message_roles、empty_message_indexes、requested_reasoning、resolved_reasoning、routing、finished_atcreated_at、status、stream、requested_model、mapped_model、account_id、api_key_id、provider、prompt_tokens、completion_tokens、cache_read_tokens、cache_write_tokens、credits、latency_ms、ttfb_ms、error_kind、attempt_counterror_kind特意留下:错误分布图读它,删了会让那天的错误分类凭空消失。credits留下还顺带解决了原先依赖request_usage_details回退取积分的问题——回退行被删之后,汇总数字仍然来自主表。第二段
PurgeRequestLogs:超过CLI2API_LOG_TOTALS_RETENTION_DAYS(默认 730 天)才整行删除。配套三处:
CLI2API_LOG_MAX_ROWS改为只统计detail_expired = 0的行。只剩汇总的行不计入,否则行数上限会继续按最老优先删,把刚保住的汇总又删掉。exportWindowHours()跟随TotalsRetentionDays()而不是明细保留期,否则数据库里还有历史,费用面板仍然只给 7 天。RequestLog增加detail_expired字段,GET /api/logs/requests与/api/logs/requests/<id>都会带出,控制台可以区分「本来没记录 attempt」和「明细已被保留策略清理」。迁移
026_request_log_detail_expiry.sql只加一列和一个索引,不重写既有行,schema_migrations的 checksum 表已同步。测试
internal/store/request_log_detail_expiry_test.go:TestExpireRequestLogDetailKeepsAggregatesExact— 造一条 30 天前、一条 1 小时前的日志,各带 attempt / usage detail / stream diagnostic。执行过期后断言:子表只剩在窗口内那条的明细;过期行被标记detail_expired且明细列已清空、error_kind仍在;SummarizeRequestLogs、SummarizeModelCosts、SummarizeRequestUsage的 requests / tokens / credits / 延迟 / 模型排行 / 错误分布逐项不变。这条是本次改动的核心断言。TestExpireRequestLogDetailIsIdempotent— 连续执行两次,第二次影响 0 行,不会重复清理。TestPurgeRequestLogsCapIgnoresTotalsOnlyRows— 3 条只剩汇总 + 2 条带明细,maxRows=2时删除 0 行;再用 30 天窗口才删掉那 3 条。另有
TestTotalsRetentionDaysDefaultsAndOverride(含「汇总窗口不得短于明细窗口」的回退)与改写后的TestExportWindowHoursFollowsTotalsRetention。docker run golang:1.25 go test ./internal/...全绿。兼容性
CLI2API_LOG_RETENTION_DAYS语义不变(仍管明细)。新增CLI2API_LOG_TOTALS_RETENTION_DAYS可选。本 PR 由 AI 辅助生成。