Skip to content

fix(memory): never overwrite an archived original with a same-named backup - #2320

Open
thejesh23 wants to merge 1 commit into
HKUDS:mainfrom
thejesh23:fix/memory-archive-no-overwrite
Open

thejesh23 wants to merge 1 commit into
HKUDS:mainfrom
thejesh23:fix/memory-archive-no-overwrite

Conversation

@thejesh23

Copy link
Copy Markdown

Summary

  • Memory GC and the compression pass both back up an entry to archive/<basename>, overwriting whatever backup already has that name. The archive is documented as archive-only / never deleted, but two cases lose an original for good:
    • with VT_MEMORY_HIERARCHY, user/x.md and project/x.md are different entries with the same basename, so archiving the second renames it over the first;
    • compressing the same entry twice (raw → daily → digest) replaces the raw original's backup with the daily summary.

Why

# agent/src/memory/lifecycle.py  _execute_gc_action (main)
dest = archive_dir / entry.path.name
entry.path.rename(dest)                 # silently replaces an existing archive/<name>

# agent/src/memory/compression.py  archive_original (main)
archive_path = self._archive_dir / entry_path.name
shutil.copy2(str(entry_path), str(tmp_path))
os.replace(str(tmp_path), str(archive_path))   # same

Two GC-eligible entries with the same title, one user, one project, hierarchy on:

live files: ['project/position_sizing.md', 'user/position_sizing.md']
gc actions: [('Position sizing', 'archive'), ('Position sizing', 'archive')]
main:   archive: [('position_sizing.md', 'PROJECT rule: Kelly/4 for crypto book')]
branch: archive: [('position_sizing.1.md', 'PROJECT rule: Kelly/4 for crypto book'),
                  ('position_sizing.md', 'USER pref: never exceed 2% risk')]

On main the user preference is gone from both the live tree and the archive. consolidation.py and the memory.consolidate scheduled job promise the opposite ("low-importance memories are moved to <memory_dir>/archive/, never deleted").

Changes

  • agent/src/memory/compression.py: new free_archive_path(archive_dir, name). The first backup keeps name, so the layout is unchanged when there's no clash. Later ones get <stem>.1<suffix>, <stem>.2<suffix>, … archive_original uses it, and its error-path temp cleanup now removes the temp file it actually created.
  • agent/src/memory/lifecycle.py: _execute_gc_action uses the same helper for both the archive and the (currently disabled) delete branch.
  • Tests: two in agent/tests/test_memory_gc.py (hierarchy basename clash; GC after a compression backup of the same entry) and two in agent/tests/memory/test_compression.py (raw backup kept through daily → digest; two categories' entries with one basename).

archive/ behaviour, main vs this branch:

case main this PR
first backup of x.md archive/x.md archive/x.md (unchanged)
second entry with basename x.md (other category) overwrites archive/x.md archive/x.1.md
same entry compressed again raw backup replaced by the daily text raw kept in x.md, daily in x.1.md
GC of an entry whose compression backup exists backup replaced both kept

Out of scope: archive restore tooling (there's none in-tree; nothing reads archive/ back), and moving to a per-category archive layout.

Test Plan

  • All 4 new tests fail on main (e.g. an archived original was overwritten: ['sizing.md']) and pass with the fix.
  • pytest agent/tests/test_memory_gc.py agent/tests/memory: 127 passed, 1 skipped.
  • Full suite: pytest --ignore=agent/tests/e2e_backtest --ignore=agent/tests/test_e2e_harness_v2.py on Python 3.12, macOS, lockfile deps: 22658 passed, 197 skipped, 30 failed. main in the same environment: 22654 passed, 197 skipped, and the same 30 failures (identical test IDs): 25 in test_history_source.py, 3 needing the optional statsmodels extra, and 2 needing the package installed (test_cli_package_bootstrap, test_governance::test_collect_key_package_versions_reflects_real_environment).
  • bash tools/ci_grep_gates.sh, the gate tests, the CI syntax checks, ruff check on the changed files (2 pre-existing unused imports in test_compression.py are on main and untouched), git diff --check. New lines are Black-formatted with --line-ranges; I didn't reformat existing code.

Checklist

  • No changes to protected areas (src/agent/, src/session/, src/providers/)
  • No hardcoded values
  • Code follows CONTRIBUTING.md (DCO sign-off on the commit)
  • Documentation updated (if user-facing change): n/a, the documented "never deleted" contract is now what happens

Risk: file naming inside <memory_dir>/archive/ only, and only when a name is already taken; live memory files, the index and FTS are untouched. Rollback is a single-commit revert.

…ackup

Memory GC and the compression pass both archived to archive/<basename>.
With VT_MEMORY_HIERARCHY, user/x.md and project/x.md share a basename, so
archiving the second renamed it over the first; compressing an entry a
second time (daily -> digest) replaced the raw original's backup with the
daily summary. The archive is documented as archive-only, never deleted.

Pick a free archive name instead: the first backup keeps <name>, later
ones get <stem>.1<suffix>, <stem>.2<suffix>, ...

Signed-off-by: thejesh23 <35212698+thejesh23@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant