Skip to content

feat: catch-up, digest, change highlights, fuzzy re-anchoring, citations and promote (collaboration M10b) - #158

Closed
maxsam4 wants to merge 16 commits into
0xPolygon:stack/collab-m6-pagesfrom
maxsam4:collab-m10b-catchup
Closed

maxsam4 wants to merge 16 commits into
0xPolygon:stack/collab-m6-pagesfrom
maxsam4:collab-m10b-catchup

Conversation

@maxsam4

@maxsam4 maxsam4 commented Nov 19, 2026

Copy link
Copy Markdown

Stacked on #154 (M5, suggestions) → #152 (M4) → #151 (M3) → #149 (M2c) → #148 (M2b) → #146 (M2a). It also includes, merged in:

Review only the M10b commits: everything after the three merge commits (ecb9661 M10a, 0e6fcaa M5's final head, 65c159b M6).

This is M10b of the collaboration plan (docs/design/collaboration-plan.md in #144).

Summary

Awareness of change: what happened while you were away, where it happened, and where claims come from.

  • Catch-up ("since you last looked"). A thin view over what Primer already keeps; no new store.
    • Edits come from the update log (doc_updates: who saved, which blocks; it survives compaction), shown as the blocks read now through getLiveDoc({ since }) and deletedIds.
    • Everything else comes from Primer's own lifecycle events (namespace primer, audience readers), joined to their rows: comment.created, thread.resolved, M10a's decision.made / decision.cleared and task.*, suggestion.created / suggestion.decided, version.published.
    • It is grouped per person (edits, comments, resolved threads, decisions, suggestions, tasks). The caller's own actions, and those of the person an agent acts for, are left out.
    • document_views.last_doc_version: opening a document records the visit and returns the previous one as lastView, read in the same statement, so a refetch never loses it. GET /api/documents/:docId/catchup?since=&sinceVersion=.
    • The web shows a dismissible "Since you last looked" panel and highlights the changed blocks.
  • Digest. GET /api/me/digest?since= and MCP primer_get_digest return the catch-up of every document in the caller's library that changed since (at most 50).
    • An agent gets its person's library, as primer_list_documents and tasks treat agents.
    • Every candidate passes the read gate again, so a document the caller lost access to, or that was deleted, never appears.
    • AidenHQ pushes it to Slack or Telegram; Primer sends nothing there.
  • Email digest. POST /api/me/digest/email { since } emails a person their digest on request (there is no scheduler and no "last sent" state). The digest is laid out as blocks and rendered by exportHtml, the static renderer that shares and print use. People's text goes in as plain text. Notifier.notifyDigest addresses and sends it; agents have no mailbox.
  • Highlights since the last checkpoint. A "Changes since checkpoint" toggle. Checkpoints in the version list now carry their docVersion; the web reads live?since= with it.
    • Highlights are a stylesheet keyed on BlockNote's data-id, never a document write. Only plain ids reach it, because ids come from the client-writable Y.Doc.
  • Comment marks are not edits. Persistence looks at what a transaction did. One that only adds or removes comment-mark formatting touches no block, whoever made it: the server anchoring a thread, or BlockNote in an open editor flagging a resolved thread. The browser check found the second case. Delta reads benefit too.
  • Fuzzy re-anchoring. findMarkedQuote / findQuote gain an opt-in fuzzy stage, used only by reanchorThread (primer_reanchor_thread, and a new Re-anchor button on orphaned threads).
    • Matching is exact first, then the best run of whole words with similarity ≥ 0.8 in the target block, or in the whole tab once that block is gone.
    • It uses the one bounded similarity from lib/similarity.ts (banded, threshold-cut), with a 2,000-character quote cap and a 20M character-pair budget.
    • A second, separate window within 0.05 of the best is ambiguous and refused, never guessed.
    • Target and quote default to the thread's own. The anchor records the text actually marked and its block.
    • Creating a thread stays exact, nothing re-anchors automatically, and legacy threadCarry.ts is untouched.
  • Evidence provenance. A citation inline chip { url?, docId?, versionNumber?, quote }, numbered in reading order by a CSS counter (editor, static pages, email).
    • Agents add citations through json block input. applyOps pins a Primer citation to the cited document's latest version and refuses a document the agent can't read.
    • GET /api/documents/:docId/citations lists them and names a cited document only after a read check.
  • Promote a conversation. primer_promote_conversation and POST /api/documents/from-conversation create one quote block per message, the author in bold above the text.
    • Without documentId it goes through the one publish path as a new private document. With documentId it goes through the ops pipeline at the end of a live document, so it is a suggestion on someone else's document.
    • "Promote to document" on a comment thread (POST /api/threads/:threadId/promote) does the same for anyone who can read the thread.
  • Plan-first (M10b.6) shipped with M4's runs (per-run counters, approval_required, approval on the document queue). This PR adds the test M4 lacked: small batches add up to the threshold.
  • M10a follow-up. This is the first tree with both M4's generic ops and M10a's choices. An agent's block input that writes decisionId, a decided* render slot or proposedBy is refused with invalid_op. Picks stay with people, on the pick route.
  • Page text (with M6).
    • Text edited inside an HTML page lives in page/<blockId>/<pid> fragments, which the update log records by name. Delta reads, catch-up and highlights map a sub-fragment (page text, embed or widget label) to its block, so a page whose text changed reads as a changed block, not as nothing or as a deleted id.
    • Fuzzy re-anchoring works inside page text: a sub-fragment is searched whole.
  • UI follow-up from M5. A republish that only suggests makes no version (AddVersionResponse with only reconciliation). The "Add version" dialog now stays open, says how many blocks were suggested, conflicted or unchanged, and offers to open the document, instead of navigating as if a version had been added.
  • Merges. M10a is merged into M5 with both sides kept. Its migrations were regenerated after M5's: one 0022, plus this PR's 0023 (document_views.last_doc_version). The working status M4 replaced with awareness stays removed; M10a's side still carried primer_set_status and the banner. M3–M5's web strings moved to i18n, so the M10d lint passes on the combined tree.

Security

  • Client-writable ids. Every lookup keyed by an id read from the Y.Doc or from an event payload is validated as a UUID and scoped by documentId: catch-up's comment, thread and task joins, and citations' cited documents, which also go through the caller's read check. Block ids reach CSS only when they match ^[\w-]{1,128}$.
  • Digest only shows what the recipient can read. It is gated per document, at read time. Tested with a recipient who opened a private document and was then removed from it.
  • Untrusted text. Agents get block text, quotes, comment excerpts, task titles, decision rationales and citation quotes fenced with the shared fenceUntrusted, with untrusted: true. Promoted message text is escaped (every ASCII punctuation character, leading spaces dropped), so it can never become a heading, link, image, HTML or code block.
  • Decisions. Agents can't write a choice's decision props through ops.

Deviations

  • Plan-first uses M4's existing code and thresholds: approval_required at 60 edits / 20 removed blocks. The plan named plan_required at 20 / 5. The test is named for M4's numbers.
  • "Since you last looked" uses lastView from the document read, taken once per visit, rather than a separate "seen" write.
  • Page-text highlights mark the whole page block, not the individual element inside the sandboxed frame; per-element highlighting would need a bridge change.
  • M5's final head was merged forward, not rebased: it was a fast-forward of the commit this branch started from. M6 was merged in the same way, keeping both sides; its migrations come first and this PR's column is regenerated after them as 0023.
  • The email digest is sent on request (POST /api/me/digest/email), not on a schedule. It answers 503 when this Primer has no mail sender.
  • edits counts saves, not blocks: typing one sentence was 29.

Test plan

  • New tests:
    • catch-up lists only changes after my last view
    • the digest only includes documents the recipient can still read (REST and email; the MCP tool ran in the browser check)
    • an agent's digest is fenced
    • orphaned thread re-anchors to the reworded sentence
    • a near-tie is refused
    • blocks edited after the checkpoint are highlighted
    • the catch-up panel
    • an unsafe id never reaches the stylesheet
    • citation to a Primer doc pins the cited version
    • a thread on page text re-anchors, and the edit reads as a change to the page
    • a thread becomes a document with one quote per message, and the REST route
    • an Add version that only suggests shows its suggestions
    • message text stays literal
    • small batches add up to the plan threshold
    • an agent can't write decision props
  • All six CI gates ran locally on the final head: codegen drift, migrate, typecheck, build, test (api 1015, web 368, mcp 125, blocks 3), audit.

Browser check

Two people, A and B (an editor), with Chrome against a dev API and web on a throwaway Postgres:

  1. B opens the document once, then leaves. No catch-up panel is shown on that first visit.
  2. A works while B is away:
    • types a sentence into a paragraph;
    • decides a choice ("EU first", with a rationale);
    • comments on a sentence and resolves the thread.
  3. B returns. The panel reads "Since you last looked … Mock User: edits: 29, comments: 1, resolved threads: 1, decisions: 1", then lists the decision, the comment and the resolution.
    • Only the typed paragraph and the choice are highlighted. The resolved thread's block is not, which was the bug the check found and the comment-mark fix addresses.
    • "Changes since checkpoint" highlights the same blocks against the upload's checkpoint.
    • Dismiss removes the panel and the highlights.
  4. A comments on "We ship the beta to five pilot customers in March." An agent replaces that block with "Plan: we ship the beta to 5 pilot customers in March, then widen." through primer_apply_ops.
    • The thread is flagged orphaned.
    • B opens Comments and clicks Re-anchor. The thread is no longer orphaned, and its mark sits on "we ship the beta to 5 pilot customers in March,".
  5. Citations. The agent adds a claim with a citation. B's editor shows the numbered chip "[1] example.com" with the quote as its title, and GET …/citations lists it.
  6. Promote. B clicks "Promote to document" on the thread and lands on a new document holding one quote block per comment.
  7. Digest. A also comments on a private document B never had. B's GET /api/me/digest lists only "Launch plan" and never mentions the other document. primer_get_digest over MCP returns fenced data (untrusted: true).
  8. There were no console errors.

🤖 Generated with Claude Code

maxsam4 and others added 16 commits November 19, 2026 23:33
…0xPolygon#154)

M10b builds on both: its catch-up reads M10a's decision and task events
and M5's suggestions. Both sides kept; M10a's migrations regenerated after
M5's (one 0022 migration). The working status M4 replaced with awareness
stays removed (M10a's side still carried primer_set_status and the banner),
and the comment, suggestion and delete-guard strings from M3 to M5 move to
i18n so the M10d lint passes on the combined tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A comment whose sentence was reworded or moved used to stay orphaned until
someone quoted the new text exactly. Re-anchoring (primer_reanchor_thread,
POST /api/threads/:id/reanchor, and a new Re-anchor button on orphaned
threads) now matches exactly first, then the best run of whole words at
least 80% alike in the target block, or in the whole tab once that block
is gone. It uses the one bounded similarity from the republish merge, with
a quote-size cap and a character-pair budget, and refuses a near-tie
between two places instead of guessing. Target and quote default to the
thread's own, so the button sends nothing. Creating a thread stays exact,
nothing re-anchors automatically, and legacy threadCarry is untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
People coming back to a document had no way to see what others did while
they were away, and AidenHQ had nothing to push to Slack or Telegram. A
catch-up is a thin view over what Primer already keeps: the update log
(who saved which blocks, which survives compaction) for edits, shown as
the blocks read now, and Primer's own lifecycle events (comments, resolved
threads, M10a's decision.made/cleared and task.*, suggestion.created and
decided, published versions) joined to their rows, grouped per person. The
caller's own actions are left out; agents get people's text fenced.

- document_views.last_doc_version: opening a document records the visit
  and returns the previous one as lastView in the same statement, so a
  refetch never loses "since you last looked".
- GET /api/documents/:docId/catchup, GET /api/me/digest and MCP
  primer_get_digest return the same data; the digest gates every document
  through the read check again, so a revoked or deleted one never shows.
- POST /api/me/digest/email sends a person the digest, laid out as blocks
  and rendered by exportHtml, the static renderer shares and print use.
- A comment mark is an annotation: it is stored as touching no blocks, so
  delta reads and catch-up don't show commenting as editing.
- The version list carries each checkpoint's docVersion, for highlights.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Opening a live document now shows "Since you last looked": what each
person did since the viewer's previous visit (taken once per visit from
the document's lastView), with the changed blocks highlighted until it is
dismissed. A "Changes since checkpoint" toggle highlights the blocks
edited after the latest checkpoint (its docVersion, then a delta read).
Highlights are a stylesheet keyed on BlockNote's data-id, never a write to
the document, and only plain block ids reach it: ids come from the
client-writable Y.Doc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agents and people had no way to tie a claim to its evidence. A citation
is a custom inline chip (url or Primer docId + versionNumber, and the
supporting quote), numbered in reading order by a CSS counter in the
editor, static pages and email. Agents add citations through json block
input; applyOps pins a Primer citation to the cited document's latest
version, so the claim stays tied to what was read, and refuses a document
the agent can't read. GET /api/documents/:docId/citations lists them with
what each cites, naming a cited document only after a read check: its id
comes from the client-writable Y.Doc. The authoring skill covers citing,
re-anchoring and catching up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decisions made in Slack, Telegram or a comment thread had no path into a
document. primer_promote_conversation and POST
/api/documents/from-conversation turn messages into one quote block each,
the author in bold above the text, through the one publish path (a new
private document) or, with documentId, through the ops pipeline at the end
of a live document, so an agent on someone else's document suggests. Every
ASCII punctuation character is escaped and leading spaces dropped, so
untrusted message text stays literal and never becomes a heading, link,
image, HTML or code block. "Promote to document" on a comment thread does
the same for anyone who can read the thread.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
M10b.6 (plan-first for big agent edits) shipped with M4's runs: per-run
edit and removal counters, approval_required past the threshold, and
approval on the document queue (refused after a cancel). What the plan
pins and no test did: splitting a large change into small batches does
not avoid approval. Batches of three add up until the edit past the
threshold is held.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This tree is the first with both M4's generic ops and M10a's choice
blocks. Decisions are made only by a person through the pick route (and
proposed through primer_propose_pick), which checks rights and records
who decided; an agent's json block input naming decisionId, a decided*
render slot or proposedBy is now refused invalid_op, as the M10a
follow-up asked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The browser check showed a block highlighted as edited after someone only
resolved a thread on it: BlockNote in an open editor rewrites a resolved
thread's comment mark, from that person's connection. Tagging the
server's annotation writes could not catch that, so persistence now looks
at what a transaction did: when it only inserted and removed comment-mark
formatting, it touches no block, whoever made it. The pipeline goes back
to its plain origin. The scope test lists primer_promote_conversation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… citation chips have height

- MCP document reads record last_doc_version with viewed_at, as the
  REST read does, so the two halves of lastView never disagree.
- POST /api/me/digest/email answers 503 when this Primer sends no email,
  instead of reporting it sent.
- The citation chip sat in a sup with zero line height (the reset gives
  sup line-height 0), so it had no box; it gets its own line height in the
  editor and in static pages.
- The promote route is tested over HTTP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
M10b's page-text parts need M6's page fragments. Both sides kept: M6's
migrations replace this branch's 0022 and the catch-up column is
regenerated after them (0023); M6's i18n keys for the M3–M5 strings win,
and this branch's own keys (catch-up, thread actions, checkpoint
changes) are added to them. The events line in the skill reference that
the M10a merge had duplicated is M6's single line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since M5, a republish by an agent in suggest mode makes no version: the
response carries only a reconciliation. The dialog still closed and
opened the document as if a version had been added. It now stays open
and says what happened (how many blocks were suggested, conflicted or
unchanged), with a button to open the document where people decide them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With M6 merged, text people edit inside an HTML page lives in the page's
own fragments (page/<blockId>/<pid>), which the update log records by
fragment name, not block id. Delta reads, catch-up and the highlights
built on them now map a sub-fragment (page text, embed, widget label) to
the block it belongs to, so a page whose text changed reads as a changed
block instead of nothing (or a deleted id). Fuzzy re-anchoring already
searches a sub-fragment whole; a test pins it for a retyped page
sentence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@maxsam4

maxsam4 commented Nov 19, 2026

Copy link
Copy Markdown
Author

Replaced by #179, part of the native stacked PR chain (#162 → #182). Review history stays here.

@maxsam4 maxsam4 closed this Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant