Skip to content

feat: live Markdown documents in BlockNote, with checkpoints (M2b) - #169

Open
maxsam4 wants to merge 16 commits into
stack/06-m9-recordsfrom
stack/07-m2b-editor
Open

maxsam4 wants to merge 16 commits into
stack/06-m9-recordsfrom
stack/07-m2b-editor

Conversation

@maxsam4

@maxsam4 maxsam4 commented Nov 19, 2026

Copy link
Copy Markdown

Replaces #148 (review history there).

One link of a linear stack, one milestone per PR. The base is stack/06-m9-records (#168), so this diff is this milestone's own work plus the merge resolutions it needs against the links below it.

Summary

M2b of the collaboration plan: Markdown and .txt documents become live documents edited in BlockNote, with checkpoints as immutable render bundles.

  • Import keeps every byte (live/import.ts). Each top-level group of the file (by markdown-it's line map) goes markdown-it → HTML → tryParseHTMLToBlocks, giving the richest stock block: headings, lists, check lists, tables, code with its language, quotes and images. A group becomes a source block holding its slice verbatim, with a sanitized preview that has no scripts, when:

    • it holds raw HTML or a footnote;
    • it is lines no token covers, such as reference definitions;
    • a fidelity check finds its text, links, images or checkboxes changed;
    • it is a code fence in a language the editor lacks.

    Front matter becomes documents.metadata. Import refuses if the groups' slices and gaps don't add back up to the input.

  • Export equals the input (live/export.ts). A group is re-emitted from its source lines while its blocks are present, contiguous and unchanged by subtree version, and while the reference definitions it uses are too. Anything else is serialised, keeping the original gap. An untouched round trip is byte-identical, and a one-block edit gives a one-block diff.

  • Only signed source is trusted. Any editor's socket can write the sources and meta maps. Import therefore signs each entry and the meta with an HMAC over the document id and every field, and export re-emits only entries that verify. Everything else is serialised from the blocks. The key is random, kept in Primer's storage, with no new config.

  • One publish path (lib/publish.ts). Markdown and text build a detached Y.Doc, then commit the document, its first update and checkpoint 1 with its bundle in one transaction. A legacy document that gets a Markdown version becomes live: its open threads are re-anchored by exact quote with comment marks, and the rest stay listed on the new checkpoint without an anchor. Publishing onto a live document is refused with republish_unsupported.

  • Checkpoints (live/checkpoint.ts, live/bundle.ts). The state is frozen on the document's queue, and the bundle is rendered outside the queue while editing goes on. The bundle holds:

    • state.ydoc;
    • export.md or export.txt, and export.html;
    • static.html, the version's content;
    • threads.json, the open threads with their quotes and comment counts at capture;
    • files.json;
    • manifest.json.

    Restore is a new edit through the pipeline.

  • Routes, matching M2a's six upcoming scope rows:

    • POST /api/documents/:docId/checkpoints;
    • POST /api/versions/:versionId/restore;
    • GET /api/versions/:versionId/ydoc;
    • GET /api/documents/:docId/export?format=&version=;
    • POST /api/documents/:docId/files;
    • GET /api/files/:fileId?doc=<docId>. It serves a file only when it belongs to the named document and the viewer can read that document, since file ids come from editor-writable block props. It sends nosniff and a sandboxing CSP, and serves an attachment unless the file is passive media. A bundle's files.json lists only the document's own files.
  • Purge deletes a purged document's bundles and uploaded files too.

  • Web. BlockNote with the Ariakit UI (slash menu, toolbars, side menu, table handles, file panel), live cursors, and BlockNote's version history over checkpoints.

    • The editor is a lazy chunk.
    • commentMarkExtension is registered.
    • Files are stored as primer-blob:// ids.
    • The header has "Save checkpoint" and Export (Markdown and HTML).
    • Checkpoints open read-only with their discussion.
    • Compare diffs checkpoint exports.
    • A connection indicator shows "Reconnecting…", "Unsaved changes" (after 5 s) and "Changes not saved" with Retry (close code 4500).
  • Removed: the Markdown-to-HTML version renderer and its stylesheet. markdown-it and gray-matter stay, and the ---js guard and its tests are kept.

The scope rows in auth/scopes.ts still carry upcoming: true. This lane doesn't own that file, so a later lane clears the flags. scopes.test passes either way.

Test evidence

  • CI order run locally, all green: db:migrate:test, codegen-drift-check, typecheck, -r build, test (api 90 files / 869 tests, web 31 / 341, blocks 2 / 3, mcp 5 / 124), and audit --audit-level=high (one finding below high).
  • New tests:
    • no text is dropped from real-world samples;
    • raw HTML and footnotes become source blocks with a rendered preview;
    • front matter is parsed into metadata and kept raw;
    • tables, task lists and code fences map to stock blocks;
    • every sample renders to static HTML;
    • untouched round trip is byte-identical;
    • editing one block changes only its lines;
    • editing a child item re-serialises its list group;
    • editing a reference definition re-serialises the groups that use it;
    • .txt round trip is byte-identical;
    • a source slice or front matter rewritten by a client never reaches the export;
    • markdown upload creates a live document with checkpoint 1;
    • legacy Markdown doc converts and re-anchors open threads by quote;
    • checkpoint bundle is frozen at one docVersion while edits continue;
    • restore is a new edit that keeps later history;
    • checkpoint view keeps its discussion context;
    • compare diffs the Markdown exports;
    • files are served by the document read check and never run;
    • live document renders the editor and legacy renders the iframe;
    • connection indicator shows reconnecting, unsaved and save-failed states;
    • a file id from another document is neither served nor bundled;
    • a purged live document takes its bundles and files with it;
    • ensureSigningKey makes the key only when it is missing;
    • every Markdown sample also round-trips with CRLF line endings.
  • Real browser (Chrome, pnpm dev, mock auth), read from the DOM:
    • A Markdown upload (front matter, table, task list, code fence, raw HTML) opens in the editor as heading, paragraph, heading, table, check list, code block, source and paragraph blocks.
    • Opened but untouched, the export equals the upload byte for byte.
    • Two tabs sync edits both ways.
    • The slash menu lists the stock blocks, and inserting a table and typing in it syncs to the other tab.
    • "Save checkpoint" reports "Checkpoint v2 saved".
    • Checkpoint 1's export is identical to the upload. Checkpoint 2 equals the live export, in which only the edited lines differ.
    • The checkpoint view shows its read-only banner.
    • A reload keeps the edits.
    • Opening without typing writes no update, for the owner or for a read-only reader (a second persona). The reader sees the content, can't edit it, and gets no indicator.
    • History lists Current and the checkpoints. Selecting checkpoint 1 previews it read-only while another tab stays live; closing returns the live text and editing. Restore from the row menu converges both tabs, and the export is checkpoint 1's.
    • No page errors.
  • Known gap: versions a document had before it went live appear in History, but they have no frozen state, so selecting one reports "not found".
  • The browser check found two bugs, both fixed and covered by tests:
    • a ```ts fence failed to render (BlockNote wants typescript);
    • 0.57 takes collaboration through withCollaboration.

🤖 Generated with Claude Code

maxsam4 and others added 16 commits November 19, 2026 21:08
A live document must never lose content silently. Each top-level group of a
Markdown file becomes the richest stock BlockNote blocks (markdown-it renders
it, tryParseHTMLToBlocks parses it), unless a fidelity check finds its text,
links, images or checkboxes changed; such a group, raw HTML, footnotes and
lines no token covers (reference definitions) become source blocks that keep
the slice verbatim with a sanitized preview. Front matter becomes metadata.
Every group remembers its exact lines and gap, and import refuses if they
don't add back up to the input, so export can re-emit untouched groups.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Output should equal input: an untouched document exports the uploaded bytes,
and a one-block edit gives a one-block diff. A group is re-emitted from its
source lines while its blocks are all present, contiguous and unchanged by
subtree version (comment marks don't count), and while every reference
definition it uses is too; anything else is serialised, keeping the original
gap after it. Static HTML for bundles is BlockNote's full HTML with its
stylesheet inlined.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The `sources` and `meta` maps live in the Y.Doc, which every editor's socket
can write, so an editor could plant text in a source slice or the front
matter that the editor never shows but downloads and agents read. Import now
signs each entry and the meta with an HMAC over the document id and every
field, keyed by a random server key kept in Primer's storage (no new
config). Export type-checks entries and re-emits one verbatim only when its
MAC verifies and its blocks are unchanged; anything else is serialised from
the blocks. The static rendering re-renders source previews from their
Markdown instead of trusting the stored preview.

Also adds `anchorThreads`, which re-anchors threads by quote when a legacy
document becomes live (used by the next commit).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Markdown and text publishes, over REST and MCP alike, now create live
documents edited in BlockNote. The import is built as a detached Y.Doc and
committed with the document row, its first update (the system's) and
checkpoint 1 with its render bundle, so nobody can open a live document half
made. A legacy document that gets a Markdown version becomes live the same
way: its open threads are re-anchored by exact quote with comment marks, and
the rest stay listed on the new checkpoint without an anchor (orphaned).
Publishing onto a live document is refused with republish_unsupported until
republish merges arrive with suggestions. HTML stays on the version path.

The bundle and checkpoint modules land here because checkpoint 1 is a bundle;
their routes and tests follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A checkpoint freezes the live state on the document's queue (so nothing
applied later slips in) and renders its bundle outside it while editing goes
on: the Y.Doc state, the export in the document's format, the static HTML
served as the version's content, the files it references and threads.json,
the open threads with their quotes and comment counts at capture. Open
threads are also placed on the checkpoint, anchored where their comment mark
is, so its discussion stays listed. Restore applies the bundle's content
through the pipeline as a new edit, writing only what differs, so later
history is kept.

Routes (the scope rows M2a landed): POST checkpoints, POST restore, GET a
checkpoint's ydoc, GET export (live or ?version=n, Markdown, text or HTML),
and document files stored as opaque primer-blob:// ids, served with nosniff,
a sandboxing CSP and attachment disposition outside a passive media list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Compare diffs what a live document's checkpoints export (Markdown or text,
exactly as downloaded) instead of the text of a rendered page; versions a
document had before it went live still diff their page text. BlockNote's
version history gets a VersionStorage over checkpoints: listing, the frozen
Y.Doc of a bundle for its preview, saving a named checkpoint, renaming and
restoring as a new edit. Versions are never deleted, so `remove` is omitted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live documents open in BlockNote with its stock blocks and Ariakit UI (slash
menu, formatting toolbar, side menu, table handles, file panel), synced over
/collab with collaboration cursors, and loaded as a lazy chunk so legacy
documents don't pay for it. The comment mark is registered so commented text
survives until comments ship. Files upload to the document as primer-blob://
ids resolved to an authorised link at read time. The header saves checkpoints
and exports Markdown or HTML; checkpoints open read-only as rendered pages
with their discussion. A connection indicator says "Reconnecting…", "Unsaved
changes" after five seconds without acknowledgement, and "Changes not saved"
with Retry when the server could not store an update.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Markdown and text publish live documents now, so the page renderer for
Markdown versions, its HTML wrapper and stylesheet go: one way to publish
Markdown. markdown-it stays as the one Markdown parser (the import's
tokenizer) and gray-matter as the front matter reader, with the `---js`
engine still disabled and its tests kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A ```ts fence imported with language "ts", which BlockNote's code block
refuses ("Language ts is not supported"), so rendering the checkpoint failed
and the upload with it; the editor would refuse it too. Import now maps
aliases to BlockNote's ids (ts → typescript, sh → shellscript) and keeps a
fence in a language the editor doesn't know verbatim as a source block, so
its info string isn't lost. Every sample now also renders to static HTML in
the tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BlockNote 0.57 takes collaboration through `withCollaboration`, not an
editor option; passed as an option it was ignored, so the editor showed an
empty paragraph instead of the synced document. The provider was also made
in a memo and destroyed by an effect, so React's development remount reused
a destroyed connection; one effect now makes and tears it down. The
connection indicator starts from the socket's current status instead of
waiting for the next change. The jsdom test now seeds the Y.Doc and checks
the editor shows it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The key was re-made on any failure to read it, so one transient storage
error after a restart would have replaced it and turned every existing
document's export lossy for good. Only a missing key (ENOENT, which both
storage backends raise) is made now; any other failure is thrown, and a key
of the wrong size fails loudly. The process-local fallback used before the
key loads no longer stands in for the stored key afterwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The purge deleted each version's storage and original keys only, so a purged
live document left its checkpoint bundles and uploaded files in storage.
It now deletes every bundle file and every document file before the rows go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the plain-text sample had CRLF; every Markdown sample, front matter
included, now also round-trips with Windows line endings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
File ids reach the server from block props, which any editor can write, so
an id copied from another document must never resolve. GET /api/files/:id
now takes the referencing document (`?doc=`), serves the file only when it
belongs to that document and the viewer can read it, and answers 404 for a
malformed id instead of failing. A bundle's files.json lists only the
document's own files. The editor resolves primer-blob:// links with its
document id.

Other ids read from the Y.Doc were audited: thread ids from comment marks
are matched only against the document's own threads, tab ids name fragments
only, checkpoint ids come from the URL and are checked against their
document, and source entries are signed per document.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat: live Markdown documents in BlockNote, with checkpoints (M2b)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant