Skip to content

pampa: raw-json — lossless pampa-native JSON reader/writer (bd-en2hvrwn)#400

Merged
cscheid merged 1 commit into
mainfrom
feature/bd-en2hvrwn-raw-json-pampa-native
Jul 17, 2026
Merged

pampa: raw-json — lossless pampa-native JSON reader/writer (bd-en2hvrwn)#400
cscheid merged 1 commit into
mainfrom
feature/bd-en2hvrwn-raw-json-pampa-native

Conversation

@cscheid

@cscheid cscheid commented Jul 17, 2026

Copy link
Copy Markdown
Member

Closes #11.

Adds pampa -t raw-json / -f raw-json: a pampa-specific JSON format whose contract is that write-then-read is the identity on the full pampa AST, including everything Pandoc-shaped JSON cannot represent:

  • standalone Inline::Attr (the exact repro from feature: "raw" JSON format #11 — previously error Q-3-32)
  • NoteReference, the four CriticMarkup inlines, Shortcode, CaptionBlock
  • full-fidelity ConfigValue metadata: Path/Glob/Expr kinds, merge ops, scalar YAML types, and map entry order

Design

A raw mode on the shared streaming JSON writer/reader (JsonConfig::raw / SourceInfoDeserializer.raw) rather than a fork or a serde-derived parallel format: the source-info pool, astContext encoding, and all standard node arms are the same code path as -t json, so Arc-shared source-info DAGs keep their structural sharing across the roundtrip. Because the raw arms live inside the same exhaustive matches, a new AST variant fails compilation until its raw behavior is decided.

The output is unmistakably not pandoc -t json: the envelope's first key is "pampa-json-format": {"version": 1} (version-checked on read) and pandoc-api-version is deliberately absent. Each reader rejects the other format with a targeted error pointing at the right -f flag.

Contract, tag vocabulary, and versioning policy: claude-notes/designs/raw-json-format.md. Plan and decision history: claude-notes/plans/2026-07-17-raw-json-format.md.

Fidelity fixes in the shared reader (improve -f json too)

The new corpus sweep caught two pre-existing write-only sidecars that were silently dropped on read-back, now restored:

  • Link/Image target_source (targetS)
  • Citation.id_source (citationIdS)

Testing

  • 21 new integration tests (TDD, written failing first): identity roundtrips for every extension construct, meta fidelity, marker/version rejection in both directions, shared-Substring-parent preservation, Concat/Generated chains, and an identity sweep over the existing tests/writers/json/*.md corpus.
  • Full workspace suite green (10099 tests); cargo xtask verify green on all legs except the 20 "live" hub-mcp.test.ts tests, which require wss://sync.automerge.org (down at time of writing; quarto-hub-mcp has no pampa dependency).
  • End-to-end through the real binary, output inspected:
$ echo 'Hello. {#free-floating-attribute} Here?' | pampa -t raw-json
{"pampa-json-format":{"version":1},"blocks":[{"c":[{"c":"Hello.","s":1,"t":"Str"},
{"s":2,"t":"Space"},{"a":{…},"c":["free-floating-attribute",[],[]],"s":3,"t":"Attr"},…

Three write → read generations byte-identical; -f raw-json -t qmd reproduces the input exactly. (Byte-stability is not part of the contract — semantic identity is; see the design doc.)

🤖 Generated with Claude Code

…wn, GH #11)

Adds -t raw-json / -f raw-json: a pampa-specific format whose contract is
that write-then-read is the identity on the full pampa AST, including
everything Pandoc JSON cannot represent — standalone Inline::Attr (the
GH #11 repro), NoteReference, the CriticMarkup inlines, Shortcode,
CaptionBlock, and full-fidelity ConfigValue metadata (Path/Glob/Expr
kinds, merge ops, scalar YAML types, map entry order).

Implementation: a raw mode on the shared streaming JSON writer/reader
(JsonConfig::raw / SourceInfoDeserializer.raw) rather than a parallel
code path — the source-info pool, astContext encoding, and all standard
node arms are the same battle-tested code as -t json, so Arc-shared
source-info DAGs survive roundtrip. The envelope self-identifies with a
first-key "pampa-json-format" marker (version-checked on read) and
deliberately omits pandoc-api-version; each reader rejects the other
format with a targeted error.

Fidelity fixes in the shared reader, found by the new corpus sweep
(these improve -f json roundtrips too):
- Link/Image target_source (targetS sidecar) was written but dropped on
  read; now restored.
- Citation.id_source (citationIdS) likewise.

Tests: 21 new integration tests (TDD; written and failing first),
including identity roundtrips for every extension construct, meta
fidelity, marker/version rejection in both directions, shared-parent
source-info preservation, and an identity sweep over the existing
tests/writers/json/*.md corpus. E2E verified through the binary: the
issue #11 document renders, three write-read generations are
byte-identical, and -f raw-json -t qmd reproduces the input exactly.

Design doc: claude-notes/designs/raw-json-format.md
Plan: claude-notes/plans/2026-07-17-raw-json-format.md

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@posit-snyk-bot

posit-snyk-bot commented Jul 17, 2026

Copy link
Copy Markdown

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues
Licenses 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@cscheid
cscheid merged commit b78bc18 into main Jul 17, 2026
8 checks passed
@cscheid
cscheid deleted the feature/bd-en2hvrwn-raw-json-pampa-native branch July 17, 2026 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feature: "raw" JSON format

2 participants