diff --git a/claude-notes/designs/raw-json-format.md b/claude-notes/designs/raw-json-format.md new file mode 100644 index 000000000..7b53a6249 --- /dev/null +++ b/claude-notes/designs/raw-json-format.md @@ -0,0 +1,147 @@ +# The pampa raw-json format + +**Status:** Active (version 1). Shipped 2026-07-17 (bd-en2hvrwn, GH #11). +**Entry points:** `crates/pampa/src/writers/raw_json.rs`, +`crates/pampa/src/readers/raw_json.rs` (thin wrappers over the shared JSON +machinery in `writers/json.rs` / `readers/json.rs` with the raw mode flag). +**CLI:** `pampa -t raw-json` / `pampa -f raw-json`. +**Plan/history:** `claude-notes/plans/2026-07-17-raw-json-format.md`. + +## Contract + +> `raw_json::write` followed by `raw_json::read` is the **identity** +> (structural equality, `PartialEq`) on the pampa AST — `Pandoc` (blocks + +> full `ConfigValue` meta) and the roundtrip-relevant `ASTContext` +> projection — for every AST the rest of pampa can produce. + +What is explicitly promised and not promised: + +- **Promised:** every AST extension roundtrips (standalone `Inline::Attr`, + `NoteReference`, `Insert`/`Delete`/`Highlight`/`EditComment`, + `Shortcode`, `CaptionBlock`, `BlockMetadata`, note definitions, custom + nodes); metadata keeps `Path`/`Glob`/`Expr` kinds, merge ops, scalar + YAML types, and map entry order; source-info graphs keep their + structural sharing (pool encoding, same as `-t json`). +- **Not promised:** byte-stability of `write ∘ read ∘ write`. The reader + rebuilds pool entries behind fresh `Arc` wrappers, and the writer's + parent-dedup is by pointer identity, so a re-serialize may intern extra + pool entries / shift ids. The contract is semantic identity, not + canonical bytes. (In practice small documents come back byte-identical; + do not rely on it.) +- **Not promised:** cross-version stability. The `version` field is a + fail-fast guard, not a compatibility contract; raw-json is a + q2-internal format. If we ever persist raw-json long-term, that is the + moment to write a compatibility policy. +- **No lenient reader.** Raw-json is strict about `s:` source refs — + every writer-produced node carries one. JSON from outside the q2 + source-tracking world goes through `-f json` (the completing reader). + +## Envelope + +```json +{ + "pampa-json-format": {"version": 1}, + "blocks": [ ... ], + "meta": { "c": [...], "s": 7, "t": "MetaMap" }, + "astContext": { + "files": [ {"line_breaks": [...], "name": "...", "total_length": N} ], + "exampleListCounter": 1, + "p": [ ...source-info pool... ] + } +} +``` + +- **`pampa-json-format` is the first key**, always — it is the format's + self-identification, visible in the first line / truncated previews. +- **`pandoc-api-version` is deliberately absent** so machine consumers of + Pandoc JSON fail fast instead of half-parsing. +- Key order is a streaming constraint: `astContext` must come last + because the source-info pool is complete only after the AST walk. +- `meta` is a single config-value node (see below), not a Pandoc-style + object; there is no `metaTopLevelKeySources` sidecar in raw mode (key + sources ride inline). +- Reader rejections (all `JsonReadError` variants, no Q-codes — the JSON + reader path predates catalog integration): Pandoc-style input → + `NotRawJson{has_pandoc_api_version: true}` ("use `-f json`"); unmarked + input → `NotRawJson{..: false}`; unknown version → + `UnsupportedRawJsonVersion`. Symmetrically, the Pandoc-superset reader + rejects raw-json input with `UnexpectedRawJsonMarker` ("use + `-f raw-json`"). + +## Node vocabulary (delta over the Pandoc-superset format) + +Everything the `-t json` format emits is unchanged (same `t`/`c` tags, +same `s`/`a`/`targetS` sidecars, same pool codes — see +`wire-format-source-info-codes.md`, which raw-json shares). Raw mode adds: + +| Tag | Shape | AST type | +|---|---|---| +| `Attr` | `{a, c: [id, classes, kvs], s, t}` | standalone `Inline::Attr` | +| `NoteReference` | `{c: id_string, s, t}` | `Inline::NoteReference` | +| `Insert` / `Delete` / `Highlight` / `EditComment` | `{a, c: [attr, [inlines]], s, t}` (Span-shaped) | CriticMarkup inlines | +| `Shortcode` | `{c: {isEscaped, keywordArgs, name, positionalArgs}, s, t}` | `Inline::Shortcode` | +| `CaptionBlock` | `{c: [inlines], s, t}` | `Block::CaptionBlock` | + +Shortcode args are tagged nodes: `String`/`Number`/`Boolean` are +`{c, t}`; a nested `Shortcode` is a full `{c, s, t}` node; `KeyValue` is +`{c: [[key, arg]...], t}` with entries **sorted by key** (the underlying +`HashMap` has no stable order; sorting keeps output deterministic, and +map equality on read is order-independent). `keywordArgs` pairs keep +`LinkedHashMap` insertion order. + +## Metadata encoding + +`meta` and every nested value is a `{c, m?, s, t}` node: + +| Tag | `c` | AST | +|---|---|---| +| `MetaString` | string | `Scalar(Yaml::String)` | +| `MetaBool` | bool | `Scalar(Yaml::Boolean)` | +| `MetaInt` | i64 | `Scalar(Yaml::Integer)` (raw only) | +| `MetaReal` | string (yaml_rust2 raw source string, byte-faithful) | `Scalar(Yaml::Real)` (raw only) | +| `MetaNull` | null | `Scalar(Yaml::Null)` (raw only) | +| `MetaPath` / `MetaGlob` / `MetaExpr` | string | `Path` / `Glob` / `Expr` (raw only) | +| `MetaInlines` / `MetaBlocks` | AST arrays | `PandocInlines` / `PandocBlocks` | +| `MetaList` | array of nodes | `Array` | +| `MetaMap` | array of `{key, key_source, value}` entries, **source order** | `Map` | + +- `m` carries a non-default merge op (`"prefer"`; the default `"concat"` + is omitted). Only raw mode emits it; the reader accepts it whenever + present. +- `MetaMap` uses an array of entries (not a JSON object) because + serde_json objects don't preserve key order on read; entry order is + part of the contract. +- A `Scalar` holding a non-scalar YAML value (Array/Hash/Alias/BadValue) + has no faithful encoding and fails the write loudly with **Q-3-57** + rather than degrade silently. + +## ASTContext + +`astContext.files` and the pool `p` are identical to the Pandoc-superset +format (writer emits, `read_ast_context` + `SourceInfoDeserializer` +restore; structural sharing preserved). Raw mode additionally carries +`exampleListCounter`. Not carried: `parent_source_info` (parse-time-only +state) and file *contents* (line-break tables only, matching `-t json`; +the contract is about the AST, not re-deriving source text). + +## Versioning policy + +- The version is a single integer in the marker; the reader accepts + exactly the versions it knows (currently: 1). +- Additive, ignorable keys (a new optional sidecar) do not need a bump — + readers ignore unknown keys, matching the existing JSON reader's + behavior. +- Any change a version-1 reader would misread (tag shape change, meaning + change, removal) bumps the version. Wire-shape questions for the shared + pool codes follow `wire-format-source-info-codes.md` — raw-json is a + second consumer of those codes and inherits that policy. + +## Maintenance rule (how the roundtrip stays true) + +The raw arms live inside the same exhaustive `match`es as the +Pandoc-superset arms, so **adding a new `Block`/`Inline` variant fails +compilation until its raw behavior is decided** — decide it by adding a +native raw tag (and reader arm + roundtrip test), not a desugaring. The +roundtrip corpus (`tests/integration/test_raw_json_roundtrip.rs`, +including the `tests/writers/json/*.md` sweep) is the semantic guard; +extend it with every new construct. diff --git a/claude-notes/designs/wire-format-source-info-codes.md b/claude-notes/designs/wire-format-source-info-codes.md index 590ed5ffc..f7779d6eb 100644 --- a/claude-notes/designs/wire-format-source-info-codes.md +++ b/claude-notes/designs/wire-format-source-info-codes.md @@ -8,6 +8,11 @@ (first formal TS reader); [q2-preview plan 5](../plans/2026-05-04-q2-preview-plan-5-wire-format.md) (`Synthetic` / `Derived` variants). +> **Second consumer:** the pampa `raw-json` format (bd-en2hvrwn, GH #11; +> see [raw-json-format.md](raw-json-format.md)) emits the same +> `astContext.p` pool through the same writer code. Code allocations +> here govern both formats. + ## Purpose The source-info pool is a JSON-encoded structure emitted by the Pandoc diff --git a/claude-notes/plans/2026-07-17-raw-json-format.md b/claude-notes/plans/2026-07-17-raw-json-format.md new file mode 100644 index 000000000..a560ef88e --- /dev/null +++ b/claude-notes/plans/2026-07-17-raw-json-format.md @@ -0,0 +1,312 @@ +# Pampa-native "raw JSON" reader/writer (GH issue #11) + +**Status:** Draft v2 — iterating with Carlos before implementation. +**GitHub issue:** https://github.com/quarto-dev/q2/issues/11 +**Braid strand:** bd-en2hvrwn +**Related strands:** k-42 (ASTContext serialization — see "k-42 status" below), +k-gv05 (nondeterministic sourceInfoPool IDs). +**Related docs:** `claude-notes/designs/wire-format-source-info-codes.md` +(source-info pool wire codes — raw-json **shares** this wire format, see below). + +## Overview + +`pampa -t json` / `-f json` speak a Pandoc-compatible superset: real Pandoc +`t`/`c` tags plus q2 sidecars (`s` pool ids, `a` attr sources, `astContext`). +Because the node vocabulary is Pandoc's, the writer **cannot represent +pampa's AST extensions** — it desugars or hard-errors on them: + +| Construct | Today's behavior | +|---|---| +| `Inline::Attr` (standalone attribute) | **Error Q-3-32** (the issue #11 repro) | +| `Inline::NoteReference` | Error Q-3-31, desugared to `Span.footnote-ref` | +| `Inline::Insert/Delete/Highlight/EditComment` (CriticMarkup) | Errors Q-3-33..36, desugared to `Span.critic-*` | +| `Inline::Shortcode` | Silently desugared to a `Span` | +| `Block::CaptionBlock` (orphaned) | Error Q-3-21, rendered as `Plain` | +| `ConfigValue` metadata | Partially lossy (see "Metadata fidelity" below) | +| `Block::BlockMetadata`, note definitions, `Custom` nodes | Representable, roundtrip today | + +**Goal:** a new, pampa-specific format (working name: `raw-json`) whose +contract is: + +> `write(ast) |> read` is the identity (structural equality) on the pampa +> AST — blocks, meta, and the roundtrip-relevant parts of ASTContext — for +> **every** AST the rest of pampa can produce, including all extensions +> above. + +Secondary goals: + +1. **Unmistakably not `pandoc -t json` output**, to humans and machines. +2. **Freedom to drift** from Pandoc's schema when the AST needs it. + +Non-goals (v1): not a replacement for `-t json` (which stays the +Pandoc-superset / filter-facing format); not the filter wire format; no +TS/hub-client reader; no cross-version stability promise (the version field +is a fail-fast guard, not a compat contract); no WASM / quarto-core pipeline +changes. + +## Design: extend the existing JSON machinery with a "raw" mode + +**(Decision v2, 2026-07-17.)** Draft v1 recommended a serde-derived shape +(the AST types all derive `Serialize`/`Deserialize`). Carlos flagged the +fatal flaw: derived serde inlines every `SourceInfo` tree per node, and +**read-back cannot reconstruct the Arc-shared DAG** — sharing that q2's +source-location machinery depends on. A code study confirmed the existing +`-t json` machinery already solves exactly this, bidirectionally, and should +be reused: + +### What the existing code already provides (reuse inventory) + +- **Writer pool** — `SourceInfoSerializer` (`writers/json.rs:271-447`): + interns `SourceInfo` into a flat topologically-ordered pool (`astContext.p`), + dedups shared `Substring` parents / `Generated` anchors by `Arc::as_ptr` + (~93% size reduction), `perf.intern` gauge under `QUARTO_PERF_STATS=1`. +- **Reader pool** — `SourceInfoDeserializer` (`readers/json.rs:102-480`): + rebuilds each pool entry exactly once (forward-reference guard), children + clone earlier entries — the clone shares the entry's *inner* Arcs, so the + reconstructed graph preserves structural sharing with no blowup. (Each + edge gets a fresh outer `Arc` wrapper; see "Roundtrip contract" for the + consequence.) +- **ASTContext** — writer emits `astContext.files` (name, line-break table, + total length) + `metaTopLevelKeySources`; reader `read_ast_context` + (`readers/json.rs:1191-1253`) rebuilds `SourceContext` + `filenames`. +- **Node arms** — exhaustive `match` over every `Block`/`Inline` variant in + writer and reader, including the q2-only representable ones + (`BlockMetadata`, note definitions, `Custom` via wrapper nodes). + +### The raw mode + +Rather than forking the ~8600 lines of writer+reader (or building a parallel +serde format), **add a mode to the existing code path**: + +- **Writer**: a `raw` flag on `JsonConfig` (or a sibling entry point + `writers/raw_json.rs` delegating to shared internals). In raw mode: + - emit the envelope marker first, omit `pandoc-api-version`; + - the extension arms emit **native tags** (`t: "Attr"`, `t: + "NoteReference"`, `t: "Insert"`, `t: "Delete"`, `t: "Highlight"`, + `t: "EditComment"`, `t: "Shortcode"`, `t: "CaptionBlock"`) instead of + desugaring or erroring — these arms already exist as the Q-3-3X + defensive branches; raw mode is the *easy* branch of each; + - metadata is written faithfully (below). + - v1 implements raw mode on the **streaming** writer only (the production + path); the `Value`-building `write_pandoc` twin exists for the HTML + writer's source map and doesn't need raw mode. +- **Reader**: dispatch on the envelope marker. Raw mode accepts the native + extension tags (new arms next to the existing ones, sharing all node + helpers and the pool deserializer). The Pandoc-mode reader continues to + reject them, keeping the two formats honest. +- **Sharing property**: because the raw arms live inside the same exhaustive + `match`es, **adding a new AST variant fails compilation until its raw + behavior is decided** — the roundtrip guarantee is enforced by the + compiler at the arm level and by the roundtrip corpus at the semantic + level. (This was the main advantage claimed for the serde option; the + shared-arms design keeps most of it.) + +Maintenance cost vs. v1's serde option: every new variant needs a writer arm +and reader arm — but it needs those for `-t json` anyway; the raw arm is the +trivial one. In exchange we keep pool interning, Arc sharing on read-back, +compact output, and one battle-tested code path for both formats. + +### Metadata fidelity (raw mode) + +Current `-t json` meta encoding (`write_config_value`, `writers/json.rs:1662-1716`) +is lossy in specific, now-catalogued ways; raw mode fixes each: + +| Loss in `-t json` | raw mode | +|---|---| +| `Path`/`Glob`/`Expr` → `MetaInlines` (tag dropped) | tagged natively (`t: "Path"` etc.) | +| `merge_op` dropped | carried | +| Scalar `Integer`/`Real`/`Bool`/`Null` → `MetaString`/`MetaBool` (YAML type collapsed) | scalar type preserved | +| Top-level map keys **sorted** (insertion order lost) | entry order preserved (`ConfigValue` maps are `Vec`-backed) | +| `key_source` | already carried today; kept | + +Reader side already reconstructs `ConfigValue` directly +(`read_config_value_top_level`), so raw arms slot in naturally. + +## Self-identification (the "not pandoc JSON" marker) + +Top-level envelope, marker emitted first: + +```json +{ + "pampa-json-format": {"version": 1}, + "astContext": { "files": [...], "p": [...] }, + "meta": { ... }, + "blocks": [ ... ] +} +``` + +- The marker is a **top-level key, not a document-metadata entry** — it + describes the file, not the document. First position makes it visible in + truncated previews / first line of output. +- **`pandoc-api-version` is deliberately absent** — machine consumers of + Pandoc JSON fail fast instead of half-parsing. +- The raw reader **requires** the marker: missing marker + present + `pandoc-api-version` → targeted diagnostic ("this looks like Pandoc-style + JSON; use `-f json`"); missing both → "not a pampa raw JSON document"; + wrong `version` → version-mismatch error. New Q-3-XX codes for each. +- Open bikeshed: also carry `producer: "pampa "` for debuggability + (leaning yes — cheap). + +## Format name + +Working name **`raw-json`** (`-f raw-json`, `-t raw-json`), matching issue +#11. Alternative: `pampa-json`. Keep the name in one constant either way. + +## Roundtrip contract, precisely + +- `read(write(doc)) == doc` — structural equality (`PartialEq`) on + (`Pandoc`, ASTContext projection), for any `doc` pampa can produce. +- **Byte-stability of `write ∘ read ∘ write` is NOT promised.** The reader + rebuilds pool entries with fresh outer `Arc` wrappers per edge, so a + re-serialize can intern a handful of extra pool entries / shift ids (the + writer's dedup is by pointer identity, by design — see + `provenance-contract.md` and k-gv05's pointer-reuse lesson). The contract + is semantic identity, not canonical bytes. If canonical bytes ever matter + (caching), that's a follow-up strand (content-hash interning). +- ASTContext roundtrip scope: `files` (names + line-break tables — **no + embedded file contents**, matching today), and the source-info pool. + Known non-roundtripped fields, documented as out of scope v1: + `example_list_counter` (reader resets to 1 today — carry it in the raw + envelope? cheap, leaning yes), `parent_source_info` (parse-time-only + state, not meaningful post-parse). + +## k-42 status (checked 2026-07-17) + +k-42 asks "investigate how to serialize ASTContext properly in JSON +readers/writers" (motivated by the `filenames`/`source_context` duplication). +The serialization question is **already implemented and shipping** — both +directions, via `astContext.files` + `read_ast_context`. What remains live in +k-42 is only the duplication cleanup (store names once internally). This plan +adds a second consumer of that code but doesn't change the duplication +either way. Action: comment findings on k-42; keep it open for the cleanup; +`related` link to bd-en2hvrwn already in place. + +## Phases + +### Phase 0 — decisions (this document) + +- [x] Wire shape: reuse existing pooled machinery via raw mode (v2 decision, + supersedes v1's serde-derived Option A) +- [x] Format name: **`raw-json`** (matches issue #11) +- [x] Marker shape: **`"pampa-json-format": {"version": 1}`** — no + `producer` field; the key name itself identifies the producer, and a + version-bearing string would churn snapshots on every release +- [x] Meta fidelity table above; native tags: `Attr`, `NoteReference`, + `Insert`, `Delete`, `Highlight`, `EditComment`, `Shortcode`, + `CaptionBlock`; meta kind tags: `Path`, `Glob`, `Expr` +- [x] `example_list_counter` carried in the raw envelope's `astContext` +- [x] Raw-reader rejections: new **`JsonReadError` variants** (targeted + messages), NOT new Q-3-XX codes — the entire JSON reader path reports + plain `JsonReadError` without catalog codes today (`main.rs:314-317`), + and a one-off catalog integration for three new errors would be + inconsistent; reader-wide Q-code integration is a separate, + pre-existing gap (file a low-priority strand). The error-corpus item + previously listed under Phase 2 is dropped — that corpus is for merr + parse errors, not reader errors. + +### Phase 1 — tests first (TDD) — DONE 2026-07-17 + +- [x] `tests/integration/test_raw_json_roundtrip.rs`: AST-level identity + roundtrip covering **every** extension: standalone `Attr`, + `NoteReference`, all four CriticMarkup inlines, `Shortcode`, + `CaptionBlock`, `BlockMetadata`, both note-definition blocks, block + + inline `Custom` nodes; `ConfigValue` metadata with `Path`/`Glob`/ + `Expr`, `merge_op`, non-string scalars, map entry order; source-info + preservation incl. shared `Substring` parents (assert sharing survives: + structural equality of parent chains) and `Concat`/`Generated` +- [x] Issue #11's exact repro: `test_raw_json_roundtrip_issue_11_document` + parses the exact document with the qmd reader and roundtrips the + parser-produced AST. (Note learned en route: a *trailing* paragraph + `Attr` is representable in Pandoc-superset JSON via the para-`attr` + hoist, bd-aeyss6p5 — only a mid-paragraph Attr exercises Q-3-32.) +- [x] Marker tests: raw reader rejects Pandoc-style JSON with the targeted + diagnostic; rejects wrong version; `-f json` rejects raw-json input + (and vice versa) +- [x] Meta fidelity regression tests pinned to the table above +- [x] ~~Fixture dirs `tests/writers/raw-json/` + `tests/readers/raw-json/`~~ + Replaced by `test_raw_json_roundtrip_writer_fixture_corpus`: sweeps + the existing `tests/writers/json/*.md` corpus through a raw-json + identity roundtrip. Identity assertions subsume snapshots for this + format, and reusing the existing corpus avoids a parallel fixture + tree. +- [x] Ran tests before implementing; failed for the expected reasons + (unresolved `writers::raw_json`, missing `UnexpectedRawJsonMarker`) + +### Phase 2 — implementation — DONE 2026-07-17 + +- [x] Raw mode through `JsonConfig::raw` + streaming writer (marker-first + envelope; native tags in the eight extension arms via + `stream_write_span_like_raw` / `stream_write_shortcode_body`; + faithful `stream_write_config_value` raw branches incl. `m` merge-op + key and Q-3-57 for non-scalar YAML in `Scalar`) +- [x] Reader: marker validation (`read_raw_pandoc`) + raw arms guarded by + `SourceInfoDeserializer.raw` + faithful meta read-back; + pool/`read_ast_context` shared untouched (astContext gains + `exampleListCounter`, read leniently in both modes) +- [x] `writers/raw_json.rs` / `readers/raw_json.rs` thin public entry + points; registered in `readers/mod.rs`, `writers/mod.rs`, `main.rs` + reader+writer arms, `options.rs` format tables +- [x] **Bonus fidelity fixes found by the corpus sweep** (shared reader, + improves `-f json` too): `Link`/`Image` `target_source` (`targetS`) + and `Citation.id_source` (`citationIdS`) were written but dropped + on read-back; now restored (`read_target_source`, + `read_opt_source_ref`) with a dedicated regression test + (`test_raw_json_roundtrip_sidecar_source_infos`). +- [x] All 21 raw-json tests green; full workspace suite green + (10099 passed) + +### Phase 3 — end-to-end + docs + +- [x] End-to-end through the real binary (output inspected): + + ``` + $ echo 'Hello. {#free-floating-attribute} Here?' | pampa -t raw-json + {"pampa-json-format":{"version":1},"blocks":[{"c":[{"c":"Hello.","s":1,"t":"Str"}, + {"s":2,"t":"Space"},{"a":{...},"c":["free-floating-attribute",[],[]],"s":3,"t":"Attr"}, + {"s":5,"t":"Space"},{"c":"Here?","s":6,"t":"Str"}],"s":0,"t":"Para"}], + "meta":{"c":[],"s":7,"t":"MetaMap"},"astContext":{"files":[...], + "exampleListCounter":1,"p":[...]}} + ``` + + Piped back through `-f raw-json -t raw-json` twice: gen1 == gen2 == + gen3 **byte-identical** (stronger than the contract requires for + this document), and `-f raw-json -t qmd` reproduces + `Hello. {#free-floating-attribute} Here?` exactly. The old + `-t json` path still errors with Q-3-32 on the same input + (asserted in tests). +- [x] Full `cargo xtask verify` (WASM leg required — pampa is in the + hub-client dependency chain). First run caught the classic + out-of-workspace trap: `wasm-quarto-hub-client/src/lib.rs` had an + exhaustive `JsonConfig` initializer (fixed with struct-update + syntax). Second run: all legs green except the 20 "live" + `hub-mcp.test.ts` tests, which require `wss://sync.automerge.org` — + down on 2026-07-17 (unrelated to this change; they are connection + timeouts in quarto-hub-mcp, which has no pampa dependency). +- [ ] Design doc `claude-notes/designs/raw-json-format.md`: contract, + envelope, versioning policy, native tag vocabulary; cross-link from + `wire-format-source-info-codes.md` (raw-json is a second consumer of + the pool codes — same allocation policy applies) +- [ ] File low-priority strand: JSON reader errors lack Q-codes + (pre-existing; noted in Phase 0 decisions) +- [x] Comment findings on k-42 (done at design time) +- [ ] Close the loop on GH issue #11 (after review/merge) + +## Notes / references (2026-07-17 code study) + +- Writer: `crates/pampa/src/writers/json.rs` (~5200 lines; streaming + `stream_write_pandoc` :3926 is the production path; Q-3-32 emit :922-934; + pool `SourceInfoSerializer` :271; meta `write_config_value` :1662). +- Reader: `crates/pampa/src/readers/json.rs` (strict `read` :1267 / lenient + `read_completing_source_info` :1293; pool `SourceInfoDeserializer` :102; + `read_ast_context` :1191; no `pandoc-api-version` check :1310; unknown + `t` tags error, unknown top-level keys silently dropped). +- `config_json.rs` is a deliberately lossy projection for `q2 get-config` — + not reusable here. +- Filters exchange the current JSON superset (`json_filter.rs:172-227`) — + untouched by this plan. +- Dispatch sites: `options.rs:240-272`, `main.rs:245`/`:488`, + `readers/mod.rs`, `writers/mod.rs`; WASM + quarto-core hardcode the + current json module and are out of scope. +- Serde derives exist across the AST types but are test-only today; v1 of + this plan (see git history of this file) explored building on them and was + rejected for losing source-info sharing on read-back. diff --git a/crates/pampa/src/main.rs b/crates/pampa/src/main.rs index a5b3567b2..4e8ce2ee3 100644 --- a/crates/pampa/src/main.rs +++ b/crates/pampa/src/main.rs @@ -317,6 +317,18 @@ fn main() { } } } + "raw-json" => { + // Full-fidelity q2-internal format: strict source-info + // references, no post-read transforms — the AST comes back + // exactly as it was written. + match readers::raw_json::read(&mut input.as_bytes()) { + Ok((pandoc, context)) => (pandoc, context), + Err(e) => { + eprintln!("Error reading raw JSON: {}", e); + std::process::exit(1); + } + } + } "commonmark" => { // Use comrak-based CommonMark reader readers::commonmark::read(&input, input_filename) @@ -496,6 +508,16 @@ fn main() { }; writers::json::write_with_config(&pandoc, &context, &mut buf, &json_config) } + "raw-json" => { + let json_config = writers::json::JsonConfig { + include_inline_locations: args + .json_source_location + .as_ref() + .is_some_and(|s| s == "full"), + ..writers::json::JsonConfig::default() + }; + writers::raw_json::write_with_config(&pandoc, &context, &mut buf, &json_config) + } "native" => writers::native::write(&pandoc, &context, &mut buf), "markdown" | "qmd" => writers::qmd::write(&pandoc, &mut buf), "html" => { diff --git a/crates/pampa/src/options.rs b/crates/pampa/src/options.rs index c55b84905..a5ad18c68 100644 --- a/crates/pampa/src/options.rs +++ b/crates/pampa/src/options.rs @@ -237,11 +237,11 @@ pub fn merge_with_defaults(defaults: Value, user_opts: &Value) -> Value { // ============================================================================= /// Supported reader format names. -pub const SUPPORTED_READER_FORMATS: &[&str] = &["qmd", "markdown", "json"]; +pub const SUPPORTED_READER_FORMATS: &[&str] = &["qmd", "markdown", "json", "raw-json"]; /// Supported writer format names. pub const SUPPORTED_WRITER_FORMATS: &[&str] = &[ - "html", "html5", "json", "native", "markdown", "qmd", "plain", + "html", "html5", "json", "raw-json", "native", "markdown", "qmd", "plain", ]; /// Check if a format is supported for reading. diff --git a/crates/pampa/src/readers/json.rs b/crates/pampa/src/readers/json.rs index 4ef7bffc4..a73b8ee53 100644 --- a/crates/pampa/src/readers/json.rs +++ b/crates/pampa/src/readers/json.rs @@ -22,11 +22,12 @@ use crate::pandoc::ast_context::ASTContext; use crate::pandoc::location::{Location, Range}; use crate::pandoc::{ - Alignment, Attr, AttrSourceInfo, Block, BlockQuote, BulletList, Caption, Cell, Citation, - CitationMode, Cite, Code, CodeBlock, ColSpec, ColWidth, CustomNode, DefinitionList, Div, Emph, - Figure, Header, HorizontalRule, Image, Inline, InlineAttr, Inlines, LineBlock, Link, - ListAttributes, ListNumberDelim, ListNumberStyle, Math, MathType, MetaBlock, Note, OrderedList, - Pandoc, Paragraph, Plain, QuoteType, Quoted, RawBlock, RawInline, Row, Slot, SmallCaps, + Alignment, Attr, AttrSourceInfo, Block, BlockQuote, BulletList, Caption, CaptionBlock, Cell, + Citation, CitationMode, Cite, Code, CodeBlock, ColSpec, ColWidth, CustomNode, DefinitionList, + Delete, Div, EditComment, Emph, Figure, Header, Highlight, HorizontalRule, Image, Inline, + InlineAttr, Inlines, Insert, LineBlock, Link, ListAttributes, ListNumberDelim, ListNumberStyle, + Math, MathType, MetaBlock, Note, NoteReference, OrderedList, Pandoc, Paragraph, Plain, + QuoteType, Quoted, RawBlock, RawInline, Row, Shortcode, ShortcodeArg, Slot, SmallCaps, SoftBreak, Space, Span, Str, Strikeout, Strong, Subscript, Superscript, Table, TableBody, TableFoot, TableHead, Underline, }; @@ -57,6 +58,18 @@ pub enum JsonReadError { }, MalformedSourceInfoPool, CircularSourceInfoReference(usize), + /// The raw-json reader was fed a document without the + /// `pampa-json-format` marker. `has_pandoc_api_version` distinguishes + /// "this is Pandoc-style JSON" (point the user at `-f json`) from + /// "this is not a pampa document at all". + NotRawJson { + has_pandoc_api_version: bool, + }, + /// The raw-json reader was fed a marker version it does not support. + UnsupportedRawJsonVersion(u64), + /// The Pandoc-superset JSON reader was fed a raw-json document + /// (the `pampa-json-format` marker is present). + UnexpectedRawJsonMarker, } impl std::fmt::Display for JsonReadError { @@ -91,6 +104,35 @@ impl std::fmt::Display for JsonReadError { id ) } + JsonReadError::NotRawJson { + has_pandoc_api_version, + } => { + if *has_pandoc_api_version { + write!( + f, + "Input is Pandoc-style JSON (it has pandoc-api-version and no pampa-json-format marker); use -f json to read it" + ) + } else { + write!( + f, + "Not a pampa raw JSON document: the pampa-json-format marker is missing" + ) + } + } + JsonReadError::UnsupportedRawJsonVersion(version) => { + write!( + f, + "Unsupported pampa raw JSON format version {} (this reader supports version {})", + version, + crate::writers::raw_json::RAW_JSON_FORMAT_VERSION + ) + } + JsonReadError::UnexpectedRawJsonMarker => { + write!( + f, + "Input is pampa raw JSON (it has the pampa-json-format marker); use the raw-json reader (-f raw-json) instead" + ) + } } } } @@ -125,6 +167,15 @@ struct SourceInfoDeserializer { /// When `None`, the deserializer is in strict mode and missing `s:` /// is an error. This is the contract for q2-internal JSON. completing_default_by: Option, + + /// When true, the node readers accept the raw-json extension tags + /// (`Attr`, `NoteReference`, CriticMarkup inlines, `Shortcode`, + /// `CaptionBlock`, and the `Meta{Path,Glob,Expr,Int,Real,Null}` meta + /// kinds). The Pandoc-superset reader leaves this off so the two + /// formats stay honest about their vocabularies. Like + /// `completing_default_by`, this is reader-level configuration + /// threaded through the recursion alongside the pool. + raw: bool, } impl SourceInfoDeserializer { @@ -133,6 +184,7 @@ impl SourceInfoDeserializer { SourceInfoDeserializer { pool: Vec::new(), completing_default_by: None, + raw: false, } } @@ -467,6 +519,7 @@ impl SourceInfoDeserializer { Ok(SourceInfoDeserializer { pool, completing_default_by: None, + raw: false, }) } @@ -712,6 +765,36 @@ fn read_attr_source( }) } +/// Read an optional source-info ref: absent or `null` → `None`, a pool +/// id → `Some(SourceInfo)`. +fn read_opt_source_ref( + value: Option<&Value>, + deserializer: &SourceInfoDeserializer, +) -> Result> { + match value { + None => Ok(None), + Some(v) if v.is_null() => Ok(None), + Some(v) => Ok(Some(deserializer.from_json_ref(v)?)), + } +} + +/// Read a `targetS` sidecar (`[url_ref?, title_ref?]`) back into a +/// `TargetSourceInfo` — the inverse of the writer's +/// `stream_write_target_source`. Absent sidecar → empty (backward +/// compatibility with JSON that predates target-source tracking). +fn read_target_source( + value: Option<&Value>, + deserializer: &SourceInfoDeserializer, +) -> Result { + let Some(arr) = value.and_then(|v| v.as_array()) else { + return Ok(crate::pandoc::attr::TargetSourceInfo::empty()); + }; + Ok(crate::pandoc::attr::TargetSourceInfo { + url: read_opt_source_ref(arr.first(), deserializer)?, + title: read_opt_source_ref(arr.get(1), deserializer)?, + }) +} + fn read_citation_mode(value: &Value) -> Result { let obj = value.as_object().ok_or_else(|| { JsonReadError::InvalidType("Expected object for CitationMode".to_string()) @@ -976,7 +1059,7 @@ fn read_inline(value: &Value, deserializer: &SourceInfoDeserializer) -> Result { @@ -1052,7 +1135,7 @@ fn read_inline(value: &Value, deserializer: &SourceInfoDeserializer) -> Result { @@ -1163,7 +1246,10 @@ fn read_inline(value: &Value, deserializer: &SourceInfoDeserializer) -> Result>>()?; @@ -1177,10 +1263,226 @@ fn read_inline(value: &Value, deserializer: &SourceInfoDeserializer) -> Result { + let c = obj + .get("c") + .ok_or_else(|| JsonReadError::MissingField("c".to_string()))?; + let id = c + .as_str() + .ok_or_else(|| { + JsonReadError::InvalidType("NoteReference content must be string".to_string()) + })? + .to_string(); + Ok(Inline::NoteReference(NoteReference { id, source_info })) + } + "Attr" if deserializer.raw => { + let c = obj + .get("c") + .ok_or_else(|| JsonReadError::MissingField("c".to_string()))?; + let attr = read_attr(c)?; + let attr_source = read_attr_source(obj.get("a"), deserializer)?; + Ok(Inline::Attr(InlineAttr { + attr, + attr_source, + source_info, + })) + } + "Insert" | "Delete" | "Highlight" | "EditComment" if deserializer.raw => { + // Span-shaped payload: c = [attr, inlines], a = attr sources. + let c = obj + .get("c") + .ok_or_else(|| JsonReadError::MissingField("c".to_string()))?; + let arr = c.as_array().ok_or_else(|| { + JsonReadError::InvalidType(format!("{} content must be array", t)) + })?; + if arr.len() != 2 { + return Err(JsonReadError::InvalidType(format!( + "{} array must have 2 elements", + t + ))); + } + let attr = read_attr(&arr[0])?; + let content = read_inlines(&arr[1], deserializer)?; + let attr_source = read_attr_source(obj.get("a"), deserializer)?; + Ok(match t { + "Insert" => Inline::Insert(Insert { + attr, + content, + source_info, + attr_source, + }), + "Delete" => Inline::Delete(Delete { + attr, + content, + source_info, + attr_source, + }), + "Highlight" => Inline::Highlight(Highlight { + attr, + content, + source_info, + attr_source, + }), + "EditComment" => Inline::EditComment(EditComment { + attr, + content, + source_info, + attr_source, + }), + _ => unreachable!("guarded by the match arm pattern"), + }) + } + "Shortcode" if deserializer.raw => { + let c = obj + .get("c") + .ok_or_else(|| JsonReadError::MissingField("c".to_string()))?; + Ok(Inline::Shortcode(read_raw_shortcode_body( + c, + source_info, + deserializer, + )?)) + } _ => Err(JsonReadError::UnsupportedVariant(format!("Inline: {}", t))), } } +/// Raw mode: read a Shortcode body object +/// `{isEscaped, keywordArgs, name, positionalArgs}` (the inverse of the +/// writer's `stream_write_shortcode_body`). `source_info` comes from the +/// enclosing `{c, s, t}` node. +fn read_raw_shortcode_body( + value: &Value, + source_info: quarto_source_map::SourceInfo, + deserializer: &SourceInfoDeserializer, +) -> Result { + let obj = value.as_object().ok_or_else(|| { + JsonReadError::InvalidType("Shortcode content must be object".to_string()) + })?; + + let is_escaped = obj + .get("isEscaped") + .and_then(|v| v.as_bool()) + .ok_or_else(|| JsonReadError::MissingField("isEscaped".to_string()))?; + let name = obj + .get("name") + .and_then(|v| v.as_str()) + .ok_or_else(|| JsonReadError::MissingField("name".to_string()))? + .to_string(); + + let positional_args = obj + .get("positionalArgs") + .and_then(|v| v.as_array()) + .ok_or_else(|| JsonReadError::MissingField("positionalArgs".to_string()))? + .iter() + .map(|arg| read_raw_shortcode_arg(arg, deserializer)) + .collect::>>()?; + + let mut keyword_args = LinkedHashMap::new(); + for pair in obj + .get("keywordArgs") + .and_then(|v| v.as_array()) + .ok_or_else(|| JsonReadError::MissingField("keywordArgs".to_string()))? + { + let pair_arr = pair.as_array().ok_or_else(|| { + JsonReadError::InvalidType("keywordArgs entry must be [key, arg]".to_string()) + })?; + if pair_arr.len() != 2 { + return Err(JsonReadError::InvalidType( + "keywordArgs entry must have 2 elements".to_string(), + )); + } + let key = pair_arr[0] + .as_str() + .ok_or_else(|| { + JsonReadError::InvalidType("keywordArgs key must be string".to_string()) + })? + .to_string(); + keyword_args.insert(key, read_raw_shortcode_arg(&pair_arr[1], deserializer)?); + } + + Ok(Shortcode { + is_escaped, + name, + positional_args, + keyword_args, + source_info, + }) +} + +/// Raw mode: read one tagged ShortcodeArg (the inverse of the writer's +/// `stream_write_shortcode_arg`). +fn read_raw_shortcode_arg( + value: &Value, + deserializer: &SourceInfoDeserializer, +) -> Result { + let obj = value + .as_object() + .ok_or_else(|| JsonReadError::InvalidType("ShortcodeArg must be object".to_string()))?; + let t = obj + .get("t") + .and_then(|v| v.as_str()) + .ok_or_else(|| JsonReadError::MissingField("t".to_string()))?; + let c = obj + .get("c") + .ok_or_else(|| JsonReadError::MissingField("c".to_string()))?; + + match t { + "String" => Ok(ShortcodeArg::String( + c.as_str() + .ok_or_else(|| { + JsonReadError::InvalidType("String arg content must be string".to_string()) + })? + .to_string(), + )), + "Number" => Ok(ShortcodeArg::Number(c.as_f64().ok_or_else(|| { + JsonReadError::InvalidType("Number arg content must be number".to_string()) + })?)), + "Boolean" => Ok(ShortcodeArg::Boolean(c.as_bool().ok_or_else(|| { + JsonReadError::InvalidType("Boolean arg content must be boolean".to_string()) + })?)), + "Shortcode" => { + let source_info = + deserializer.resolve_source_info(obj.get("s"), "ShortcodeArg.Shortcode")?; + Ok(ShortcodeArg::Shortcode(read_raw_shortcode_body( + c, + source_info, + deserializer, + )?)) + } + "KeyValue" => { + let mut map = std::collections::HashMap::new(); + for pair in c.as_array().ok_or_else(|| { + JsonReadError::InvalidType("KeyValue arg content must be array".to_string()) + })? { + let pair_arr = pair.as_array().ok_or_else(|| { + JsonReadError::InvalidType("KeyValue entry must be [key, arg]".to_string()) + })?; + if pair_arr.len() != 2 { + return Err(JsonReadError::InvalidType( + "KeyValue entry must have 2 elements".to_string(), + )); + } + let key = pair_arr[0] + .as_str() + .ok_or_else(|| { + JsonReadError::InvalidType("KeyValue key must be string".to_string()) + })? + .to_string(); + map.insert(key, read_raw_shortcode_arg(&pair_arr[1], deserializer)?); + } + Ok(ShortcodeArg::KeyValue(map)) + } + other => Err(JsonReadError::UnsupportedVariant(format!( + "ShortcodeArg: {}", + other + ))), + } +} + fn read_inlines(value: &Value, deserializer: &SourceInfoDeserializer) -> Result { let arr = value .as_array() @@ -1246,9 +1548,16 @@ fn read_ast_context(value: &Value) -> Result { } } + // Raw-json envelopes carry the example-list counter; Pandoc-superset + // documents don't have it and get the fresh-context default of 1. + let example_list_counter = obj + .get("exampleListCounter") + .and_then(|v| v.as_u64()) + .unwrap_or(1) as usize; + Ok(ASTContext { filenames, - example_list_counter: std::cell::Cell::new(1), + example_list_counter: std::cell::Cell::new(example_list_counter), source_context, parent_source_info: None, }) @@ -1307,6 +1616,13 @@ fn read_pandoc(value: &Value, completing_default_by: Option) -> Result<(Pand .as_object() .ok_or_else(|| JsonReadError::InvalidType("Expected object for Pandoc".to_string()))?; + // A pampa-json-format marker means this is a raw-json document; its + // vocabulary is a superset of ours, so reject with a pointer to the + // right reader instead of failing later on an unknown tag. + if obj.contains_key("pampa-json-format") { + return Err(JsonReadError::UnexpectedRawJsonMarker); + } + // We could validate the API version here if needed // let _api_version = obj.get("pandoc-api-version"); @@ -1361,6 +1677,72 @@ fn read_pandoc(value: &Value, completing_default_by: Option) -> Result<(Pand Ok((Pandoc { meta, blocks }, context)) } +/// Read a raw-json document (see `writers::raw_json` for the format +/// contract). Called by [`crate::readers::raw_json::read`]. +/// +/// Differences from [`read_pandoc`]: +/// - the `pampa-json-format` marker is required and its version checked; +/// - the node readers run with the raw vocabulary enabled; +/// - `meta` is a single config-value node (order-preserving entries with +/// inline key sources), not a Pandoc-style sorted object; +/// - always strict about `s:` references — raw-json is q2-internal. +pub(crate) fn read_raw_pandoc(value: &Value) -> Result<(Pandoc, ASTContext)> { + let obj = value + .as_object() + .ok_or_else(|| JsonReadError::InvalidType("Expected object for Pandoc".to_string()))?; + + match obj.get("pampa-json-format") { + None => { + return Err(JsonReadError::NotRawJson { + has_pandoc_api_version: obj.contains_key("pandoc-api-version"), + }); + } + Some(marker) => { + let version = marker + .get("version") + .and_then(|v| v.as_u64()) + .ok_or_else(|| { + JsonReadError::InvalidType( + "pampa-json-format.version must be a number".to_string(), + ) + })?; + if version != crate::writers::raw_json::RAW_JSON_FORMAT_VERSION { + return Err(JsonReadError::UnsupportedRawJsonVersion(version)); + } + } + } + + let context = if let Some(ast_context_val) = obj.get("astContext") { + read_ast_context(ast_context_val)? + } else { + ASTContext::new() + }; + + let mut deserializer = if let Some(pool_json) = obj + .get("astContext") + .and_then(|v| v.as_object()) + .and_then(|o| o.get("p")) + { + SourceInfoDeserializer::new(pool_json)? + } else { + SourceInfoDeserializer::empty() + }; + deserializer.raw = true; + + let meta = read_config_value( + obj.get("meta") + .ok_or_else(|| JsonReadError::MissingField("meta".to_string()))?, + &deserializer, + )?; + let blocks = read_blocks( + obj.get("blocks") + .ok_or_else(|| JsonReadError::MissingField("blocks".to_string()))?, + &deserializer, + )?; + + Ok((Pandoc { meta, blocks }, context)) +} + fn read_blockss(value: &Value, deserializer: &SourceInfoDeserializer) -> Result>> { let arr = value .as_array() @@ -2231,6 +2613,18 @@ fn read_block(value: &Value, deserializer: &SourceInfoDeserializer) -> Result { + let c = obj + .get("c") + .ok_or_else(|| JsonReadError::MissingField("c".to_string()))?; + let content = read_inlines(c, deserializer)?; + Ok(Block::CaptionBlock(CaptionBlock { + content, + source_info, + })) + } _ => Err(JsonReadError::UnsupportedVariant(format!("Block: {}", t))), } } @@ -2306,7 +2700,19 @@ fn read_config_value(value: &Value, deserializer: &SourceInfoDeserializer) -> Re let source_info = deserializer.resolve_source_info(obj.get("s"), &format!("ConfigValue.{}", t))?; - let merge_op = quarto_pandoc_types::MergeOp::default(); + // `m` carries a non-default merge op (raw-json only; the writer omits + // it for the default). Absent key → default, matching the writer. + let merge_op = match obj.get("m").and_then(|v| v.as_str()) { + Some("prefer") => quarto_pandoc_types::MergeOp::Prefer, + Some("concat") => quarto_pandoc_types::MergeOp::Concat, + Some(other) => { + return Err(JsonReadError::InvalidType(format!( + "Unknown merge op: {}", + other + ))); + } + None => quarto_pandoc_types::MergeOp::default(), + }; match t { "MetaString" => { @@ -2434,6 +2840,64 @@ fn read_config_value(value: &Value, deserializer: &SourceInfoDeserializer) -> Re merge_op, }) } + // Raw-json meta vocabulary (bd-en2hvrwn): faithful encodings for + // the kinds the Pandoc-style tags collapse. + "MetaInt" if deserializer.raw => { + let c = obj.get("c").and_then(|v| v.as_i64()).ok_or_else(|| { + JsonReadError::InvalidType("MetaInt content must be integer".to_string()) + })?; + Ok(ConfigValue { + value: ConfigValueKind::Scalar(yaml_rust2::Yaml::Integer(c)), + source_info, + merge_op, + }) + } + "MetaReal" if deserializer.raw => { + // yaml_rust2 reals are raw source strings; carried verbatim. + let c = obj.get("c").and_then(|v| v.as_str()).ok_or_else(|| { + JsonReadError::InvalidType("MetaReal content must be string".to_string()) + })?; + Ok(ConfigValue { + value: ConfigValueKind::Scalar(yaml_rust2::Yaml::Real(c.to_string())), + source_info, + merge_op, + }) + } + "MetaNull" if deserializer.raw => Ok(ConfigValue { + value: ConfigValueKind::Scalar(yaml_rust2::Yaml::Null), + source_info, + merge_op, + }), + "MetaPath" if deserializer.raw => { + let c = obj.get("c").and_then(|v| v.as_str()).ok_or_else(|| { + JsonReadError::InvalidType("MetaPath content must be string".to_string()) + })?; + Ok(ConfigValue { + value: ConfigValueKind::Path(c.to_string()), + source_info, + merge_op, + }) + } + "MetaGlob" if deserializer.raw => { + let c = obj.get("c").and_then(|v| v.as_str()).ok_or_else(|| { + JsonReadError::InvalidType("MetaGlob content must be string".to_string()) + })?; + Ok(ConfigValue { + value: ConfigValueKind::Glob(c.to_string()), + source_info, + merge_op, + }) + } + "MetaExpr" if deserializer.raw => { + let c = obj.get("c").and_then(|v| v.as_str()).ok_or_else(|| { + JsonReadError::InvalidType("MetaExpr content must be string".to_string()) + })?; + Ok(ConfigValue { + value: ConfigValueKind::Expr(c.to_string()), + source_info, + merge_op, + }) + } _ => Err(JsonReadError::UnsupportedVariant(format!( "MetaValue: {}", t diff --git a/crates/pampa/src/readers/mod.rs b/crates/pampa/src/readers/mod.rs index 3ad50beed..cfaca5418 100644 --- a/crates/pampa/src/readers/mod.rs +++ b/crates/pampa/src/readers/mod.rs @@ -8,3 +8,4 @@ pub mod json; pub mod qmd; pub mod qmd_error_message_table; pub mod qmd_error_messages; +pub mod raw_json; diff --git a/crates/pampa/src/readers/raw_json.rs b/crates/pampa/src/readers/raw_json.rs new file mode 100644 index 000000000..0543b93b9 --- /dev/null +++ b/crates/pampa/src/readers/raw_json.rs @@ -0,0 +1,35 @@ +/* + * raw_json.rs + * Copyright (c) 2026 Posit, PBC + */ + +//! The pampa-native `raw-json` reader (GH #11, bd-en2hvrwn) — the +//! inverse of [`crate::writers::raw_json`]. +//! +//! Validates the `pampa-json-format` envelope marker (version-checked; +//! Pandoc-style JSON is rejected with a pointer at `-f json`), then reads +//! the document with the shared JSON machinery in raw mode: the full +//! extension vocabulary is accepted and metadata is decoded as an +//! order-preserving config-value node. +//! +//! Always strict about `s:` source references — raw-json is q2-internal, +//! and every writer-produced node carries one. There is no completing +//! variant; JSON from outside the q2 source-tracking world goes through +//! the Pandoc-superset reader instead. + +use crate::pandoc::ASTContext; +use crate::pandoc::Pandoc; + +use super::json::{JsonReadError, read_raw_pandoc}; + +/// Read a raw-json document. +pub fn read(reader: &mut R) -> Result<(Pandoc, ASTContext), JsonReadError> { + let value: serde_json::Value = + serde_json::from_reader(reader).map_err(JsonReadError::InvalidJson)?; + read_raw_pandoc(&value) +} + +/// Read a raw-json document from an already-parsed [`serde_json::Value`]. +pub fn read_value(value: &serde_json::Value) -> Result<(Pandoc, ASTContext), JsonReadError> { + read_raw_pandoc(value) +} diff --git a/crates/pampa/src/writers/json.rs b/crates/pampa/src/writers/json.rs index dcd8a2a7e..a7b88a18e 100644 --- a/crates/pampa/src/writers/json.rs +++ b/crates/pampa/src/writers/json.rs @@ -60,6 +60,24 @@ pub struct JsonConfig { /// by `attribution_by_node`, including warning-path placeholders. /// Emitted as `astContext.attributionActors` when any entry is used. pub attribution_actors: Option, JsonAttributionIdentity>>>, + + /// If true, emit the pampa-native `raw-json` format (bd-en2hvrwn, + /// GH #11) instead of the Pandoc-superset shape: + /// + /// - the `pampa-json-format` envelope marker is emitted first and + /// `pandoc-api-version` is omitted; + /// - pampa AST extensions (standalone `Attr`, `NoteReference`, + /// CriticMarkup inlines, `Shortcode`, `CaptionBlock`) are written + /// with native tags instead of being desugared or rejected; + /// - metadata is written faithfully (full `ConfigValue`: Path/Glob/ + /// Expr kinds, merge ops, scalar types, entry order) as a single + /// config-value node instead of a sorted Pandoc-style meta object; + /// - `astContext` carries `exampleListCounter`. + /// + /// The contract is that write-then-read is the identity on the AST. + /// Only the streaming writer implements raw mode; use the + /// `writers::raw_json` / `readers::raw_json` entry points. + pub raw: bool, } // ============================================================================ @@ -1755,11 +1773,17 @@ fn write_blocks(blocks: &[Block], ctx: &mut JsonWriterContext) -> Value { /// Generate JSON representation of a Pandoc document. /// /// This function is used internally by the HTML writer to build the source map. +/// Raw mode (`JsonConfig::raw`) is implemented only on the streaming path; +/// this function's callers never set it (asserted below). pub(crate) fn write_pandoc( pandoc: &Pandoc, ast_context: &ASTContext, config: &JsonConfig, ) -> Result> { + debug_assert!( + !config.raw, + "raw mode is only implemented on the streaming writer; use writers::raw_json" + ); // Create the JSON writer context let mut ctx = JsonWriterContext::new(ast_context, config); @@ -2279,6 +2303,140 @@ where Ok(()) } +/// Raw mode: emit a Shortcode's body object +/// `{isEscaped, keywordArgs, name, positionalArgs}` (alphabetical). +/// +/// `keywordArgs` is an array of `[key, arg]` pairs in `LinkedHashMap` +/// insertion order (the qmd writer relies on that order for source-order +/// roundtrips; so do we). +fn stream_write_shortcode_body( + w: &mut JsonStreamWriter, + shortcode: &quarto_pandoc_types::Shortcode, + ctx: &mut JsonWriterContext, +) -> io::Result<()> { + w.begin_object()?; + w.key("isEscaped")?; + w.bool_value(shortcode.is_escaped)?; + w.key("keywordArgs")?; + w.begin_array()?; + for (key, arg) in &shortcode.keyword_args { + w.begin_array()?; + w.str_value(key)?; + stream_write_shortcode_arg(w, arg, ctx)?; + w.end_array()?; + } + w.end_array()?; + w.key("name")?; + w.str_value(&shortcode.name)?; + w.key("positionalArgs")?; + w.begin_array()?; + for arg in &shortcode.positional_args { + stream_write_shortcode_arg(w, arg, ctx)?; + } + w.end_array()?; + w.end_object()?; + Ok(()) +} + +/// Raw mode: emit one ShortcodeArg as a tagged node. +/// +/// `String`/`Number`/`Boolean` are `{c, t}`; a nested `Shortcode` is a +/// full `{c, s, t}` node (its own source info interned into the pool); +/// `KeyValue` is `{c: [[key, arg]...], t}` with entries sorted by key — +/// the underlying `HashMap` has no stable order, and sorting keeps the +/// output deterministic (map equality on read-back is order-independent). +fn stream_write_shortcode_arg( + w: &mut JsonStreamWriter, + arg: &quarto_pandoc_types::ShortcodeArg, + ctx: &mut JsonWriterContext, +) -> io::Result<()> { + use quarto_pandoc_types::ShortcodeArg; + match arg { + ShortcodeArg::String(s) => { + w.begin_object()?; + w.key("c")?; + w.str_value(s)?; + w.key("t")?; + w.str_value("String")?; + w.end_object()?; + Ok(()) + } + ShortcodeArg::Number(n) => { + w.begin_object()?; + w.key("c")?; + w.f64_value(*n)?; + w.key("t")?; + w.str_value("Number")?; + w.end_object()?; + Ok(()) + } + ShortcodeArg::Boolean(b) => { + w.begin_object()?; + w.key("c")?; + w.bool_value(*b)?; + w.key("t")?; + w.str_value("Boolean")?; + w.end_object()?; + Ok(()) + } + ShortcodeArg::Shortcode(sc) => stream_write_simple_node( + w, + "Shortcode", + &sc.source_info, + ctx, + |w: &mut JsonStreamWriter, ctx: &mut JsonWriterContext| { + stream_write_shortcode_body(w, sc, ctx) + }, + ), + ShortcodeArg::KeyValue(map) => { + let mut keys: Vec<&String> = map.keys().collect(); + keys.sort(); + w.begin_object()?; + w.key("c")?; + w.begin_array()?; + for key in keys { + w.begin_array()?; + w.str_value(key)?; + stream_write_shortcode_arg(w, &map[key], ctx)?; + w.end_array()?; + } + w.end_array()?; + w.key("t")?; + w.str_value("KeyValue")?; + w.end_object()?; + Ok(()) + } + } +} + +/// Raw mode: emit a Span-shaped extension node `{a, c: [attr, inlines], s, t}`. +/// Used for the four CriticMarkup inlines, whose payload is exactly a +/// Span's (attr + inline content) under their own tag. +fn stream_write_span_like_raw( + w: &mut JsonStreamWriter, + type_name: &str, + attr: &Attr, + content: &Vec, + source_info: &SourceInfo, + attr_source: &AttrSourceInfo, + ctx: &mut JsonWriterContext, +) -> io::Result<()> { + stream_write_attrs_node( + w, + type_name, + source_info, + attr_source, + ctx, + |w: &mut JsonStreamWriter, ctx: &mut JsonWriterContext| { + w.begin_array()?; + stream_write_attr(w, attr)?; + stream_write_inlines(w, content, ctx)?; + w.end_array()?; + Ok(()) + }, + ) +} + /// Emit an Inlines array: `[...]`. fn stream_write_inlines( w: &mut JsonStreamWriter, @@ -2866,6 +3024,17 @@ fn stream_write_inline( }, ), Inline::Shortcode(shortcode) => { + if ctx.serializer.config.raw { + return stream_write_simple_node( + w, + "Shortcode", + &shortcode.source_info, + ctx, + |w: &mut JsonStreamWriter, ctx: &mut JsonWriterContext| { + stream_write_shortcode_body(w, shortcode, ctx) + }, + ); + } let span = shortcode_to_span(shortcode.clone()); let attr = ( span.attr.0.clone(), @@ -2887,6 +3056,17 @@ fn stream_write_inline( ) } Inline::NoteReference(note_ref) => { + if ctx.serializer.config.raw { + return stream_write_simple_node( + w, + "NoteReference", + ¬e_ref.source_info, + ctx, + |w: &mut JsonStreamWriter, _ctx: &mut JsonWriterContext| { + w.str_value(¬e_ref.id) + }, + ); + } ctx.errors.push( DiagnosticMessageBuilder::error("Unprocessed note reference in JSON writer") .with_code("Q-3-31") @@ -2918,6 +3098,18 @@ fn stream_write_inline( ) } Inline::Attr(inline_attr) => { + if ctx.serializer.config.raw { + return stream_write_attrs_node( + w, + "Attr", + &inline_attr.source_info, + &inline_attr.attr_source, + ctx, + |w: &mut JsonStreamWriter, _ctx: &mut JsonWriterContext| { + stream_write_attr(w, &inline_attr.attr) + }, + ); + } ctx.errors.push( DiagnosticMessageBuilder::error( "Standalone attribute not supported in JSON format", @@ -2939,6 +3131,17 @@ fn stream_write_inline( Ok(()) } Inline::Insert(ins) => { + if ctx.serializer.config.raw { + return stream_write_span_like_raw( + w, + "Insert", + &ins.attr, + &ins.content, + &ins.source_info, + &ins.attr_source, + ctx, + ); + } ctx.errors.push( DiagnosticMessageBuilder::error("Unprocessed Insert markup in JSON writer") .with_code("Q-3-33") @@ -2968,6 +3171,17 @@ fn stream_write_inline( ) } Inline::Delete(del) => { + if ctx.serializer.config.raw { + return stream_write_span_like_raw( + w, + "Delete", + &del.attr, + &del.content, + &del.source_info, + &del.attr_source, + ctx, + ); + } ctx.errors.push( DiagnosticMessageBuilder::error("Unprocessed Delete markup in JSON writer") .with_code("Q-3-34") @@ -2997,6 +3211,17 @@ fn stream_write_inline( ) } Inline::Highlight(hl) => { + if ctx.serializer.config.raw { + return stream_write_span_like_raw( + w, + "Highlight", + &hl.attr, + &hl.content, + &hl.source_info, + &hl.attr_source, + ctx, + ); + } ctx.errors.push( DiagnosticMessageBuilder::error("Unprocessed Highlight markup in JSON writer") .with_code("Q-3-35") @@ -3026,6 +3251,17 @@ fn stream_write_inline( ) } Inline::EditComment(ec) => { + if ctx.serializer.config.raw { + return stream_write_span_like_raw( + w, + "EditComment", + &ec.attr, + &ec.content, + &ec.source_info, + &ec.attr_source, + ctx, + ); + } ctx.errors.push( DiagnosticMessageBuilder::error("Unprocessed EditComment markup in JSON writer") .with_code("Q-3-36") @@ -3406,6 +3642,17 @@ fn stream_write_block( }, ), Block::CaptionBlock(caption) => { + if ctx.serializer.config.raw { + return stream_write_simple_node( + w, + "CaptionBlock", + &caption.source_info, + ctx, + |w: &mut JsonStreamWriter, ctx: &mut JsonWriterContext| { + stream_write_inlines(w, &caption.content, ctx) + }, + ); + } ctx.errors.push( DiagnosticMessageBuilder::error("Orphaned caption block in JSON writer") .with_code("Q-3-21") @@ -3649,20 +3896,33 @@ fn stream_write_custom_inline( } /// Emit a meta node `{c, s, t}` with alphabetical key ordering. +/// Emit a config-value node: `{c, m?, s, t}` (alphabetical). +/// +/// `s` comes from the value's own `source_info`. In raw mode, a +/// non-default merge op is carried in `m` ("prefer"; the default, +/// "concat", is omitted to keep output lean). The Pandoc-superset mode +/// never emits `m` — merge ops are not representable there. fn stream_write_meta_node( w: &mut JsonStreamWriter, type_name: &str, - source_info: &SourceInfo, + value: &ConfigValue, ctx: &mut JsonWriterContext, content: FC, ) -> io::Result<()> where FC: FnOnce(&mut JsonStreamWriter, &mut JsonWriterContext) -> io::Result<()>, { - let s_id = ctx.serializer.intern(source_info); + let s_id = ctx.serializer.intern(&value.source_info); w.begin_object()?; w.key("c")?; content(w, ctx)?; + if ctx.serializer.config.raw && value.merge_op != quarto_pandoc_types::MergeOp::default() { + w.key("m")?; + w.str_value(match value.merge_op { + quarto_pandoc_types::MergeOp::Prefer => "prefer", + quarto_pandoc_types::MergeOp::Concat => "concat", + })?; + } w.key("s")?; w.u64_value(s_id as u64)?; w.key("t")?; @@ -3676,68 +3936,97 @@ fn stream_write_config_value( value: &ConfigValue, ctx: &mut JsonWriterContext, ) -> io::Result<()> { + let raw = ctx.serializer.config.raw; match &value.value { ConfigValueKind::Scalar(yaml) => match yaml { yaml_rust2::Yaml::String(s) => { - stream_write_meta_node(w, "MetaString", &value.source_info, ctx, |w, _ctx| { - w.str_value(s) - }) + stream_write_meta_node(w, "MetaString", value, ctx, |w, _ctx| w.str_value(s)) } yaml_rust2::Yaml::Boolean(b) => { - stream_write_meta_node(w, "MetaBool", &value.source_info, ctx, |w, _ctx| { - w.bool_value(*b) - }) + stream_write_meta_node(w, "MetaBool", value, ctx, |w, _ctx| w.bool_value(*b)) + } + yaml_rust2::Yaml::Integer(i) if raw => { + stream_write_meta_node(w, "MetaInt", value, ctx, |w, _ctx| w.i64_value(*i)) } yaml_rust2::Yaml::Integer(i) => { let text = i.to_string(); - stream_write_meta_node(w, "MetaString", &value.source_info, ctx, |w, _ctx| { - w.str_value(&text) - }) + stream_write_meta_node(w, "MetaString", value, ctx, |w, _ctx| w.str_value(&text)) + } + // yaml_rust2 stores reals as their raw source string; carrying + // the string keeps the raw roundtrip byte-faithful. + yaml_rust2::Yaml::Real(r) if raw => { + stream_write_meta_node(w, "MetaReal", value, ctx, |w, _ctx| w.str_value(r)) } yaml_rust2::Yaml::Real(r) => { - stream_write_meta_node(w, "MetaString", &value.source_info, ctx, |w, _ctx| { - w.str_value(r) - }) + stream_write_meta_node(w, "MetaString", value, ctx, |w, _ctx| w.str_value(r)) + } + yaml_rust2::Yaml::Null if raw => { + stream_write_meta_node(w, "MetaNull", value, ctx, |w, _ctx| w.null_value()) } yaml_rust2::Yaml::Null => { - stream_write_meta_node(w, "MetaString", &value.source_info, ctx, |w, _ctx| { - w.str_value("") - }) + stream_write_meta_node(w, "MetaString", value, ctx, |w, _ctx| w.str_value("")) } - _ => stream_write_meta_node(w, "MetaString", &value.source_info, ctx, |w, _ctx| { - w.str_value("") - }), + other if raw => { + // Raw mode must not silently degrade: a Scalar holding a + // non-scalar YAML value has no faithful encoding, so fail + // the write loudly. + ctx.errors.push( + DiagnosticMessageBuilder::error( + "Unserializable YAML scalar in raw JSON writer", + ) + .with_code("Q-3-57") + .with_location(value.source_info.clone()) + .problem(format!( + "Scalar metadata holds a non-scalar YAML value ({:?}), which raw-json cannot represent faithfully", + std::mem::discriminant(other) + )) + .add_detail("Arrays and maps should use ConfigValueKind::Array / Map, not Scalar") + .add_hint("This may indicate a bug in metadata construction") + .build(), + ); + stream_write_meta_node(w, "MetaNull", value, ctx, |w, _ctx| w.null_value()) + } + _ => stream_write_meta_node(w, "MetaString", value, ctx, |w, _ctx| w.str_value("")), }, ConfigValueKind::PandocInlines(inlines) => { - stream_write_meta_node(w, "MetaInlines", &value.source_info, ctx, |w, ctx| { + stream_write_meta_node(w, "MetaInlines", value, ctx, |w, ctx| { stream_write_inlines(w, inlines, ctx) }) } ConfigValueKind::PandocBlocks(blocks) => { - stream_write_meta_node(w, "MetaBlocks", &value.source_info, ctx, |w, ctx| { + stream_write_meta_node(w, "MetaBlocks", value, ctx, |w, ctx| { stream_write_blocks(w, blocks, ctx) }) } + ConfigValueKind::Path(p) if raw => { + stream_write_meta_node(w, "MetaPath", value, ctx, |w, _ctx| w.str_value(p)) + } ConfigValueKind::Path(p) => { let inlines = build_path_inlines(p, &value.source_info); - stream_write_meta_node(w, "MetaInlines", &value.source_info, ctx, |w, ctx| { + stream_write_meta_node(w, "MetaInlines", value, ctx, |w, ctx| { stream_write_inlines(w, &inlines, ctx) }) } + ConfigValueKind::Glob(g) if raw => { + stream_write_meta_node(w, "MetaGlob", value, ctx, |w, _ctx| w.str_value(g)) + } ConfigValueKind::Glob(g) => { let inlines = build_glob_inlines(g, &value.source_info); - stream_write_meta_node(w, "MetaInlines", &value.source_info, ctx, |w, ctx| { + stream_write_meta_node(w, "MetaInlines", value, ctx, |w, ctx| { stream_write_inlines(w, &inlines, ctx) }) } + ConfigValueKind::Expr(e) if raw => { + stream_write_meta_node(w, "MetaExpr", value, ctx, |w, _ctx| w.str_value(e)) + } ConfigValueKind::Expr(e) => { let inlines = build_expr_inlines(e, &value.source_info); - stream_write_meta_node(w, "MetaInlines", &value.source_info, ctx, |w, ctx| { + stream_write_meta_node(w, "MetaInlines", value, ctx, |w, ctx| { stream_write_inlines(w, &inlines, ctx) }) } ConfigValueKind::Array(items) => { - stream_write_meta_node(w, "MetaList", &value.source_info, ctx, |w, ctx| { + stream_write_meta_node(w, "MetaList", value, ctx, |w, ctx| { w.begin_array()?; for item in items { stream_write_config_value(w, item, ctx)?; @@ -3747,7 +4036,7 @@ fn stream_write_config_value( }) } ConfigValueKind::Map(entries) => { - stream_write_meta_node(w, "MetaMap", &value.source_info, ctx, |w, ctx| { + stream_write_meta_node(w, "MetaMap", value, ctx, |w, ctx| { w.begin_array()?; for entry in entries { w.begin_object()?; @@ -3933,16 +4222,36 @@ fn stream_write_pandoc( let res: io::Result<()> = (|| { w.begin_object()?; + if config.raw { + // The marker MUST be the first key: it is the format's + // self-identification, visible in the first line of output. + // `pandoc-api-version` is deliberately absent in raw mode so + // Pandoc-JSON consumers fail fast instead of half-parsing. + w.key("pampa-json-format")?; + w.begin_object()?; + w.key("version")?; + w.u64_value(crate::writers::raw_json::RAW_JSON_FORMAT_VERSION)?; + w.end_object()?; + } w.key("blocks")?; stream_write_blocks(w, &pandoc.blocks, &mut ctx)?; w.key("meta")?; - stream_write_config_value_as_meta(w, &pandoc.meta, &mut ctx)?; - w.key("pandoc-api-version")?; - w.begin_array()?; - w.u64_value(1)?; - w.u64_value(23)?; - w.u64_value(1)?; - w.end_array()?; + if config.raw { + // Full-fidelity meta: a single config-value node (entry order, + // key sources, merge ops, and the top-level value's own + // source info all inline). The Pandoc-style sorted object + // cannot preserve entry order on read (serde_json maps are + // BTreeMaps), hence the array-of-entries encoding. + stream_write_config_value(w, &pandoc.meta, &mut ctx)?; + } else { + stream_write_config_value_as_meta(w, &pandoc.meta, &mut ctx)?; + w.key("pandoc-api-version")?; + w.begin_array()?; + w.u64_value(1)?; + w.u64_value(23)?; + w.u64_value(1)?; + w.end_array()?; + } w.key("astContext")?; w.begin_object()?; // files (alphabetical: files, metaTopLevelKeySources?, p) @@ -3973,8 +4282,17 @@ fn stream_write_pandoc( } w.end_array()?; - // metaTopLevelKeySources (only if non-empty) - if let ConfigValueKind::Map(entries) = &pandoc.meta.value + // Roundtrip-relevant ASTContext state beyond the files table + // (raw mode only): the example-list counter. + if config.raw { + w.key("exampleListCounter")?; + w.u64_value(ast_context.example_list_counter.get() as u64)?; + } + + // metaTopLevelKeySources (only if non-empty; not in raw mode, + // where key sources ride inline in the meta entry encoding) + if !config.raw + && let ConfigValueKind::Map(entries) = &pandoc.meta.value && !entries.is_empty() { w.key("metaTopLevelKeySources")?; diff --git a/crates/pampa/src/writers/mod.rs b/crates/pampa/src/writers/mod.rs index a053e2f1d..5ec9b2886 100644 --- a/crates/pampa/src/writers/mod.rs +++ b/crates/pampa/src/writers/mod.rs @@ -14,3 +14,4 @@ pub(crate) mod json_stream; pub mod native; pub mod plaintext; pub mod qmd; +pub mod raw_json; diff --git a/crates/pampa/src/writers/raw_json.rs b/crates/pampa/src/writers/raw_json.rs new file mode 100644 index 000000000..593440666 --- /dev/null +++ b/crates/pampa/src/writers/raw_json.rs @@ -0,0 +1,63 @@ +/* + * raw_json.rs + * Copyright (c) 2026 Posit, PBC + */ + +//! The pampa-native `raw-json` writer (GH #11, bd-en2hvrwn). +//! +//! Contract: `raw_json::write` followed by `readers::raw_json::read` is +//! the identity (structural equality) on the pampa AST — including the +//! extensions the Pandoc-superset `-t json` format desugars or rejects +//! (standalone `Inline::Attr`, `NoteReference`, CriticMarkup inlines, +//! `Shortcode`, `CaptionBlock`) and full-fidelity `ConfigValue` metadata. +//! +//! The output deliberately does **not** look like `pandoc -t json` +//! output to machine consumers: the `pampa-json-format` marker is the +//! first key of the envelope and `pandoc-api-version` is absent. +//! +//! Implementation-wise this is a thin entry point over the shared +//! streaming JSON machinery in [`super::json`] with [`JsonConfig::raw`] +//! set: the source-info pool, `astContext` encoding, and all standard +//! node arms are the same battle-tested code path as `-t json`. +//! +//! Design + contract details: +//! `claude-notes/plans/2026-07-17-raw-json-format.md`. + +use crate::pandoc::ASTContext; +use crate::pandoc::Pandoc; +use quarto_error_reporting::DiagnosticMessage; + +use super::json::JsonConfig; + +/// Version of the raw-json envelope, carried in the +/// `pampa-json-format.version` marker and validated by the reader. +/// Bump on any breaking change to the wire shape; the reader rejects +/// versions it does not know. +pub const RAW_JSON_FORMAT_VERSION: u64 = 1; + +/// Write `pandoc` as raw-json with a custom configuration. +/// +/// `config.raw` is forced on; the other fields (inline locations, +/// attribution) compose with raw mode as they do with the Pandoc-superset +/// format. +pub fn write_with_config( + pandoc: &Pandoc, + context: &ASTContext, + writer: &mut W, + config: &JsonConfig, +) -> Result<(), Vec> { + let raw_config = JsonConfig { + raw: true, + ..config.clone() + }; + super::json::write_with_config(pandoc, context, writer, &raw_config) +} + +/// Write `pandoc` as raw-json with the default configuration. +pub fn write( + pandoc: &Pandoc, + context: &ASTContext, + writer: &mut W, +) -> Result<(), Vec> { + write_with_config(pandoc, context, writer, &JsonConfig::default()) +} diff --git a/crates/pampa/tests/integration/attribution_json_wire_test.rs b/crates/pampa/tests/integration/attribution_json_wire_test.rs index 2030334d1..a522ca605 100644 --- a/crates/pampa/tests/integration/attribution_json_wire_test.rs +++ b/crates/pampa/tests/integration/attribution_json_wire_test.rs @@ -148,6 +148,7 @@ fn attribution_on_path_emits_records_and_actors_table() { include_inline_locations: false, attribution_by_node: Some(Arc::new(by_node)), attribution_actors: Some(Arc::new(actors)), + ..JsonConfig::default() }; let mut buf = Vec::new(); @@ -221,6 +222,7 @@ fn attribution_off_path_is_byte_identical_to_no_attribution_default() { include_inline_locations: false, attribution_by_node: None, attribution_actors: None, + ..JsonConfig::default() }; let mut buf_explicit = Vec::new(); write_with_config(&pandoc, &context, &mut buf_explicit, &config_explicit_none) diff --git a/crates/pampa/tests/integration/main.rs b/crates/pampa/tests/integration/main.rs index de8eac9b0..54246f579 100644 --- a/crates/pampa/tests/integration/main.rs +++ b/crates/pampa/tests/integration/main.rs @@ -61,6 +61,7 @@ pub mod test_meta; pub mod test_metadata_source_tracking; pub mod test_nested_yaml_serialization; pub mod test_ordered_list_formatting; +pub mod test_raw_json_roundtrip; pub mod test_rawblock_to_config_value; pub mod test_section_divs; pub mod test_shortcode; diff --git a/crates/pampa/tests/integration/test_raw_json_roundtrip.rs b/crates/pampa/tests/integration/test_raw_json_roundtrip.rs new file mode 100644 index 000000000..58b51ca63 --- /dev/null +++ b/crates/pampa/tests/integration/test_raw_json_roundtrip.rs @@ -0,0 +1,850 @@ +/* + * test_raw_json_roundtrip.rs + * Copyright (c) 2026 Posit, PBC + * + * Roundtrip contract tests for the pampa-native `raw-json` format + * (GH issue #11, bd-en2hvrwn, plan + * claude-notes/plans/2026-07-17-raw-json-format.md). + * + * Contract under test: `raw_json::write` then `raw_json::read` is the + * identity (structural equality) on the pampa AST — including every + * extension Pandoc JSON cannot represent — plus targeted rejection + * diagnostics when the wrong format is fed to either reader. + */ + +use std::sync::Arc; + +use hashlink::LinkedHashMap; +use pampa::pandoc::ast_context::ASTContext; +use pampa::pandoc::attr::AttrSourceInfo; +use pampa::pandoc::{ + Block, CaptionBlock, CustomNode, Delete, Div, EditComment, Highlight, Inline, InlineAttr, + Insert, MetaBlock, NoteDefinitionFencedBlock, NoteDefinitionPara, NoteReference, Pandoc, + Paragraph, Shortcode, ShortcodeArg, Slot, Str, +}; +use pampa::readers; +use pampa::readers::json::JsonReadError; +use pampa::writers::{json, raw_json}; +use quarto_pandoc_types::{ConfigMapEntry, ConfigValue, ConfigValueKind, MergeOp}; +use quarto_source_map::{FileId, SourceInfo}; + +/// Original-file source info spanning `start..end` in FileId(0). +fn si(start: usize, end: usize) -> SourceInfo { + SourceInfo::original(FileId(0), start, end) +} + +fn str_inline(text: &str, start: usize, end: usize) -> Inline { + Inline::Str(Str { + text: text.to_string(), + source_info: si(start, end), + }) +} + +fn para(content: Vec, start: usize, end: usize) -> Block { + Block::Paragraph(Paragraph { + content, + source_info: si(start, end), + }) +} + +fn empty_attr() -> quarto_pandoc_types::Attr { + (String::new(), vec![], LinkedHashMap::new()) +} + +/// Write `doc` as raw-json, read it back, and return the parsed document +/// and context. +fn roundtrip(doc: &Pandoc, context: &ASTContext) -> (Pandoc, ASTContext) { + let mut out = Vec::new(); + raw_json::write(doc, context, &mut out).expect("raw-json write should succeed"); + let mut cursor = std::io::Cursor::new(out); + readers::raw_json::read(&mut cursor).expect("raw-json read should succeed") +} + +fn assert_roundtrip_identity(doc: &Pandoc, context: &ASTContext) { + let (parsed, _parsed_context) = roundtrip(doc, context); + assert_eq!(doc, &parsed, "raw-json write→read must be the identity"); +} + +// --------------------------------------------------------------------------- +// Envelope shape +// --------------------------------------------------------------------------- + +#[test] +fn test_raw_json_marker_is_first_key_and_no_pandoc_api_version() { + let doc = Pandoc { + meta: ConfigValue::default(), + blocks: vec![para(vec![str_inline("hi", 0, 2)], 0, 2)], + }; + let context = ASTContext::new(); + let mut out = Vec::new(); + raw_json::write(&doc, &context, &mut out).expect("write should succeed"); + let text = String::from_utf8(out).expect("valid UTF-8"); + + // The marker must be the first key so it is visible in truncated + // previews / the first line of output. + assert!( + text.starts_with(r#"{"pampa-json-format":{"version":1}"#), + "output must start with the pampa-json-format marker, got: {}", + &text[..text.len().min(80)] + ); + // Machine consumers of Pandoc JSON key on pandoc-api-version; it must + // be absent so they fail fast. + assert!( + !text.contains("pandoc-api-version"), + "raw-json must not contain pandoc-api-version" + ); + + let value: serde_json::Value = serde_json::from_str(&text).expect("valid JSON"); + assert_eq!(value["pampa-json-format"]["version"], 1); +} + +// --------------------------------------------------------------------------- +// Extension inlines: the constructs Pandoc JSON rejects or desugars +// --------------------------------------------------------------------------- + +/// The issue #11 repro: a standalone attribute must roundtrip through +/// raw-json (while the Pandoc-superset writer errors with Q-3-32). +/// +/// The Attr sits mid-paragraph: a *trailing* paragraph Attr is +/// representable in the Pandoc-superset format via the para-`attr` hoist +/// (bd-aeyss6p5) and would not exercise the Q-3-32 path. +#[test] +fn test_raw_json_roundtrip_standalone_attr() { + let mut kvs = LinkedHashMap::new(); + kvs.insert("key".to_string(), "value".to_string()); + let attr = ( + "free-floating".to_string(), + vec!["cls-a".to_string(), "cls-b".to_string()], + kvs, + ); + let doc = Pandoc { + meta: ConfigValue::default(), + blocks: vec![para( + vec![ + str_inline("Hello.", 0, 6), + Inline::Attr(InlineAttr::new(attr, AttrSourceInfo::empty(), si(8, 32))), + str_inline("Here?", 34, 39), + ], + 0, + 39, + )], + }; + let context = ASTContext::new(); + + // Sanity: the Pandoc-superset JSON writer refuses this document. + let mut pandoc_json = Vec::new(); + let err = json::write(&doc, &context, &mut pandoc_json) + .expect_err("pandoc-superset json must reject standalone Attr"); + assert!( + err.iter().any(|d| d.code.as_deref() == Some("Q-3-32")), + "expected Q-3-32 from the pandoc-superset writer, got: {:?}", + err.iter().map(|d| d.code.clone()).collect::>() + ); + + // The raw writer accepts it and roundtrips identically. + assert_roundtrip_identity(&doc, &context); +} + +#[test] +fn test_raw_json_roundtrip_note_reference() { + let doc = Pandoc { + meta: ConfigValue::default(), + blocks: vec![para( + vec![ + str_inline("See", 0, 3), + Inline::NoteReference(NoteReference { + id: "note-1".to_string(), + source_info: si(4, 12), + }), + ], + 0, + 12, + )], + }; + assert_roundtrip_identity(&doc, &ASTContext::new()); +} + +#[test] +fn test_raw_json_roundtrip_critic_markup() { + let mut kvs = LinkedHashMap::new(); + kvs.insert("author".to_string(), "cs".to_string()); + let attr = ("crit-1".to_string(), vec!["review".to_string()], kvs); + + let doc = Pandoc { + meta: ConfigValue::default(), + blocks: vec![para( + vec![ + Inline::Insert(Insert { + attr: attr.clone(), + content: vec![str_inline("added", 3, 8)], + source_info: si(0, 11), + attr_source: AttrSourceInfo::empty(), + }), + Inline::Delete(Delete { + attr: empty_attr(), + content: vec![str_inline("removed", 14, 21)], + source_info: si(11, 24), + attr_source: AttrSourceInfo::empty(), + }), + Inline::Highlight(Highlight { + attr: empty_attr(), + content: vec![str_inline("marked", 27, 33)], + source_info: si(24, 36), + attr_source: AttrSourceInfo::empty(), + }), + Inline::EditComment(EditComment { + attr: empty_attr(), + content: vec![str_inline("why?", 39, 43)], + source_info: si(36, 46), + attr_source: AttrSourceInfo::empty(), + }), + ], + 0, + 46, + )], + }; + assert_roundtrip_identity(&doc, &ASTContext::new()); +} + +#[test] +fn test_raw_json_roundtrip_shortcode() { + let mut keyword_args = LinkedHashMap::new(); + keyword_args.insert( + "width".to_string(), + ShortcodeArg::String("100%".to_string()), + ); + keyword_args.insert("autoplay".to_string(), ShortcodeArg::Boolean(true)); + + let nested = Shortcode { + is_escaped: false, + name: "meta".to_string(), + positional_args: vec![ShortcodeArg::String("title".to_string())], + keyword_args: LinkedHashMap::new(), + source_info: si(20, 34), + }; + + let mut kv_map = std::collections::HashMap::new(); + kv_map.insert("a".to_string(), ShortcodeArg::Number(1.5)); + kv_map.insert("b".to_string(), ShortcodeArg::String("two".to_string())); + + let doc = Pandoc { + meta: ConfigValue::default(), + blocks: vec![para( + vec![Inline::Shortcode(Shortcode { + is_escaped: true, + name: "video".to_string(), + positional_args: vec![ + ShortcodeArg::String("intro.mp4".to_string()), + ShortcodeArg::Number(2.0), + ShortcodeArg::Shortcode(nested), + ShortcodeArg::KeyValue(kv_map), + ], + keyword_args, + source_info: si(0, 40), + })], + 0, + 40, + )], + }; + assert_roundtrip_identity(&doc, &ASTContext::new()); +} + +// --------------------------------------------------------------------------- +// Extension blocks +// --------------------------------------------------------------------------- + +#[test] +fn test_raw_json_roundtrip_caption_block() { + let doc = Pandoc { + meta: ConfigValue::default(), + blocks: vec![ + Block::CaptionBlock(CaptionBlock { + content: vec![str_inline("A caption", 0, 9)], + source_info: si(0, 9), + }), + para(vec![str_inline("body", 10, 14)], 10, 14), + ], + }; + assert_roundtrip_identity(&doc, &ASTContext::new()); +} + +#[test] +fn test_raw_json_roundtrip_note_definitions_and_block_metadata() { + let doc = Pandoc { + meta: ConfigValue::default(), + blocks: vec![ + Block::NoteDefinitionPara(NoteDefinitionPara { + id: "fn-1".to_string(), + content: vec![str_inline("a footnote", 0, 10)], + source_info: si(0, 10), + }), + Block::NoteDefinitionFencedBlock(NoteDefinitionFencedBlock { + id: "fn-2".to_string(), + content: vec![para(vec![str_inline("fenced note", 11, 22)], 11, 22)], + source_info: si(11, 22), + }), + Block::BlockMetadata(MetaBlock { + meta: ConfigValue { + value: ConfigValueKind::Map(vec![ConfigMapEntry { + key: "layout".to_string(), + key_source: si(23, 29), + value: ConfigValue { + value: ConfigValueKind::Scalar(yaml_rust2::Yaml::String( + "wide".to_string(), + )), + source_info: si(31, 35), + merge_op: MergeOp::default(), + }, + }]), + source_info: si(23, 35), + merge_op: MergeOp::default(), + }, + source_info: si(23, 35), + }), + ], + }; + assert_roundtrip_identity(&doc, &ASTContext::new()); +} + +#[test] +fn test_raw_json_roundtrip_custom_nodes() { + let block_custom = CustomNode::new("Callout", empty_attr(), si(0, 30)) + .with_slot("title", Slot::Inlines(vec![str_inline("Watch out", 5, 14)])) + .with_slot( + "content", + Slot::Blocks(vec![para(vec![str_inline("body", 15, 19)], 15, 19)]), + ) + .with_data(serde_json::json!({"type": "warning", "appearance": "default"})); + + let inline_custom = CustomNode::new("Kbd", empty_attr(), si(31, 40)) + .with_slot("keys", Slot::Inline(Box::new(str_inline("Ctrl", 33, 37)))); + + let doc = Pandoc { + meta: ConfigValue::default(), + blocks: vec![ + Block::Custom(block_custom), + para(vec![Inline::Custom(inline_custom)], 31, 40), + ], + }; + assert_roundtrip_identity(&doc, &ASTContext::new()); +} + +// --------------------------------------------------------------------------- +// Metadata fidelity +// --------------------------------------------------------------------------- + +/// Everything the Pandoc-superset meta encoding loses, raw-json must keep: +/// Path/Glob/Expr kinds, merge_op, non-string scalar types, and map entry +/// order. +#[test] +fn test_raw_json_meta_fidelity() { + let entry = |key: &str, ks: (usize, usize), value: ConfigValue| ConfigMapEntry { + key: key.to_string(), + key_source: si(ks.0, ks.1), + value, + }; + let scalar = |yaml: yaml_rust2::Yaml, s: (usize, usize)| ConfigValue { + value: ConfigValueKind::Scalar(yaml), + source_info: si(s.0, s.1), + merge_op: MergeOp::default(), + }; + + let meta = ConfigValue { + value: ConfigValueKind::Map(vec![ + // Deliberately NOT in alphabetical order: order must survive. + entry( + "zebra", + (0, 5), + scalar(yaml_rust2::Yaml::String("stripes".to_string()), (7, 14)), + ), + entry( + "resources", + (15, 24), + ConfigValue { + value: ConfigValueKind::Glob("images/*.png".to_string()), + source_info: si(26, 38), + merge_op: MergeOp::Concat, + }, + ), + entry( + "logo", + (39, 43), + ConfigValue { + value: ConfigValueKind::Path("assets/logo.svg".to_string()), + source_info: si(45, 60), + // Non-default merge op (default is Concat): exercises the + // `m` key emission path. + merge_op: MergeOp::Prefer, + }, + ), + entry( + "date", + (61, 65), + ConfigValue { + value: ConfigValueKind::Expr("Sys.Date()".to_string()), + source_info: si(67, 77), + merge_op: MergeOp::default(), + }, + ), + entry( + "count", + (78, 83), + scalar(yaml_rust2::Yaml::Integer(42), (85, 87)), + ), + entry( + "ratio", + (88, 93), + scalar(yaml_rust2::Yaml::Real("0.75".to_string()), (95, 99)), + ), + entry( + "draft", + (100, 105), + scalar(yaml_rust2::Yaml::Boolean(false), (107, 112)), + ), + entry( + "abstract", + (113, 121), + scalar(yaml_rust2::Yaml::Null, (122, 123)), + ), + entry( + "alpha", + (124, 129), + ConfigValue { + value: ConfigValueKind::Array(vec![ + scalar(yaml_rust2::Yaml::Integer(1), (131, 132)), + scalar(yaml_rust2::Yaml::Integer(2), (133, 134)), + ]), + source_info: si(130, 135), + merge_op: MergeOp::Concat, + }, + ), + entry( + "title", + (136, 141), + ConfigValue { + value: ConfigValueKind::PandocInlines(vec![ + str_inline("Hello", 143, 148), + str_inline("world", 149, 154), + ]), + source_info: si(143, 154), + merge_op: MergeOp::default(), + }, + ), + ]), + source_info: si(0, 154), + merge_op: MergeOp::default(), + }; + + let doc = Pandoc { + meta, + blocks: vec![para(vec![str_inline("x", 155, 156)], 155, 156)], + }; + let context = ASTContext::new(); + let (parsed, _) = roundtrip(&doc, &context); + + // Full identity, which covers kinds, merge ops, scalar types, and + // source infos in one shot. + assert_eq!(doc.meta, parsed.meta, "meta must roundtrip identically"); + + // Entry order asserted explicitly so a future "helpful" sort is caught + // even if equality semantics ever change. + if let ConfigValueKind::Map(entries) = &parsed.meta.value { + let keys: Vec<&str> = entries.iter().map(|e| e.key.as_str()).collect(); + assert_eq!( + keys, + vec![ + "zebra", + "resources", + "logo", + "date", + "count", + "ratio", + "draft", + "abstract", + "alpha", + "title" + ], + "meta entry order must be preserved" + ); + } else { + panic!("expected Map meta after roundtrip"); + } +} + +/// A `Scalar` holding a non-scalar YAML value (Array/Hash) cannot be +/// represented faithfully; the raw writer must fail loudly rather than +/// silently degrade the value. +#[test] +fn test_raw_json_rejects_non_scalar_yaml_in_scalar() { + let doc = Pandoc { + meta: ConfigValue { + value: ConfigValueKind::Map(vec![ConfigMapEntry { + key: "bad".to_string(), + key_source: si(0, 3), + value: ConfigValue { + value: ConfigValueKind::Scalar(yaml_rust2::Yaml::Array(vec![ + yaml_rust2::Yaml::Integer(1), + ])), + source_info: si(5, 8), + merge_op: MergeOp::default(), + }, + }]), + source_info: si(0, 8), + merge_op: MergeOp::default(), + }, + blocks: vec![], + }; + let mut out = Vec::new(); + raw_json::write(&doc, &ASTContext::new(), &mut out) + .expect_err("raw-json must not silently degrade non-scalar YAML in Scalar"); +} + +// --------------------------------------------------------------------------- +// Source-info preservation +// --------------------------------------------------------------------------- + +/// Two nodes sharing one `Substring` parent through the same `Arc` must +/// come back with equal (structurally shared) parent chains. +#[test] +fn test_raw_json_preserves_shared_substring_parents() { + let parent = Arc::new(si(0, 100)); + let shared_a = SourceInfo::Substring { + parent: parent.clone(), + start_offset: 0, + end_offset: 5, + }; + let shared_b = SourceInfo::Substring { + parent: parent.clone(), + start_offset: 6, + end_offset: 11, + }; + + let concat = SourceInfo::concat(vec![(shared_a.clone(), 0), (shared_b.clone(), 5)]); + let generated = SourceInfo::generated(quarto_source_map::By::filter("f.lua".to_string(), 3)); + + let doc = Pandoc { + meta: ConfigValue::default(), + blocks: vec![para( + vec![ + Inline::Str(Str { + text: "one".to_string(), + source_info: shared_a, + }), + Inline::Str(Str { + text: "two".to_string(), + source_info: shared_b, + }), + Inline::Str(Str { + text: "cat".to_string(), + source_info: concat, + }), + Inline::Str(Str { + text: "gen".to_string(), + source_info: generated, + }), + ], + 0, + 100, + )], + }; + assert_roundtrip_identity(&doc, &ASTContext::new()); +} + +/// ASTContext state must roundtrip: filenames and the example-list counter. +#[test] +fn test_raw_json_roundtrips_ast_context() { + let context = ASTContext::with_filename("doc.qmd"); + context.example_list_counter.set(7); + + let doc = Pandoc { + meta: ConfigValue::default(), + blocks: vec![para(vec![str_inline("hi", 0, 2)], 0, 2)], + }; + let (_, parsed_context) = roundtrip(&doc, &context); + assert_eq!(parsed_context.filenames, vec!["doc.qmd"]); + assert_eq!( + parsed_context.example_list_counter.get(), + 7, + "example_list_counter must be carried in the raw envelope" + ); +} + +// --------------------------------------------------------------------------- +// Format self-identification / cross-format rejection +// --------------------------------------------------------------------------- + +#[test] +fn test_raw_json_reader_rejects_pandoc_style_json() { + // Real output of the Pandoc-superset writer. + let doc = Pandoc { + meta: ConfigValue::default(), + blocks: vec![para(vec![str_inline("hi", 0, 2)], 0, 2)], + }; + let mut pandoc_json = Vec::new(); + json::write(&doc, &ASTContext::new(), &mut pandoc_json).expect("pandoc json write"); + + let mut cursor = std::io::Cursor::new(pandoc_json); + let err = + readers::raw_json::read(&mut cursor).expect_err("raw reader must reject Pandoc-style JSON"); + let msg = err.to_string(); + assert!( + msg.contains("-f json"), + "rejection must point the user at `-f json`, got: {}", + msg + ); +} + +#[test] +fn test_raw_json_reader_rejects_unmarked_json() { + let input = br#"{"blocks": [], "meta": {}}"#; + let mut cursor = std::io::Cursor::new(&input[..]); + let err = + readers::raw_json::read(&mut cursor).expect_err("raw reader must reject unmarked JSON"); + let msg = err.to_string(); + assert!( + msg.contains("pampa-json-format"), + "rejection must name the missing marker, got: {}", + msg + ); +} + +#[test] +fn test_raw_json_reader_rejects_wrong_version() { + let input = br#"{"pampa-json-format": {"version": 999}, "blocks": [], "meta": {}}"#; + let mut cursor = std::io::Cursor::new(&input[..]); + let err = + readers::raw_json::read(&mut cursor).expect_err("raw reader must reject unknown versions"); + let msg = err.to_string(); + assert!( + msg.contains("999"), + "version rejection must name the offending version, got: {}", + msg + ); +} + +#[test] +fn test_pandoc_json_reader_rejects_raw_json() { + let doc = Pandoc { + meta: ConfigValue::default(), + blocks: vec![para(vec![str_inline("hi", 0, 2)], 0, 2)], + }; + let mut raw_out = Vec::new(); + raw_json::write(&doc, &ASTContext::new(), &mut raw_out).expect("raw write"); + + let mut cursor = std::io::Cursor::new(raw_out); + let err = readers::json::read(&mut cursor) + .expect_err("pandoc json reader must reject raw-json input"); + let msg = err.to_string(); + assert!( + msg.contains("raw-json"), + "rejection must point the user at the raw-json reader, got: {}", + msg + ); + assert!( + matches!(err, JsonReadError::UnexpectedRawJsonMarker), + "expected UnexpectedRawJsonMarker, got: {:?}", + err + ); +} + +// --------------------------------------------------------------------------- +// Stability across generations +// --------------------------------------------------------------------------- + +/// AST-level idempotence: read(write(read(write(doc)))) == read(write(doc)). +/// (Byte-level stability of the pool is explicitly NOT promised — see the +/// plan's roundtrip-contract section.) +#[test] +fn test_raw_json_ast_stable_across_generations() { + let parent = Arc::new(si(0, 50)); + let doc = Pandoc { + meta: ConfigValue::default(), + blocks: vec![para( + vec![ + Inline::Str(Str { + text: "a".to_string(), + source_info: SourceInfo::Substring { + parent: parent.clone(), + start_offset: 0, + end_offset: 1, + }, + }), + Inline::Str(Str { + text: "b".to_string(), + source_info: SourceInfo::Substring { + parent: parent.clone(), + start_offset: 1, + end_offset: 2, + }, + }), + ], + 0, + 50, + )], + }; + let context = ASTContext::new(); + let (gen1, ctx1) = roundtrip(&doc, &context); + let (gen2, _) = roundtrip(&gen1, &ctx1); + assert_eq!(gen1, gen2, "second-generation roundtrip must be stable"); +} + +// --------------------------------------------------------------------------- +// End-to-end from qmd: the issue #11 document +// --------------------------------------------------------------------------- + +/// Parse the exact document from GH issue #11 with the qmd reader — which +/// produces a standalone `Inline::Attr` — and roundtrip the parser-produced +/// AST (real Substring/Concat source infos) through raw-json. +#[test] +fn test_raw_json_roundtrip_issue_11_document() { + let input = "Hello. {#free-floating-attribute} Here?\n"; + let (doc, context, diagnostics) = readers::qmd::read( + input.as_bytes(), + false, + "", + &mut std::io::sink(), + true, + None, + ) + .expect("qmd parse should succeed"); + assert!( + diagnostics.is_empty(), + "expected clean parse, got: {:?}", + diagnostics + ); + + // Confirm the fixture exercises what we think it does. + let has_standalone_attr = doc.blocks.iter().any(|b| match b { + Block::Paragraph(p) => p.content.iter().any(|i| matches!(i, Inline::Attr(_))), + _ => false, + }); + assert!( + has_standalone_attr, + "issue #11 fixture should produce a standalone Inline::Attr; blocks: {:?}", + doc.blocks + ); + + assert_roundtrip_identity(&doc, &context); +} + +/// Corpus sweep: every document in the existing json-writer fixture +/// corpus (`tests/writers/json/*.md`), parsed by the real qmd reader, +/// must roundtrip identically through raw-json. Identity assertions +/// subsume snapshots here, so raw-json has no parallel snapshot corpus — +/// this sweep provides the breadth instead. +#[test] +fn test_raw_json_roundtrip_writer_fixture_corpus() { + let dir = std::path::Path::new("tests/writers/json"); + let mut count = 0; + for entry in std::fs::read_dir(dir).expect("fixture corpus dir must exist") { + let path = entry.expect("readable dir entry").path(); + if path.extension().and_then(|e| e.to_str()) != Some("md") { + continue; + } + let content = std::fs::read(&path).expect("readable fixture"); + let (doc, context, _diagnostics) = readers::qmd::read( + &content, + false, + &path.to_string_lossy(), + &mut std::io::sink(), + true, + None, + ) + .unwrap_or_else(|e| panic!("qmd parse failed for {}: {:?}", path.display(), e)); + + let (parsed, _) = roundtrip(&doc, &context); + assert_eq!( + doc, + parsed, + "raw-json roundtrip must be the identity for fixture {}", + path.display() + ); + count += 1; + } + assert!(count > 0, "corpus sweep found no fixtures — path wrong?"); +} + +// --------------------------------------------------------------------------- +// Interop sanity: constructs that already roundtrip through pandoc json +// must also roundtrip through raw-json +// --------------------------------------------------------------------------- + +/// Sidecar source infos that the shared reader used to drop on read-back +/// (found by the corpus sweep): Link/Image `target_source` and Citation +/// `id_source` must survive. +#[test] +fn test_raw_json_roundtrip_sidecar_source_infos() { + let doc = Pandoc { + meta: ConfigValue::default(), + blocks: vec![para( + vec![ + Inline::Link(pampa::pandoc::Link { + attr: empty_attr(), + content: vec![str_inline("a link", 1, 7)], + target: ("./hello".to_string(), "out".to_string()), + source_info: si(0, 46), + attr_source: AttrSourceInfo::empty(), + target_source: pampa::pandoc::attr::TargetSourceInfo { + url: Some(si(9, 16)), + title: Some(si(17, 22)), + }, + }), + Inline::Cite(pampa::pandoc::Cite { + citations: vec![pampa::pandoc::Citation { + id: "knuth1984".to_string(), + prefix: vec![str_inline("see", 48, 51)], + suffix: vec![str_inline("p. 42", 62, 67)], + mode: pampa::pandoc::CitationMode::NormalCitation, + note_num: 0, + hash: 0, + id_source: Some(si(52, 61)), + }], + content: vec![str_inline("[@knuth1984]", 47, 68)], + source_info: si(47, 68), + }), + ], + 0, + 68, + )], + }; + assert_roundtrip_identity(&doc, &ASTContext::new()); +} + +#[test] +fn test_raw_json_roundtrip_standard_constructs() { + let mut kvs = LinkedHashMap::new(); + kvs.insert("style".to_string(), "border: 1px".to_string()); + let doc = Pandoc { + meta: ConfigValue::default(), + blocks: vec![ + para( + vec![ + str_inline("plain", 0, 5), + Inline::Emph(pampa::pandoc::Emph { + content: vec![str_inline("emph", 6, 10)], + source_info: si(5, 11), + }), + ], + 0, + 11, + ), + Block::Div(Div { + attr: ("d1".to_string(), vec!["note".to_string()], kvs), + content: vec![para(vec![str_inline("inner", 12, 17)], 12, 17)], + source_info: si(11, 18), + attr_source: AttrSourceInfo::empty(), + }), + Block::CodeBlock(pampa::pandoc::CodeBlock { + attr: ( + "cb".to_string(), + vec!["rust".to_string()], + LinkedHashMap::new(), + ), + text: "fn main() {}".to_string(), + source_info: si(19, 31), + attr_source: AttrSourceInfo::empty(), + }), + ], + }; + assert_roundtrip_identity(&doc, &ASTContext::new()); +} diff --git a/crates/quarto-core/src/pipeline.rs b/crates/quarto-core/src/pipeline.rs index 6d153dff5..84b36b694 100644 --- a/crates/quarto-core/src/pipeline.rs +++ b/crates/quarto-core/src/pipeline.rs @@ -1018,6 +1018,7 @@ pub async fn render_qmd_to_preview_ast( include_inline_locations: true, attribution_by_node, attribution_actors, + ..Default::default() }; let mut buf = Vec::new(); pampa::writers::json::write_with_config(&ast.ast, &ast_context, &mut buf, &json_config) diff --git a/crates/wasm-quarto-hub-client/src/lib.rs b/crates/wasm-quarto-hub-client/src/lib.rs index 172a35cfb..ae7d15d52 100644 --- a/crates/wasm-quarto-hub-client/src/lib.rs +++ b/crates/wasm-quarto-hub-client/src/lib.rs @@ -920,6 +920,7 @@ pub async fn parse_qmd_to_ast_with_attribution( include_inline_locations: true, attribution_by_node, attribution_actors, + ..Default::default() }; let ast_json = match pampa::writers::json::write_with_config(