[docs] xtable-spark-runtime README + Spark quickstart docs#847
[docs] xtable-spark-runtime README + Spark quickstart docs#847vinishjail97 wants to merge 4 commits into
Conversation
Documents the xtable-spark-runtime bundle: a spark-submit quickstart with examples, the CLI options, a source/target format matrix, and Spark 3.4/3.5 compatibility (Delta auto-switches to the Spark-free Delta Kernel on 3.5+). Adds a "Run XTable on Apache Spark" docs page (linked from Installation and the sidebar) and a module README. Note: depends on the xtable-spark-runtime module (PR apache#843); intended to merge after the 0.4.0 release once the module lands on main. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-datasetconfig, TPC-DS example - Add "Adding to an existing Spark job" section: drop the thin bundle on an existing cluster's classpath (engines are provided) and run XTableSparkSync against a table the job already writes to add other formats; no data rewrite. - Document the --datasetconfig multi-table mode and add it to the options table (mutually exclusive with --basepath/--sourceformat/--targets). - Add examples/tpcds-dataset.yaml: a 24-table TPC-DS Hudi -> Iceberg,Delta config. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename "bundle jar" to "runtime jar" throughout to match the module name and Iceberg's convention; keep "(shaded, relocated)" once in the developer README. Drop em-dashes from prose. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Normalize the ASF header to the canonical wrapping in style/text-license-header and prefix the example comment with ## so Spotless treats it as content instead of folding it into the license-header region. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
|
||
| ## Supported formats | ||
|
|
||
| Paimon and Parquet are read-only sources (there is no corresponding write target). |
There was a problem hiding this comment.
The bundle IT covers Hudi↔Iceberg and Hudi↔Delta, but the matrix also marks Iceberg↔Delta and Paimon/Parquet sources as supported. Are those validated somewhere, or is this the core capability matrix rather than what this bundle is tested for? A line distinguishing the two would help.
Separately, the README describes the IT as "one case per direction across Hudi, Iceberg, and Delta" — that reads a bit broader than the actual directions(), which has no Iceberg↔Delta case.
|
|
||
| | Spark version | Hudi / Iceberg | Delta implementation | | ||
| | --- | :---: | --- | | ||
| | 3.4.x | ✅ | Delta Standalone (`delta-core`) | |
There was a problem hiding this comment.
Minor naming thing — the delta-core in brackets is unambiguous, but "Delta Standalone" usually names io.delta:delta-standalone, a separate JVM library now deprecated in favor of Delta Kernel. The 3.4 path here is delta-core, so is the "Delta Standalone" label intentional? Delta's own naming is confusing (delta-core was renamed delta-spark in 3.0), so feel free to ignore if this is the convention we want.
|
|
||
| | Spark version | Hudi / Iceberg | Delta | | ||
| | --- | :---: | --- | | ||
| | 3.4.x | ✅ | Delta Standalone (`delta-core`) | |
There was a problem hiding this comment.
Same minor naming question as the docs page — "Delta Standalone" is a separate (deprecated) artifact from the delta-core in brackets.
2 participants
What
Documentation for the
xtable-spark-runtimebundle (added in #843):Run XTable on Apache Spark(website/docs/spark-runtime.md) — aspark-submitquickstart with worked examples, the full CLI option table, a source/target format matrix, and Spark version compatibility.xtable-spark-runtime/README.md) — build, usage, compatibility, and how to run the bundle validation IT.Spark quickstart (excerpt)
The docs also cover Spark version compatibility: Hudi/Iceberg sync is Spark-free, and a Delta source/target auto-switches from
delta-core(Spark 3.4) to the Spark-free Delta Kernel on Spark 3.5+.Note on merge order
This documents the
xtable-spark-runtimemodule, which currently lives on the #843 branch (targetingbranch-0.4) and is not yet onmain. Intended to merge after the 0.4.0 release, once the module lands onmain.🤖 Generated with Claude Code