[836] xtable-spark-runtime: thin drop-in Spark bundle (Hudi 0.x)#843
Open
vinishjail97 wants to merge 8 commits into
Open
[836] xtable-spark-runtime: thin drop-in Spark bundle (Hudi 0.x)#843vinishjail97 wants to merge 8 commits into
vinishjail97 wants to merge 8 commits into
Conversation
Ports the xtable-spark-runtime packaging slice onto the Hudi 0.x line
(main-hudi-0x, Hudi 0.14 / Spark 3.4). New thin, relocated bundle that
runs an incremental XTable sync inside a Spark job; engines are provided
by the cluster, never bundled.
Module:
- xtable-spark-runtime_${scala.binary.version}: xtable-core compile;
Spark/Hadoop and the engines (Hudi/Iceberg/Delta) provided. Curated
shade allowlist (xtable modules + guava/protobuf/commons-cli relocated);
avro/parquet/jackson NOT relocated (exchanged with the engines). Thin
~3.7 MB bundle.
- XTableSparkSync: standalone spark-submit entry point (Apache Commons CLI).
- XTableSyncService / TableSyncSpec: build an INCREMENTAL ConversionConfig
and run ConversionController.sync; target metadata path = source data
path (required by Hudi; Iceberg data lives under <basePath>/data).
Hudi 0.14 specifics (vs the Hudi 1.x variant on main):
- No hudi-hadoop-common (that split is 1.x); hudi-common provides FSUtils.
- Engine classpath uses the hudi-spark bundle (its regenerated Avro model
classes link on Avro 1.12, which Iceberg 1.9.2 requires) plus
hudi-java-client for the Hudi target's Java write client, with the raw
hudi-common excluded so the bundle's clean DecimalWrapper wins.
ConversionTargetFactory: make ServiceLoader discovery resilient so a
subset of engines works when others are absent (warn + skip on
LinkageError/ServiceConfigurationError; name-based Delta-Kernel check).
ITXTableSparkRuntimeBundle: spark-submits the shaded jar for one case per
direction across Hudi, Iceberg and Delta (source and target), engines on
a flat classpath, asserting data-equivalence over comparable scalar
columns. Requires a Spark 3.4 SPARK_HOME; skipped otherwise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
vinishjail97
commented
Jul 10, 2026
…sing, RFC) Publish the shaded jar as the MAIN artifact with a dependency-reduced POM (shadedArtifactAttached=false, createDependencyReducedPom=true), matching iceberg-spark-runtime / hudi-spark-bundle. Resolving the coordinate via a Maven dependency or --packages now yields the relocated bundle and pulls no un-relocated transitive deps; --jars is equivalent. The thin main jar + separate -bundle classifier (which re-introduced the cluster guava clash) is gone. IT findBundleJar() now picks the shaded main jar. Pass release/scripts/validate_shaded_license_coverage.sh (the existing allowlist gate): the shade <includes> must equal the runtime dependency tree, so jackson / scala-library / log4j-1.2-api - which every Spark runtime supplies - are declared provided (dropped from the tree, kept off the shaded jar) and excluded from the IT's flat engine classpath so Spark's own copies win. avro/parquet stay on the flat classpath (the engine's newer avro must win over Spark 3.4's). Bundled-dependency licensing: add META-INF/LICENSE-bundled and NOTICE-bundled (wired via IncludeResourceTransformer) attributing the only bundled third-party - guava's closure (Apache-2.0) and commons-cli (Apache-2.0), plus checker-qual (MIT). Remove the dead protobuf-java allowlist entry and relocation: protobuf resolves as provided and was never bundled. RFC-3: scope v1 to the CLI (XTableSparkSync); mark the config-only XTableSyncListener on-ramp (and XTableSparkConfig / PlanTargetResolver) as a deferred follow-up. Fix the activation example to the shipped CLI and document the required engine Avro version + flat-classpath placement. XTableSyncService: normalize sourceFormat with Locale.ROOT. Root pom: exclude the shade-generated dependency-reduced-pom.xml from spotless (no license header; CI runs clean install). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consolidate repeated rationale in the bundle pom (no functional change): - Tell the hudi-common DecimalWrapper / Avro-1.8-1.9 story once (on the hudi-spark bundle dep); the hudi-java-client exclusion just points to it. - State "engine Avro must win on a flat classpath" once (engine-classpath plugin); the avro dep and excludeGroupIds comments reference it. - Explain jackson/scala-library are Spark-supplied once. - Fix a stale "Spark 3.5" reference to Spark 3.4 (this is the Hudi 0.x line). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add main-hudi-0x to the push and pull_request branch filters of both workflows (same change as apache#841) so CI runs on PRs targeting the Hudi 0.x release line, including this one. A pull_request workflow's branch filter takes effect from the PR's own branch, so this makes CI fire on apache#843. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
vinishjail97
marked this pull request as ready for review
July 10, 2026 01:29
…k-submit stdout off-thread - ConversionTargetFactory: hasNext() resolves provider classes lazily and can throw ServiceConfigurationError too, so move it inside the existing ServiceConfigurationError|LinkageError guard alongside next(). - ITXTableSparkRuntimeBundle: drain spark-submit stdout on a background thread so a hung process is caught by waitFor(10min) instead of blocking forever on readOutput() reading until EOF.
main-hudi-0x was deleted and replaced by branch-0.4 (the 0.4.0 release line), so update the push/pull_request branch filters accordingly.
…istro ITXTableSparkRuntimeBundle spark-submits the shaded bundle jar and self-skips unless SPARK_HOME is set, so the main CI never runs it. Add a path-filtered workflow that installs a matching Spark 3.4 distribution from the Apache archive, sets SPARK_HOME, and runs the failsafe IT.
…filter) The workflow is intended to be a required status check on branch-0.4. A required check whose workflow is skipped by a path filter never reports, leaving PRs blocked on a pending check, so run it unconditionally on the covered branches. The Spark distro is cached, so the added cost is the reactor build plus the ~20s IT.
vinothchandar
left a comment
vinothchandar
left a comment
Member
There was a problem hiding this comment.
Thanks @vinishjail97 ~ I think this will make life 10x better for users.
| push: | ||
| branches: | ||
| - "main" | ||
| - "branch-0.4" |
Member
There was a problem hiding this comment.
I was a little confused at first to see the version number. But it seems you already cut the release branch
| workflow_dispatch: | ||
|
|
||
| env: | ||
| SPARK_VERSION: "3.4.3" |
Member
There was a problem hiding this comment.
This cannot be 3.5 or something.?
| Arrays.stream(cmd.getOptionValue(TARGETS).split(",")) | ||
| .map(String::trim) | ||
| .filter(s -> !s.isEmpty()) | ||
| .map(s -> s.toUpperCase(Locale.ROOT)) |
Member
There was a problem hiding this comment.
The resolved target list is never validated to be non-empty. An input like --targets "," (or all-blank entries) passes through trim/filter and yields an empty targetFormats, so the sync silently produces no target tables and does nothing. Consider failing fast with a clear message when the resulting list is empty.
| .dataPath(cmd.getOptionValue(DATA_PATH)) | ||
| .namespace(namespace == null ? null : namespace.split("\\.")) | ||
| .partitionSpec(cmd.getOptionValue(PARTITION_SPEC)) | ||
| .sourceFormat(cmd.getOptionValue(SOURCE_FORMAT).toUpperCase(Locale.ROOT)) |
Member
There was a problem hiding this comment.
An unsupported --sourceFormat isn't rejected here; it surfaces only later in XTableSyncService.sourceProviderFor(...) as an UnsupportedOperationException, after the SparkSession has already been created via getOrCreate(). Consider validating the source format (and required args) up front in the CLI so an invalid value fails fast without spinning up Spark.
| on: | ||
| push: | ||
| branches: | ||
| - "main" |
Member
There was a problem hiding this comment.
The Spark version is hardcoded to 3.4.3. Since this module targets the Hudi 0.x / Spark 3.4 line, that's reasonable, but consider parameterizing it (matrix or workflow input) or at least adding a comment explaining why 3.4.x is pinned, so the validation stays easy to bump. (A maintainer also raised this above.)
| <description> | ||
| Thin, relocated, drop-in Spark bundle that runs incremental XTable metadata sync. v1 ships | ||
| the standalone spark-submit entry point (XTableSparkSync); the config-only, driver-side | ||
| QueryExecutionListener on-ramp is a planned follow-up (see rfc/rfc-3). |
Member
There was a problem hiding this comment.
So right now, the sync needs to be manually invoked?
| <scope>provided</scope> | ||
| </dependency> | ||
| <!-- | ||
| XTable's Hudi TARGET writes via the Java client (HoodieJavaEngineContext), which the Spark |
Member
There was a problem hiding this comment.
+1 . we should keep it this way, and publish the support matrix of what format versions works for source -> target? (could we add docs for that?)
| - @vinishjail97 | ||
|
|
||
| ## Approvers | ||
| - Anyone from the XTable community can approve/add feedback. |
| is **deferred to a follow-up PR** — those classes are not in v1. The listener design is retained | ||
| below as the target so the entry point and config schema stay aligned. | ||
|
|
||
| ### Activation — v1 (CLI entry point) |
Member
There was a problem hiding this comment.
Can we also document how to call this from code i.e from within pySpark or spark code? These could go into the quickstart as well. Can we revamp that
| ## Approvers | ||
| - Anyone from the XTable community can approve/add feedback. | ||
|
|
||
| ## Status |
Member
There was a problem hiding this comment.
the rfc looks good, given the implementation is in the same PR as well, you can look at my comments on those
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports
xtable-spark-runtimeonto the Hudi 0.x line (branch-0.4, Hudi 0.14 / Spark 3.4) — a thin, relocated Spark bundle that runs an incremental XTable sync in-job; engines are provided by the cluster, never bundled. Hudi-0.x counterpart of #838; part of the 0.4.0 on-ramp.xtable-spark-runtime(~3.7 MB) withXTableSparkSyncspark-submitentry point; resilientConversionTargetFactoryServiceLoader discovery so a subset of engines works when others are absent.hudi-hadoop-common(1.x split); engine classpath uses thehudi-sparkbundle so Avro model classes link on Avro 1.12 (Iceberg 1.9.2), plushudi-java-client.ITXTableSparkRuntimeBundlecovers each direction across Hudi/Iceberg/Delta. It'sSPARK_HOME-gated, so a dedicated Spark Runtime Bundle Validation workflow installs Spark 3.4.3, setsSPARK_HOME, and runs it (green); the standard CI build skips it.