[SPARK-55716][SQL] Support NOT NULL constraint enforcement for V1 file source table inserts

[yaooqinn]

What changes were proposed in this pull request?

V1 file-based DataSource writes (Parquet, ORC, JSON, etc.) silently accept null values into NOT NULL columns. This PR adds opt-in NOT NULL constraint enforcement controlled by spark.sql.fileSource.insert.enforceNotNull.

Changes:

1. CreateDataSourceTableCommand: Preserves user-specified nullability by recursively merging nullability flags from the user schema into the resolved dataSource.schema. Previously it stored dataSource.schema directly, which is all-nullable due to DataSource.resolveRelation() calling dataSchema.asNullable (SPARK-13738).

2. PreprocessTableInsertion: Restores nullability flags from the catalog schema before null checks, ensuring AssertNotNull is injected when needed. Gated behind spark.sql.fileSource.insert.enforceNotNull.

3. New config: spark.sql.fileSource.insert.enforceNotNull (default false) — when set to true, enables NOT NULL constraint enforcement for V1 file-based tables, consistent with the behavior for other data sources and V2 catalog tables.

4. SparkGetColumnsOperation: Fixed IS_NULLABLE to respect column.nullable instead of always returning "YES".

Why are the changes needed?

DataSource.resolveRelation() calls dataSchema.asNullable (added in SPARK-13738 for read safety), which strips all NOT NULL constraints recursively. CreateDataSourceTableCommand then stores this all-nullable schema in the catalog, permanently losing NOT NULL information. As a result, PreprocessTableInsertion never injects AssertNotNull for V1 file source tables.

Note: InsertableRelation (e.g., SimpleInsertSource) does NOT have this problem because it preserves the original schema (SPARK-24583).

Does this PR introduce any user-facing change?

No change in default behavior. Users can opt in to NOT NULL enforcement for V1 file source tables by setting spark.sql.fileSource.insert.enforceNotNull to true.

How was this patch tested?

• Added 7 new tests in InsertSuite covering top-level, nested struct, array, and map null constraint enforcement.
• Fixed 3 existing interval column test assertions in SparkMetadataOperationSuite.

Was this patch authored or co-authored using generative AI tooling?

Yes, co-authored with GitHub Copilot.

[yaooqinn]

cc @dongjoon-hyun @cloud-fan @gengliangwang, Is the fix direction correct? is this a genuine bug or design choice. I haven't found any public discussions on this area.

[dongjoon-hyun]

Hi, @yaooqinn

• Apache Spark didn't claim to support SQL CONSTRAINT for v1 yet, did it? IIUC, it should be handled explicitly by the user in FROM SELECT clause of that INSERT statement.
• SPARK-51207: SPIP: Constraints in DSv2 was a fairly new feature of Apache Spark 4.1.0 only.

For me, this PR seems to introduce a new feature instead of bugs.

cc @aokolnychyi , @peter-toth , too.

[yaooqinn]

Apache Spark didn't claim to support SQL CONSTRAINT for v1 yet, did it? IIUC, it should be handled explicitly by the user in FROM SELECT clause of that INSERT statement.

@dongjoon-hyun
I'm not sure,StoreAssignmentPolicy is introduced in 3.0, in the meanwhile NOT NULL keywords are long-term support（but may not be fully or well documented） features. It looks like that V2 support is an addition based on that. In legacy v1, it partially work for some datasources with no asNullable called

[dongjoon-hyun]

IIUC, StoreAssignmentPolicy is supposed to check type casting only. For a single nullable type, I'm not sure that was achieved by SPARK-28730.

• [SPARK-28730][SQL] Configurable type coercion policy for table insertion #25453

[dongjoon-hyun]

Anyway, let's wait for @gengliangwang and @cloud-fan 's comment. Maybe, I lost the track of code change.

[yaooqinn]

IIUC, StoreAssignmentPolicy is supposed to check type casting only.

@dongjoon-hyun
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala#L447

FYI, Null constraints for table output relies on it too

[dongjoon-hyun]

Got it. So, from Apache Spark 3.5.0 via the following? Then, cc @aokolnychyi , too.

• [SPARK-42855][SQL] Use runtime null checks in TableOutputResolver #40655
• [SPARK-42151][SQL] Align UPDATE assignments with table attributes #40308

[cloud-fan]

This is probably a too breaking change, and many users may already treat this bug as a feature.

If users need strict not null enforcement, they should migrate to DS v2.

[gengliangwang]

+1 with @cloud-fan . At least, the default behavior should not be changed in 4.2.

[yaooqinn]

Yeah, I agree with you that this is too breaking. However, many of our clients and developers here have raised doubts about this strange behavior of Spark. There are no clear docs for this.

they should migrate to DS v2.

Do you mean by using built-in sources with v2 code path or suggesting users migrate to third-party formats?

[cloud-fan]

I think using plain file source table is not recommended now, people should switch to lakehouse table formats.

[dongjoon-hyun]

Since DSv2 migration is an independent topic from this DSv1 bug, I made a PR for DSv2 specifically, @cloud-fan , @gengliangwang , @yaooqinn .

• [SPARK-55748][SQL] Use DSv2 for avro|csv|json|kafka|orc|parquet|text by default #54547

[yaooqinn]

Hi @dongjoon-hyun builtin file sources with DSv2 code path have the same issue based on my unit tests

[yaooqinn]

@cloud-fan Instead of making NOT NULL as a default behavior, would you mind if we give it a shot that provide an alternative option for users if they want built-in parquet and delta lake to behave the same on NOT NULL constraints?

For steaming source, we have such an option for toggling asNullable.