[GH-3044] Fix aggregate function registration in sessions cloned from FunctionRegistry.builtin

[jiayuasu]

Did you read the Contributor Guide?

• Yes, I have read the Contributor Rules and Contributor Development Guide

Is this PR related to a ticket?

• Yes, and the PR name follows the format [GH-XXX] my subject. Closes In Sedona 1.9.0 aggregate functions fail on the second call within a Spark-Connect session #3044

What changes were proposed in this PR?

Sedona registers aggregate functions in two places: the real UDAF via udf.register in the session registry, and an entry in the global FunctionRegistry.builtin that throws UnsupportedOperationException on invocation (added in #2420 for permanent VIEW support). Every new SparkSession clones FunctionRegistry.builtin into its session registry, so any session created after the first SedonaContext.create() in the JVM starts out holding the throwing placeholder under the aggregate names.

The functionExists() guard added in #2642 mistook that cloned placeholder for a real registration and skipped udf.register, so the real UDAF was never installed and the session failed with Aggregate function ST_Envelope_Aggr cannot be used as a regular function. Spark Connect creates a new server-side session per client session, which is why the reported repro fails on the second session (1.8.1 re-registered unconditionally, which silently overwrote the cloned placeholder).

Two commits:

1. Re-register over the placeholder. registerAggregateFunction now probes the existing entry by invoking its builder: the placeholder throws, a real entry builds an expression. The real UDAF is re-registered only when the entry is not invocable, so the Simply the warning messages of SedonaContext.create() #2640 behavior (no re-registration noise on repeated create()) is preserved. Also extends the scalar skip check to cover Sedona expression classes under org.apache.spark.sql.sedona_sql, which the org.apache.sedona prefix never matched.
2. Back the builtin entry with the real builder. FunctionRegistry.builtin now receives the actual ScalaAggregator builder instead of a throwing placeholder, so cloned registries resolve Sedona aggregates by construction (even in sessions that never ran SedonaContext.create). The required Spark API is package-private and moved between versions, so it is loaded reflectively: UserDefinedAggregator.scalaAggregator on Spark 3.4/3.5, the ScalaAggregator companion apply on Spark 4.x. If neither is found, the previous placeholder behavior is kept and the probe from commit 1 still applies.

The first commit is intentionally minimal and self-contained so it can be cherry-picked to a 1.9.1 patch release on its own.

How was this patch tested?

Two new tests in aggregateFunctionTestScala:

• ST_Envelope_Aggr, ST_Envelope_Agg, and ST_Union_Aggr in a newSession() after SedonaContext.create — reproduces In Sedona 1.9.0 aggregate functions fail on the second call within a Spark-Connect session #3044 without Spark Connect; fails on current master with the exact exception from the report.
• ST_Envelope_Aggr in a newSession() that never ran SedonaContext.create — verifies the builtin entry is the real builder.

Full suite run locally on Spark 3.4 / Scala 2.12 / JDK 11 and Spark 4.0 / Scala 2.13 / JDK 17 (30/30 passing), covering both reflection paths.

Did this PR include necessary documentation updates?

• No, this PR does not affect any public API so no need to change the documentation.

[Copilot]

Pull request overview

Fixes Sedona SQL aggregate function resolution/registration for SparkSessions that are created by cloning FunctionRegistry.builtin (notably impacting Spark Connect), ensuring Sedona aggregate UDAFs are correctly invocable in later sessions and can also be resolved purely from the cloned built-in registry.

Changes:

• Update aggregate registration to detect and overwrite non-invocable placeholder entries in a session’s function registry.
• Register Sedona aggregates into FunctionRegistry.builtin using a real aggregate builder when possible (via Spark-version-specific reflective access), falling back to the prior placeholder only when necessary.
• Add regression tests covering aggregate usage in a newSession() both with and without calling SedonaContext.create() on that session.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
spark/common/src/main/scala/org/apache/sedona/sql/UDF/AbstractCatalog.scala	Fixes session/builtin aggregate registration by probing invocability and (when possible) registering real built-in aggregate builders via reflection across Spark versions.
spark/common/src/test/scala/org/apache/sedona/sql/aggregateFunctionTestScala.scala	Adds regression tests reproducing #3044 and validating aggregate resolution in cloned/new sessions.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.