Skip to content

feat: upgrade Deequ to 2.0.21, Spark 3.5 only (drops Spark 3.1–3.4) #283

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sudsali wants to merge 7 commits into
base: master
Choose a base branch
from
+66 −54
Open

feat: upgrade Deequ to 2.0.21, Spark 3.5 only (drops Spark 3.1–3.4) #283

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
700ca4b
feat: upgrade Deequ to 2.0.21 (Spark 3.5 only)
sudsali Jul 9, 2026
bdbd98f
test: expect Histogram.tailCount row on Deequ 2.0.21
sudsali Jul 9, 2026
eeadb2e
fix: align contract surface + tests with Spark-3.5-only bump
sudsali Jul 9, 2026
fb67fea
chore: regenerate poetry.lock for the pyspark >=3.5,<3.6 constraint
sudsali Jul 9, 2026
98e3efb
fix: correct README version note + align Dockerfile pyspark with lock
sudsali Jul 9, 2026
fdeac1c
docs: derive Sphinx release from package version, not a hardcoded sta…
sudsali Jul 9, 2026
c607cc0
docs: target 1.7.0 for the Spark-3.5-only release
sudsali Jul 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 0 additions & 9 deletions .github/workflows/base.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,6 @@ jobs:
fail-fast: false
matrix:
include:
- PYSPARK_VERSION: "3.1.3"
PYTHON_VERSION: "3.9"
JAVA_VERSION: "11"
- PYSPARK_VERSION: "3.2"
PYTHON_VERSION: "3.9"
JAVA_VERSION: "11"
- PYSPARK_VERSION: "3.3"
PYTHON_VERSION: "3.9"
JAVA_VERSION: "11"
- PYSPARK_VERSION: "3.5"
PYTHON_VERSION: "3.9"
JAVA_VERSION: "17"
Expand Down
11 changes: 5 additions & 6 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@ ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update
RUN apt-get install -y software-properties-common
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt-get install -y python3.8 python3-pip
RUN apt-get install -y python3.8-distutils
RUN apt-get install -y openjdk-11-jdk
RUN apt-get install -y python3.9 python3-pip
RUN apt-get install -y python3.9-distutils
RUN apt-get install -y openjdk-17-jdk

# Update symlink to point to latest
RUN rm /usr/bin/python3 && ln -s /usr/bin/python3.8 /usr/bin/python3
RUN rm /usr/bin/python3 && ln -s /usr/bin/python3.9 /usr/bin/python3
RUN python3 --version
RUN pip3 --version
RUN java -version
Expand All @@ -21,8 +21,7 @@ COPY pyproject.toml /python-deequ
COPY poetry.lock /python-deequ
WORKDIR python-deequ

RUN poetry install -vvv
RUN poetry add pyspark==3.5.0 -vvv
RUN poetry install --extras pyspark -vvv

ENV SPARK_VERSION=3.5
COPY . /python-deequ
Expand Down
10 changes: 6 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ You can install [PyDeequ via pip](https://pypi.org/project/pydeequ/).
pip install pydeequ
```

> **Supported Spark version:** PyDeequ 1.7.0+ tracks the Deequ 2.0.21 JVM library, which is published for **Spark 3.5 only**. For Spark 3.1–3.4, use PyDeequ 1.6.0 (the last release supporting them, on Deequ 2.0.8).

### Set up a PySpark session
```python
from pyspark.sql import SparkSession, Row
Expand Down Expand Up @@ -214,8 +216,8 @@ Install Java Now open favourite terminal and enter the following:
List the Apache Spark versions:
$ sdk list spark

To install For Spark 3
$ sdk install spark 3.0.2
To install For Spark 3.5
$ sdk install spark 3.5.1
```

### Poetry
Expand Down Expand Up @@ -246,8 +248,8 @@ $ poetry run pytest
If you have issues installing the dependencies listed above, another way to run the tests and verify your changes is through Docker. There is a Dockerfile that will install the required dependencies and run the tests in a container.

```
docker build . -t spark-3.3-docker-test
docker run spark-3.3-docker-test
docker build . -t spark-3.5-docker-test
docker run spark-3.5-docker-test
```

## Contributing
Expand Down
8 changes: 6 additions & 2 deletions docs/source/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
#
import os
import sys
from importlib.metadata import PackageNotFoundError, version
from recommonmark.parser import CommonMarkParser
sys.path.insert(0, os.path.abspath('../..'))

Expand All @@ -22,8 +23,11 @@
copyright = 'Copyright 2020, Amazon'
author = 'Calvin Wang, Chris Ghyzel, Joan Aoanan, Veronika Megler'

# The full version, including alpha/beta/rc tags
release = '0.0.4'
# Track the installed package version so the docs don't drift from releases.
try:
release = version("pydeequ")
except PackageNotFoundError:
release = "0.0.0"


# -- General configuration ---------------------------------------------------
Expand Down
25 changes: 13 additions & 12 deletions poetry.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 3 additions & 4 deletions pydeequ/configs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,10 @@
import re


# Deequ 2.0.10+ only publishes a spark-3.5 build; support for Spark 3.1/3.2/3.3

@github-actions github-actions Bot Jul 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DESIGN: The dependency constraint pyspark >=2.4.7,<4.0.0 no longer reflects the Spark-3.5-only support; users on Spark 3.1–3.4 will encounter an import-time RuntimeError with no pip-level guard.

configs.py lines 42-43 run at module import; lines 34-39 raise RuntimeError for non-"3.5" versions. pyproject declares pyspark = ">=2.4.7,<4.0.0" (unchanged by diff). A user on Spark 3.3 installing pydeequ will hit RuntimeError at import pydeequ.

Refutation trail (why this survived the Critic's disprove pass)

Hypothesis (Investigator): Dropping all keys except "3.5" leaves the declared pyspark = ">=2.4.7,<4.0.0" dependency floor inconsistent with actual runtime support, so installs on Spark <3.5 import-fail with RuntimeError.

Disprove attempt (Critic): The pyproject shows pyspark = { version = ">=2.4.7,<4.0.0", optional = true }, unchanged by the diff. configs.py lines 32-39 now raise RuntimeError for any Spark version whose major.minor != "3.5". So a user on an allowed pyspark (e.g. 3.3) hits a hard RuntimeError at import pydeequ (line 43 executes at module load). The dependency metadata and the coord map now disagree; the PR narrows runtime support without narrowing the declared pyspark range.

The Critic's default verdict is OVERTURNED. UPHELD findings are those it tried — and failed — to refute.

# was dropped upstream, so pydeequ tracks Spark 3.5 only on Deequ 2.0.21.
SPARK_TO_DEEQU_COORD_MAPPING = {
"3.5": "com.amazon.deequ:deequ:2.0.8-spark-3.5",
"3.3": "com.amazon.deequ:deequ:2.0.8-spark-3.3",
"3.2": "com.amazon.deequ:deequ:2.0.8-spark-3.2",
"3.1": "com.amazon.deequ:deequ:2.0.8-spark-3.1"
"3.5": "com.amazon.deequ:deequ:2.0.21-spark-3.5",
}


Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ classifiers = [
python = ">=3.9,<4"
numpy = ">=1.14.1"
pandas = ">=0.23.0"
pyspark = { version = ">=2.4.7,<4.0.0", optional = true }
pyspark = { version = ">=3.5,<3.6", optional = true }

[tool.poetry.dev-dependencies]
pytest = "^8.0"
Expand Down
5 changes: 5 additions & 0 deletions tests/test_analyzers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -411,6 +411,9 @@ def test_fail_Histogram(self):
)

def test_Histogram_maxBins(self):
# Deequ 2.0.10+ emits a trailing Histogram.tailCount metric when the
# number of distinct values (3) exceeds maxDetailBins (2): tailCount =
# 3 - 2 = 1, so the final Row(value=1.0) is the rolled-up tail.
self.assertEqual(
self.Histogram_maxBins("b", maxDetailBins=2),
[
Expand All @@ -419,6 +422,7 @@ def test_Histogram_maxBins(self):
Row(value=0.3333333333333333),
Row(value=1.0),
Row(value=0.3333333333333333),
Row(value=1.0),
],
)
self.assertEqual(
Expand All @@ -429,6 +433,7 @@ def test_Histogram_maxBins(self):
Row(value=0.3333333333333333),
Row(value=1.0),
Row(value=0.3333333333333333),
Row(value=1.0),
],
)

Expand Down
15 changes: 0 additions & 15 deletions tests/test_anomaly_detection.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
# -*- coding: utf-8 -*-
import os
import unittest

import pytest
Expand Down Expand Up @@ -282,13 +281,6 @@ def HoltWinters(self, analyzer_func, test, df_prev, df_curr=None):
print(df.collect())
return df.select("check_status").collect()

@pytest.mark.xfail(
os.environ.get("SPARK_VERSION", "").startswith(("3.1", "3.2")),
reason=(
"Not supported in Spark < 3.3: breeze.stats.DescriptiveStats "
"is in unnamed module of loader 'app'"
)
)
def test_BatchNormalStrategy(self):

# Interval is inclusive, so meet the requirements upper value is up to 9
Expand Down Expand Up @@ -357,13 +349,6 @@ def test_OnlineNormalStrategy(self):
[Row(check_status="Success")],
)

@pytest.mark.xfail(
os.environ.get("SPARK_VERSION", "").startswith(("3.1", "3.2")),
reason=(
"Not supported in Spark < 3.3: breeze.stats.DescriptiveStats "
"is in unnamed module of loader 'app'"
)
)
def test_holtWinters(self):
# must have 15 points of data
self.assertEqual(self.HoltWinters(Size(), 1, self.df_1), [Row(check_status="Success")])
Expand Down
28 changes: 27 additions & 1 deletion tests/test_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
import pytest
from pydeequ.configs import _extract_major_minor_versions
from pydeequ.configs import (
_extract_major_minor_versions,
_get_deequ_maven_config,
_get_spark_version,
)


@pytest.mark.parametrize(
Expand All @@ -13,3 +17,25 @@
)
def test_extract_major_minor_versions(full_version, major_minor_version):
assert _extract_major_minor_versions(full_version) == major_minor_version


def test_supported_spark_resolves_to_deequ_coord(monkeypatch):
# _get_spark_version is lru_cached and reads SPARK_VERSION at call time.
monkeypatch.setenv("SPARK_VERSION", "3.5")
_get_spark_version.cache_clear()
try:
assert _get_deequ_maven_config() == "com.amazon.deequ:deequ:2.0.21-spark-3.5"
finally:
_get_spark_version.cache_clear()


@pytest.mark.parametrize("unsupported", ["3.3", "3.2", "3.1", "3.4"])
def test_unsupported_spark_raises(monkeypatch, unsupported):
# Spark versions Deequ no longer publishes a build for must fail loudly.
monkeypatch.setenv("SPARK_VERSION", unsupported)
_get_spark_version.cache_clear()
try:
with pytest.raises(RuntimeError, match="incompatible Spark version"):
_get_deequ_maven_config()
finally:
_get_spark_version.cache_clear()
Loading