Skip to content

Commit 24b9d2c

Browse files
Yicong-HuangHyukjinKwon
authored andcommitted
[SPARK-55965][PYTHON] Add warning when pandas >= 3.0.0 is used with PySpark
### What changes were proposed in this pull request? Add a `FutureWarning` in `require_minimum_pandas_version()` when pandas >= 3.0.0 is detected. ### Why are the changes needed? PySpark does not yet fully support pandas >= 3.0.0. This adds a warning to inform users. ### Does this PR introduce _any_ user-facing change? Yes. Users with pandas >= 3.0.0 will see a `FutureWarning` when using PySpark pandas-related features. ### How was this patch tested? Existing tests. Closes #54760 from Yicong-Huang/SPARK-55965. Authored-by: Yicong Huang <17627829+Yicong-Huang@users.noreply.github.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
1 parent fa87249 commit 24b9d2c

File tree

1 file changed

+10
-0
lines changed

1 file changed

+10
-0
lines changed

python/pyspark/sql/pandas/utils.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,16 @@ def require_minimum_pandas_version() -> None:
5656
"current_version": str(pandas.__version__),
5757
},
5858
)
59+
if LooseVersion(pandas.__version__) >= LooseVersion("3.0.0"):
60+
import warnings
61+
62+
warnings.warn(
63+
"PySpark does not yet fully support pandas >= 3.0.0. "
64+
"Some features may not work correctly. "
65+
"It is recommended to use pandas < 3.0.0 for now.",
66+
FutureWarning,
67+
stacklevel=2,
68+
)
5969

6070

6171
def require_minimum_pyarrow_version() -> None:

0 commit comments

Comments
 (0)