Add CUDA version compatibility check #1412

Andy-Jost · 2025-12-18T19:06:15Z

Summary

Adds a version compatibility check that warns users when cuda-bindings was compiled against a newer CUDA major version than the installed driver supports.

Changes

cuda-bindings

Added check_cuda_version_compatibility() function in cuda/bindings/utils/_version_check.py
Compares compile-time CUDA_VERSION vs runtime cuDriverGetVersion()
Exported from cuda.bindings.utils
Added comprehensive unit tests in tests/test_version_check.py

cuda-core

Device.__new__ calls check_cuda_version_compatibility() after cuInit succeeds
Imports the function directly from cuda.bindings.utils

Rationale

When cuda-bindings is built against CUDA 13 headers but the user's driver only supports CUDA 12, many features will silently fail or behave unexpectedly. This check provides early, clear feedback:

UserWarning: cuda-bindings was built against CUDA 13.0, but the installed driver 
only supports CUDA 12.8. Some features may not work correctly. Consider updating 
your NVIDIA driver. Set CUDA_PYTHON_DISABLE_VERSION_CHECK=1 to suppress this warning.

Design

Provided by cuda-bindings: The version check implementation lives in cuda.bindings.utils since it checks cuda-bindings' compile-time version
Invoked by cuda-core: Called when Device first triggers CUDA initialization
Runs once: Uses a module-level flag to ensure the check runs only once per process
Non-blocking: Warning only, does not prevent operation
Suppressible: Set CUDA_PYTHON_DISABLE_VERSION_CHECK=1 to disable
Major version only: No warning for minor version differences (handled by graceful degradation per PR Handle unsupported device attributes gracefully #1409)

Future Work

We could not find a suitable place to invoke the version check automatically within cuda-bindings itself (e.g., hooking into cuInit), so the check is currently triggered by cuda-core. This may be revisited in the future.

Test Coverage

7 tests in cuda-bindings covering:

No warning when driver is newer
No warning when same major version
Warning when compile major > driver major
Warning only issued once per process
Suppression via environment variable
Silent handling of driver errors
Silent handling of missing CUDA_VERSION attribute

Related Work

Complements PR Handle unsupported device attributes gracefully #1409 which handles graceful degradation for minor version mismatches

copy-pr-bot · 2025-12-18T19:06:18Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Andy-Jost · 2025-12-18T19:06:28Z

/ok to test 7ce325c

Andy-Jost · 2025-12-18T19:08:17Z

/ok to test 1962e35

github-actions · 2025-12-18T19:20:22Z

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-1412/
https://nvidia.github.io/cuda-python/pr-preview/pr-1412/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-1412/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-1412/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

cuda_core/cuda/core/_utils/cuda_utils.pyx

cuda_bindings/docs/source/environment_variables.rst

leofang · 2025-12-18T20:31:03Z

cuda_core/cuda/core/_utils/cuda_utils.pyx

+            f"but the installed driver only supports CUDA {runtime_major}.{runtime_minor}. "
+            f"Some features may not work correctly. Consider updating your NVIDIA driver. "


This statement is nerve wrecking considering that we support CUDA Minor Version Compatibility. In a few cases, we check the minor version in cuda-core to ensure what we need is available at run time, but only when we actually need them. I think we should rephrase this warning.

This warning is only issued in the case of a MAJOR version mismatch.

Andy-Jost · 2025-12-19T00:10:25Z

/ok to test 73e5a43

Warn when cuda-bindings was compiled against a newer CUDA major version than the installed driver supports. This helps users understand why certain features may not work correctly. The check runs once after cuInit and can be suppressed via CUDA_PYTHON_DISABLE_VERSION_CHECK=1.

Andy-Jost · 2025-12-19T00:48:39Z

/ok to test ddd92bd

rparolin · 2025-12-19T23:52:44Z

cuda_bindings/cuda/bindings/utils/_version_check.py

+    # Get runtime driver version
+    err, runtime_version = driver.cuDriverGetVersion()
+    if err != driver.CUresult.CUDA_SUCCESS:
+        return  # Can't check, skip silently


Not being able to query from the driver version is worthy of a warning to the user instead of silently eating it.

rparolin · 2025-12-19T23:56:15Z

cuda_bindings/cuda/bindings/utils/_version_check.py

+        )
+
+
+def _reset_version_compatibility_check():


Is there a real use case where production code would call this function? If it’s only used by tests, then it seems reasonable to rely implicitly on the global and move the function into the test code instead.

rparolin

We should issue a warning to the user if we can't fetch the driver version.

Andy-Jost added this to the cuda.core beta 11 milestone Dec 18, 2025

Andy-Jost added P0 High priority - Must do! feature New feature or request cuda.core Everything related to the cuda.core module labels Dec 18, 2025

Andy-Jost self-assigned this Dec 18, 2025

Andy-Jost requested review from leofang, rparolin and rwgk December 18, 2025 19:06

Andy-Jost force-pushed the runtime-version-check branch from 7ce325c to 1962e35 Compare December 18, 2025 19:08

mdboom approved these changes Dec 18, 2025

View reviewed changes

Andy-Jost mentioned this pull request Dec 18, 2025

fix: derive CUDA major version from headers for build #1395

Open

2 tasks

leofang requested changes Dec 18, 2025

View reviewed changes

cuda_core/cuda/core/_utils/cuda_utils.pyx Outdated Show resolved Hide resolved

leofang reviewed Dec 18, 2025

View reviewed changes

cuda_bindings/docs/source/environment_variables.rst Show resolved Hide resolved

leofang reviewed Dec 18, 2025

View reviewed changes

Andy-Jost force-pushed the runtime-version-check branch 2 times, most recently from 8a0d248 to 73e5a43 Compare December 19, 2025 00:09

Andy-Jost added 2 commits December 18, 2025 16:47

Merge remote-tracking branch 'origin/main' into runtime-version-check

ddd92bd

Andy-Jost force-pushed the runtime-version-check branch from 73e5a43 to ddd92bd Compare December 19, 2025 00:47

leofang self-requested a review December 19, 2025 01:22

rparolin reviewed Dec 19, 2025

View reviewed changes

rparolin requested changes Dec 20, 2025

View reviewed changes

		f"but the installed driver only supports CUDA {runtime_major}.{runtime_minor}. "
		f"Some features may not work correctly. Consider updating your NVIDIA driver. "

		)


		def _reset_version_compatibility_check():

Add CUDA version compatibility check #1412

Are you sure you want to change the base?

Add CUDA version compatibility check #1412

Conversation

Andy-Jost commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

cuda-bindings

cuda-core

Rationale

Design

Future Work

Test Coverage

Related Work

Uh oh!

copy-pr-bot bot commented Dec 18, 2025

Uh oh!

Andy-Jost commented Dec 18, 2025

Uh oh!

Andy-Jost commented Dec 18, 2025

Uh oh!

github-actions bot commented Dec 18, 2025

Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

Uh oh!

Uh oh!

leofang Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

Andy-Jost Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

Andy-Jost commented Dec 19, 2025

Uh oh!

Andy-Jost commented Dec 19, 2025

Uh oh!

rparolin Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

rparolin Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

rparolin left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Andy-Jost commented Dec 18, 2025 •

edited

Loading