Fixing BaseSettings ImportError When Using vaex

Problem

When importing the vaex library (version 4.16.0) in a SageMaker notebook environment, you encounter the following error:

PydanticImportError: `BaseSettings` has been moved to the `pydantic-settings` package.
See https://docs.pydantic.dev/2.0.2/migration/#basesettings-has-moved-to-pydantic-settings for more details.

This error occurs due to recent changes in pydantic version 2, where the BaseSettings class has been moved to a separate package. Since vaex (or one of its dependencies) tries to import BaseSettings from the old location, you need to manually resolve the dependency conflict.

Solutions

Install pydantic-settings (Recommended)

The most reliable solution is to install the pydantic-settings package:

pip install pydantic-settings

This creates a smooth upgrade path by providing the required module while maintaining compatibility with other packages.

Use Pydantic's v1 Compatibility Import (Alternative)

If you need to maintain other dependencies that require pydantic v1:

# Replace old import:
# from pydantic import BaseSettings
from pydantic.v1 import BaseSettings  # Use v1 compatibility layer

Update Third-Party Libraries

For library maintainers migrating their codebase, replace imports:

# Replace old import:
# from pydantic import BaseSettings
from pydantic_settings import BaseSettings  # New import path

Verify Installation in SageMaker

After installing dependencies, validate your environment in SageMaker:

# Check installed versions
import vaex
import pydantic
print(f"vaex version: {vaex.__version__}")
print(f"pydantic version: {pydantic.version.VERSION}")

Explanation

The error occurs because pydantic version 2 introduced breaking changes:

• BaseSettings moved to the separate pydantic-settings package
• Applications using this class must either:
  • Install the new package OR
  • Use pydantic's v1 compatibility layer

<warning> **Version Conflicts** Downgrading `pydantic` (e.g., `pip install "pydantic<2"`) may cause other dependency conflicts. Installing `pydantic-settings` is preferred. </warning> <tip> **Environment Management** In SageMaker notebooks, always restart the kernel after installing new packages to ensure environment changes take effect. </tip>

Best Practices

1. Explicit Version Pinning
   Add requirements to your requirements.txt:

   pydantic>=2.0
   pydantic-settings>=2.0

2. Test Compatibility
   Verify library compatibility before deployment:

   pip install pytest
   pytest tests/test_pydantic_compatibility.py

3. Monitor Updates
   Check vaex GitHub for pydantic v2 compatibility updates. Newer versions are likely to resolve this dependency natively.