Fixing "TypeError: Descriptors cannot not be created directly" in Python

Problem Overview

The "TypeError: Descriptors cannot not be created directly" error commonly occurs when working with Google's Protocol Buffers (protobuf) library in Python. This issue typically arises due to version incompatibilities between the protobuf package and other libraries that depend on it, such as TensorFlow, Ray, Keras, or various Google Cloud services.

The error message explicitly states:

TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

Root Cause

This error emerged when protobuf version 4.21.0 was released, which introduced breaking changes that affected many popular Python libraries. The protobuf library is a dependency for numerous data serialization and machine learning frameworks, making this a widespread issue.

Solutions

Primary Solution: Downgrade Protobuf

The most reliable fix is to downgrade the protobuf package to a compatible version (3.20.x or lower):

pip install --upgrade "protobuf<=3.20.1"

For a forced reinstallation:

pip install 'protobuf<=3.20.1' --force-reinstall

Alternatively, pin a specific 3.20.x version:

pip install protobuf==3.20.3

Version Compatibility

Ensure the downgraded protobuf version is compatible with your other dependencies. Some users have reported success with specific combinations:

• TensorFlow 1.15.5 + protobuf 3.15.0
• TensorFlow 2.8 + protobuf 3.20.*
• wandb 0.12.17 + protobuf 3.15.0

Alternative Solution: Update Dependent Packages

If downgrading protobuf causes issues with other dependencies, consider updating the packages that depend on protobuf:

# Update TensorFlow
pip install tensorflow==2.12.0

# Update other specific packages that might be causing the issue
pip install streamlit --upgrade
pip install pychromecast --upgrade
pip install google-cloud-audit-log==0.2.4

Environment Variable Workaround

As suggested in the error message, you can set an environment variable as a temporary workaround:

export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python

Performance Impact

This workaround forces pure-Python parsing, which may significantly impact performance. Use this only as a temporary solution.

Conda Installation Method

If using Conda alongside pip, you might need to coordinate installations:

pip uninstall protobuf
conda install 'protobuf=3.20.1'

Verify Installation Location

Ensure the correct protobuf version is installed in your Python environment:

python -c "import protobuf; print(protobuf.__version__)"

Check that this matches the version shown in:

pip list | grep protobuf

Prevention Strategies

To avoid similar issues in the future:

1. Pin versions in your requirements.txt file:

   protobuf==3.20.*

2. Add protobuf as the last dependency in your requirements file to ensure it overrides any conflicting versions installed by other packages.

3. Use virtual environments to isolate project dependencies and prevent conflicts.

4. Regularly update your dependencies to maintain compatibility.

Common Affected Packages

This error has been reported with various packages including:

• TensorFlow and Keras
• Ray framework
• Streamlit
• Google Cloud libraries (google-cloud-logging, google-cloud-audit-log)
• wandb (Weights & Biases)
• pychromecast

Conclusion

The "Descriptors cannot not be created directly" error is primarily a version compatibility issue between protobuf and its dependent packages. The most reliable solution is downgrading protobuf to version 3.20.x or lower. If this isn't feasible, updating the dependent packages or using the environment variable workaround are alternative approaches.

Always ensure your development environment has consistent package versions to prevent such compatibility issues.