Segmentation fault or Floating point exception(core dumped) error when importing pyrosetta with Tensorflow #88

ncoish · 2024-07-10T19:51:36Z

ncoish
Jul 10, 2024

Originally posted by my colleague on the RosettaCommons forums here.

A very unusual problem is happening. If I use both Pyrosetta and Tensorflow in the same python script, depending on which I import first, I get one of two errors:

importing tensorflow followed by pyrosetta -> Segmentation Fault(core dumped)
importing pyrosetta followed by tensorflow -> Floating point exception(core dumped)

My environment includes:

python 3.11
Linux - Fedora 40
Installing the release wheel through pip from an internal mirror

This happens for all versions of Pyrosetta that I've tested (2023.45+release and 2024.24+release), and with all Tensorflow versions I've tested (2.14, 2.15, 2.16). Several colleagues are running into the same problem on their machines.

Additionally, one of my colleagues who runs into this problem does not have this issue when installed dl_binder_design's dependencies through conda, which appears to include both pyrosetta and tensorflow

See the below quote for details from the original reported issue, including a core dump indicating an issue during import coming from _pywrap_tf2.so

Hi my workflow involves importing pyrosetta and tensorflow together in a single python script. And whenever I import both modules and run the program, I'm keep getting segmentation fault (core dumped) error. Here is my code:

import tensorflow as tf
from pyrosetta import *

If I put them other way around

from pyrosetta import *
import tensorflow as tf

I get Floating point exception (core dumped)

Here is my system information:

OS type: Fedora Linux 38 (64 bit)

Python: 3.11.4

Pyrosetta: pyrosetta-2023.45+release.a6d9ba8-cp311-cp311-linux_x86_64.whl

Below is the output I get from gdb debugger:

Thread 1 "python" received signal SIGFPE, Arithmetic exception.
0x00007fff7b1a4831 in std::__detail::_Map_base<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, void*>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, void*> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::hash<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::__cxx11::basic_string<char, std::char_traits, std::allocator >&&) () from /home/ju/.cache/pypoetry/virtualenvs/hercules-An2atp8I-py3.11/lib/python3.11/site-packages/tensorflow/python/platform/_pywrap_tf2.so
(gdb) backtrace
#0 0x00007fff7b1a4831 in std::__detail::_Map_base<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, void*>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, void*> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::hash<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::__cxx11::basic_string<char, std::char_traits, std::allocator >&&) () from /home/ju/.cache/pypoetry/virtualenvs/hercules-An2atp8I-py3.11/lib/python3.11/site-packages/tensorflow/python/platform/_pywrap_tf2.so
#1 0x00007fff7b1a467f in pybind11::detail::local_internals::local_internals() () from /home/ju/.cache/pypoetry/virtualenvs/hercules-An2atp8I-py3.11/lib/python3.11/site-packages/tensorflow/python/platform/_pywrap_tf2.so
#2 0x00007fff7b1a6207 in pybind11::detail::loader_life_support::get_stack_tls_key() () from /home/ju/.cache/pypoetry/virtualenvs/hercules-An2atp8I-py3.11/lib/python3.11/site-packages/tensorflow/python/platform/_pywrap_tf2.so
#3 0x00007fff7b1a237c in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) () from /home/ju/.cache/pypoetry/virtualenvs/hercules-An2atp8I-py3.11/lib/python3.11/site-packages/tensorflow/python/platform/_pywrap_tf2.so
#4 0x000000000055505b in ?? ()
#5 0x000000000052f0ec in _PyObject_MakeTpCall ()
#6 0x000000000053d28e in _PyEval_EvalFrameDefault ()
#7 0x0000000000613134 in ?? ()
#8 0x0000000000612797 in PyEval_EvalCode ()
#9 0x000000000062c513 in ?? ()
#10 0x000000000054adab in ?? ()
#11 0x00000000005458d7 in _PyEval_EvalFrameDefault ()
#12 0x0000000000565f23 in _PyFunction_Vectorcall ()
#13 0x0000000000553b10 in ?? ()
#14 0x0000000000583dc1 in PyObject_CallMethodObjArgs ()
#15 0x0000000000583860 in PyImport_ImportModuleLevelObject ()
#16 0x0000000000598d96 in ?? ()
#17 0x000000000054adab in ?? ()
#18 0x00000000005458d7 in _PyEval_EvalFrameDefault ()
#19 0x0000000000565f23 in _PyFunction_Vectorcall ()
#20 0x0000000000553b10 in ?? ()
#21 0x0000000000583dc1 in PyObject_CallMethodObjArgs ()
#22 0x0000000000583860 in PyImport_ImportModuleLevelObject ()
#23 0x0000000000598d96 in ?? ()
#24 0x000000000054adab in ?? ()
#25 0x00000000005458d7 in _PyEval_EvalFrameDefault ()
#26 0x0000000000565f23 in _PyFunction_Vectorcall ()
#27 0x0000000000553b10 in ?? ()
#28 0x0000000000583dc1 in PyObject_CallMethodObjArgs ()
#29 0x0000000000583860 in PyImport_ImportModuleLevelObject ()
#30 0x00000000005428f7 in _PyEval_EvalFrameDefault ()
#31 0x0000000000613134 in ?? ()
#32 0x0000000000612797 in PyEval_EvalCode ()
#33 0x00000000006335ab in ?? ()
#34 0x000000000062f7f4 in ?? ()
#35 0x0000000000643e95 in ?? ()
#36 0x0000000000643504 in _PyRun_SimpleFileObject ()
#37 0x0000000000643317 in _PyRun_AnyFileObject ()
#38 0x000000000063dea9 in Py_RunMain ()
#39 0x0000000000603b0d in Py_BytesMain ()
#40 0x00007ffff7c29d90 in __libc_start_call_main (main=main@entry=0x603a60, argc=argc@entry=2, argv=argv@entry=0x7fffffffdde8) at ../sysdeps/nptl/libc_start_call_main.h:58
#41 0x00007ffff7c29e40 in __libc_start_main_impl (main=0x603a60, argc=2, argv=0x7fffffffdde8, init=, fini=, rtld_fini=, stack_end=0x7fffffffddd8) at ../csu/libc-start.c:392
#42 0x0000000000603995 in _start ()

Answered by ncoish

Jul 11, 2024

@lyskov I actually have an update on this: Despite running on Fedora, as a last-ditch effort I tried installing the Ubuntu wheel (which is separate from the general "linux" wheel), and this error does not occur! Additionally things seem to run fine for me, even though I'm not on Ubuntu. Do you happen to know what the difference is between these packages? Would you expect the Ubuntu wheel to give me any trouble?

In either case I think this isolates the issue to the general "linux" wheel.

To answer your question, I have not tried Python 3.10, but I can give that a try tomorrow if it will help you debug. Otherwise it seems like I have a working alternative.

View full answer

ncoish · 2024-07-10T22:05:50Z

ncoish
Jul 10, 2024
Author

Some additional context.
After installing the pyrosetta debug wheel (https://graylab.jhu.edu/download/PyRosetta4/archive/release/PyRosetta4.Debug.python311.linux.wheel/pyrosetta-2024.24+release.ca096da-cp311-cp311-linux_x86_64.whl)
Along with tensorflow 2.13.1 to match the current conda package from dl_binder_design, I get the following outputs when switching the import order. I hope these are helpful:

importing tensorflow first:

Python 3.11.6 (main, Oct 18 2023, 12:05:05) [GCC 13.2.1 20230728 (Red Hat 13.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow
2024-07-10 17:57:50.895347: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-07-10 17:57:50.932679: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-07-10 17:57:50.933167: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-07-10 17:57:51.922082: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
>>> import pyrosetta
=slt=bcl::util=> caught signal: 11 cleaning! Call Stack
/home/ncoish/.cache/pypoetry/virtualenvs/farfalle-JYS8fO-N-py3.11/lib/python3.11/site-packages/pyrosetta/rosetta.so: bcl::util::CleanableInterface::Cleanables::SignalHandler(int)
/home/ncoish/.cache/pypoetry/virtualenvs/farfalle-JYS8fO-N-py3.11/lib/python3.11/site-packages/pyrosetta/rosetta.so: bind_std_type_traits(std::function<pybind11::module_& (std::string const&)>&)
/home/ncoish/.cache/pypoetry/virtualenvs/farfalle-JYS8fO-N-py3.11/lib/python3.11/site-packages/pyrosetta/rosetta.so: PyInit_rosetta
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: _PyEval_EvalFrameDefault
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: PyObject_CallMethodObjArgs
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: PyImport_ImportModuleLevelObject
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: _PyEval_EvalFrameDefault
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: PyEval_EvalCode
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: _PyEval_EvalFrameDefault
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: PyObject_CallMethodObjArgs
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: PyImport_ImportModuleLevelObject
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: _PyEval_EvalFrameDefault
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: PyEval_EvalCode
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: _PyRun_InteractiveLoopObject
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: _PyRun_AnyFileObject

Segmentation fault (core dumped)

Importing Pyrosetta first:

Python 3.11.6 (main, Oct 18 2023, 12:05:05) [GCC 13.2.1 20230728 (Red Hat 13.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyrosetta
>>> import tensorflow
2024-07-10 17:56:57.742970: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-07-10 17:56:57.945343: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
=slt=bcl::util=> caught signal: 8 cleaning! Call Stack
/home/ncoish/.cache/pypoetry/virtualenvs/farfalle-JYS8fO-N-py3.11/lib/python3.11/site-packages/pyrosetta/rosetta.so: bcl::util::CleanableInterface::Cleanables::SignalHandler(int)
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: _PyObject_MakeTpCall
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: _PyEval_EvalFrameDefault
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: PyEval_EvalCode
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: _PyEval_EvalFrameDefault
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: PyObject_CallMethodObjArgs
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: PyImport_ImportModuleLevelObject
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: _PyEval_EvalFrameDefault
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: PyObject_CallMethodObjArgs
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: PyImport_ImportModuleLevelObject
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: _PyEval_EvalFrameDefault
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: PyEval_EvalCode
/home/ncoish/.pyenv/versions/3.11.6/lib/libpython3.11.so.1.0: _PyEval_EvalFrameDefault

Floating point exception (core dumped)

4 replies

lyskov Jul 11, 2024
Maintainer

@ncoish thank you for info! Have you tried to use different Python versions? I wonder if 3.10 might work (since it works in Colab env).
Thanks,

ncoish Jul 11, 2024
Author

@lyskov I actually have an update on this: Despite running on Fedora, as a last-ditch effort I tried installing the Ubuntu wheel (which is separate from the general "linux" wheel), and this error does not occur! Additionally things seem to run fine for me, even though I'm not on Ubuntu. Do you happen to know what the difference is between these packages? Would you expect the Ubuntu wheel to give me any trouble?

In either case I think this isolates the issue to the general "linux" wheel.

To answer your question, I have not tried Python 3.10, but I can give that a try tomorrow if it will help you debug. Otherwise it seems like I have a working alternative.

Answer selected by lyskov

lyskov Jul 11, 2024
Maintainer

Oh that is very interesting! The main difference between Linux and Ubuntu binaries is the compiler used to build it (much newer version on Ubuntu). I think this will be super helpful workaround.

And yes, - if Ubuntu binaries works for you, - use them! Glad to hear that this is now working.

lyskov Jul 11, 2024
Maintainer

(I was planning to deprecate 'linux' (which is CentOS-7) binaries soon anyway since whole distro now is no longer supported.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segmentation fault or Floating point exception(core dumped) error when importing pyrosetta with Tensorflow #88

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Segmentation fault or Floating point exception(core dumped) error when importing pyrosetta with Tensorflow #88

ncoish Jul 10, 2024

Replies: 1 comment · 4 replies

ncoish Jul 10, 2024 Author

lyskov Jul 11, 2024 Maintainer

ncoish Jul 11, 2024 Author

lyskov Jul 11, 2024 Maintainer

lyskov Jul 11, 2024 Maintainer

ncoish
Jul 10, 2024

Replies: 1 comment 4 replies

ncoish
Jul 10, 2024
Author

lyskov Jul 11, 2024
Maintainer

ncoish Jul 11, 2024
Author

lyskov Jul 11, 2024
Maintainer

lyskov Jul 11, 2024
Maintainer