Error on train 0 model

[OmarAshkar]

Hello,
I am trying to use the tool. I run "Make patches", but the next step fails with

ERROR: train_autoencoder (job 18) failed. 

I have build with docker container.

Any help is appreciated.

Best Regards,
Omar

[nanli-emory]

Hi @OmarAshkar, could you provide more information about your run environments such as the version of OS, python, and docker? what GPUs do you use? It will help us to narrow down the issues. Thank you.

[VolodymyrChapman]

TL;DR Downgrade protobuf to 3.20.x or below
I recently built the environment from the CUDA 11 requirements.txt on Ubuntu 22.04 and this same step failed due to protobuf (cannot find the exact reason for the error now) but the associated guidance in the error was to downgrade protobuf to version <=3.20.x .
I performed:
$ pip install protobuf==3.20.0
After which, the environment worked in all functions (training model 0, embedding, superpixel generation and retraining of DL). My working environment:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
absl-py                   2.0.0                    pypi_0    pypi
albumentations            0.4.3                    pypi_0    pypi
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
cachetools                4.2.4                    pypi_0    pypi
certifi                   2023.7.22                pypi_0    pypi
chardet                   4.0.0                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
config                    0.4.2                    pypi_0    pypi
contourpy                 1.1.1                    pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
flask                     1.1.2                    pypi_0    pypi
flask-restless            0.17.0                   pypi_0    pypi
flask-sqlalchemy          2.4.4                    pypi_0    pypi
fonttools                 4.43.1                   pypi_0    pypi
google-auth               1.35.0                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
grpcio                    1.59.0                   pypi_0    pypi
idna                      2.10                     pypi_0    pypi
imageio                   2.9.0                    pypi_0    pypi
imgaug                    0.2.6                    pypi_0    pypi
importlib-metadata        6.8.0                    pypi_0    pypi
importlib-resources       6.1.0                    pypi_0    pypi
itsdangerous              1.1.0                    pypi_0    pypi
jinja2                    3.0.3                    pypi_0    pypi
joblib                    1.3.2                    pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_2    conda-forge
libgomp                   13.2.0               h807b86a_2    conda-forge
libnsl                    2.0.0                hd590300_1    conda-forge
libsqlite                 3.43.2               h2797004_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
llvmlite                  0.39.1                   pypi_0    pypi
markdown                  3.5                      pypi_0    pypi
markupsafe                2.1.3                    pypi_0    pypi
matplotlib                3.7.3                    pypi_0    pypi
mimerender                0.6.0                    pypi_0    pypi
ncurses                   6.4                  hcb278e6_0    conda-forge
networkx                  3.1                      pypi_0    pypi
numba                     0.56.4                   pypi_0    pypi
numpy                     1.20.3                   pypi_0    pypi
oauthlib                  3.2.2                    pypi_0    pypi
opencv-python-headless    4.5.3.56                 pypi_0    pypi
openssl                   3.1.3                hd590300_0    conda-forge
packaging                 23.2                     pypi_0    pypi
pillow                    8.1.2                    pypi_0    pypi
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
protobuf                  3.20.0                   pypi_0    pypi
pyasn1                    0.5.0                    pypi_0    pypi
pyasn1-modules            0.3.0                    pypi_0    pypi
pynndescent               0.5.10                   pypi_0    pypi
pyparsing                 3.1.1                    pypi_0    pypi
python                    3.8.18          hd12c33a_0_cpython    conda-forge
python-dateutil           2.8.2                    pypi_0    pypi
python-mimeparse          1.6.0                    pypi_0    pypi
pywavelets                1.4.1                    pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
readline                  8.2                  h8228510_1    conda-forge
requests                  2.25.1                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
scikit-image              0.18.1                   pypi_0    pypi
scikit-learn              0.24.0                   pypi_0    pypi
scipy                     1.6.0                    pypi_0    pypi
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
six                       1.16.0                   pypi_0    pypi
sqlalchemy                1.3.22                   pypi_0    pypi
tensorboard               2.4.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
threadpoolctl             3.2.0                    pypi_0    pypi
tifffile                  2023.7.10                pypi_0    pypi
tk                        8.6.13               h2797004_0    conda-forge
torch                     1.8.1+cu111              pypi_0    pypi
torchaudio                0.8.1                    pypi_0    pypi
torchvision               0.9.1+cu111              pypi_0    pypi
ttach                     0.0.2                    pypi_0    pypi
typing-extensions         4.8.0                    pypi_0    pypi
umap-learn                0.5.1                    pypi_0    pypi
urllib3                   1.26.17                  pypi_0    pypi
werkzeug                  2.0.3                    pypi_0    pypi
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zipp                      3.17.0                   pypi_0    pypi