PathMe and DiffuPy Versioning

[jfbeltran]

Hi, I've been trying to set up DiffuPath in a clean conda environment and get a cascade of conflicts with diffupy and pathme, probably due to their individual versioning.

Here is an environment with python3 -m pip install diffupath:

// SET UP ENVIRONMENT //
(base)$ conda create --name=diffupath_env pybel=0.13.2 click=7.0 tqdm=4.31.1 numpy=1.16.3 scipy=1.2.1 scikit-learn=0.21.3 pandas=0.24.2 openpyxl=3.0.2 plotly=4.5.3 matplotlib=3.1.2
[installation succeeds without issues]
(base)$ conda activate diffupath_env

// INSTALL DIFFUPATH //
(diffupath_env)$ python3 -m pip install diffupath
...
Installing collected packages: zipp, MarkupSafe, Werkzeug, pyrsistent, Jinja2, itsdangerous, importlib-metadata, attrs, PyYAML, mistune, jsonschema, flask, wtforms, typing-extensions, pymysql, passlib, mypy-extensions, marshmallow, future, flask-cors, flasgger, easy-config, dataclasses, configparser, wcwidth, typing-inspect, torch, stringcase, requests-toolbelt, pysolr, pyhgnc, pillow, marshmallow-enum, isodate, ijson, enum34, bio2bel, torchvision, rdflib, prompt-toolkit, ndex2, flask-admin, dataclasses-json, compath-utils, click-default-group, bio2bel-hgnc, bio2bel-chebi, bel-repository, pykeen, pybel-tools, pybel-cx, mock, bio2bel-wikipathways, bio2bel-reactome, bio2bel-kegg, pathme, matplotlib-venn, diffupy, chart-studio, biokeen, diffupath
Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 PyYAML-5.3.1 Werkzeug-1.0.1 attrs-20.3.0 bel-repository-0.0.8 bio2bel-0.2.1 bio2bel-chebi-0.2.1 bio2bel-hgnc-0.2.2 bio2bel-kegg-0.2.5 bio2bel-reactome-0.2.3 bio2bel-wikipathways-0.2.3 biokeen-0.0.14 chart-studio-1.0.0 click-default-group-1.2.2 compath-utils-0.2.1 configparser-5.0.1 dataclasses-0.6 dataclasses-json-0.5.2 diffupath-0.0.3 diffupy-0.0.5 easy-config-0.2.0 enum34-1.1.10 flasgger-0.9.5 flask-1.1.2 flask-admin-1.5.7 flask-cors-3.0.9 future-0.18.2 ijson-3.1.3 importlib-metadata-3.1.1 isodate-0.6.0 itsdangerous-1.1.0 jsonschema-3.2.0 marshmallow-3.9.1 marshmallow-enum-1.5.1 matplotlib-venn-0.11.5 mistune-0.8.4 mock-4.0.2 mypy-extensions-0.4.3 ndex2-3.3.1 passlib-1.7.4 pathme-0.1.13 pillow-8.0.1 prompt-toolkit-3.0.8 pybel-cx-0.1.3 pybel-tools-0.7.5 pyhgnc-0.2.4 pykeen-0.0.26 pymysql-0.10.1 pyrsistent-0.17.3 pysolr-3.9.0 rdflib-4.2.2 requests-toolbelt-0.9.1 stringcase-1.2.0 torch-1.7.0 torchvision-0.8.1 typing-extensions-3.7.4.3 typing-inspect-0.6.0 wcwidth-0.2.5 wtforms-2.3.3 zipp-3.4.0

// TEST DIFFUPATH //
(diffupath_env)$ python3
Python 3.7.8 | packaged by conda-forge | (default, Nov 17 2020, 23:45:15)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from diffupath.diffuse import run_diffusion
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'diffupath.diffuse'

After removing the environment, I try again from a clean one and attempt to use the latest github version.
Here is an environment with python3 -m pip install git+https://github.com/multipaths/diffupath.git:

// SET UP ENVIRONMENT //
(base)$ conda create --name=diffupath_env pybel=0.13.2 click=7.0 tqdm=4.31.1 numpy=1.16.3 scipy=1.2.1 scikit-learn=0.21.3 pandas=0.24.2 openpyxl=3.0.2 plotly=4.5.3 matplotlib=3.1.2
[installation succeeds without issues]
(base)$ conda activate diffupath_env

// INSTALL DIFFUPATH //
(diffupath_env)$ python3 -m pip install git+https://github.com/multipaths/diffupath.git
...
Installing collected packages: zipp, MarkupSafe, Werkzeug, pyrsistent, Jinja2, itsdangerous, importlib-metadata, attrs, PyYAML, mistune, jsonschema, flask, wtforms, typing-extensions, pymysql, passlib, mypy-extensions, marshmallow, future, flask-cors, flasgger, easy-config, dataclasses, configparser, wcwidth, typing-inspect, torch, stringcase, requests-toolbelt, pysolr, pyhgnc, pillow, marshmallow-enum, isodate, ijson, enum34, bio2bel, torchvision, rdflib, prompt-toolkit, ndex2, flask-admin, dataclasses-json, compath-utils, click-default-group, bio2bel-hgnc, bio2bel-chebi, bel-repository, pykeen, pybel-tools, pybel-cx, patsy, mock, bio2bel-wikipathways, bio2bel-reactome, bio2bel-kegg, statsmodels, seaborn, pathme, matplotlib-venn, diffupy, chart-studio, biokeen, diffupath
Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 PyYAML-5.3.1 Werkzeug-1.0.1 attrs-20.3.0 bel-repository-0.0.8 bio2bel-0.2.1 bio2bel-chebi-0.2.1 bio2bel-hgnc-0.2.2 bio2bel-kegg-0.2.5 bio2bel-reactome-0.2.3 bio2bel-wikipathways-0.2.3 biokeen-0.0.14 chart-studio-1.0.0 click-default-group-1.2.2 compath-utils-0.2.1 configparser-5.0.1 dataclasses-0.6 dataclasses-json-0.5.2 diffupath-0.0.4.dev0 diffupy-0.0.5 easy-config-0.2.0 enum34-1.1.10 flasgger-0.9.5 flask-1.1.2 flask-admin-1.5.7 flask-cors-3.0.9 future-0.18.2 ijson-3.1.3 importlib-metadata-3.1.1 isodate-0.6.0 itsdangerous-1.1.0 jsonschema-3.2.0 marshmallow-3.9.1 marshmallow-enum-1.5.1 matplotlib-venn-0.11.5 mistune-0.8.4 mock-4.0.2 mypy-extensions-0.4.3 ndex2-3.3.1 passlib-1.7.4 pathme-0.1.13 patsy-0.5.1 pillow-8.0.1 prompt-toolkit-3.0.8 pybel-cx-0.1.3 pybel-tools-0.7.5 pyhgnc-0.2.4 pykeen-0.0.26 pymysql-0.10.1 pyrsistent-0.17.3 pysolr-3.9.0 rdflib-4.2.2 requests-toolbelt-0.9.1 seaborn-0.11.0 statsmodels-0.12.1 stringcase-1.2.0 torch-1.7.0 torchvision-0.8.1 typing-extensions-3.7.4.3 typing-inspect-0.6.0 wcwidth-0.2.5 wtforms-2.3.3 zipp-3.4.0

// TEST DIFFUPATH //
(diffupath_env)$ python3
Python 3.7.8 | packaged by conda-forge | (default, Nov 17 2020, 23:45:15)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from diffupath.diffuse import run_diffusion
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/workdir/users/jfb294/miniconda3/envs/diffupath_env/lib/python3.7/site-packages/diffupath/diffuse.py", line 16, in <module>
    from diffupy.process_network import get_kernel_from_network_path, process_graph_from_file, get_kernel_from_graph, \
ImportError: cannot import name 'get_kernel_from_graph' from 'diffupy.process_network' (/workdir/users/jfb294/miniconda3/envs/diffupath_env/lib/python3.7/site-packages/diffupy/process_network.py)

The above can be solved by installing the latest version of diffupy:

// UPDATE DIFFUPY & TEST DIFFUPATH //
(diffupath_env)$ python -m pip install git+https://github.com/multipaths/DiffuPy.git
(diffupath_env)$ python3
Python 3.7.8 | packaged by conda-forge | (default, Nov 17 2020, 23:45:15)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from diffupath.diffuse import run_diffusion
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/workdir/users/jfb294/miniconda3/envs/diffupath_env/lib/python3.7/site-packages/diffupath/diffuse.py", line 18, in <module>
    from google_drive_downloader import GoogleDriveDownloader
ModuleNotFoundError: No module named 'google_drive_downloader'

This one can be fixed by installing the google drive downloader:

// INSTALL GOOGLEDRIVEDOWNLOADER & TEST DIFFUPATH //
(diffupath_env)$ conda install googledrivedownloader
(diffupath_env)$ python3
Python 3.7.8 | packaged by conda-forge | (default, Nov 17 2020, 23:45:15)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from diffupath.diffuse import run_diffusion
INFO:rdflib:RDFLib Version: 4.2.2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/workdir/users/jfb294/miniconda3/envs/diffupath_env/lib/python3.7/site-packages/diffupath/diffuse.py", line 19, in <module>
    from pathme.cli import generate_universe, universe
ImportError: cannot import name 'generate_universe' from 'pathme.cli' (/workdir/users/jfb294/miniconda3/envs/diffupath_env/lib/python3.7/site-packages/pathme/cli.py)

And that's where I'm stuck!
The error remains the same if I also update PathMe using
python3 -m pip install git+https://github.com/PathwayMerger/PathMe.git

I can't seem to find any version of PathMe that still has a generate_universe function.

If you have a way to push a fix, or let me know which versions of pathme, diffupy, and diffupath you use in a working triplet, I'd appreciate it a ton!

[jmarinllao]

Hello!

Thank you for reporting! We recently changed the generate_universe function in PathMe, in order to allow multiple-species generation but the update was not merged yet, same with the Google Drive Downloader requirement with another new feature in diffuPy.

We will push the merged updates by tomorrow. I'll further ping you here when it's fixed.

Apologies for the versioning issues!

[jfbeltran]

No problem at all! I took a look at the change history for all three modules and could tell you're in pretty active development. Definitely would love a ping when you've pushed the merged updates, and thanks for working on these, they are fantastic modules to have in the python environment!!

[jmarinllao]

Hello again!

Sorry for the delay, I had some mandatory extra work on early Dec. I hope the update would be still useful.

Finally, the updated version is available for PathMe and also some updates for MultiPaths.
Download all from source again (PathMe, diffuPy and DiffuPath) and try it out.

If you wish, take a look at this usability notebooks for DiffuPath and DiffuPy.

Also if you would like to check further dataset preprocessing or evaluations.

We will be updating extra features regarding other species networks and further network filtering/customization. If you would be interested on them, we could further let you know by here when all is tested out.

If you have any issue with the updates let us know,
Sorry again for the delay and have a nice Christmas,

Regards,

Josep