Installation issue

[Sally-K]

At Royal Botanic Garden Edinburgh are trying to install Panoptes using the Anaconda version of python and it seems that our version of python 3 (3.8) is incompatible with the dependencies that need to be installed prior to installing the panoptes aggregator itself (see Anaconda output below). Cam Allen suggested logging this as an issue in the repo.

(base) C:\>conda install -c conda-forge python-levenshtein hdbscan
Collecting package metadata (current_repodata.json): done
Solving environment: done

Package Plan

environment location: C:\ProgramData\Anaconda3b

 added / updated specs:
   - hdbscan
   - python-levenshtein

The following NEW packages will be INSTALLED:

 hdbscan            conda-forge/win-64::hdbscan-0.8.27-py38h347fdf6_0
 python-levenshtein conda-forge/win-64::python-levenshtein-0.12.2-py38h294d835_0
 python_abi         conda-forge/win-64::python_abi-3.8-1_cp38

The following packages will be SUPERSEDED by a higher-priority channel:

 conda              pkgs/main::conda-4.10.1-py38haa95532_1 --> conda-forge::conda-4.10.1-py38haa244fe_0

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

(base) C:\>conda install wxpython psutil
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: -
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
Examining @/win-64::__win==0=0:  40%|                                                           

| 2/5 [00:00<00:00, 87.20it/s]/failed

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

 - wxpython -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.5,<3.6.0a0']

Your python: python=3.8

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.

The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package vs2008_runtime conflicts for:
psutil -> python[version='>=2.7,<2.8.0a0'] -> vs2008_runtime[version='>=9.0.30729.1,<10.0a0']
wxpython -> python[version='>=2.7,<2.8.0a0'] -> vs2008_runtime[version='>=9.0.30729.1,<10.0a0']

(base) C:\>

[CKrawczyk]

It looks like the base conda build of wxpython does not support 3.8 or higher, but the version in conda-forge does.

Try running

1. conda install -c conda-forge wxpython
2. conda install psutil
3. pip install "panoptes_aggregation[gui]" (if you want the GUI) or pip install panoptes_aggregation (if you just want the command line interface)

If this works I will update the docs to reflect this change.

[Sally-K]

Hi Coleman,

Your suggestion worked, Please go ahead and add this to the documentation for other users.

Many thanks,

Sally

[bollwyvl]

Have started this PR:

conda-forge/staged-recipes#20584

It translates the various [extras] into panoptes-aggregation-with-<extra>, so a modern suggestion might be:

mamba install -c conda-forge panoptes-aggregation-with-gui

(written mostly in c++, mamba is, crutially, mostly bug-for-bug compatible with conda, but quite a bit faster... though conda has been making some strides)

While it is working locally, it may be a while before it makes it through the whole vetting process: certainly time for anyone interested here to sign up to co-maintain! Just leave a note to that effect on that PR.

[CKrawczyk]

@bollwyvl That is great :D

Do you know if there is an easy way to keep the conda-forge version up to date when dependencies are bumped? I am also looking to switch this package over to PDM in the near future, would that cause issues with the conda-forge version?

[bollwyvl]

easy way to keep the conda-forge version up to date when dependencies are bumped

We have pip check in the mix to catch gross issues, and I usually try to run the full test suite, which often reveals things (e.g. the joblib issue) which emerge after releases (or were tightly pinned when tested upstream).

There are some semi-automated approaches, but they don't handle [extras] very well. Indeed, neither does pip, though the situation is getting better.

About the best thing a conda-forge recipe can do is go back and re-parse the as-published package metadata (DIST-INFO) and compare it to the as-written recipe and look for differences. Indeed, we should probably write this as a helper.

Upstream, it really helps to avoid the ^ and ~ version operators, as these are not interpreted the same way in all communities, which leaves conda-forge in a bit of a pickle. Sticking with >=x,<y is much easier to reason about.

switch this package over to PDM

Really only mis-configured poetry projects have caused build issues in the past. flit, hatch and pdm seem to have been mostly working without issue unless pip changes its behavior (happens more than you'd think).

The challenge with exotic build backends that do a lot more (like heavyweight dependency management lock files) is that downstream users almost certainly won't be using the specific one a project picks.

[bollwyvl]

The feedstock is now up:

https://github.com/conda-forge/panoptes-aggregation-feedstock

Builds should be generally available within the hour:

conda install -c conda-forge panoptes-aggregation

Or one of the extras:

conda install -c conda-forge panoptes-aggregation-with-gui
conda install -c conda-forge panoptes-aggregation-with-online

Happy for any feedback, or offers to co-maintain! I've started the issue about (not) installing panoptes-aggregation-with-online on Windows:

conda-forge/panoptes-aggregation-feedstock#1

The next time a release shows up on PyPI, a bot should create a PR, though sometimes it gets confused if the build system changes. Usually these are quick to resolve, however.

[CKrawczyk]

The challenge with exotic build backends that do a lot more (like heavyweight dependency management lock files) is that downstream users almost certainly won't be using the specific one a project picks.

I was planning on using the lock file only for the CI and Docker builds (that Zooniverse deploys), and only keeping the min package versions specified in the main package. This should make the package easier to install into existing environments with fewer dependency conflicts.

Happy for any feedback, or offers to co-maintain!

I don't know much about the conda build system at the moment, but I am happy to learn (might be a few weeks before I have time to look at this closely).