poetry: S3 feature is incompatible with latest boto3 version

[iainelder]

Bug Report

Issue names must follow the pattern command: description where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug.

This time the bug isn't in a DVC command but in how its dependencies are declared. Feel free to rename the issue to match the preferred format.

Description

I've installed the latest version of the boto3 package for my application.

I want to use it together with DVC's S3 feature.

When I try to install the S3 feature, I get a SolverProblemError from poetry.

Reproduce

$ mkdir tmp
$ cd tmp
$ poetry init

This command will guide you through creating your pyproject.toml config.

Package name [tmp]:  
Version [0.1.0]:  
Description []:  
Author [Iain Samuel McLean Elder <[email protected]>, n to skip]:  
License []:  
Compatible Python versions [^3.8]:  

Would you like to define your main dependencies interactively? (yes/no) [yes] no
Would you like to define your development dependencies interactively? (yes/no) [yes] no
Generated file

[tool.poetry]
name = "tmp"
version = "0.1.0"
description = ""
authors = ["Iain Samuel McLean Elder <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.8"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"


Do you confirm generation? (yes/no) [yes] 
$ poetry add boto3
Creating virtualenv tmp-pXV3aO_m-py3.8 in /home/isme/.cache/pypoetry/virtualenvs
Using version ^1.23.1 for boto3

Updating dependencies
Resolving dependencies... (0.5s)

Writing lock file

Package operations: 7 installs, 0 updates, 0 removals

  • Installing six (1.16.0)
  • Installing jmespath (1.0.0)
  • Installing python-dateutil (2.8.2)
  • Installing urllib3 (1.26.9)
  • Installing botocore (1.26.1)
  • Installing s3transfer (0.5.2)
  • Installing boto3 (1.23.1)
$ poetry add dvc[s3]
Using version ^2.10.2 for dvc

Updating dependencies
Resolving dependencies... (9.8s)

  SolverProblemError

      Because aiobotocore (2.0.1) depends on botocore (>=1.22.8,<1.22.9)
   and no versions of aiobotocore match >2.0.1,<2.1.0, aiobotocore (>=2.0.1,<2.1.0) requires botocore (>=1.22.8,<1.22.9).
      And because s3fs (2021.11.1) depends on aiobotocore (>=2.0.1,<2.1.0)
   and no versions of s3fs match >2021.11.1,<2022.1.0 || >2022.1.0,<2022.2.0 || >2022.2.0,<2022.3.0 || >2022.3.0, s3fs (>=2021.11.1,<2022.1.0 || >2022.1.0,<2022.2.0 || >2022.2.0,<2022.3.0 || >2022.3.0) requires botocore (>=1.22.8,<1.22.9).
  (1) So, because s3fs (2022.2.0) depends on aiobotocore (>=2.1.0,<2.2.0)
   and s3fs (2022.1.0) depends on aiobotocore (>=2.1.0,<2.2.0), s3fs (>=2021.11.1,<2022.3.0 || >2022.3.0) requires botocore (>=1.22.8,<1.22.9) or aiobotocore (>=2.1.0,<2.2.0).
  
      Because no versions of aiobotocore match >2.1.0,<2.1.1 || >2.1.1,<2.1.2 || >2.1.2,<2.2.0
   and aiobotocore (2.1.0) depends on botocore (>=1.23.24,<1.23.25), aiobotocore (>=2.1.0,<2.1.1 || >2.1.1,<2.1.2 || >2.1.2,<2.2.0) requires botocore (>=1.23.24,<1.23.25).
      And because aiobotocore (2.1.1) depends on botocore (>=1.23.24,<1.23.25)
   and aiobotocore (2.1.2) depends on botocore (>=1.23.24,<1.23.25), aiobotocore (>=2.1.0,<2.2.0) requires botocore (>=1.23.24,<1.23.25).
      And because s3fs (>=2021.11.1,<2022.3.0 || >2022.3.0) requires botocore (>=1.22.8,<1.22.9) or aiobotocore (>=2.1.0,<2.2.0) (1), s3fs (>=2021.11.1,<2022.3.0 || >2022.3.0) requires botocore (>=1.22.8,<1.22.9 || >=1.23.24,<1.23.25)
      And because s3fs (2022.3.0) depends on aiobotocore (>=2.2.0,<2.3.0), s3fs (>=2021.11.1) requires botocore (>=1.22.8,<1.22.9 || >=1.23.24,<1.23.25) or aiobotocore (>=2.2.0,<2.3.0).
      Because no versions of aiobotocore match >2.2.0,<2.3.0
   and aiobotocore (2.2.0) depends on botocore (>=1.24.21,<1.24.22), aiobotocore (>=2.2.0,<2.3.0) requires botocore (>=1.24.21,<1.24.22).
      Thus, s3fs (>=2021.11.1) requires botocore (>=1.22.8,<1.22.9 || >=1.23.24,<1.23.25 || >=1.24.21,<1.24.22).
      Because no versions of dvc match >2.10.2,<3.0.0
   and dvc (2.10.2) depends on s3fs (>=2021.11.1), dvc (>=2.10.2,<3.0.0) requires s3fs (>=2021.11.1).
      Thus, dvc (>=2.10.2,<3.0.0) requires botocore (>=1.22.8,<1.22.9 || >=1.23.24,<1.23.25 || >=1.24.21,<1.24.22).
      And because boto3 (1.23.1) depends on botocore (>=1.26.1,<1.27.0)
   and no versions of boto3 match >1.23.1,<2.0.0, dvc (>=2.10.2,<3.0.0) is incompatible with boto3 (>=1.23.1,<2.0.0).
      So, because tmp depends on both boto3 (^1.23.1) and dvc (^2.10.2), version solving failed.

  at ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│ 
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

Expected

No error message. I expect that the S3 feature will be installed with whatever version version of boto3 I'm using in my application.

Environment information

$ poetry run dvc doctor
DVC version: 2.10.2 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-5.13.0-41-generic-x86_64-with-glibc2.29
Supports:
	webhdfs (fsspec = 2022.3.0),
	http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme0n1p3
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/nvme0n1p3
Repo: dvc, git

[iainelder]

Possibly related: fsspec/s3fs#615

[daavoo]

Can also reproduce with pip resolver:

pip install boto3==1.23.1 'dvc[s3]==2.10.2'

ERROR: Cannot install aiobotocore[boto3]==2.0.1, aiobotocore[boto3]==2.1.0, aiobotocore[boto3]==2.1.1, aiobotocore[boto3]==2.1.2, aiobotocore[boto3]==2.2.0, aiobotocore[boto3]==2.3.0, aiobotocore[boto3]==2.3.1, aiobotocore[boto3]==2.3.2 and boto3==1.23.1 because these package versions have conflicting dependencies.

The conflict is caused by:
    boto3 1.23.1 depends on botocore<1.27.0 and >=1.26.1
    aiobotocore[boto3] 2.3.2 depends on botocore<1.24.22 and >=1.24.21
    boto3 1.23.1 depends on botocore<1.27.0 and >=1.26.1
    aiobotocore[boto3] 2.3.1 depends on botocore<1.24.22 and >=1.24.21
    boto3 1.23.1 depends on botocore<1.27.0 and >=1.26.1
    aiobotocore[boto3] 2.3.0 depends on botocore<1.24.22 and >=1.24.21
    boto3 1.23.1 depends on botocore<1.27.0 and >=1.26.1
    aiobotocore[boto3] 2.2.0 depends on botocore<1.24.22 and >=1.24.21
    boto3 1.23.1 depends on botocore<1.27.0 and >=1.26.1
    aiobotocore[boto3] 2.1.2 depends on botocore<1.23.25 and >=1.23.24
    boto3 1.23.1 depends on botocore<1.27.0 and >=1.26.1
    aiobotocore[boto3] 2.1.1 depends on botocore<1.23.25 and >=1.23.24
    boto3 1.23.1 depends on botocore<1.27.0 and >=1.26.1
    aiobotocore[boto3] 2.1.0 depends on botocore<1.23.25 and >=1.23.24
    boto3 1.23.1 depends on botocore<1.27.0 and >=1.26.1
    aiobotocore[boto3] 2.0.1 depends on botocore<1.22.9 and >=1.22.8

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

[daavoo]

Unfortunately, it seems that until aiobotocore updates the pinned boto3 requirement (https://github.com/aio-libs/aiobotocore/blob/master/setup.py#L19) , the most recent compatible version would be 1.21.21:

poetry add boto3==1.21.21
poetry add 'dvc[s3]'

[iainelder]

Another thread discussing the problems with s3fs's dependency on aiobotocore: fsspec/s3fs#357

From what I gather the aiobotocore developers are unlikely to relax the version constraints because it's hacking the internals in ways that are likely to break from version to version.

s3fs depends on aiobotocore, which also makes a problem for anyone that depends on s3fs.

At this point it seems like the easier solution for me is just to install dvc globally using pipx. It's the easiest way to avoid this path into version hell :-)

Except I tried that and it didn't work for me either! See #7777 .

As for this specific issue, if the way to solve it is to wait for aiobotocore and botocore proper to sort themselves out then I suppose you can just close this.

[daavoo]

@iainelder Could you try installing globally via a system package manager (i.e. https://dvc.org/doc/install/linux#install-from-repository ) ?

[iainelder]

Wow, how did I miss that in the documentation?

Installing from a Debian repository is my preferred way of installing anything, as long as the versions are kept up to date. The version I see in your repo matches the version on PyPI: 2.10.2.

So that solves my installation problem and makes the problem reported here irrelevant. Thanks so much for your help @daavoo !