Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow slightly unbalanced consumption in tee #56

Open
dcsommer opened this issue Apr 5, 2021 · 1 comment
Open

Allow slightly unbalanced consumption in tee #56

dcsommer opened this issue Apr 5, 2021 · 1 comment
Labels
enhancement New feature or request

Comments

@dcsommer
Copy link

dcsommer commented Apr 5, 2021

The implementation of aioitertools.itertools.tee currently has the caveat "all iterators are dependent on the first iterator." This caveat can cause periodic stalls in my pipeline where the consumers all consume at roughly the same speed, with small imbalances.

A small refactor of tee could remove this limitation if the producer was a separate scheduled task (instead of the first consumer). Would you be open to a pull request for a change along these lines?

@amyreese
Copy link
Member

amyreese commented Apr 6, 2021

A PR would be great, but I would like to see a couple constraints:

  • Some form of configurable limit to the amount that any one iterator can get ahead of the rest (ie, a size limit on queues). Maybe something like tee(itr: ..., n: int = 2, *, limit: int = 100), where limit=0 would be unbounded.
  • The task reading items from the original source ensure that it yields the event loop frequently. Just doing await asyncio.sleep(0) after each item would be sufficient.

By default, this would still cause consumption to be limited by the slowest iterator, but no longer dependent on the first iterator specifically.

@amyreese amyreese added the enhancement New feature or request label Apr 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants