You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The implementation of aioitertools.itertools.tee currently has the caveat "all iterators are dependent on the first iterator." This caveat can cause periodic stalls in my pipeline where the consumers all consume at roughly the same speed, with small imbalances.
A small refactor of tee could remove this limitation if the producer was a separate scheduled task (instead of the first consumer). Would you be open to a pull request for a change along these lines?
The text was updated successfully, but these errors were encountered:
A PR would be great, but I would like to see a couple constraints:
Some form of configurable limit to the amount that any one iterator can get ahead of the rest (ie, a size limit on queues). Maybe something like tee(itr: ..., n: int = 2, *, limit: int = 100), where limit=0 would be unbounded.
The task reading items from the original source ensure that it yields the event loop frequently. Just doing await asyncio.sleep(0) after each item would be sufficient.
By default, this would still cause consumption to be limited by the slowest iterator, but no longer dependent on the first iterator specifically.
The implementation of aioitertools.itertools.tee currently has the caveat "all iterators are dependent on the first iterator." This caveat can cause periodic stalls in my pipeline where the consumers all consume at roughly the same speed, with small imbalances.
A small refactor of
tee
could remove this limitation if the producer was a separate scheduled task (instead of the first consumer). Would you be open to a pull request for a change along these lines?The text was updated successfully, but these errors were encountered: