Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TODO: multithreaded fetching #5

Open
luckman212 opened this issue Sep 10, 2024 · 2 comments
Open

TODO: multithreaded fetching #5

luckman212 opened this issue Sep 10, 2024 · 2 comments

Comments

@luckman212
Copy link
Owner

luckman212 commented Sep 10, 2024

Add configurable # of download threads to increase speed. Not sure what Fastmail's ratelimits are, so 1-4 would probably be a good starting point.

@alexpovel
Copy link

Could perhaps use async! TaskGroup will allow the creation of a bunch of tasks to be run concurrently (similar to Go's WaitGroup or Rust's std::thread::scope if you're familiar) leveraging a context manager, awaiting them all. A task could be "backup an email". An asyncio.Semaphore can then help remain below rate limits, as a first approximation... it would at least help only run n (1-4) tasks concurrently.

All of that is single-threaded, but so are threads in Python (currently :-) ). In any case, it's more than fast enough, and you can easily reach hundreds of concurrent requests per second (but that's what the semaphore is for, limiting it).

@luckman212
Copy link
Owner Author

@alexpovel Thank you for the suggestions. I plan to put some work into this next weekend. Hopefully will have a new version available for testing within a week or so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants