Skip to content

Commit 2e1bee5

Browse files
tadlytadly
committed
reworked README and added separate example
1 parent addd910 commit 2e1bee5

File tree

2 files changed

+119
-99
lines changed

2 files changed

+119
-99
lines changed

‎README.md

+15-99
Original file line numberDiff line numberDiff line change
@@ -3,17 +3,17 @@ pyloader - A simple python downloader
33
[![Build Status](https://travis-ci.org/linuxwhatelse/pyloader.svg?branch=master)](https://travis-ci.org/linuxwhatelse/pyloader)
44
[![pypi](https://img.shields.io/pypi/v/lwe-pyloader.svg)](https://pypi.python.org/pypi/lwe-pyloader)
55

6-
**pyloader** is a simple, easy to use, multi-threaded downloader with queuing support.
6+
**pyloader** is a simple, easy to use, multi-threaded downloader with queuing support.
77

8-
It is **NOT** a command-line utility but instead something you can (if you want) implement
9-
in one (or more, I don't care :)) of your applications.
8+
It is **NOT** a command-line utility but instead something you can (if you want) implement
9+
in one (or more, I don't care :)) of your applications.
1010

11-
I was in need for such a thing and that's why I wrote it myself after only finding command-line utilities.
12-
(I haven't spent a lot of time searching though)
11+
I wrote project-specific downloader a few times now and finally decided to create a proper module for it as
12+
I couldn't find an existing one (Haven't spent that much time searching though).
1313

1414
## Important notice
1515
As of right now, this is **Beta**, so treat it as such ;)
16-
I added some unittests but there're still many more to go
16+
I added some unittests but there're still more to go!
1717

1818
## ToDo
1919
Things to implement:
@@ -26,106 +26,22 @@ What you need:
2626
* The great python [requests](https://github.com/kennethreitz/requests) module
2727

2828
## Installation
29-
Just run:
29+
### From pypi (recommanded)
3030
```bash
3131
pip install lwe-pyloader
3232
```
33-
33+
### From source
34+
```bash
35+
git clone https://github.com/linuxwhatelse/pyloader
36+
cd pyloader
37+
python setup.py install
38+
```
3439
## Usage
35-
The source has been commented quite well so at any point in time you might just:
40+
The source has been commented quite well so at any point you might just:
3641
```python
3742
import pyloader
3843
help(pyloader)
3944
```
4045

41-
But to get you started, here are some examples :)
42-
```python
43-
import pyloader
44-
45-
def progress_callback(progress):
46-
# !!!IMPORTANT NOTE!!!
47-
# The callback will NOT be called within a separate thread
48-
# (to ensure consistancy) and WILL block the download
49-
# for as long as your callback runs!
50-
# Think twice about what you do in here.
51-
# Usually you just want to persist/visualize data.
52-
53-
# Use ``help(pyloader.Progress)`` to know what's available
54-
print(progress.percent)
55-
56-
# To cancel the download
57-
return True
58-
# If you don't want to cancel,
59-
return False
60-
# or return nothing at all
61-
62-
def url_resolver(item):
63-
# Here you would resolve the url to something
64-
# that can actually be downloaded.
65-
# Useful in case resource-urls would expire after
66-
# a set amount of time.
67-
return item.url
68-
69-
if __name__ == '__main__':
70-
# Create a loader instance
71-
dl = pyloader.Loader.get_loader()
72-
dl.configure(
73-
max_concurrent=3,
74-
update_interval=3,
75-
progress_cb=progress_callback,
76-
url_resolve_cb=url_resolver,
77-
daemon=False
78-
)
79-
80-
# Start the loader
81-
# Make sure you know how the `daemon` flag
82-
# affects the liftime of your program
83-
dl.start()
84-
85-
# Create a downloadable item
86-
# Make sure to read the docstrings via
87-
# help(pyloader.DLable)
88-
# for available arguments
89-
item = pyloader.DLable(
90-
url = 'http://download.thinkbroadband.com/5MB.zip',
91-
target_dir = '~/Downloads/',
92-
)
93-
94-
# Queue an item or...
95-
dl.queue(item)
96-
# ...alternatively you can force a download,
97-
# ignoring ``max_concurrent``
98-
dl.download(item)
99-
100-
# If you don't use a callback (to, if necessary, cancel a download),
101-
# you can stop one via:
102-
dl.stop(item)
103-
# or via its uid
104-
dl.stop(item.uid)
105-
106-
# You can also clear the queue by
107-
dl.clear_queue()
108-
# and active items as well
109-
dl.clear_active()
110-
# though you'll never need the second one as there
111-
# are little to none cases where items are actually
112-
# stuck within the active queue
113-
114-
# To check the downloaders state you can:
115-
print(dl.is_alive) # True if both necessary main threads are still alive and kicking
116-
print(dl.is_active) # True if items are queued and/or downloading
117-
118-
print(dl.queued) # Returns the amount of queued items
119-
print(dl.active) # Returns the amount of active/downloading items
120-
121-
print(dl.max_concurrent) # The amount of maximum concurrent allowed downloads
122-
dl.max_concurrent = 5 # Set the amount up to 5 and starts new downloads (if any)
123-
124-
# To stop all downloads and end the downloader just do:
125-
dl.exit()
126-
127-
# Important to note here, queued/active items will **NOT** be persisted upon exit (or any other point in time)
128-
# It's up to you to keep track of the items ;)
129-
```
130-
Well, that should be enough to get you going (I hope)
46+
To get you started though, check the included [examples](examples).
13147
Happy coding! :)

‎examples/simple.py

+104
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
import time
2+
import pyloader
3+
4+
5+
def progress_callback(progress):
6+
"""!!! IMPORTANT !!!
7+
8+
This callback will NOT be called from a subsequent thread but will use
9+
the downloads thread instead.
10+
This means the download blocks for as long as the callback is executed.
11+
12+
Usually you only want to persist/visualize the progress.
13+
"""
14+
# Use `help(pyloader.Progress)` to see all the goodies
15+
print(progress.dlable.file_name, '{0:.2f}%'.format(progress.percent))
16+
17+
# `return True` if the download should be canceled
18+
return False
19+
20+
21+
def url_resolver(item):
22+
"""At times you may have urls that would expire while the item is queued.
23+
24+
This callback will be called right before the download starts and allowes
25+
you to alter the `DLable` instance (and the url that goes along with it).
26+
"""
27+
# item.url = 'http://new.url'
28+
return item
29+
30+
31+
if __name__ == '__main__':
32+
# Create a loader instance
33+
loader = pyloader.Loader.get_loader()
34+
loader.configure(
35+
max_concurrent=3,
36+
update_interval=1,
37+
progress_cb=progress_callback,
38+
url_resolve_cb=url_resolver,
39+
daemon=False
40+
)
41+
42+
# Start the loader
43+
# Make sure you know how the `daemon` flag
44+
# affects the liftime of your program
45+
loader.start()
46+
47+
# Create downloadable items
48+
item1 = pyloader.DLable(
49+
url='http://download.thinkbroadband.com/5MB.zip',
50+
target_dir='./',
51+
file_name='item1.zip'
52+
)
53+
54+
item2 = pyloader.DLable(
55+
url='http://download.thinkbroadband.com/5MB.zip',
56+
target_dir='./',
57+
file_name='item2.zip'
58+
)
59+
# Queue an item or...
60+
loader.queue(item1)
61+
62+
# ...alternatively you can force a download,
63+
# ignoring `max_concurrent`
64+
loader.download(item2)
65+
66+
# If you don't use a callback (which would allow you to cancel a
67+
# download if necessary), you can `stop` one too
68+
loader.stop(item1)
69+
# or via its uid
70+
loader.stop(item1.uid)
71+
72+
# You can also clear all queued items
73+
loader.clear_queued()
74+
75+
# True if both necessary main threads are still alive and kicking
76+
print('Is alive:', loader.is_alive())
77+
78+
# True if items are queued and/or downloading
79+
print('Is active:', loader.is_active())
80+
81+
# Amount of queued items
82+
print('Queued items:', loader.queued)
83+
84+
# Amount of active/downloading items
85+
print('Active items:', loader.active)
86+
87+
# The amount of maximum concurrent allowed downloads
88+
print('Max concurrent:', loader.max_concurrent)
89+
# Change the amount of max concurrent downloads
90+
# Will also trigger new downloads if necessary
91+
loader.max_concurrent = 5
92+
93+
# How often the progress callback will be called in seconds
94+
print('Update interval:', loader.update_interval)
95+
# Change the interval
96+
loader.update_interval = 0.5
97+
98+
# Wait for all downloads to finish
99+
while loader.is_active():
100+
time.sleep(0.25)
101+
102+
# Exit downloader and stop all downloads
103+
# `pyloader` does NOT persist anything. This is up to you!
104+
loader.exit()

0 commit comments

Comments
 (0)