File tree 2 files changed +24
-1
lines changed
2 files changed +24
-1
lines changed Original file line number Diff line number Diff line change @@ -12,7 +12,15 @@ so that one can get started easily.
12
12
13
13
## Scraped site
14
14
15
- This spider returns quotes from [ quotes.toscrape.com] ( https://quotes.toscrape.com ) .
15
+ This project contains two spiders. The ` quotes ` spider returns quotes from
16
+ [ quotes.toscrape.com] ( https://quotes.toscrape.com ) .
17
+
18
+ The ` static ` spider returns a single dummy quote without accessing the network.
19
+ This can be used for testing. There are several settings and environment variables
20
+ that modify its behaviour:
21
+ - spider setting ` STATIC_TEXT ` - quote text (default _ To be, or not to be_ )
22
+ - spider setting ` STATIC_AUTHOR ` - quote author (default _ Shakespeare_ )
23
+ - environment variable ` STATIC_TAGS ` - quote tags (default _ static_ )
16
24
17
25
## Running locally
18
26
@@ -33,6 +41,7 @@ $ scrapy list
33
41
```
34
42
> ```
35
43
> quotes
44
+ > static
36
45
> ```
37
46
```sh
38
47
$ scrapy crawl quotes
@@ -67,6 +76,7 @@ docker run --rm example scrapy list
67
76
```
68
77
> ```
69
78
> quotes
79
+ > static
70
80
> ```
71
81
72
82
```sh
Original file line number Diff line number Diff line change
1
+ import os
2
+ from scrapy import Spider
3
+
4
+ class StaticSpider (Spider ):
5
+ name = "static"
6
+ start_urls = ["file:///dev/null" ]
7
+
8
+ def parse (self , response ):
9
+ yield {
10
+ 'text' : self .settings .get ('STATIC_TEXT' , 'To be or not to be' ),
11
+ 'author' : self .settings .get ('STATIC_AUTHOR' , 'Shakespeare' ),
12
+ 'tags' : os .getenv ('STATIC_TAGS' , 'static' )
13
+ }
You can’t perform that action at this time.
0 commit comments