Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document how to handle action failures #157

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
+613 −5
Open

Document how to handle action failures #157

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
7cc483b
Document how to handle action failures
Gallaecio Dec 29, 2023
47197ee
Remove an empty line
Gallaecio Dec 29, 2023
00b2c2a
Fix the ZyteAPIFailedActionsRetryMiddleware example code
Gallaecio Dec 29, 2023
20f3715
Merge remote-tracking branch 'scrapy-plugins/main' into action-failur…
Gallaecio Jan 17, 2024
deedfff
Implement retries on action error
Gallaecio Jan 17, 2024
75faf68
Make mypy happy
Gallaecio Jan 17, 2024
cdc0aa1
Do not assume Zyte API responses have actions
Gallaecio Jan 17, 2024
f19e157
Backport get_retry_request for older Scrapy versions
Gallaecio Jan 17, 2024
62c9ae6
Fix test failures due to a nameless spider class
Gallaecio Jan 17, 2024
ea2da8f
Update docs/reference/settings.rst
Gallaecio Feb 22, 2024
e1eb2b1
Merge remote-tracking branch 'scrapy-plugins/main' into action-failur…
Gallaecio Dec 23, 2024
12c7907
Address black-introduced issue
Gallaecio Dec 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Loading
Loading status checks…
Fix test failures due to a nameless spider class
@Gallaecio
Gallaecio committed Jan 17, 2024
commit 62c9ae6072509e90242553ad2f953a47a6043869
14 changes: 7 additions & 7 deletions tests/test_middlewares.py
View file
Original file line number Diff line number Diff line change
@@ -436,7 +436,7 @@ class CrawleraSpider(Spider):
)
@ensureDeferred
async def test_action_error_retry_enabled(settings, meta, enabled):
crawler = get_crawler(settings_dict=settings)
crawler = get_crawler(NamedSpider, settings_dict=settings)
await crawler.crawl()

middleware = create_instance(
@@ -483,7 +483,7 @@ async def test_action_error_retry_enabled(settings, meta, enabled):
)
@ensureDeferred
async def test_action_error_retry_times(settings, meta, max_retries):
crawler = get_crawler(settings_dict=settings)
crawler = get_crawler(NamedSpider, settings_dict=settings)
await crawler.crawl()

middleware = create_instance(
@@ -533,7 +533,7 @@ async def test_action_error_retry_times(settings, meta, max_retries):
)
@ensureDeferred
async def test_action_error_retry_priority_adjust(settings, meta, priority):
crawler = get_crawler(settings_dict=settings)
crawler = get_crawler(NamedSpider, settings_dict=settings)
await crawler.crawl()

middleware = create_instance(
@@ -592,7 +592,7 @@ async def test_action_error_handling_no_retries(
settings, expected, setup_errors, caplog
):
settings["ZYTE_API_ACTION_ERROR_RETRY_ENABLED"] = False
crawler = get_crawler(settings_dict=settings)
crawler = get_crawler(NamedSpider, settings_dict=settings)
await crawler.crawl()

middleware = create_instance(
@@ -657,7 +657,7 @@ async def test_action_error_handling_no_retries(
@ensureDeferred
async def test_action_error_handling_retries(settings, expected, setup_errors, caplog):
settings["RETRY_TIMES"] = 1
crawler = get_crawler(settings_dict=settings)
crawler = get_crawler(NamedSpider, settings_dict=settings)
await crawler.crawl()

middleware = create_instance(
@@ -690,7 +690,7 @@ async def test_action_error_handling_retries(settings, expected, setup_errors, c

@ensureDeferred
async def test_process_response_non_zyte_api():
crawler = get_crawler()
crawler = get_crawler(NamedSpider)
await crawler.crawl()

middleware = create_instance(
@@ -707,7 +707,7 @@ async def test_process_response_non_zyte_api():

@ensureDeferred
async def test_process_response_no_action_error():
crawler = get_crawler()
crawler = get_crawler(NamedSpider)
await crawler.crawl()

middleware = create_instance(

Unchanged files with check annotations Beta

View file
scrapy_zyte_api/_middlewares.py
stats = spider.crawler.stats
retry_times = request.meta.get("retry_times", 0) + 1
if max_retry_times is None:
max_retry_times = request.meta.get("max_retry_times")
if max_retry_times is None:
max_retry_times = settings.getint("RETRY_TIMES")

Check warning on line 81 in scrapy_zyte_api/_middlewares.py

Codecov / codecov/patch

scrapy_zyte_api/_middlewares.py#L79-L81 

Added lines #L79 - L81 were not covered by tests
if retry_times <= max_retry_times:
logger.debug(
"Retrying %(request)s (failed %(retry_times)d times): %(reason)s",
new_request.meta["retry_times"] = retry_times
new_request.dont_filter = True
if priority_adjust is None:
priority_adjust = settings.getint("RETRY_PRIORITY_ADJUST")

Check warning on line 92 in scrapy_zyte_api/_middlewares.py

Codecov / codecov/patch

scrapy_zyte_api/_middlewares.py#L92 

Added line #L92 was not covered by tests
new_request.priority = request.priority + priority_adjust
if callable(reason):
reason = reason()

Check warning on line 96 in scrapy_zyte_api/_middlewares.py

Codecov / codecov/patch

scrapy_zyte_api/_middlewares.py#L96 

Added line #L96 was not covered by tests
if isinstance(reason, Exception):
reason = global_object_name(reason.__class__)

Check warning on line 98 in scrapy_zyte_api/_middlewares.py

Codecov / codecov/patch

scrapy_zyte_api/_middlewares.py#L98 

Added line #L98 was not covered by tests
stats.inc_value(f"{stats_base_key}/count")
stats.inc_value(f"{stats_base_key}/reason_count/{reason}")