[Bug] crawl4ai tool not able to scrape data

[GauravKasat]

from agno.agent import Agent
from agno.tools.crawl4ai import Crawl4aiTools
import warnings
warnings.simplefilter("ignore") # Ignore all warnings
import os
os.environ["OPENAI_API_KEY"] = "api-key"

agent = Agent(tools=[Crawl4aiTools(max_length=None)], show_tool_calls=True)
agent.print_response("Tell me the price and rating mentioned on https://www.amazon.in/Apple-iPhone-Pro-Max-256/dp/B0CHWV2WYK?th=1")

Its not able to bring down the price and rating specified for my use case

[isikepalaku]

add cannot run in docker as agent:

2025-02-10 16:26:08 from api.routes.v1_router import v1_router
2025-02-10 16:26:08 File "/app/api/routes/v1_router.py", line 3, in
2025-02-10 16:26:08 from api.routes.playground import playground_router
2025-02-10 16:26:08 File "/app/api/routes/playground.py", line 7, in
2025-02-10 16:26:08 from agents.trend_kejahatan import get_crime_trend_agent
2025-02-10 16:26:08 File "/app/agents/trend_kejahatan.py", line 10, in
2025-02-10 16:26:08 from agno.tools.crawl4ai import Crawl4aiTools
2025-02-10 16:26:08 File "/usr/local/lib/python3.12/site-packages/agno/tools/crawl4ai.py", line 7, in
2025-02-10 16:26:08 from crawl4ai import AsyncWebCrawler, CacheMode
2025-02-10 16:26:08 File "/usr/local/lib/python3.12/site-packages/crawl4ai/init.py", line 3, in
2025-02-10 16:26:08 from .async_webcrawler import AsyncWebCrawler, CacheMode
2025-02-10 16:26:08 File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_webcrawler.py", line 14, in
2025-02-10 16:26:08 from .async_database import async_db_manager
2025-02-10 16:26:08 File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_database.py", line 24, in
2025-02-10 16:26:08 os.makedirs(DB_PATH, exist_ok=True)
2025-02-10 16:26:08 File "", line 225, in makedirs
2025-02-10 16:26:08 PermissionError: [Errno 13] Permission denied: '/app/.crawl4ai'

[dirkbrnd]

@isikepalaku It looks like your app does not have the required folder permissions locally? Is this running in Docker?

[isikepalaku]

@isikepalaku It looks like your app does not have the required folder permissions locally? Is this running in Docker?

I run it on docker

[dirkbrnd]

It could be related to the user you set in Docker?