feature: web crawler ingestion

[densumesh]

Description

We want to be able to ingest any blog/doc site into Trieve

• Get Firecrawl up and running to the point where api access works #2297
• feature: chunker/chunkers to target almost all blog/doc site #2301
• Set up a worker to poll firecrawl and retrieve chunks #2303
• Set up server to accept urls to scrape for datasets
• Be able to demonstrate to the user the current state of scraping/when their job has been scheduled for
• Modify dashboard to have first class experience for scraping

Target(s)

server, dashboard

Requirement to close

Community channels

Matrix is preferred. Reach out on discord or Matrix for further assistance.

• discord
• matrix