Skip to content

AI Plugin that can be used to scrape useful information from a given URL

License

Notifications You must be signed in to change notification settings

craigomatic/webscraper-aiplugin

Folders and files

NameName
Last commit message
Last commit date

Latest commit

5def7af · Sep 2, 2023

History

29 Commits
Jul 28, 2023
Aug 21, 2023
Aug 11, 2023
Jul 28, 2023
Jul 28, 2023
Jul 31, 2023
Jul 31, 2023
Jul 26, 2023
Aug 2, 2023
Sep 2, 2023
Aug 10, 2023
Aug 18, 2023
Aug 15, 2023
Aug 21, 2023
Sep 2, 2023

Repository files navigation

Webscraper AIplugin

AI Plugin that can be used to scrape useful information from a given URL.

Configuration

Local dev

Open local.settings.json and add values for each of the following fields:

CompletionConfig:AIService

One of AzureOpenAI or OpenAI

CompletionConfig:Endpoint

Your Azure endpoint if using Azure OpenAI, or empty if using OpenAI

CompletionConfig:DeploymentOrModelId

Your Azure deployment name if using Azure OpenAI, or model name if using OpenAI

CompletionConfig:Key

Your API key

Deploy to Azure

You'll need to have all the same values listed above entered into Configuration -> Application Settings

Usage

Run it locally with this command:

func start

You can test functionality using the swagger UI, which will default to: http://localhost:7071/api/swagger/ui

Note that the first time the function runs, it will attempt to install the Playwright dependencies (chromium, etc) by running a powershell script and that requests to the endpoint will result in 503s while this is happening.

Once you've verified it locally using swagger, the next step is to try running it in the context of other AI.

Some options I like for this are:

  1. Chat Copilot
  2. sk-researcher

About

AI Plugin that can be used to scrape useful information from a given URL

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages