Skip to content

Commit

Permalink
adding images video
Browse files Browse the repository at this point in the history
  • Loading branch information
pelikhan committed Jan 28, 2025
1 parent 3a70874 commit 95b243f
Show file tree
Hide file tree
Showing 2 changed files with 115 additions and 0 deletions.
2 changes: 2 additions & 0 deletions docs/src/content/docs/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,8 @@ Include images in prompts, we'll crop/resize/resize then for you.
defImages(images, { autoCrop: true, details: "low" })
```

<YouTube id="https://youtu.be/XbWgDn7NdTg" />

</Card>

<Card title="Videos" icon="seti:video">
Expand Down
113 changes: 113 additions & 0 deletions docs/src/content/docs/reference/scripts/images.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
---
title: Images
description: Learn how to add images to prompts for AI models supporting visual inputs, including image formats and usage.
keywords: images in prompts, AI model images, visual inputs, image formats, OpenAI Vision
sidebar:
order: 10
---
import { YouTube } from "astro-embed"

Images can be added to the prompt for models that support this feature (like `gpt-4o`).
Use the `defImages` function to declare the images. Supported images will vary
with models but typically include `PNG`, `JPEG`, `WEBP`, and `GIF`. Both local files and URLs are supported.

```js
defImages(env.files)
```

<YouTube id="https://youtu.be/XbWgDn7NdTg" posterQuality="high" />

Read more about [OpenAI Vision](https://platform.openai.com/docs/guides/vision/limitations).

## URLs

Public URLs (that do not require authentication) will be passed directly to OpenAI.

```js wrap
defImages(
"https://github.com/microsoft/genaiscript/blob/main/docs/public/images/logo.png?raw=true"
)
```

Local files are loaded and encoded as a data uri.

## Buffer, Blob, ReadableStream

The `defImages` function also supports [Buffer](https://nodejs.org/api/buffer.html),
[Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob), [ReadableStream](https://nodejs.org/api/stream.html).

This example takes a screenshot of bing.com and adds it to the images.

```js wrap
const page = await host.browse("https://bing.com")
const screenshot = await page.screenshot() // returns a node.js Buffer
defImages(screenshot)
```

## Detail

OpenAI supports a "low" / "high" field. An image in "low" detail
will be downsampled to 512x512 pixels.

```js 'detail: "low"'
defImages(img, { detail: "low" })
```

## Cropping

You can crop a region of interest from the image.

```js "crop: { x: 0, y: 0, w: 512, h: 512 }" wrap
defImages(img, { crop: { x: 0, y: 0, w: 512, h: 512 } })
```

## Auto crop

You can also automatically remove uniform color on the edges of the image.

```js "autoCrop" wrap
defImages(img, { autoCrop: true })
```

## Greyscale

You can convert the image to greyscale.

```js "greyscale" wrap
defImages(img, { greyscale: true })
```

## Rotate

You can rotate the image.

```js "rotate: 90"
defImages(img, { rotate: 90 })
```

## Scale

You can scale the image.

```js "scale: 0.5"
defImages(img, { scale: 0.5 })
```

## Flip

You can flip the image.

```js "flip: { horizontal: true; vertical: true }" wrap
defImages(img, { flip: { horizontal: true; vertical: true } })
```

## Max width, max height

You can specify a maximum width, maximum height. GenAIScript will resize
the image to fit into the constraints.

```js "maxWidth: 800" "maxHeight: 800"
defImages(img, { maxWidth: 800 })
// and / or
defImages(img, { maxHeight: 800 })
```

0 comments on commit 95b243f

Please sign in to comment.