Skip to content

Commit 38a06e7

Browse files
PatrikKozakPatrikKozak
authored
feat: adds support for both client-side and server-side remote URL uploads fetching (payloadcms#10004)
### What? The `pasteURL` feature for Upload fields has been updated to support both **client-side** and **server-side** URL fetching. Previously, users could only paste URLs from the same domain as their Payload instance (internal) or public domains, which led to **CORS** errors when trying to fetch files from external URLs. Now, users can choose between **client-side fetching** (default) and **server-side fetching** using the new `pasteURL` option in the Upload collection config. ### How? - By default, Payload will attempt to fetch the file client-side directly in the browser. - To enable server-side fetching, you can configure the new `pasteURL` option with an `allowList` of trusted domains. - The new `/api/:collectionSlug/paste-url` endpoint is used to fetch files server-side and stream them back to the browser. #### Example ``` import type { CollectionConfig } from 'payload' export const Media: CollectionConfig = { slug: 'media', upload: { // pasteURL: false, // Can now disable the pasteURL option entirely by passing "false". pasteURL: { allowList: [ { hostname: 'payloadcms.com', // required pathname: '', port: '', protocol: 'https', // defaults to https - options: "https" | "http" search: '' }, { hostname: 'example.com', pathname: '/images/*', }, ], }, }, } ``` ### Why This update provides more flexibility for users to paste URLs into Upload fields without running into **CORS errors** and allows Payload to securely fetch files from trusted domains.
1 parent 28b7c04 commit 38a06e7

File tree

13 files changed

+493
-52
lines changed

13 files changed

+493
-52
lines changed

Diff for: ‎docs/upload/overview.mdx

+79-20
Original file line numberDiff line numberDiff line change
@@ -88,26 +88,27 @@ export const Media: CollectionConfig = {
8888

8989
_An asterisk denotes that an option is required._
9090

91-
| Option | Description |
92-
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
93-
| **`adminThumbnail`** | Set the way that the [Admin Panel](../admin/overview) will display thumbnails for this Collection. [More](#admin-thumbnails) |
94-
| **`bulkUpload`** | Allow users to upload in bulk from the list view, default is true |
95-
| **`cacheTags`** | Set to `false` to disable the cache tag set in the UI for the admin thumbnail component. Useful for when CDNs don't allow certain cache queries. |
96-
| **`crop`** | Set to `false` to disable the cropping tool in the [Admin Panel](../admin/overview). Crop is enabled by default. [More](#crop-and-focal-point-selector) |
97-
| **`disableLocalStorage`** | Completely disable uploading files to disk locally. [More](#disabling-local-upload-storage) |
98-
| **`displayPreview`** | Enable displaying preview of the uploaded file in Upload fields related to this Collection. Can be locally overridden by `displayPreview` option in Upload field. [More](/docs/fields/upload#config-options). |
99-
| **`externalFileHeaderFilter`** | Accepts existing headers and returns the headers after filtering or modifying. |
100-
| **`filesRequiredOnCreate`** | Mandate file data on creation, default is true. |
101-
| **`filenameCompoundIndex`** | Field slugs to use for a compound index instead of the default filename index. |
102-
| **`focalPoint`** | Set to `false` to disable the focal point selection tool in the [Admin Panel](../admin/overview). The focal point selector is only available when `imageSizes` or `resizeOptions` are defined. [More](#crop-and-focal-point-selector) |
103-
| **`formatOptions`** | An object with `format` and `options` that are used with the Sharp image library to format the upload file. [More](https://sharp.pixelplumbing.com/api-output#toformat) |
104-
| **`handlers`** | Array of Request handlers to execute when fetching a file, if a handler returns a Response it will be sent to the client. Otherwise Payload will retrieve and send back the file. |
105-
| **`imageSizes`** | If specified, image uploads will be automatically resized in accordance to these image sizes. [More](#image-sizes) |
106-
| **`mimeTypes`** | Restrict mimeTypes in the file picker. Array of valid mimetypes or mimetype wildcards [More](#mimetypes) |
107-
| **`resizeOptions`** | An object passed to the the Sharp image library to resize the uploaded file. [More](https://sharp.pixelplumbing.com/api-resize) |
108-
| **`staticDir`** | The folder directory to use to store media in. Can be either an absolute path or relative to the directory that contains your config. Defaults to your collection slug |
109-
| **`trimOptions`** | An object passed to the the Sharp image library to trim the uploaded file. [More](https://sharp.pixelplumbing.com/api-resize#trim) |
110-
| **`withMetadata`** | If specified, appends metadata to the output image file. Accepts a boolean or a function that receives `metadata` and `req`, returning a boolean. |
91+
| Option | Description |
92+
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
93+
| **`adminThumbnail`** | Set the way that the [Admin Panel](../admin/overview) will display thumbnails for this Collection. [More](#admin-thumbnails) |
94+
| **`bulkUpload`** | Allow users to upload in bulk from the list view, default is true |
95+
| **`cacheTags`** | Set to `false` to disable the cache tag set in the UI for the admin thumbnail component. Useful for when CDNs don't allow certain cache queries. |
96+
| **`crop`** | Set to `false` to disable the cropping tool in the [Admin Panel](../admin/overview). Crop is enabled by default. [More](#crop-and-focal-point-selector) |
97+
| **`disableLocalStorage`** | Completely disable uploading files to disk locally. [More](#disabling-local-upload-storage) |
98+
| **`displayPreview`** | Enable displaying preview of the uploaded file in Upload fields related to this Collection. Can be locally overridden by `displayPreview` option in Upload field. [More](/docs/fields/upload#config-options). |
99+
| **`externalFileHeaderFilter`** | Accepts existing headers and returns the headers after filtering or modifying. |
100+
| **`filesRequiredOnCreate`** | Mandate file data on creation, default is true. |
101+
| **`filenameCompoundIndex`** | Field slugs to use for a compound index instead of the default filename index. |
102+
| **`focalPoint`** | Set to `false` to disable the focal point selection tool in the [Admin Panel](../admin/overview). The focal point selector is only available when `imageSizes` or `resizeOptions` are defined. [More](#crop-and-focal-point-selector) |
103+
| **`formatOptions`** | An object with `format` and `options` that are used with the Sharp image library to format the upload file. [More](https://sharp.pixelplumbing.com/api-output#toformat) |
104+
| **`handlers`** | Array of Request handlers to execute when fetching a file, if a handler returns a Response it will be sent to the client. Otherwise Payload will retrieve and send back the file. |
105+
| **`imageSizes`** | If specified, image uploads will be automatically resized in accordance to these image sizes. [More](#image-sizes) |
106+
| **`mimeTypes`** | Restrict mimeTypes in the file picker. Array of valid mimetypes or mimetype wildcards [More](#mimetypes) |
107+
| **`pasteURL`** | Controls whether files can be uploaded from remote URLs by pasting them into the Upload field. **Enabled by default.** Accepts `false` to disable or an object with an `allowList` of valid remote URLs. [More](#uploading-files-from-remote-urls) |
108+
| **`resizeOptions`** | An object passed to the the Sharp image library to resize the uploaded file. [More](https://sharp.pixelplumbing.com/api-resize) |
109+
| **`staticDir`** | The folder directory to use to store media in. Can be either an absolute path or relative to the directory that contains your config. Defaults to your collection slug |
110+
| **`trimOptions`** | An object passed to the the Sharp image library to trim the uploaded file. [More](https://sharp.pixelplumbing.com/api-resize#trim) |
111+
| **`withMetadata`** | If specified, appends metadata to the output image file. Accepts a boolean or a function that receives `metadata` and `req`, returning a boolean. |
111112

112113

113114
### Payload-wide Upload Options
@@ -327,6 +328,64 @@ fetch('api/:upload-slug', {
327328
})
328329
```
329330

331+
## Uploading Files from Remote URLs
332+
333+
The `pasteURL` option allows users to fetch files from remote URLs by pasting them into an Upload field. This option is **enabled by default** and can be configured to either **allow unrestricted client-side fetching** or **restrict server-side fetching** to specific trusted domains.
334+
335+
By default, Payload uses **client-side fetching**, where the browser downloads the file directly from the provided URL. However, **client-side fetching will fail if the URL’s server has CORS restrictions**, making it suitable only for internal URLs or public URLs without CORS blocks.
336+
337+
To fetch files from **restricted URLs** that would otherwise be blocked by CORS, use **server-side fetching** by configuring the `pasteURL` option with an `allowList` of trusted domains. This method ensures that Payload downloads the file on the server and streams it to the browser. However, for security reasons, only URLs that match the specified `allowList` will be allowed.
338+
339+
#### Configuration Example
340+
341+
Here’s how to configure the pasteURL option to control remote URL fetching:
342+
343+
```
344+
import type { CollectionConfig } from 'payload'
345+
346+
export const Media: CollectionConfig = {
347+
slug: 'media',
348+
upload: {
349+
pasteURL: {
350+
allowList: [
351+
{
352+
hostname: 'payloadcms.com', // required
353+
pathname: '',
354+
port: '',
355+
protocol: 'https',
356+
search: ''
357+
},
358+
{
359+
hostname: 'example.com',
360+
pathname: '/images/*',
361+
},
362+
],
363+
},
364+
},
365+
}
366+
```
367+
368+
##### Accepted Values for `pasteURL`
369+
370+
| Option | Description |
371+
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
372+
| **`undefined`** | Default behavior. Enables client-side fetching for internal or public URLs. |
373+
| **`false`** | Disables the ability to paste URLs into Upload fields. |
374+
| **`allowList`** | Enables server-side fetching for specific trusted URLs. Requires an array of objects defining trusted domains. See the table below for details on `AllowItem`. |
375+
376+
##### `AllowItem` Properties
377+
378+
_An asterisk denotes that an option is required._
379+
380+
| Option | Description | Example |
381+
| ---------------- | ---------------------------------------------------------------------------------------------------- | ------------- |
382+
| **`hostname`** * | The hostname of the allowed URL. This is required to ensure the URL is coming from a trusted source. | `example.com` |
383+
| **`pathname`** | The path portion of the URL. Supports wildcards to match multiple paths. | `/images/*` |
384+
| **`port`** | The port number of the URL. If not specified, the default port for the protocol will be used. | `3000` |
385+
| **`protocol`** | The protocol to match. Must be either `http` or `https`. Defaults to `https`. | `https` |
386+
| **`search`** | The query string of the URL. If specified, the URL must match this exact query string. | `?version=1` |
387+
388+
330389
## Access Control
331390

332391
All files that are uploaded to each Collection automatically support the `read` [Access Control](/docs/access-control/overview) function from the Collection itself. You can use this to control who should be allowed to see your uploads, and who should not.

Diff for: ‎packages/payload/src/collections/endpoints/getFileFromURL.ts

+81
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
import type { PayloadHandler } from '../../config/types.js'
2+
3+
import executeAccess from '../../auth/executeAccess.js'
4+
import { APIError } from '../../errors/APIError.js'
5+
import { Forbidden } from '../../errors/Forbidden.js'
6+
import { getRequestCollectionWithID } from '../../utilities/getRequestEntity.js'
7+
import { isURLAllowed } from '../../utilities/isURLAllowed.js'
8+
9+
// If doc id is provided, it means we are updating the doc
10+
// /:collectionSlug/paste-url/:doc-id?src=:fileUrl
11+
12+
// If doc id is not provided, it means we are creating a new doc
13+
// /:collectionSlug/paste-url?src=:fileUrl
14+
15+
export const getFileFromURLHandler: PayloadHandler = async (req) => {
16+
const { id, collection } = getRequestCollectionWithID(req, { optionalID: true })
17+
18+
if (!req.user) {
19+
throw new Forbidden(req.t)
20+
}
21+
22+
const config = collection?.config
23+
24+
if (id) {
25+
// updating doc
26+
const accessResult = await executeAccess({ req }, config.access.update)
27+
if (!accessResult) {
28+
throw new Forbidden(req.t)
29+
}
30+
} else {
31+
// creating doc
32+
const accessResult = await executeAccess({ req }, config.access?.create)
33+
if (!accessResult) {
34+
throw new Forbidden(req.t)
35+
}
36+
}
37+
try {
38+
if (!req.url) {
39+
throw new APIError('Request URL is missing.', 400)
40+
}
41+
42+
const { searchParams } = new URL(req.url)
43+
const src = searchParams.get('src')
44+
45+
if (!src || typeof src !== 'string') {
46+
throw new APIError('A valid URL string is required.', 400)
47+
}
48+
49+
const validatedUrl = new URL(src)
50+
51+
if (
52+
typeof config.upload?.pasteURL === 'object' &&
53+
!isURLAllowed(validatedUrl.href, config.upload.pasteURL.allowList)
54+
) {
55+
throw new APIError(`The provided URL (${validatedUrl.href}) is not allowed.`, 400)
56+
}
57+
58+
// Fetch the file with no compression
59+
const response = await fetch(validatedUrl.href, {
60+
headers: {
61+
'Accept-Encoding': 'identity',
62+
},
63+
})
64+
65+
if (!response.ok) {
66+
throw new APIError(`Failed to fetch file from ${validatedUrl.href}`, response.status)
67+
}
68+
69+
const decodedFileName = decodeURIComponent(validatedUrl.pathname.split('/').pop() || '')
70+
71+
return new Response(response.body, {
72+
headers: {
73+
'Content-Disposition': `attachment; filename="${decodedFileName}"`,
74+
'Content-Length': response.headers.get('content-length') || '',
75+
'Content-Type': response.headers.get('content-type') || 'application/octet-stream',
76+
},
77+
})
78+
} catch (error) {
79+
throw new APIError(`Error fetching file: ${error.message}`, 500)
80+
}
81+
}

Diff for: ‎packages/payload/src/collections/endpoints/index.ts

+6
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ import { findByIDHandler } from './findByID.js'
1313
import { findVersionByIDHandler } from './findVersionByID.js'
1414
import { findVersionsHandler } from './findVersions.js'
1515
import { getFileHandler } from './getFile.js'
16+
import { getFileFromURLHandler } from './getFileFromURL.js'
1617
import { previewHandler } from './preview.js'
1718
import { restoreVersionHandler } from './restoreVersion.js'
1819
import { updateHandler } from './update.js'
@@ -46,6 +47,11 @@ export const defaultCollectionEndpoints: Endpoint[] = [
4647
method: 'post',
4748
path: '/access/:id?',
4849
},
50+
{
51+
handler: getFileFromURLHandler,
52+
method: 'get',
53+
path: '/paste-url/:id?',
54+
},
4955
{
5056
handler: findVersionsHandler,
5157
method: 'get',

Diff for: ‎packages/payload/src/uploads/types.ts

+19
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,14 @@ export type ImageSize = {
8383

8484
export type GetAdminThumbnail = (args: { doc: Record<string, unknown> }) => false | null | string
8585

86+
export type AllowList = Array<{
87+
hostname: string
88+
pathname?: string
89+
port?: string
90+
protocol?: 'http' | 'https'
91+
search?: string
92+
}>
93+
8694
export type UploadConfig = {
8795
/**
8896
* The adapter name to use for uploads. Used for storage adapter telemetry.
@@ -175,6 +183,17 @@ export type UploadConfig = {
175183
* @default undefined
176184
*/
177185
modifyResponseHeaders?: ({ headers }: { headers: Headers }) => Headers
186+
/**
187+
* Controls the behavior of pasting/uploading files from URLs.
188+
* If set to `false`, fetching from remote URLs is disabled.
189+
* If an allowList is provided, server-side fetching will be enabled for specified URLs.
190+
* @default true (client-side fetching enabled)
191+
*/
192+
pasteURL?:
193+
| {
194+
allowList: AllowList
195+
}
196+
| false
178197
/**
179198
* Sharp resize options for the original image.
180199
* @link https://sharp.pixelplumbing.com/api-resize#resize

0 commit comments

Comments
 (0)