Respect HTTP Accept header

[sergiodxa]

Proposal

Make Remix respect the HTTP Accept header and if it's not text/html send the loader data instead, even if the route is a UI route.

Background

The Accept header let clients (like browsers or another app) let the server know what type of responses it can handle. Then the server can send a different response type based on that.

Right now, you can use it in a resource route quite simple:

export async function loader({ request }: LoaderArgs) {
  let accept = request.headers.get("Accept")
  let data = await getDataSomehow()
  if (accept.includes("application/json")) return json(data);
  if (accept.includes("application/pdf")) return pdf(renderToPDF(data));
  throw badRequest("Invalid accept type") // or fallback to json
}

If a UI route (a route with a export default) wants to support this it's not possible, Remix will always attempt to render the app, the only way to skip it is with the special _data search param.

Use Case

• Let developer use the same route loader without creating a separate resource route
• Improve web standards support
• Allow the same URL to send different representations of the same data (e.g. HTML, JSON, XML or PDF)

API/Examples

The idea would be for Remix to detect the header and if the Accept is not text/html treat the request as a resource route request.

This would mean that Remix will call any leaf route matching the request and it will send the loader response, if the developer wants to support more than JSON it should manually check the Accept header to return something different based on that.

If the request comes with the text/html inside Accept Remix should treat the request a document one and call all matching routes.

Remix must still use the _data search param to ensure the URL for JSON and the URL for document made by Remix itself are different and avoid any issue with CDNs not respecting Accept header.

For requests with the Accept not being text/html it should be documented somewhere that CDNs may cache the responses so developers can take care of that and avoid setting Cache-Control or use a CDN which correctly handle Accept and Vary.

Prior Art

Rails works this way, you can use a single controller to send different response types like HTML or JSON, they even give you a function to do the conditions easier

respond_to do |format|
  # Rails take care of using the correct template, Remix this will happen outside the loader
  format.html
  # You could send an XML response
  format.xml  { render xml: @users }
  # Our you could send a JSON response
  format.json { render json: @users }
end

This could either be implemented by Remix or left to devs to do it, maybe Remix Utils could provide a helper

let data = await getDataSomehow();
return respondTo(request.headers, {
  json() {
    return json(data);
  },
  xml() {
    let string = createXMLFromJSON(data);
    return xml(string);
  },
  pdf() {
    let pdfFile = renderPDF(data);
    return pdf(pdfFile);
  },
  "application/custom-type": () => {
    return new Response(parseDataSomehow(data), {
      headers: { "Content-Type": "application/custom-type" }
    });
  },
  default() {
    // set a default for unknown Accept values, you could throw here or return another one
    return json(data);
  }
});

[machour]

This was discussed in Roadmap #5, around 20'55
Ryan and Michael are incline to not doing that, because of CDN caching, and avoidable "complexity".

The workaround is to implement a myroute[.ext].ts routes, and serve that different version of content from there, or continue to use the _data param.

[sergiodxa]

That workaround doesn't work if you want to migrate a current app to Remix without breaking URLs, also

Remix must still use the _data search param to ensure the URL for JSON and the URL for document made by Remix itself are different and avoid any issue with CDNs not respecting Accept header.

I mentioned that Remix should still use _data search param to avoid this CDN caching issue, but supporting this can still be useful for app developers who know their CDN either supports Accept header or they're not caching at the CDN so this is not an issue for them.

[jamesarosen]

I mentioned that Remix should still use _data search param to avoid this CDN caching issue

Some CDNs (notably Amazon CloudFront) do not vary on query-params by default and others will refuse to cache anything that has query-params. Rails calls this out in their static assets guide:

The query string strategy has several disadvantages:
Not all caches will reliably cache content where the filename only differs by query parameters. Steve Souders recommends, "...avoiding a querystring for cacheable resources". He found that in this case 5-20% of requests will not be cached. Query strings in particular do not work at all with some CDNs for cache invalidation.

[sergiodxa]

Using that search param is how it works right now

[jamesarosen]

Building on this topic, the Form Validation docs show this:

export const action = async ({ request }: ActionArgs) => {
  const formData = await request.formData();
  const [errors, project] = await createProject(formData);

  if (errors) {
    const values = Object.fromEntries(formData);
    return json({ errors, values });
  }

  return redirect(`/projects/${project.id}`);
};

That absolutely works, but it delivers a pretty lousy experience for non-JS browsers. Who wants to see a page with nothing but a JSON-encoded error message?

Adapting the Rails pattern mentioned above, this might become

return respondTo(request.headers, {
  json: () => json({ errors, values }),
  html: () => {
    return redirectBackOrDefault('/', {
      headers: { 'set-cookie': flash(errors) }
    }
  },
})

[machour]

@jamesarosen if a user don't have JS enabled, the form will trigger a full page reload and the json() data will correctly populate the page.

[jamesarosen]

the form will trigger a full page reload

This form

<form method="POST" action="/projects/123">
  <button type="submit">Save</button>
</form>

will certainly trigger a full page load, but the page it loads will be the response to POST /projects/123, which will be JSON from

return json({ errors, values });

In order to re-render the form with an error message, we would need to change the action handler to save the errors in session flash and redirect:

if (errors) {
  session.flash("formErrors", JSON.stringify(errors));
}

redirect(`/projects/${project.id}`);

[sergiodxa]

If /projects/123 is a resource route then yes, you will need a redirect, otherwise Remix should render the component of /projects/123 when using a <form> instead of <Form>.

It's also possible to know if a fetcher.Form was submitted with or without JS and change the response from json to a redirect, check https://sergiodxa.com/articles/progressively-enhance-the-usefetcher-hook-in-remix.

If this is not a case, try to create a reproduction repo because this may be a bug.

[jamesarosen]

Remix should render the component of /projects/123 when using a <form> instead of <Form>.

I think this is the key disconnect. I'm glad you put your finger on it. Yes, Remix's front-end components enrich this behavior. I was thinking of the HTTP APIs in isolation, not necessarily consumed by the Remix UI.

[kael]

FYI, SvelteKit supports content-negotiation by using two different files formats under the same route :

To determine which, SvelteKit applies the following rules:

* PUT/PATCH/DELETE/OPTIONS requests are always handled by +server.js since they do not apply to pages

* GET/POST/HEAD requests are treated as page requests if the accept header prioritises text/html 
  (in other words, it's a browser page request), else they are handled by +server.js.

* Responses to GET requests will include a Vary: Accept header, so that proxies and browsers cache HTML and  
  JSON responses separately.

[tmcw]

👋 Another vote in support of this. My usecase is pretty similar to how deno.land supports a page like this: https://deno.land/[email protected]/bytes/bytes_list.ts - which renders HTML for browsers but returns TypeScript for Deno and other clients.

The options for this kind of problem - content negotiation - in Remix are lackluster. I can't implement a system where you add .ts to the end of URLs to get the TypeScript version because the router doesn't support regex routes and so can't differentiate between /route/$param.ts and /route/$param[.ts].ts. And I can't support a way to redirect away from /route/$param.ts to a specific TypeScript-serving resource route, because as this discussion mentions, it's impossible to have a UI route that does content negotiation.

[kael]

it's impossible to have a UI route that does content negotiation.

I've used a routes redirect from /user/$id.js to /user/$id[.json].js for that case.

[nickretallack]

TIL that the behavior of an action changes based on whether on not there is a default export in the same route file. This seems like an unfortunate counter-intuitive gotcha. Please consider changing this, or you'll have more people tripping on this problem and getting upset with the framework.

The _data parameter work-around seems very bad. The information you pass to the _data parameter is redundant with the URL that's already in the request. What happens if those two don't match? Which one wins? Why do we have to construct this redundant information anyway? Something like _format=json would be preferable. But if you're really going to claim to be the framework that embraces web standards, you should support accept headers.

Let people handle caching in their own way. It would still be possible to use other methods to fix the caching problem even if you did support accept headers.

Also, even just having a way to make an action always return JSON regardless of whether there is a default export in that file would be nice too. Maybe call it apiAction or something. Then you wouldn't be tripped up with it suddenly changing behavior when you add a default export later.

[sergiodxa]

The reason they use .data instead of Accept header is because CDN cache, some (really popular) CDN ignore HTTP headers for cache, so if your CDN cache /something and that URL can return JSON or HTML it will cache one and return it always, by adding .data or ?_data it will avoid avoid this issue since now the URL is different.

[nickretallack]

I was frustrated to find that when I enable v3 types the .data suffix now gives me turbo-stream when the server returns JSON, but remix doesn't seem to expose a way to directly access or parse this from a POST response. fetcher.submit doesn't return data and fetcher.load doesn't accept a method. Is there any way to access the parser? Do I have to watch useActionData?

It's wild how many hoops I have to go through just to get the data that I'm returning from the server. I'm not even trying to do an API here, by the way. I just want to submit a form, get the result, and send that result to an external API (it's Telegram.WebApp.switchInlineQuery in case you're curious). It seems my options are:

• Put the action at a different route from the form that submits to it.
• Get the response with useActionData and watch that with a useEffect to send it to the second step, triggering a needless re-render in the process.
• Pass the secret _data parameter in v2, or find a way to parse turbo-stream in v3.

Or, remix could support automatic content negotiation and it could just work with no changes to the developer interface.

[nickretallack]

Btw, which CDNs are you talking about that don't support the Accept header? Is it just Microsoft Azure? Can we make a special mode just for them? :P

Regardless, nobody should be caching POST responses, since you must hit the server for those. I guess POST is affected by this policy for consistency with GET?

[sergiodxa]

Cloudflare, Cloudflare doesn't support the Vary header needed to tell it to cache a response based on the Accept header. And Cloudflare is probably the most popular one.

Netlify CDN uses a custom Netlify-Vary, and it even ignores search params by default, you need to configure it to make /foo?bar=1 and /foo?bar=2 different keys in their CDN.

[nickretallack]

Hm. I see. It takes custom workers to make it consider the Accept header.

And I see that the .data approach also avoids using querystring arguments that might cause problems with other CDNs.

As for the .data approach, would you consider exposing some way to do a fetch("thing.data", {method: "POST"}) that would parse the turbo-stream POST response data the same way that useActionData does? I really just want to fetch data from the server without having to do an extra re-render and watch that data with useEffect, and it seems unfortunate to sometimes make things into "resource routes" when Remix is happily consuming the data from non-resource routes all the time, but just making it inconvenient to access that data. Having fetcher.submit() return a promise for the response data instead of void would be one way to provide that.