Add core support for page level A/B testing

[Auspicus]

Summary

There was some recent discussion around page level A/B testing (swapping out the template depending on the variant) using edge rewrites. We came to the conclusion that whilst this is possible without modifications to Gatsby core it would be nice to provide some sort of API to allow for this in the createPage action. Similar to the Script component, having a recommended way to handle this will ensure better community direction and less fragmentation. First class Gatsby Cloud support would also be amazing!

There's been some similar discussions around this:

• Control gatsby router behaviour #36063
• Disable Client-Side Routing? #4337
• https://github.com/wardpeet/gatsby-plugin-static-site
• [docs][guides] Personalization in Gatsby #31219
• https://www.gatsbyjs.com/docs/ab-testing/

I have also written an article on our discussion here https://daviddisch.com/blog/using-netlify-edge-functions-for-request-rewrites-to-run-page-level-a-b-tests-on-a-gatsby-site

Detailed design

It's still unclear how the API would work but during the createPage call, to do this currently, you need to use matchPath to set the expected path for the template. eg.

createPage({
  path: '/home-a',
  component: path.resolve('templates/home-a.tsx'),
  matchPath: '/',
})

createPage({
  path: '/home-b',
  component: path.resolve('templates/home-b.tsx'),
  matchPath: '/',
})

Because these two pages have the same matchPath they will both be selected for use at / but the first one takes precedent so we have to swap the page-data.json files out. Similar to the defer API, if the user was allowed to pass a percentage for the chance the variant would be shown I think that would feel very comprehensive. eg.

createPage({
  path: '/',
  component: path.resolve('templates/home-a.tsx'),
  variantChance: 0.5,
})

createPage({
  path: '/',
  component: path.resolve('templates/home-b.tsx'),
  variantChance: 0.5,
})

However, without knowing what affinity the user is expecting it's hard for Gatsby to do anything for us here (ie. should we use cookies to persist the test session? should we use a session token?)

Perhaps this affinity is configured in gatsby-config similar to trailingSlash or other global settings:

{
  pageVariantAffinity: 'cookie' | 'token'
}

Finally, the API could provide some async function that would determine the A/B variant based on user data:

// maybe in `gatsby-node.ts`
const selectUserVariant = async (userData: UserData): Variant => {
  // determine variant based on user data ie.
  if (userData.cookies['test_bucket'] === 'a') {
     return Variant.A
  }
  if (userData.cookies['test_bucket'] === 'b') {
     return Variant.B
  }

  // or you could do an async call to your personalization engine to determine variant
  const r = await fetch('https://mypersonalizationengine.com/api/v1/variant')
  // use response data to determine variant
}

Drawbacks / alternatives

Perhaps the scope of A/B testing is beyond the touch of Gatsby. This will seriously complicate the internals of pages and will probably have implications on a lot of core functionality like routing, caching, etc.

Rollout

...

What's next?

• Decide if this feature is worth adding and the added complexity is worth the gains
• Decide upon a draft API