Code splitting and i18n files

[slorber]

Hi,

The focus of this issue/question is not related to making i18n work, but is related to making it fast (ie, make code splitting work in an optimized way for your translation bundles).

There are already issues regarding how to translate your Gatsby website with various tools/libs/CMS. There are also some nice blog posts on how to create one page per lang (hint, loop over your locales and call createPages, or read this post

But so far, all the solutions mentioned focus on making it work, but they make it work in an unoptimized way.

Here I'll focus mostly on splitting per language. If you have to split translations by page, the best of probably to use a CMS and inject page specific data from context/query. What I focus on is mostly translations that are "global" into your app, like the translations appearing in your main layout that you reuse everywhere.

Bad solution 1: importing all locale files in your layout

If you do this in your layout component:

import enData from 'react-intl/locale-data/en'
import ptData from 'react-intl/locale-data/pt'
import en from '../i18n/en.json'
import pt from '../i18n/pt.json'

Then, all the translations will always be downloaded by your user inside the js bundle, for any page of your site.

This means that if you have 50 languages and a very large static site, this solution will download a lot of useless bytes. The more your static site grows, the more there is a delay between the static page being rendered and the static page being interactive (ie you can't click on the buttons yet).

This is to me a big problem and makes this solution unsuitable

Bad solution 2: providing locale messages with pageContext

If you do this in gatsby-node.js

["fr","en"].map(locale => {
  createPage({
    path: `/${locale}/${page.path}`
    context: {
      messages: getMessagesForLocale(locale);
    }
  })
})

Then you can use the messages of your locale from the context.

The problem is that the french locale messages will be in the context of every french page. It means, everytime you navigate to another french page, you will redownload everytime those translations as it's included into all pages js bundles.

Possible solution

@KyleAMathews mentioned here that we could:

• Create on layout per lang
• Create on "wrapper component" per page, per locale

I can see how this would solve the code splitting issues, and how webpack would be able to move some translation files into "common chunks", but this is a dev maintainance burden not really sustainable.

This means that for 2 locales and 2 pages you would have:

• EnLayout
• FrLayout
• EnPage1
• FrPage1
• EnPage2
• FrPage2

Now for 3 locales and 3 pages you would have:

• EnLayout
• FrLayout
• DeLayout
• EnPage1
• FrPage1
• DePage1
• EnPage2
• FrPage2
• DePage2
• EnPage3
• FrPage3
• DePage3

Now, imagine that you have 10 pages and 10 locales (which I would not consider a large international website), you end up with more than 1000 entry point files to maintain, most of them look almost similar.

Possible solution: using the layout plugin

This comment of @KyleAMathews (and some other parts of the thread), is helpful and

However, I'm not even sure this solution works in v2, because the plugin doc that not mention we can use different layouts in createPages while according to this comment it seems v1 allowed to do this:

posts.edges.map(({ node }) => {
  const id = node.contentfulid;
  const locale = node.node_locale;

  return createPage({
    path: `/${locale}/blog/posts/${id}`,
    layout: locale,
    component: path.resolve('./src/templates/post-page.jsx'),
    context: {
      id,
      locale,
    },
  });
});

Is it still possible to use different layout (in term of layout plugin) from createPages?

What should we do?

I'd like to know what is the current recommended way to solve this problem, that is suitable for a large but realistic website (something like 50 pages and 20 locales) and optimized.

cc @KyleAMathews @deltaskelta @szimek @pbrandone

[universse]

https://github.com/kentcdodds/babel-plugin-preval / https://github.com/kentcdodds/preval.macro might be a nice solution to this. preval.macro works out of the box with gatsby. See https://www.gatsbyjs.org/docs/babel-plugin-macros/

First during createPages, you inject locale into page context.

posts.edges.map(({ node }) => {
  const id = node.contentfulid;
  const locale = node.node_locale;

  return createPage({
    path: `/${locale}/blog/posts/${id}`,
    component: path.resolve('./src/templates/post-page.jsx'),
    context: {
      id,
      locale,
    },
  });
});

Then, in your layouts/index.js (I'm assuming you are using gatsby-plugin-layout)

import preval from 'preval.macro'

export default function Layout ({ pageContext: { locale }, children }) {
  // with preval.macro you can use node apis (like fs.readFile) and have the data at build time
  const translations = preval`
    const fs = require('fs')
   
    // here interpolate pageContext.locale to to read the correct translation file. Also, adjust file path accordingly

    const translations = fs.readFileSync('./${locale}.json')
    module.exports = translations
  `
  
  // LanguageProvider use Context api
  return ( 
    <LanguageProvider value={translations}>
      {children}
    </LanguageProvider>
}

This way you only need 1 layout file instead of 1 for each language.

Then in your component, read the translations from LanguageContext.

const translations = useContext(LanguageContext)

One last runtime optimization. Since translations is a new object every render, it can be memoized with useMemo hook. Useful if you are managing state here.

const translations = useMemo(
  () => preval`
    const fs = require('fs')
    // here interpolate pageContext.locale to to read the correct translation file.
    const translations = fs.readFileSync('./${locale}.json')
    module.exports = translations
  `,
  []
)

[slorber]

That looks like a great solution if this works :)

I thought of using macros but wasn't sure it would work because I thought macros would not be able to get access to the locale variable but I may be wrong.

Have you tried this approach and confirm it works or it's an idea to test?

Edit, got confirmation that it won't work: https://twitter.com/sebastienlorber/status/1123963405486960642

[universse]

Hm. Ok last time I made use of this technique, I didn't interpolate variables from react props. Thought it could work here since gatsby statically renders things. Take a look here https://codesandbox.io/s/6l7n7pvkr3

[slorber]

Yeah this works because the locale is hardcoded/static, but don't think it's suitable to solve the problem.

[szimek]

I'd love to see an official solution to this issue.

After migrating from v1 to v2, we've started to build the app separately for each language as a workaround for this issue, and some other ones, like huge size of pages-manifest.js file and lack of assetPrefix option back then.

[slorber]

Hi and thanks for the feedback @szimek . I did not talk about your growing pages manifest but yes that looks like a potential issue for scalability I didn't think about.

Even after reading your comment here I'm not totally sure what you mean by using asset prefix? Do you create one prefix per locale? What problem does it solve for you?

[slorber]

The kind of API I would like in Gatsby would enable this.

const AppLocaleList = ["fr_FR","en-US"];
const TranslatedPages = [
    {
      path: '/items',
      component: path.resolve(`src/page/items.tsx`),
    },
    {
      path: '/item/:itemId',
      matchPath: '/item/:itemId',
      component: path.resolve(`src/page/item.tsx`),
    },
];

TranslatedPages.forEach(translatedPage => {
    AppLocaleList.forEach(locale => {
      const prefix = getLocalePrefix(locale);
      const messages = AppLocalesMap[locale].messages;
      const page = {
        path: prefix + translatedPage.path,
        component: [
          path.resolve(`src/layouts/${locale}_layout.tsx`)
          translatedPage.component,
        ],
      };
      if (translatedPage.matchPath) {
        page.matchPath = prefix + translatedPage.matchPath;
      }
      createPage(page);
    });
  });

The ability to pass a component array would be convenient to create multiple "wrappers", applied in order, which all have the right imports. This would enable to optimize bundles on multiple axes and be flexible enough for multiple further usecases (we could imagine setting up an authenticated/non-authenticated wrapper automatically the same way).

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contributefor more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[tricoder42]

Hey everyone, I'm working on automatic splitting of message files in lingui/js-lingui#503. Ideally I would like to have a webpack plugin, which:

1. Generates message files for each chunk, e.g. chunk.0.[locale].js (I've already done this, because I can track what message is included in what module)
2. When chunk.0.js is about to load, request chunk.0.[locale].js at the same time. That requires runtime function, which decide what locale to load. It also requires Webpack Plugin API available in Webpack 5. @sokra from webpack pointed me to LocalizationPlugin example which uses new API.

I want also experiment with per-locale bundles. That means that instead of having code, which loads translations at runtime, each chunk is generated for every locale with translations baked in. I believe this would work great with static sites served under [locale]/ url prefix and it's actually easy to do with new version of LinguiJS.

For example, having page like this:

// src/main.js
function Page() {
  return (
    <div>
      <h1><Trans>GatsbyJS Documentation</Trans></h1>
      <p><Plural value={numSites} one="GatsbyJS is used on # site" other="GatsbyJS is used on # sites" /></p>
    </div>
  )
}

At build time, the english translations would be replaced directly in the code:

// build/main.en.js
function Page() {
  return (
    <div>
      <h1>GatsbyJS Documentation</h1>
      <p><Plural value={numSites} one="GatsbyJS is used on # site" other="GatsbyJS is used on # sites" /></p>
    </div>
  )
}

the same applies for other locales, like Czech for example:

// build/main.cs.js
function Page() {
  return (
    <div>
      <h1>Dokumentace GatsbyJS</h1>
      <p>
        <Plural 
          value={numSites}
          one="GatsbyJS používá # web"
          few="GatsbyJS používají # weby"
          other="GatsbyJS používá # webů"
        />
      </p>
    </div>
  )
}

• static messages are replaced as they are (even including variables)
• plural komponent is kept, as it need to format message based on runtime value. But it doesn't load translations from catalog, all plural rules are injected directly into code (notice there's extra few prop, which isn't in English)
• date, numbers are all formated at runtime, but the app has locale data for the single locale and there're no extra requests to server

---

That said, both solutions are waiting for next version of some libraries, but it would be great to get a feedback or help from you guys.

[danilobuerger]

@slorber did you make any progress on finding a good solution?

[slorber]

Hi @danilobuerger unfortunately I've not yet solved this problem and currently not working on this anymore.

In my understanding, it will be better to pass heavy text content directly through context (probably easier with a cms integration, as you can assign a translation directly to a specific page).

Yet I'm still interested to a solution to this problem, as having local non-cms keys still make sense in a few usecases (for translating your global site layout for example)

I've seen a blog post published recently (still in my reading list), maybe it will give some infos on this topic, or its author @ondrabus has an opinion?
https://www.gatsbyjs.org/blog/2020-02-19-how-to-build-multilingual-sites-with-gatsby/

[danilobuerger]

Alright. I am currently experimenting with i18next (but this could be any framework) and lazy json loading. It seems that those will be code splitted separately and loaded only once (not again on page nav, like the page context option would) and only the current locale (unlike the import all solution at the top). I will report back (if I don't forget) once I manage to get it into production!

[danilobuerger]

I think there a two acceptable solutions on where the translation data should end up:

• Persist only the translation strings which the current page / locale needs in the page context. This is a variation of your outlined Bad solution 2, but only persists what is really needed. This would be equivalent to how the remark plugin usually works, persisting the currently needed markdown (and only that) to the page data. This approach would make sense if you have lots of pages with different content.

• Persist each locales translation strings to a separate file (this could be n files, arbitrarily splitting the translations, for example by namespaces) and load that file along with the page data. When navigating (and keeping the locale the same) this file would not be requested again. This approach would make sense if you have lots of common translation strings.

Since we have SSR and the first page would already be localized by it, the translation file could be loaded after the page load but before navigating. This could be quicker than having a subset of translation strings in the page data. So I am going to investigate the second option.

The approach could be as follows:

1. SSR:

• Render the page in the current locale
• Collect the used translation strings
• Render the collected translation strings into the body as a script tag to make them available for hydration

2. Browser:

• Hydrate the translation strings from a special window var. This will prevent a content flash, as the browser rendering will take over but didn't load the translation file yet
• Use webpacks lazy import to request the whole translation file

3. Changing Locale:

• Use suspend to wait while loading the new translation file

[slorber]

Thanks @danilobuerger that seems a cool idea, I didn't think about it but it could work. Somehow it's the same strategy that is applied to extract the critical CSS of css-in-js libs, but instead of css, it's translation strings

[LekoArts]

cc @pieh - we wanted to maybe work on a PoC here

[ddennis]

I found https://dev.to/jospohr/proper-i18n-with-gatsby-i18next-and-sanity-io-5g8a which seems to solve most of the problems with "bad solution 1" and "bad solution 2". repo here

Does anybody have insights to why this should not be the way to go?
I would assume the sourcing from Sanity can just be replaced with whatever alternativ if needed.

any feedback would be much appreciated.

[LekoArts]

The person is still passing the data via pageContext, or did I miss something? (Which is "Bad solution 2")

[ddennis]

The person is still passing the data via pageContext, or did I miss something? (Which is "Bad solution 2")

Sorry - Yes you are right. But only the translation keys for the specific/requested page.
but is it not the solution as described by danilobuerger

Persist each locales translation strings to a separate file (this could be n files, arbitrarily splitting the translations, for example by namespaces) and load that file along with the page data. When navigating (and keeping the locale the same) this file would not be requested again. This approach would make sense if you have lots of common translation strings.

and thanks 👍

[ddennis]

@LekoArts i saw you have been working on https://github.com/LekoArts/gatsby-theme-i18n implementing examples - would it make sense to start using your solution? or should we wait for a more official release?

[LekoArts]

This is currently WIP and not released yet. We'll have some announcements soon though.

[nsavareika]

I think there a two acceptable solutions on where the translation data should end up:

• Persist only the translation strings which the current page / locale needs in the page context. This is a variation of your outlined Bad solution 2, but only persists what is really needed. This would be equivalent to how the remark plugin usually works, persisting the currently needed markdown (and only that) to the page data. This approach would make sense if you have lots of pages with different content.
• Persist each locales translation strings to a separate file (this could be n files, arbitrarily splitting the translations, for example by namespaces) and load that file along with the page data. When navigating (and keeping the locale the same) this file would not be requested again. This approach would make sense if you have lots of common translation strings.

Since we have SSR and the first page would already be localized by it, the translation file could be loaded after the page load but before navigating. This could be quicker than having a subset of translation strings in the page data. So I am going to investigate the second option.

The approach could be as follows:

1. SSR:

• Render the page in the current locale
• Collect the used translation strings
• Render the collected translation strings into the body as a script tag to make them available for hydration

1. Browser:

• Hydrate the translation strings from a special window var. This will prevent a content flash, as the browser rendering will take over but didn't load the translation file yet
• Use webpacks lazy import to request the whole translation file

1. Changing Locale:

• Use suspend to wait while loading the new translation file

Another possible solution to prevent a content flash on initial load is to perform hydration only after translation file has loaded. Gatsby browser api provides replaceHydrateFunction which allows replacing ReactDOM.hydrate/ ReactDOM.render function with the custom one. You could use it to load translation file depending on the requested locale, place translations inside window var, and only then perform the rendering. Then you won’t have to store script with translations on every generated html page. Example:

exports.replaceHydrateFunction = () => {
	return (element, container, callback) => {
		const language = getLanguageFromPath(window.location.pathname);

		import(`./src/translations/${language}.json`).then(translations => {
                      window.__translations = translations.default;
                      ReactDOM.render(element, container, callback);
                });
	};
};