Profiling slow createPages in develop/build

[zslabs]

👋 Are there any tools available to help me dig a bit deeper into why my createPages is taking so long? Without cache and about 300 pages, it's taking on average about 90 seconds, which is the majority of my bootstrap time. The other 20s or so is always around transforming source nodes. Refreshing content after changes to .md files is also quite slow.

I noticed a decrease when removing the gatsbyRemarkPlugins, but blindly removing and rebuilding is a pretty time-consuming task. If there was a breakdown of how long each plugin was adding to the boot process, similar to Atom and VSCode, that'd be immensely helpful.

Here's my gatsby.config.esm:

export default {
  siteMetadata: {
    title: 'Site',
    description: "...",
    siteUrl: 'https://www.domain.com',
    twitter: '@twitter',
  },
  // for avoiding CORS while developing Netlify Functions locally
  // read more: https://www.gatsbyjs.org/docs/api-proxy/#advanced-proxying
  developMiddleware: (app) => {
    app.use(
      '/.netlify/functions/',
      createProxyMiddleware({
        target: 'http://localhost:9000',
        pathRewrite: {
          '/.netlify/functions/': '',
        },
      })
    )
  },
  plugins: [
    {
      resolve: 'gatsby-plugin-compile-es6-packages',
      options: {
        modules: ['chaoskit'],
      },
    },
    {
      resolve: 'gatsby-plugin-mdx-frontmatter',
    },
    {
      resolve: 'gatsby-plugin-launchdarkly',
      options: {
        clientSideID: '...',
        options: {
          bootstrap: 'localstorage', // Caches flag values in localstorage
        },
      },
    },
    {
      resolve: 'gatsby-plugin-emotion',
      options: {
        sourceMap: false,
      },
    },
    {
      resolve: 'gatsby-plugin-react-helmet',
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'data',
        path: path.join(__dirname, '/src/data'),
      },
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'people',
        path: path.join(__dirname, '/src/meta/people'),
      },
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'media',
        path: path.join(__dirname, '/src/assets/media'),
      },
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'posts',
        path: path.join(__dirname, '/src/posts'),
      },
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'tutorials',
        path: path.join(__dirname, '/src/tutorials'),
      },
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'caseStudies',
        path: path.join(__dirname, '/src/case-studies'),
      },
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'staticPages',
        path: path.join(__dirname, '/src/static-pages'),
      },
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'press-media',
        path: path.join(__dirname, '/src/media'),
      },
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'whitepaper-landing-pages',
        path: path.join(__dirname, '/src/whitepaper-landing-pages'),
      },
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'sign-up-landing-pages',
        path: path.join(__dirname, '/src/sign-up-landing-pages'),
      },
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'demo-column-variation-landing-pages',
        path: path.join(__dirname, '/src/demo-column-variation-landing-pages'),
      },
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'webinars',
        path: path.join(__dirname, '/src/webinars'),
      },
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'events',
        path: path.join(__dirname, '/src/events'),
      },
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'article-ctas',
        path: path.join(__dirname, '/src/article-ctas'),
      },
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'docArticles',
        path: path.join(__dirname, '/src/docs/articles'),
      },
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'docSections',
        path: path.join(__dirname, '/src/docs/sections'),
      },
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'mdxDocs',
        path: path.join(__dirname, '/src/mdx-docs'),
      },
    },
    {
      resolve: 'gatsby-plugin-mdx',
      options: {
        extensions: ['.md', '.mdx'],
        gatsbyRemarkPlugins: [
          {
            resolve: 'gatsby-remark-smartypants',
          },
          {
            resolve: 'gatsby-remark-custom-blocks',
            options: {
              blocks: {
                alertPrimary: {
                  classes: 'Alert--primary',
                  title: 'optional',
                },
                alertDanger: {
                  classes: 'Alert--danger',
                  title: 'optional',
                },
                alertWarning: {
                  classes: 'Alert--warning',
                  title: 'optional',
                },
              },
            },
          },
          {
            resolve: 'gatsby-remark-responsive-iframe',
          },
        ],
      },
    },
    {
      resolve: 'gatsby-plugin-mdx-embed',
    },
    {
      resolve: 'gatsby-plugin-google-tagmanager',
      options: {
        id: '...',
      },
    },
    {
      resolve: 'gatsby-plugin-segment-js',
      options: {
        prodKey: process.env.SEGMENT_KEY,
      },
    },
    /*
    {
      resolve: 'gatsby-plugin-webpack-bundle-analyser-v2',
    },
    */
    {
      resolve: 'gatsby-plugin-feed',
      options: {
        feeds: [
          {
            serialize: ({ query: { site, allMdx } }) =>
              allMdx.edges.map((edge) =>
                Object.assign({}, edge.node.frontmatter, {
                  description: edge.node.excerpt,
                  date: edge.node.frontmatter.date,
                  url: `${site.siteMetadata.siteUrl}/${edge.node.fields.fullUrl}`,
                  guid: `${site.siteMetadata.siteUrl}/${edge.node.fields.fullUrl}`,
                })
              ),
            query: `
              {
                allMdx(
                  limit: 50,
                  sort: { fields: [frontmatter___date], order: DESC }
                  filter: {
                    fileAbsolutePath: { regex: "/src/posts/" }
                    frontmatter: { title: { ne: "BLUEPRINT" } }
                  }
                ) {
                  edges {
                    node {
                      excerpt(pruneLength: 120)
                      frontmatter {
                        title
                        date
                      }
                      fields {
                        fullUrl
                      }
                    }
                  }
                }
              }
            `,
            title: 'RSS Feed',
            output: '/rss.xml',
          },
        ],
      },
    },
    {
      resolve: 'gatsby-plugin-manifest',
      options: {
        name: 'Domain',
        short_name: 'Domain',
        start_url: '/',
        background_color: '#fff',
        theme_color: '#000',
        display: 'minimal-ui',
        icon: './src/assets/media/favicon-base.png',
      },
    },
    {
      resolve: 'gatsby-plugin-sitemap',
      options: {
        query: `
        {
          site {
            siteMetadata {
              siteUrl
            }
          }

          allSitePage {
            edges {
              node {
                path
              }
            }
          }
        }`,
      },
    },
    {
      resolve: 'gatsby-plugin-react-helmet-canonical-urls',
      options: {
        siteUrl: 'https://www.domain.com',
      },
    },
    {
      resolve: 'gatsby-plugin-netlify-cms',
      options: {
        modulePath: `${__dirname}/src/cms/cms.js`,
        manualInit: true,
      },
    },
    {
      resolve: 'gatsby-plugin-netlify', // Make sure to keep it last in the array
    },
  ],
  mapping: {
    // Team
    'Mdx.frontmatter.team.employees.employee': 'Mdx.frontmatter.title',
    // Leadership VPs
    'Mdx.frontmatter.vps.employee': 'Mdx.frontmatter.title',
    // Community
    'Mdx.frontmatter.expert.tutorials.tutorial': 'Mdx.frontmatter.title',
    // Press featured items
    'Mdx.frontmatter.featuredPress.items.link': 'Mdx.frontmatter.link',
    // Static featured articles
    'Mdx.frontmatter.articles.title': 'Mdx.frontmatter.title',
    // Industries case studies
    'Mdx.frontmatter.industryCaseStudies.title': 'Mdx.frontmatter.title',
    // Article CTAs
    'Mdx.frontmatter.defaultBlogCTA': 'Mdx.frontmatter.title__articleCTA',
    'Mdx.frontmatter.defaultTutorialCTA': 'Mdx.frontmatter.title__articleCTA',
    'Mdx.frontmatter.articleCTA': 'Mdx.frontmatter.title__articleCTA',
  },
}

[KyleAMathews]

That'd be a nice feature — tracking execution time / plugin.

Which plugins do implement createPages? I assume your gatsby-node.js file does? You can add console.time there to track things + in other plugins that implement createPages

[zslabs]

👋 @KyleAMathews Appreciate the response and thanks for the recommendation on console.time(). I had previously added in checks around each query (example one from blog content below) and it was coming in around 30 seconds, which definitely seems odd:

{
  allMdx(
    sort: { fields: [frontmatter___date], order: DESC }
    filter: {
      fileAbsolutePath: { regex: "/src/posts/" }
      frontmatter: { title: { ne: "BLUEPRINT" } }
    }
  ) {
    edges {
      node {
        fileAbsolutePath
        excerpt(pruneLength: 120)
        timeToRead
        rawBody
        body
        frontmatter {
          title
          description
          canonical
          date
          dateModified
          slug
          featuredImage
          socialMediaImage
          categories
          author
          hideFromResources
          topics
        }
        tableOfContents(maxDepth: 3)
        fields {
          slug
          fullUrl
          categoryUrls
          authorUrl
          featuredImage
        }
      }
    }
  }
}

I need the rawBody in order to create content chunks for Algolia (I handle this in a onPostBuild hook), but that in itself seems fairly light-weight. My guess is it's something with gatsby-plugin-mdx since my previous "deleting plugins and rebuilding" did show a slight decrease in time spent building.

[KyleAMathews]

Ah that could be it. MDX is... slow — as it has to do babel transforms + bundling etc.

[pvdz]

Yeah it's probably MDX. 900 pages should be super fast to source, but MDX does like a double babel parse at bootstrap time which can explain this.

There's no easy way to get this info unfortunately. You can hook into the tracing (https://www.gatsbyjs.com/docs/performance-tracing/) but it's very noisy.

I tend to use something like timeSum (https://gist.github.com/pvdz/793f89bca3b27501479488b4abbb8972) in serial mode to figure out where time is spent.

Can you verify what happens when you (as a test) switch to regular markdown? Just roughly, you don't need to a full port.

[zslabs]

Thanks both! Does the initial boot-time (and slow reloads) lend itself at all to lazily querying pages in development as they are needed? I believe I recall seeing an issue around that and figured aside from this specific usecase, boot-times could be improved quite a bit if what Gatsby was listening for was on-demand.

FWIW with this, I removed the 3 remark plugins I had and saw the createPages time go down by about 25 seconds and source and transform nodes decrease by 4 seconds. Reloading still appears to query EVERYTHING again though.