creating a shiki wrapper for nextjs, and rsc

[saadi925]

Clear and concise description of the problem

The Shiki is Vue and Nuxt first , it also works fine with the next js , but there are a few problems with it when using in nextjs. it just does not support it the way it works with vue. and we have too import different functions and each have their own set of rules.

Suggested solution

Therefore , I am working on a shiki nextjs, react , rsc wrapper , i am following the best practices, using type guards, and i want to know what suggestions community will give. i will also write the docs (in nextjs with turborepo,fuma),
i am following a pattern of single config. you only have to create a function generateShiki.
actions have the actions like 'codeToHtml' | 'codeToHast' | 'codeToTokens' | 'codeToTokensBase' | 'codeToTokensWithThemes';
etc
there are some base options , in each action we can overide the default options , and each action have it's appropriate options.
to use a theme in the base options we have to load it in the custom highlight options,
actions is an array on action , if we remove an action and invoke their function will be undefined.

import generateShiki, { RenderCode } from '@repo/ui/shiki/react'
import oneDark from 'shiki/themes/one-dark-pro.mjs';
import oneLight from 'shiki/themes/one-light.mjs';
export default async function Page() {
  const result = await generateShiki({
    code: ` const x = 10
console.log(x)`
    , baseOptions: {
      lang: 'javascript',
      theme: "one-dark-pro",
    },
    customHighlighterOptions: {
      themes: [oneDark, oneLight],
    },

    actions: [
      {
        action: 'codeToHtml',
        options : {
          decorations : []
        }
      },
      {
        action: 'codeToTokensWithThemes',
        options: {
          themes: {
            light: "one-light",
            dark: "one-dark-pro"
          }
        }
      }

    ],
  });
  const html = await result.renderCode?.();
  const hast = await result.getHast?.();
  const tokensBase = await result.getTokensBase?.();
  const tokensFull = await result.getTokens?.();
  const tokens_theme = await result.getTokensWithThemes?.();
// the function will be undefined , whose action is not listed in the actions array , using only what it needs 
  return (
    <main>
      <RenderCode className="" code={html} />

    </main>
  );
}

we do not have to write that much to get started , it is an example so i did it intentionally to explain the structure.
we can start with just writting

import generateShiki, { disposeHighlighter, RenderCode } from '@repo/ui/shiki/react'
export default async function Page() {
 const result = await generateShiki({
   code: `console.log('Hello, world!')`
   , baseOptions: {
     lang: 'javascript',
     theme: "nord",
   },
   actions: [
     {
       action: 'codeToHtml',
     },
   ],
 });
 const html = result.renderCode?.();
 disposeHighlighter()
 return (
   <main>
     <RenderCode code={html || ""} />
   </main>
 );
}

Validations

• Follow our Code of Conduct
• Read the Contributing Guide.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

Contributes

• If this feature request is accepted, I am willing to submit a PR to fix this issue

[repraze]

I've been experimenting with shiki and next.js with app router for the last few days. So far I have been unable to get SSR working for dynamic pages. Only static build or client side highlighting works.

When creating a highlighter at runtime, it fails to dynamically import the languages. (From other issues, it seems next.js might not support the kind of import shiki is doing) I've tried to bypass that by using the core highlighter and playing with the imports, but no luck so far.

Would this wrapper/enhancement help with that use case? Otherwise next.js seems to work for me on client side / static site generation.

[saadi925]

yeah

I've been experimenting with shiki and next.js with app router for the last few days. So far I have been unable to get SSR working for dynamic pages. Only static build or client side highlighting works.

When creating a highlighter at runtime, it fails to dynamically import the languages. (From other issues, it seems next.js might not support the kind of import shiki is doing) I've tried to bypass that by using the core highlighter and playing with the imports, but no luck so far.

Would this wrapper/enhancement help with that use case? Otherwise next.js seems to work for me on client side / static site generation.

yeah i have been working on it , it could work even now with ssr for dynamic pages .

i have created this wrapper , but i am still working on it ,
this will be an optimized single function . which could be used for csr, static site and ssr ,

/**
 * Base options for code highlighting
 */
export interface BaseOptions {
  lang: sh.BundledLanguage;
  theme: sh.BundledTheme;
}

/**
 * Action options for different code highlighting methods
 */
// Union of all action types
export type ShikiAction =
  | CodeToHtmlAction
  | CodeToHastAction
  | CodeToTokensAction
  | CodeToTokensBaseAction
  | CodeToTokensWithThemesAction;


/**
 * Props for the generateShiki function
 */
export interface ShikiConfig {
  baseOptions: BaseOptions;
  actions: ShikiAction[];
  customHighlighterOptions?: HighlighterCoreOptions;
}

/**
 * Result type for the generateShiki function
 */

// Result Types
export type ShikiResult = {
  codeToHtml?: (code: string, options?: CodeToHtmlAction["options"]) => string
  getTokensBase?: (code: string, options?: CodeToTokensBaseAction["options"]) => Promise<sh.ThemedToken[][]>
  getTokens?: (code: string, options?: CodeToTokensAction["options"]) => Promise<sh.TokensResult>
  getHast?: (code: string, options?: CodeToHastAction["options"]) => Promise<any>
  getTokensWithThemes?: (code: string, options?: CodeToTokensWithThemesAction["options"]) => Promise<sh.ThemedTokenWithVariants[][]>;
}

Example

i am also working on dynamic languages import without loosing optimizations
this is a server component
the only loaded language is javascript and we are loading tsx language dynamically without loading the full bundle . it's just loading that particular language

import generateShiki, { disposeHighlighter, RenderCode, ShikiConfig } from '@repo/ui/shiki/react'
import minDark from "shiki/themes/min-dark.mjs"
import oneDark from "shiki/themes/one-dark-pro.mjs"
export default async function Page() {

const config : ShikiConfig  ={
  baseOptions: {
    lang: 'javascript',
    theme:"min-dark"
  },
  customHighlighterOptions  : {
    themes : [minDark, oneDark]
  },
  actions : [
    {
      "action": "codeToHtml",
    }
  ]
}
  const hl = await generateShiki(config);
// example code
 const code1 =   `
import React from 'react'
import { ThemeProvider } from 'next-themes'
`
const c =  await hl.codeToHtml?.(code1, {
  "lang" :"tsx",
})

if a language is not found it moves back to the default one , instead of error
  disposeHighlighter()
  return (
    <main >
      <div 
      dangerouslySetInnerHTML={{
        __html : c || ""
      }}
      className={`
      py-2 mx-12 mt-2 px-4 
      rounded-md overflow-x-auto 
      bg-[#1f1f1f]
      `} />
    </main>
  );
}

also see this article must read last lines , final thoughts
shiki_nextjs

[repraze]

Sounds promising, but could be hard to maintain the mapping on top of shiki.

I found that importing from @shikijs/core helps with next js runtime since shiki bundling is the issue on the server. The only thing left on my side would be loading the langs/themes around dynamic imports.

import {createHighlighterCore} from "@shikijs/core";
import getWasm from "@shikijs/core/wasm-inlined";

async function getHighlighter (){
    const highlighter = await createHighlighterCore({loadWasm: getWasm});

    highlighter.loadLanguage(/* can't use import("shiki/langs/javascript.mjs") */);

    return highlighter;
}

[saadi925]

Sounds promising, but could be hard to maintain the mapping on top of shiki.

I found that importing from @shikijs/core helps with next js runtime since shiki bundling is the issue on the server. The only thing left on my side would be loading the langs/themes around dynamic imports.

import {createHighlighterCore} from "@shikijs/core";
import getWasm from "@shikijs/core/wasm-inlined";

async function getHighlighter (){
    const highlighter = await createHighlighterCore({loadWasm: getWasm});

    highlighter.loadLanguage(/* can't use import("shiki/langs/javascript.mjs") */);

    return highlighter;
}

i am not doing any mapping on it , the mappings are already in the shiki package , the are using it
it is very great,
they have the mappings in the langs of all the imports i am importing it dynamically on server side (demand) .
the above article link states that it is not possible and easy but it is . i am using it , it's working fine.

[fuma-nama]

Cannot quite understand the need of this feature, it's working great on the latest Next.js 14 release.
I believe Next.js has fixed the issues with Shiki even if there was an issue on older versions.

Note that Next.js added shiki to serverExternalPackages by default

[asmyshlyaev177]

That approach works locally with NextJS14 Server components, but not on Vercel.

import {
  transformerNotationHighlight,
  transformerNotationWordHighlight,
} from '@shikijs/transformers';
import { type HighlighterCore } from 'shiki';
import { createHighlighterCore } from 'shiki/core';
import tsLang from 'shiki/langs/typescript.mjs';
import githubTheme from 'shiki/themes/github-dark.mjs';
import getWasm from 'shiki/wasm';

export const createHighlighter = async () =>
  await createHighlighterCore({
    themes: [githubTheme],
    langs: [tsLang],
    loadWasm: getWasm,
  });

export let highlighter: HighlighterCore

export const highlight = async (content: string) => {
  if (!highlighter) {
    highlighter = await createHighlighter()
  }
  return highlighter?.codeToHtml?.(content, {
    lang: 'typescript',
    theme: 'github-dark',
    transformers: [
      transformerNotationHighlight(),
      transformerNotationWordHighlight(),
    ],
  }) || '';
}

and use like this

import { highlight } from '../highlighter';

export const File = async ({
  content,
}: {
  content: string;
}) => {

  const text = await highlight(content);

  return (
      <code>
        <pre
          dangerouslySetInnerHTML={{ __html: text }}
          className=" dark:bg-gray-800 p-4"
        ></pre>
      </code>
  );
};

@fuma-nama need to use "Fine-grained bundle", with edge function bundle size is very limited, from docs couldn't find a proper way to do it with Next.js.

[fuma-nama]

Production build works as expected: https://shiki-test-eight.vercel.app
It uses revalidate = 0 which forced the page into dynamic rendered.

edge function bundle size

You're not supposed to use Shiki in edge runtime, it uses Node.js APIs. And honestly, you have no reasons to use Shiki under edge environments like Middleware, for normal pages, serverless is absolutely enough

[asmyshlyaev177]

Production build works as expected: https://shiki-test-eight.vercel.app It uses revalidate = 0 which forced the page into dynamic rendered.

edge function bundle size

You're not supposed to use Shiki in edge runtime, it uses Node.js APIs. And honestly, you have no reasons to use Shiki under edge environments like Middleware, for normal pages, serverless is absolutely enough

Ok, better to add more exhaustive docs for Next.js, still source of confusion.

[fuma-nama]

Hmm I see, maybe open a separate issue? Would be nice to have a Next.js guide under the integrations section.

At least from what I see, the original issue of OP has been solved on newer versions of Next.js.

[Marcus-zh]

I've been experimenting with shiki and next.js with app router for the last few days. So far I have been unable to get SSR working for dynamic pages. Only static build or client side highlighting works.在过去的几天里，我一直在尝试 shiki 和 next.js 与 app router 的尝试。到目前为止，我一直无法让 SSR 适用于动态页面。只有静态构建或客户端高亮显示有效。

When creating a highlighter at runtime, it fails to dynamically import the languages. (From other issues, it seems next.js might not support the kind of import shiki is doing) I've tried to bypass that by using the core highlighter and playing with the imports, but no luck so far.在运行时创建荧光笔时，它无法动态导入语言。（从其他问题来看，似乎 next.js 可能不支持 Shiki 正在进行的那种导入）我试图通过使用核心荧光笔和玩导入来绕过这个问题，但到目前为止还没有运气。

Would this wrapper/enhancement help with that use case? Otherwise next.js seems to work for me on client side / static site generation.此包装器/增强功能是否有助于该用例？否则next.js似乎对我在客户端/静态站点生成有用。

I think you can see this
The issue is #658 and code is this

[antfu]

For Shiki itself, we don't have a plan to maintain wrappers for each specific metaframeworks. We'll leave that to community solutions. Marking this as not planned to make it clear. Thanks for bringing it up.