Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(core): custom caching docs #27953

Open
wants to merge 4 commits into
base: docs/powerpack
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 29 additions & 13 deletions docs/generated/manifests/menus.json
Original file line number Diff line number Diff line change
Expand Up @@ -325,29 +325,29 @@
},
{
"name": "Powerpack Features",
"path": "/features/powerpack-features",
"id": "powerpack-features",
"path": "/features/powerpack",
"id": "powerpack",
"isExternal": false,
"children": [
{
"name": "Run Language-Agnostic Conformance Rules",
"path": "/features/powerpack-features/conformance",
"path": "/features/powerpack/conformance",
"id": "conformance",
"isExternal": false,
"children": [],
"disableCollapsible": false
},
{
"name": "Define Code Ownership at the Project Level",
"path": "/features/powerpack-features/owners",
"path": "/features/powerpack/owners",
"id": "owners",
"isExternal": false,
"children": [],
"disableCollapsible": false
},
{
"name": "Change the Remote Cache Storage Location",
"path": "/features/powerpack-features/custom-caching",
"path": "/features/powerpack/custom-caching",
"id": "custom-caching",
"isExternal": false,
"children": [],
Expand Down Expand Up @@ -474,29 +474,29 @@
},
{
"name": "Powerpack Features",
"path": "/features/powerpack-features",
"id": "powerpack-features",
"path": "/features/powerpack",
"id": "powerpack",
"isExternal": false,
"children": [
{
"name": "Run Language-Agnostic Conformance Rules",
"path": "/features/powerpack-features/conformance",
"path": "/features/powerpack/conformance",
"id": "conformance",
"isExternal": false,
"children": [],
"disableCollapsible": false
},
{
"name": "Define Code Ownership at the Project Level",
"path": "/features/powerpack-features/owners",
"path": "/features/powerpack/owners",
"id": "owners",
"isExternal": false,
"children": [],
"disableCollapsible": false
},
{
"name": "Change the Remote Cache Storage Location",
"path": "/features/powerpack-features/custom-caching",
"path": "/features/powerpack/custom-caching",
"id": "custom-caching",
"isExternal": false,
"children": [],
Expand All @@ -507,23 +507,23 @@
},
{
"name": "Run Language-Agnostic Conformance Rules",
"path": "/features/powerpack-features/conformance",
"path": "/features/powerpack/conformance",
"id": "conformance",
"isExternal": false,
"children": [],
"disableCollapsible": false
},
{
"name": "Define Code Ownership at the Project Level",
"path": "/features/powerpack-features/owners",
"path": "/features/powerpack/owners",
"id": "owners",
"isExternal": false,
"children": [],
"disableCollapsible": false
},
{
"name": "Change the Remote Cache Storage Location",
"path": "/features/powerpack-features/custom-caching",
"path": "/features/powerpack/custom-caching",
"id": "custom-caching",
"isExternal": false,
"children": [],
Expand Down Expand Up @@ -4876,6 +4876,14 @@
"children": [],
"disableCollapsible": false
},
{
"name": "Custom Task Runners",
"path": "/deprecated/custom-task-runners",
"id": "custom-task-runners",
"isExternal": false,
"children": [],
"disableCollapsible": false
},
{
"name": "Workspace Executors",
"path": "/deprecated/workspace-executors",
Expand Down Expand Up @@ -4975,6 +4983,14 @@
"children": [],
"disableCollapsible": false
},
{
"name": "Custom Task Runners",
"path": "/deprecated/custom-task-runners",
"id": "custom-task-runners",
"isExternal": false,
"children": [],
"disableCollapsible": false
},
{
"name": "Workspace Executors",
"path": "/deprecated/workspace-executors",
Expand Down
56 changes: 39 additions & 17 deletions docs/generated/manifests/nx.json
Original file line number Diff line number Diff line change
Expand Up @@ -439,7 +439,7 @@
"tags": ["nx-release"]
},
{
"id": "powerpack-features",
"id": "powerpack",
"name": "Powerpack Features",
"description": "Features of Nx that are available with a powerpack license.",
"mediaImage": "",
Expand All @@ -453,7 +453,7 @@
"file": "shared/features/powerpack/conformance",
"itemList": [],
"isExternal": false,
"path": "/features/powerpack-features/conformance",
"path": "/features/powerpack/conformance",
"tags": ["conformance"]
},
{
Expand All @@ -464,7 +464,7 @@
"file": "shared/features/powerpack/owners",
"itemList": [],
"isExternal": false,
"path": "/features/powerpack-features/owners",
"path": "/features/powerpack/owners",
"tags": ["owners"]
},
{
Expand All @@ -475,12 +475,12 @@
"file": "shared/features/powerpack/custom-caching",
"itemList": [],
"isExternal": false,
"path": "/features/powerpack-features/custom-caching",
"path": "/features/powerpack/custom-caching",
"tags": ["custom-caching"]
}
],
"isExternal": false,
"path": "/features/powerpack-features",
"path": "/features/powerpack",
"tags": []
},
{
Expand Down Expand Up @@ -643,8 +643,8 @@
"path": "/features/manage-releases",
"tags": ["nx-release"]
},
"/features/powerpack-features": {
"id": "powerpack-features",
"/features/powerpack": {
"id": "powerpack",
"name": "Powerpack Features",
"description": "Features of Nx that are available with a powerpack license.",
"mediaImage": "",
Expand All @@ -658,7 +658,7 @@
"file": "shared/features/powerpack/conformance",
"itemList": [],
"isExternal": false,
"path": "/features/powerpack-features/conformance",
"path": "/features/powerpack/conformance",
"tags": ["conformance"]
},
{
Expand All @@ -669,7 +669,7 @@
"file": "shared/features/powerpack/owners",
"itemList": [],
"isExternal": false,
"path": "/features/powerpack-features/owners",
"path": "/features/powerpack/owners",
"tags": ["owners"]
},
{
Expand All @@ -680,45 +680,45 @@
"file": "shared/features/powerpack/custom-caching",
"itemList": [],
"isExternal": false,
"path": "/features/powerpack-features/custom-caching",
"path": "/features/powerpack/custom-caching",
"tags": ["custom-caching"]
}
],
"isExternal": false,
"path": "/features/powerpack-features",
"path": "/features/powerpack",
"tags": []
},
"/features/powerpack-features/conformance": {
"/features/powerpack/conformance": {
"id": "conformance",
"name": "Run Language-Agnostic Conformance Rules",
"description": "Conformance",
"mediaImage": "",
"file": "shared/features/powerpack/conformance",
"itemList": [],
"isExternal": false,
"path": "/features/powerpack-features/conformance",
"path": "/features/powerpack/conformance",
"tags": ["conformance"]
},
"/features/powerpack-features/owners": {
"/features/powerpack/owners": {
"id": "owners",
"name": "Define Code Ownership at the Project Level",
"description": "Owners",
"mediaImage": "",
"file": "shared/features/powerpack/owners",
"itemList": [],
"isExternal": false,
"path": "/features/powerpack-features/owners",
"path": "/features/powerpack/owners",
"tags": ["owners"]
},
"/features/powerpack-features/custom-caching": {
"/features/powerpack/custom-caching": {
"id": "custom-caching",
"name": "Change the Remote Cache Storage Location",
"description": "Custom Caching",
"mediaImage": "",
"file": "shared/features/powerpack/custom-caching",
"itemList": [],
"isExternal": false,
"path": "/features/powerpack-features/custom-caching",
"path": "/features/powerpack/custom-caching",
"tags": ["custom-caching"]
},
"/features/ci-features": {
Expand Down Expand Up @@ -6677,6 +6677,17 @@
"path": "/deprecated/workspace-generators",
"tags": []
},
{
"id": "custom-task-runners",
"name": "Custom Task Runners",
"description": "",
"mediaImage": "",
"file": "shared/deprecated/custom-task-runners",
"itemList": [],
"isExternal": false,
"path": "/deprecated/custom-task-runners",
"tags": []
},
{
"id": "workspace-executors",
"name": "Workspace Executors",
Expand Down Expand Up @@ -6814,6 +6825,17 @@
"path": "/deprecated/workspace-generators",
"tags": []
},
"/deprecated/custom-task-runners": {
"id": "custom-task-runners",
"name": "Custom Task Runners",
"description": "",
"mediaImage": "",
"file": "shared/deprecated/custom-task-runners",
"itemList": [],
"isExternal": false,
"path": "/deprecated/custom-task-runners",
"tags": []
},
"/deprecated/workspace-executors": {
"id": "workspace-executors",
"name": "Workspace Executors",
Expand Down
6 changes: 3 additions & 3 deletions docs/generated/manifests/tags.json
Original file line number Diff line number Diff line change
Expand Up @@ -581,7 +581,7 @@
"file": "shared/features/powerpack/conformance",
"id": "conformance",
"name": "Run Language-Agnostic Conformance Rules",
"path": "/features/powerpack-features/conformance"
"path": "/features/powerpack/conformance"
}
],
"owners": [
Expand All @@ -590,7 +590,7 @@
"file": "shared/features/powerpack/owners",
"id": "owners",
"name": "Define Code Ownership at the Project Level",
"path": "/features/powerpack-features/owners"
"path": "/features/powerpack/owners"
}
],
"custom-caching": [
Expand All @@ -599,7 +599,7 @@
"file": "shared/features/powerpack/custom-caching",
"id": "custom-caching",
"name": "Change the Remote Cache Storage Location",
"path": "/features/powerpack-features/custom-caching"
"path": "/features/powerpack/custom-caching"
}
],
"intro": [
Expand Down
7 changes: 6 additions & 1 deletion docs/map.json
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@
},
{
"name": "Powerpack Features",
"id": "powerpack-features",
"id": "powerpack",
"description": "Features of Nx that are available with a powerpack license.",
"itemList": [
{
Expand Down Expand Up @@ -1398,6 +1398,11 @@
"id": "workspace-generators",
"file": "shared/deprecated/workspace-generators"
},
{
"name": "Custom Task Runners",
"id": "custom-task-runners",
"file": "shared/deprecated/custom-task-runners"
},
{
"name": "Workspace Executors",
"id": "workspace-executors",
Expand Down
12 changes: 12 additions & 0 deletions docs/shared/deprecated/custom-task-runners.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# tasksRunnerOptions and cacheDirectory

As of Nx 20, custom task runners (defined via `tasksRunnerOptions`) and the `cacheDirectory` property in `nx.json` are deprecated. They will cease to function in Nx 21. In Nx 20, the local cache is stored in a database, rather than using the file system. This has two benefits:

1. Cache reads and writes are faster.
2. The local cache is more secure since other processes with access to the file system can no longer read or modify the cache.

For most organizations, this feature is a net positive. If you are currently using a custom task runner, you are mostly likely using it to define your own custom [remote cache](/ci/features/remote-cache) storage location. You have several options moving forward:

1. Use [Nx Cloud](/nx-cloud) for your remote cache
2. Use an [Nx Powerpack]() plugin to store your remote cache on an [AWS S3 bucket](/nx-api/powerpack-s3-cache) or a [network drive]()
3. Use the deprecated custom task runner feature until Nx 21
32 changes: 32 additions & 0 deletions docs/shared/features/powerpack/custom-caching.md
Original file line number Diff line number Diff line change
@@ -1 +1,33 @@
# Change the Remote Cache Storage Location

The recommended way to enable the [remote cache]() is to use [Nx Replay]() and have Nx Cloud share the task cache across your organization. For those organizations that are unable to use Nx Cloud, Nx offers official plugins that are enabled by [Nx Powerpack]() to customize your remote cache location in a fast and secure manner.

In Nx 20, the local cache mechanism is getting faster and more secure by storing the cache in a database instead of directly on the file system. The new cache is faster because it avoids the unavoidably slow speed of interacting with the file system. The new cache is more secure because it eliminates the possibility that other processes with file system access could read or modify the cache. The [`@nx/powerpack-s3-cache`](/nx-api/powerpack-s3-cache) and [`@nx/powerpack-shared-fs-cache`]() plugins enable you to leverage that speed and security with a custom remote cache storage location.

{% callout type="note" title="Custom Task Runners are Deprecated in Nx 20" %}
If you have a custom coded method for changing the remote cache location that uses either `tasksRunnerOptions` or `cacheDirectory`, you'll need to choose a migration path. Those properties are [deprecated in Nx 20](/deprecated/custom-task-runners) and will be removed in Nx 21.
{% /callout %}

## Custom Remote Cache Storage Plugins Require Nx Powerpack

The `@nx/powerpack-s3-cache` and `@nx/powerpack-shared-fs-cache` plugins require an Nx Powerpack license to function. [Activating Powerpack](/recipes/installation/activate-powerpack) is a simple process.

{% call-to-action title="Buy a Powerpack License" icon="nx" description="Unlock all the features of Nx" url="https://nx.app/nx-powerpack/purchase" /%}

## Choose Your Storage Solution

Read the individual plugin documentation for specific set up instructions.

{% cards cols="2" lgCols="2" mdCols="2" smCols="2" %}

{% link-card title="AWS S3 Bucket Remote Cache" type="Nx Plugin" url="/nx-api/powerpack-s3-cache" icon="aws" /%}

{% link-card title="Shared Network Drive Remote Cache" type="Nx Plugin" url="/nx-api/powerpack-s3-cache" icon="network" /%}

{% /cards %}

## Switch to Nx Cloud

These custom remote cache storage solutions only provide the remote cache functionality of Nx Cloud. If you want to leverage [distributed task execution](), [re-running flaky tasks]() or [automatically splitting tasks](), you'll need to [connect to Nx Cloud]() and use Nx Cloud's remote cache solution instead.

{% call-to-action title="Connect to Nx Cloud" icon="nxcloud" description="Enable task distribution and Atomizer" url="/ci/intro/connect-to-nx-cloud" /%}
Loading
Loading