Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to prime URL metrics #1850

Open
wants to merge 52 commits into
base: trunk
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
57f9958
Add settings page for Optimization Detective
b1ink0 Jan 23, 2025
71cd706
Add initial admin script and UI elements for URL priming
b1ink0 Jan 27, 2025
da1d275
Implement batch URL priming functionality with REST API endpoint
b1ink0 Jan 27, 2025
bfed250
Include breakpoints in URL processing
b1ink0 Jan 27, 2025
6ec82e3
Add debug mode and better dynamic height generation for breakpoints
b1ink0 Jan 28, 2025
af0172e
Add verification token auth for protecting REST API, Add message pass…
b1ink0 Jan 29, 2025
bff9973
Refactor to improve control flow and add support for pause and resume…
b1ink0 Jan 29, 2025
c155fc1
Move batch URL fetching logic to helper function
b1ink0 Jan 30, 2025
7640ef7
Add support to filter URL whose metrics are already populated
b1ink0 Jan 30, 2025
f05f6c0
Add support to filter breakpoints whose metrics are already populated
b1ink0 Jan 31, 2025
0631daf
Refactor by moving filtering batch URls and computing standard brekpo…
b1ink0 Jan 31, 2025
5d93fdc
Fix sending empty batch URLs caused by filtering the batch URLs
b1ink0 Jan 31, 2025
3118fb3
Save last batch cursor in options table to improve performance
b1ink0 Jan 31, 2025
d8cfa02
Improve verification token handling by using a transient check before…
b1ink0 Feb 3, 2025
8879f60
Add beforeunload event listener to prevent page navigation during pro…
b1ink0 Feb 3, 2025
72b18fd
Add new REST API routes for URL metrics breakpoints and verification …
b1ink0 Feb 3, 2025
f5bba48
Add block editor support for priming URL metrics
b1ink0 Feb 3, 2025
b0422eb
Merge branch 'trunk' into add/url-metrics-priming
b1ink0 Feb 4, 2025
db1fd81
Add missing since and access annotations, Add new JavaScripts to webp…
b1ink0 Feb 4, 2025
63564c4
Remove usage of fetch and Fix URL metrics not priming due to usage of…
b1ink0 Feb 4, 2025
4fd36b6
Merge branch 'trunk' into add/url-metrics-priming
b1ink0 Feb 4, 2025
82a16b5
Fix detect.js getting stuck on 'onLCP' due to IFRAME being offscreen …
b1ink0 Feb 5, 2025
a50f23e
Improve responsiveness of IFRAME in debug mode
b1ink0 Feb 5, 2025
9201c86
Merge branch 'WordPress:trunk' into add/url-metrics-priming
b1ink0 Feb 6, 2025
163d24e
Skip check of already submitted URL metrics
b1ink0 Feb 7, 2025
57d77ea
Merge branch 'trunk' into add/url-metrics-priming
b1ink0 Feb 7, 2025
2bc9f49
Fix `IFRAME` positioning
b1ink0 Feb 7, 2025
d17dede
Add URL priming functionality for classic editor
b1ink0 Feb 7, 2025
97b888f
Refactor to prime URL metrics only when the post is saved
b1ink0 Feb 10, 2025
931861b
Move URL metrics success message to the end of the detection process
b1ink0 Feb 10, 2025
0a7c1f1
Add instructions and important information section to settings page, …
b1ink0 Feb 11, 2025
969f6b5
Pause URL metrics processing when tab visibility chages, Add abort fu…
b1ink0 Feb 11, 2025
c1e245d
Implement task processing with abort functionality and visibility han…
b1ink0 Feb 12, 2025
b7eeef7
Add settings link to plugin action links
b1ink0 Feb 13, 2025
41f5085
Merge branch 'trunk' into add/url-metrics-priming
b1ink0 Feb 14, 2025
42bfb98
Add minification support for classic editor JavaScript file
b1ink0 Feb 14, 2025
7487ae8
Add function to disable admin-based URL priming feature when reached …
b1ink0 Feb 18, 2025
49f9eab
Merge branch 'trunk' into add/url-metrics-priming
b1ink0 Mar 4, 2025
fed3012
Merge branch 'trunk' into add/url-metrics-priming
b1ink0 Mar 14, 2025
591a013
Merge branch 'trunk' into add/url-metrics-priming
b1ink0 Mar 18, 2025
17e24c4
Refactor REST API endpoints to use class
b1ink0 Mar 18, 2025
32f1e9e
Merge branch 'trunk' into add/url-metrics-priming
b1ink0 Mar 31, 2025
53cc462
Refactor large URL metrics processing function into multiple smaller …
b1ink0 Mar 31, 2025
f9fd604
Refactor to avoid creation of new `AbortController` for each task
b1ink0 Apr 1, 2025
8de014a
Add URL metrics priming CLI
b1ink0 Apr 3, 2025
ce69218
Improve task processing with `AbortController` and improve logging
b1ink0 Apr 3, 2025
7908b89
Refactor to use custom event for communication and improve spinner me…
b1ink0 Apr 3, 2025
3df2f36
Refactor to use global varible to pass verification token to page
b1ink0 Apr 3, 2025
14f8be3
Add environment checks
b1ink0 Apr 4, 2025
fdcb05f
Merge branch 'trunk' into add/url-metrics-priming
b1ink0 Apr 11, 2025
a2ebf17
Update js-lint workflow to install npm packages for optimization-dete…
b1ink0 Apr 11, 2025
3a795e9
Improve communication of response of URL Metric sending request
b1ink0 Apr 11, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/js-lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,9 @@ jobs:
node-version-file: '.nvmrc'
cache: npm
- name: npm install
run: npm ci
run: |
npm ci
npm ci --prefix plugins/optimization-detective/priming-cli
- name: JS Lint
run: npm run lint-js
- name: TypeScript compile
Expand Down
102 changes: 100 additions & 2 deletions plugins/optimization-detective/detect.js
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,14 @@ const storageLockTimeSessionKey = 'odStorageLockTime';
*/
const compressionDebounceWaitDuration = 1000;

/**
* Verification token for skipping the storage lock check while priming URL Metrics.
*
* @see {detect}
* @type {?string}
*/
let odPrimeUrlMetricsVerificationToken = null;

/**
* Checks whether storage is locked.
*
Expand All @@ -73,6 +81,10 @@ const compressionDebounceWaitDuration = 1000;
* @return {boolean} Whether storage is locked.
*/
function isStorageLocked( currentTime, storageLockTTL ) {
if ( odPrimeUrlMetricsVerificationToken ) {
return false;
}

if ( storageLockTTL === 0 ) {
return false;
}
Expand Down Expand Up @@ -503,6 +515,38 @@ function debounceCompressUrlMetric() {
}, compressionDebounceWaitDuration );
}

/**
* Notifies about the URL Metric request status.
*
* @param {Object} status - The status details.
* @param {boolean} status.success - Indicates if the request succeeded.
* @param {string} [status.error] - An error message if the request failed.
* @param {Object} [options] - Options for where to dispatch the message.
* @param {boolean} [options.toParent=true] - Whether to send the message to the parent window.
* @param {boolean} [options.toLocal=true] - Whether to dispatch a custom event locally.
*/
function notifyStatus( status, options = { toParent: true, toLocal: true } ) {
const message = {
type: 'OD_PRIME_URL_METRICS_REQUEST_STATUS',
success: status.success,
...( status.error && { error: status.error } ),
};

// This will be used when URL metrics are primed using a IFRAME.
if ( options.toParent && window.parent && window.parent !== window ) {
window.parent.postMessage( message, '*' );
}

// This will be used when URL metrics are primed using Puppeteer script.
if ( options.toLocal ) {
document.dispatchEvent(
new CustomEvent( message.type, {
detail: { ...status },
} )
);
}
}

/**
* @typedef {{timestamp: number, creationDate: Date}} UrlMetricDebugData
* @typedef {{groups: Array<{url_metrics: Array<UrlMetricDebugData>}>}} CollectionDebugData
Expand Down Expand Up @@ -595,6 +639,34 @@ export default async function detect( {
return;
}

try {
if (
win.parent &&
win.location.origin === win.parent.location.origin
) {
/** @type {HTMLIFrameElement|null} */
const urlPrimeIframeElement = win.parent.document.querySelector(
'iframe#od-prime-url-metrics-iframe'
);
if (
urlPrimeIframeElement &&
urlPrimeIframeElement.dataset.odPrimeUrlMetricsVerificationToken
) {
odPrimeUrlMetricsVerificationToken =
urlPrimeIframeElement.dataset
.odPrimeUrlMetricsVerificationToken;
}
}
} catch ( e ) {
// Ignoring error caused possibly due to cross-origin iframe access.
}

// Only available when page is loaded by Puppeteer script.
if ( win.__odPrimeUrlMetricsVerificationToken ) {
odPrimeUrlMetricsVerificationToken =
win.__odPrimeUrlMetricsVerificationToken;
}
Comment on lines +642 to +668
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest the query parameter be supplied consistently by adding a URL query parameter instead. The cross-domain issue could be a common one, where wp-admin is on a different origin than the frontend. Then the token would be supplied in the same way for Puppeteer as for non-Puppeteer.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, and as opposed to passing the parameter as a query parameter it could also be supplied in the URL fragment instead. The benefit here would be it wouldn't get bust any page caches, although caching would be unlikely since the user is already logged-in. But another reason is that sometimes I've noticed that requests with unrecognized query parameters get redirected with those stripped out. For example:

https://joost.blog/wordpress-leadership/?odPrimeUrlMetricsVerificationToken=abc123

Notice how that redirects with the query parameter removed. Compare with:

https://joost.blog/wordpress-leadership/#odPrimeUrlMetricsVerificationToken=abc123

The URL target here now remains.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The URL fragment approach seems very good. I had previously thought of using a query parameter, but I was also worried about the same problem you described. Additionally, we would have had to remove the query parameter from the currentUrl. I was also considering using postMessage to send the verification token, as cross-origin access can't reach the dataset of the iframe. Using a URL fragment will eliminate the complex logic required for postMessage.


// Abort if the client already submitted a URL Metric for this URL and viewport group.
const alreadySubmittedSessionStorageKey =
await getAlreadySubmittedSessionStorageKey(
Expand All @@ -604,6 +676,7 @@ export default async function detect( {
logger
);
if (
! odPrimeUrlMetricsVerificationToken &&
null !== alreadySubmittedSessionStorageKey &&
alreadySubmittedSessionStorageKey in sessionStorage
) {
Expand Down Expand Up @@ -922,6 +995,10 @@ export default async function detect( {

// Wait for the page to be hidden.
await new Promise( ( resolve ) => {
if ( odPrimeUrlMetricsVerificationToken ) {
resolve();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parent and IFRAME communication is handled via postMessage. A message is sent to the parent, and the promise resolves immediately.

If the promise isn't resolved immediately, navigating to a new URL causes the code following the promise to never execute. This is because changing the iframe.src does not trigger events like pagehide, pageswap, or visibilitychange.

// Wait for the page to be hidden.
await new Promise( ( resolve ) => {

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder. Do we even need to post a message here? As soon as the iframe is destroyed won't it automatically cause the URL Metric to be sent, right?

Copy link
Contributor Author

@b1ink0 b1ink0 Feb 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is we need to signal the parent that we can move to next URL or breakpoint using postMessage as the load event can't be used. Check this comment for detailed explanation #1850 (comment) .

Will it makes sense to send the postMessage after the navigator.sendBeacon then?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think it makes sense to send the message after the beacon is sent, definitely.

}

win.addEventListener( 'pagehide', resolve, { once: true } );
win.addEventListener( 'pageswap', resolve, { once: true } );
doc.addEventListener(
Expand Down Expand Up @@ -1092,6 +1169,12 @@ export default async function detect( {
);
}
url.searchParams.set( 'hmac', urlMetricHMAC );
if ( odPrimeUrlMetricsVerificationToken ) {
url.searchParams.set(
'prime_url_metrics_verification_token',
odPrimeUrlMetricsVerificationToken
);
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Authentication for REST API

  • WP Nonce Limitation: The default WordPress (WP) nonce does not function correctly when generated for the parent page and then passed to an iframe for REST API requests.

  • Custom Token Authentication: To address this, I have added a custom token-based authentication mechanism. This generates a time-limited token used to authenticate REST API requests made via the iframe.

In #1835 PR, WP nonces are introduced for REST API requests for logged-in users. This may allow us to eliminate the custom token authentication if URL metrics are collected exclusively from logged-in users.


const headers = {
'Content-Type': 'application/json',
Expand All @@ -1104,7 +1187,22 @@ export default async function detect( {
method: 'POST',
body: payloadBlob,
headers,
keepalive: true, // This makes fetch() behave the same as navigator.sendBeacon().
keepalive: odPrimeUrlMetricsVerificationToken ? false : true, // Setting keepalive to true makes fetch() behave the same as navigator.sendBeacon().
} );
await fetch( request );

if ( ! odPrimeUrlMetricsVerificationToken ) {
await fetch( request );
} else {
try {
const response = await fetch( request );
if ( ! response.ok ) {
throw new Error(
`Failed to send URL Metric. Status: ${ response.status }`
);
}
notifyStatus( { success: true } );
} catch ( err ) {
notifyStatus( { success: false, error: err.message } );
}
}
}
25 changes: 25 additions & 0 deletions plugins/optimization-detective/detection.php
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,31 @@ function_exists( 'gzdecode' ) &&
return $result;
}

/**
* Registers the REST API endpoint for priming URL Metrics.
*
* @since n.e.x.t
* @access private
*/
function od_register_rest_url_metric_priming_endpoint(): void {
$endpoint_controller = new OD_REST_URL_Metrics_Priming_Endpoint();
register_rest_route(
OD_REST_URL_Metrics_Store_Endpoint::ROUTE_NAMESPACE,
$endpoint_controller::PRIME_URLS_ROUTE,
$endpoint_controller->get_registration_args_prime_urls()
);
register_rest_route(
OD_REST_URL_Metrics_Store_Endpoint::ROUTE_NAMESPACE,
$endpoint_controller::PRIME_URLS_BREAKPOINTS_ROUTE,
$endpoint_controller->get_registration_args_prime_urls_breakpoints()
);
register_rest_route(
OD_REST_URL_Metrics_Store_Endpoint::ROUTE_NAMESPACE,
$endpoint_controller::PRIME_URLS_VERIFICATION_TOKEN_ROUTE,
$endpoint_controller->get_registration_args_prime_urls_verification_token()
);
}

/**
* Triggers post update actions for page caches to invalidate their caches related to the supplied cache purge post ID.
*
Expand Down
Loading
Loading