Add ability to prime URL metrics #1850

b1ink0 · 2025-02-05T17:55:23Z

Summary

Relevant technical choices

This PR introduces a new mechanism for priming URL metrics across the site. It uses a newly added submenu page in the Tools menu and automatically primes URL metrics when a post is saved in the block editor.

Demos

Settings page with debugging off:

Settings.Page.mp4

Settings page with debugging on:

Settings.Page.With.Debug.mp4

Saving post in Block Editor:

Block.Editor.mp4

…ing between parent and iframe

… on frontend

…ints logic to dedicated functions

… generating a new UUID

…cessing

…token

…ack config

… visibility hidden

…or not visible

codecov · 2025-02-05T18:04:15Z

Codecov Report

Attention: Patch coverage is 16.50485% with 258 lines in your changes missing coverage. Please review.

Project coverage is 64.43%. Comparing base (e845dd8) to head (c1e245d).
Report is 38 commits behind head on trunk.

Files with missing lines	Patch %	Lines
plugins/optimization-detective/helper.php	1.07%	184 Missing ⚠️
...lugins/optimization-detective/storage/rest-api.php	50.00%	49 Missing ⚠️
plugins/optimization-detective/settings.php	0.00%	24 Missing ⚠️
plugins/optimization-detective/load.php	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##            trunk    #1850      +/-   ##
==========================================
- Coverage   66.36%   64.43%   -1.94%     
==========================================
  Files          88       89       +1     
  Lines        6975     7318     +343     
==========================================
+ Hits         4629     4715      +86     
- Misses       2346     2603     +257

Flag	Coverage Δ
multisite	`64.43% <16.50%> (-1.94%)`	⬇️
single	`35.73% <0.00%> (-1.76%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

b1ink0 · 2025-02-06T08:22:30Z

plugins/optimization-detective/detect.js

+			'prime_url_metrics_verification_token',
+			odPrimeUrlMetricsVerificationToken
+		);
+	}


Authentication for REST API

WP Nonce Limitation: The default WordPress (WP) nonce does not function correctly when generated for the parent page and then passed to an iframe for REST API requests.

Custom Token Authentication: To address this, I have added a custom token-based authentication mechanism. This generates a time-limited token used to authenticate REST API requests made via the iframe.

In #1835 PR, WP nonces are introduced for REST API requests for logged-in users. This may allow us to eliminate the custom token authentication if URL metrics are collected exclusively from logged-in users.

b1ink0 · 2025-02-06T08:26:22Z

plugins/optimization-detective/prime-url-metrics.js

+			};
+
+			// Load the iframe
+			iframe.src = task.url;


Currently if the IFRAME shares the same origin as the parent, then it allows it to access the parent session. This ensures that the user session in the page loaded within the iframe (which is a frontend page) matches the logged-in user of the WordPress dashboard.

But if the WordPress admin dashboard and the frontend have different origins, WP nonces won’t work for REST API authentication because the iframe will not recognize the logged-in session. As the different origin does not allow iframe to access parents session. For context I am talking about the REST nonce introduced in #1835.

b1ink0 · 2025-02-06T08:30:29Z

plugins/optimization-detective/prime-url-metrics-block-editor.js

+	iframe.style.transform = 'scale(0.05)';
+	iframe.style.transformOrigin = '0 0';
+	iframe.style.pointerEvents = 'none';
+	iframe.style.opacity = '0.000001';


As the detect.js requires the iframe to be visible in the viewport to resolve the onLCP promise. Traditional methods like moving the iframe off-screen using translate, setting visibility: hidden, or opacity: 0 cause the promise to hang.

performance/plugins/optimization-detective/detect.js

Lines 496 to 503 in 6ca5c4b

// Obtain at least one LCP candidate. More may be reported before the page finishes loading.

await new Promise( ( resolve ) => {

onLCP(

( /** @type LCPMetric */ metric ) => {

lcpMetricCandidates.push( metric );

resolve();

},

{

I am using a workaround using the following CSS to keep the iframe minimally visible and functional:

position: fixed; top: 0px; left: 0px; transform: scale(0.05); transform-origin: 0px 0px; pointer-events: none; opacity: 1e-6; z-index: -9999;

b1ink0 · 2025-02-06T08:33:27Z

plugins/optimization-detective/detect.js

+				'OD_PRIME_URL_METRICS_REQUEST_SUCCESS',
+				'*'
+			);
+			resolve();


Parent and IFRAME communication is handled via postMessage. A message is sent to the parent, and the promise resolves immediately.

If the promise isn't resolved immediately, navigating to a new URL causes the code following the promise to never execute. This is because changing the iframe.src does not trigger events like pagehide, pageswap, or visibilitychange.

performance/plugins/optimization-detective/detect.js

Lines 568 to 569 in 6ca5c4b

// Wait for the page to be hidden.

await new Promise( ( resolve ) => {

I wonder. Do we even need to post a message here? As soon as the iframe is destroyed won't it automatically cause the URL Metric to be sent, right?

The problem is we need to signal the parent that we can move to next URL or breakpoint using postMessage as the load event can't be used. Check this comment for detailed explanation #1850 (comment) .

Will it makes sense to send the postMessage after the navigator.sendBeacon then?

Yes, I think it makes sense to send the message after the beacon is sent, definitely.

b1ink0 · 2025-02-06T09:36:05Z

plugins/optimization-detective/settings.php

+			<h2><?php esc_html_e( 'Prime URL Metrics', 'optimization-detective' ); ?></h2>
+			<div id="od-prime-url-metrics-container">
+				<button id="od-prime-url-metrics-control-button" class="button button-primary"><?php esc_html_e( 'Start', 'optimization-detective' ); ?></button>
+				<progress id="od-prime-url-metrics-progress" value="0" max="0"></progress>


Currently I have only added one button which can start/pause/resume URL metrics priming and a progress bar indicating the current batch's progress.

ui.mp4

I would like to get suggestions on the UI to improve it.

b1ink0 · 2025-02-06T09:39:00Z

plugins/optimization-detective/helper.php

+ * @param array<string, int> $cursor Cursor to resume from.
+ * @return array<string, mixed> Batch of URLs to prime metrics for and the updated cursor.
+ */
+function od_get_batch_for_iframe_url_metrics_priming( array $cursor ): array {


I have added a batch function with a cursor mechanism to fetch URLs in manageable chunks. The current default batch size is set to 10, which seems too low. I would like to get suggestion on optimal batch size.

This function also got complex because of the need of handling multiple layers, including providers, provider subtypes, and pages of each subtype for batching.

b1ink0 · 2025-02-06T09:45:31Z

plugins/optimization-detective/helper.php

+ * @param string[] $urls Array of exact URLs, as stored in post_title of od_url_metrics.
+ * @return array<string, OD_URL_Metric_Group_Collection> Map of URL to its OD_URL_Metric_Group_Collection.
+ */
+function od_get_metrics_by_post_title( array $urls ): array {


Since WP sitemaps only provide post URLs, a custom WHERE clause was needed to be added to WP_Query to filter out URLs that are already primed. This is done based on the post_title.

b1ink0 · 2025-02-06T09:46:44Z

plugins/optimization-detective/storage/rest-api.php

+
+	register_rest_route(
+		OD_REST_API_NAMESPACE,
+		OD_PRIME_URLS_ROUTE,


Currently, the helper function and REST API endpoints has been added to existing files. Should we consider moving it to a new folder named admin for better organization?

b1ink0 · 2025-02-06T09:48:15Z

plugins/optimization-detective/helper.php

+ * @param array<string> $urls Array of URLs to filter.
+ * @return array<int, array{url: string, breakpoints: array<int, array{width: int, height: int}>}> Filtered batch of URLs.
+ */
+function od_filter_batch_urls_for_iframe_url_metrics_priming( array $urls ): array {


What would be the good naming conventions for these helper function for priming URLs?

westonruter · 2025-02-07T05:27:55Z

plugins/optimization-detective/prime-url-metrics-block-editor.js

+			const handleMessage = ( event ) => {
+				if ( event.data === 'OD_PRIME_URL_METRICS_REQUEST_SUCCESS' ) {
+					cleanup();
+					resolve();
+				}
+			};


Instead of this, what about listening to the load event on the iframe?

The reason is that the load event fires when the document has been fully processed. However, load does not wait for the async and await operations used in detect.js. From my testing, I found that the detect function gets stuck on await import(webVitalsLibrarySrc), waiting for the promise to resolve. However, before this promise is fulfilled, the load event is triggered, causing the IFRAME to navigate to the next URL or a breakpoint change. Changing the IFRAME's src destroys the JavaScript execution, preventing the code from progressing past await import(webVitalsLibrarySrc). Because of this reason I have used the postMessage for communication.

That makes sense, yes. In fact, the plugin explicitly waits for load event to even start doing anything:

performance/plugins/optimization-detective/detect.js

Lines 416 to 423 in e845dd8

// Wait until the resources on the page have fully loaded.

await new Promise( ( resolve ) => {

if ( doc.readyState === 'complete' ) {

resolve();

} else {

win.addEventListener( 'load', resolve, { once: true } );

}

} );

So it makes sense that the iframe's load event would fire before we do any detection.

…Improve UI to show current batch and task

…nctionality

…dling

b1ink0 added 23 commits January 23, 2025 23:58

Add settings page for Optimization Detective

57f9958

Add initial admin script and UI elements for URL priming

71cd706

Implement batch URL priming functionality with REST API endpoint

da1d275

Include breakpoints in URL processing

bfed250

Add debug mode and better dynamic height generation for breakpoints

6ec82e3

Add verification token auth for protecting REST API, Add message pass…

af0172e

…ing between parent and iframe

Refactor to improve control flow and add support for pause and resume…

bff9973

… on frontend

Move batch URL fetching logic to helper function

c155fc1

Add support to filter URL whose metrics are already populated

7640ef7

Add support to filter breakpoints whose metrics are already populated

f05f6c0

Refactor by moving filtering batch URls and computing standard brekpo…

0631daf

…ints logic to dedicated functions

Fix sending empty batch URLs caused by filtering the batch URLs

5d93fdc

Save last batch cursor in options table to improve performance

3118fb3

Improve verification token handling by using a transient check before…

d8cfa02

… generating a new UUID

Add beforeunload event listener to prevent page navigation during pro…

8879f60

…cessing

Add new REST API routes for URL metrics breakpoints and verification …

72b18fd

…token

Add block editor support for priming URL metrics

f5bba48

Merge branch 'trunk' into add/url-metrics-priming

b0422eb

Add missing since and access annotations, Add new JavaScripts to webp…

db1fd81

…ack config

Remove usage of fetch and Fix URL metrics not priming due to usage of…

63564c4

… visibility hidden

Merge branch 'trunk' into add/url-metrics-priming

4fd36b6

Fix detect.js getting stuck on 'onLCP' due to IFRAME being offscreen …

82a16b5

…or not visible

Improve responsiveness of IFRAME in debug mode

a50f23e

b1ink0 mentioned this pull request Feb 5, 2025

Add ability to prime URL metrics across a site upon installation of Optimization Detective #1311

Open

Merge branch 'WordPress:trunk' into add/url-metrics-priming

9201c86

b1ink0 commented Feb 6, 2025

View reviewed changes

b1ink0 added [Type] Enhancement A suggestion for improvement of an existing feature [Plugin] Optimization Detective Issues for the Optimization Detective plugin labels Feb 6, 2025

westonruter reviewed Feb 7, 2025

View reviewed changes

b1ink0 added 9 commits February 7, 2025 22:42

Skip check of already submitted URL metrics

163d24e

Merge branch 'trunk' into add/url-metrics-priming

57d77ea

Fix IFRAME positioning

2bc9f49

Add URL priming functionality for classic editor

d17dede

Refactor to prime URL metrics only when the post is saved

97b888f

Move URL metrics success message to the end of the detection process

931861b

Add instructions and important information section to settings page, …

0a7c1f1

…Improve UI to show current batch and task

Pause URL metrics processing when tab visibility chages, Add abort fu…

969f6b5

…nctionality

Implement task processing with abort functionality and visibility han…

c1e245d

…dling

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability to prime URL metrics #1850

Add ability to prime URL metrics #1850

b1ink0 commented Feb 5, 2025 •

edited

Loading

codecov bot commented Feb 5, 2025 •

edited

Loading

b1ink0 Feb 6, 2025

b1ink0 Feb 6, 2025

b1ink0 Feb 6, 2025 •

edited

Loading

b1ink0 Feb 6, 2025

westonruter Feb 7, 2025

b1ink0 Feb 7, 2025 •

edited

Loading

westonruter Feb 7, 2025

b1ink0 Feb 6, 2025

b1ink0 Feb 6, 2025 •

edited

Loading

b1ink0 Feb 6, 2025

b1ink0 Feb 6, 2025

b1ink0 Feb 6, 2025 •

edited

Loading

westonruter Feb 7, 2025

b1ink0 Feb 7, 2025

westonruter Feb 7, 2025

	// Obtain at least one LCP candidate. More may be reported before the page finishes loading.
	await new Promise( ( resolve ) => {
	onLCP(
	( /** @type LCPMetric */ metric ) => {
	lcpMetricCandidates.push( metric );
	resolve();
	},
	{

	// Wait for the page to be hidden.
	await new Promise( ( resolve ) => {

	// Wait until the resources on the page have fully loaded.
	await new Promise( ( resolve ) => {
	if ( doc.readyState === 'complete' ) {
	resolve();
	} else {
	win.addEventListener( 'load', resolve, { once: true } );
	}
	} );

Add ability to prime URL metrics #1850

Are you sure you want to change the base?

Add ability to prime URL metrics #1850

Conversation

b1ink0 commented Feb 5, 2025 • edited Loading

Summary

Relevant technical choices

Demos

codecov bot commented Feb 5, 2025 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

b1ink0 Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

b1ink0 Feb 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

b1ink0 Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

b1ink0 Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

b1ink0 commented Feb 5, 2025 •

edited

Loading

codecov bot commented Feb 5, 2025 •

edited

Loading

b1ink0 Feb 6, 2025 •

edited

Loading

b1ink0 Feb 7, 2025 •

edited

Loading

b1ink0 Feb 6, 2025 •

edited

Loading

b1ink0 Feb 6, 2025 •

edited

Loading