Add ImageData Float16Array support #11143

ccameron-chromium · 2025-03-19T12:44:54Z

Add the ImageDataPixelFormat enum type, which can take on values of "rgba-unorm8" (the current default of 8-bit unsigned normalized RGBA) and "rgba-float16" (the new option for 16-bit floating point RGBA).

Add ImageDataPixelFormat to the ImageDataSettings dictionary used by ImageData constructors, to allow creation of ImageData objects that are backed by Float16Array.

Add ImageDataPixelFormat as an attribute to ImageData.

Add a new ImageDataArray type which is the union of Uint8ClampedArray and Float16Array. Change the ImageData data attribute to be this union type instead of Uint8ClampedArray.

Fix the Canvas Pixel ArrayBuffer section to correctly describe the pixel layout of an ImageBuffer (there was a bug in the spec specifying that an ImageData's pixel data starts at the beginning of its data attributes's backing ArrayBuffer, when what was intended was for it to start at the beginning of its data attribute).

Fixes #10856

At least two implementers are interested (and none opposed):
Tests are written and can be reviewed and commented upon at:
- Staging at: https://chromium-review.googlesource.com/6372220
Implementation bugs are filed:
- Chromium: crbug.com/40277213
- Gecko: https://bugzilla.mozilla.org/show_bug.cgi?id=1958830
- WebKit: https://bugs.webkit.org/show_bug.cgi?id=291196
MDN issue is filed:
- Add Float16Array-backed ImageData support mdn/content#39023
The top of this comment includes a clear commit message to use.

(See WHATWG Working Mode: Changes for more details.)

/canvas.html ( diff )
/infrastructure.html ( diff )

Add the ImageDataPixelFormat enum type, which can take on values of "rgba-unorm8" (the current default of 8-bit unsigned normalized RGBA) and "rgba-float16" (the new option for 16-bit floating point RGBA). Add ImageDataPixelFormat to the ImageDataSettings dictionary used by ImageData constructors, to allow creation of ImageData objects that are backed by Float16Array. Add ImageDataPixelFormat as an attribute to ImageData. Add a new ImageDataArray type which is the union of Uint8ClampedArray and Float16Array. Change the ImageData data attribute to be this union type instead of Uint8ClampedArray. Fix the Canvas Pixel ArrayBuffer section to correctly describe the pixel layout of an ImageBuffer (there was a bug in the spec specifying that an ImageData's pixel data starts at the beginning of its data attributes's backing ArrayBuffer, when what was intended was for it to start at the beginning of its data attribute). Fixes whatwg#10856

source

Kaiido · 2025-03-21T07:56:29Z

source

@@ -70609,30 +70614,61 @@ try {
  <p>To <dfn>initialize an <code>ImageData</code> object</dfn> <var>imageData</var>, given a
  positive integer number of rows <var>rows</var>, a positive integer number of pixels per row


This isn't new, but aren't the arguments inverted here?
All the call-sites do Initialize imageData given sw, sh, settings set to [...], but here the algorithm expects number of rows as the first argument and then pixels per row.
Though maybe this should be fixed separately and I'm sure there is some modernization that could get in there.

I swapped the ordering of the parameters, since that seems fairly urgent to not have wrong (when we do a modernization pass, we should also change the names to width and height).

domfarolino · 2025-03-22T22:51:14Z

Add ImageDataPixelFormat as an attribute to ImageData.

This PR also adds the pixelFormat readonly attribute to ImageData, but I'm not completely sure why. The explainer indicates that it is for feature detection, but couldn't you just as easily type-check the underlying data? Is this member just added as a convenience mechanism for that, or is it necessary for some other reason? My concern is that it isn't necessary, but that it does add complexity just to validate/synchronize the type of the input source with the pixelFormat descriptor in the settings object.

I could imagine this being important if you wanted to accept a Float16Array yet force it to be treated like a Uint8ClampedArray by passing in pixelFormat = "rgba-unorm8", but we don't seem to support that.

ccameron-chromium · 2025-03-23T17:03:33Z

Add ImageDataPixelFormat as an attribute to ImageData.

This PR also adds the pixelFormat readonly attribute to ImageData, but I'm not completely sure why. [...] My concern is that it isn't necessary, but that it does add complexity just to validate/synchronize the type of the input source with the pixelFormat descriptor in the settings object.

Indeed this is partially for future-proofing. This is intended to handle the situation where there isn't a 1:1 relationship between the TypedArray type and the actual pixel format (for now, there is indeed a 1:1 relationship). A simple example (but not the most useful) is that one could have an rg-unorm8 format, which also uses Uint8ClampedArray, but only has two channels (red-green). There could also be things like grey-alpha as well, etc.

It's also there because it felt odd for the ImageDataSettings to have a pixelFormat for use by constructors that would then disappear and not be query-able.

And it's also a good way of checking the actual layout (as opposed to the sub-type of the ArrayBuffer). Cause it will always be the case that pixelFormat implies a specific ArrayBuffer type.

(And feature detection was sort of an afterthought).

I could imagine this being important if you wanted to accept a Float16Array yet force it to be treated like a Uint8ClampedArray by passing in pixelFormat = "rgba-unorm8", but we don't seem to support that.

Indeed, we would not want to support anything like that (float to uint8, etc).

domfarolino · 2025-03-24T02:33:11Z

It's also there because it felt odd for the ImageDataSettings to have a pixelFormat for use by constructors that would then disappear and not be query-able.

Right, I think it's a reasonable member of ImageData if it needs to be passed in via settings—it was just the passing in via settings that I was questioning in the first place. I think your explanation of different pixel format types mapping to the same underlying ArrayBuffer type makes sense and is compelling, thanks!

source

annevk · 2025-03-24T13:36:31Z

source

@@ -70491,7 +70497,7 @@ try {
  constructor steps are:</p>

  <ol>
-   <li><p>Let <var>length</var> be the number of bytes in <var>data</var>.</p></li>
+   <li><p>Let <var>length</var> be the length of <var>data</var>.</p></li>


This should reference some actual algorithm. It seems currently we only have https://webidl.spec.whatwg.org/#buffersource-byte-length so maybe we need to add this to Web IDL first.

I've changed all of the math to be in term of "bytes per pixel", since that's more natural (I had been leaning on the fact that all pixel formats so far are 4 elements per pixel, which works so far, but would require re-writing later down the line), and that made this issue go away.

That said, should we open an issue on Web IDL to add length to TypedArray? (Or should we wait until something will use it).

source

ccameron-chromium · 2025-03-30T20:21:06Z

Any further thoughts on this?

annevk · 2025-03-31T06:24:08Z

source

+   If <var>settings</var> was given and
+   <var>settings</var>["<code data-x="dom-ImageDataSettings-pixelFormat">pixelFormat</code>"]
+   equals "<span data-x="dom-ImageDataPixelFormat-rgba-float16">rgba-float16</span>" then
+   <var>bytesPerPixel</var> is equal to 8.</p></li>


This seems wrong. settings is always given after bindings have been resolved so this shouldn't be this conditional unless I'm missing something.

Indeed, settings was conditional in the constructor, not here.

That said, it's easier to re-work this so that settings is not optional, and have the one function that didn't specify settings create its own ImageDataSettings. I've done that, and non-optional-ized settings. The diff turned out to be fairly small.

source

annevk · 2025-03-31T06:25:39Z

source

-    Pixel <code data-x="idl-ArrayBuffer">ArrayBuffer</code></span> must have the correct size to
-    store <var>rows</var> × <var>pixelsPerRow</var> pixels.</p>
+    <p>If <var>source</var> was given:</p>
+    <ol>


Newline before ol.

This does not appear to be done?

Oops (I did something here, I assume you!). Should be fixed now (along with the next ol, which also missed it).

source

annevk · 2025-03-31T06:28:15Z

source

+  <p>The <code>ImageDataPixelFormat</code> enumeration is used to specify type of the
+  <code data-x="dom-imagedata-data">data</code> attribute of an <code>ImageData</code>
+  and the arrangement and numerical representation of the color components for each pixel.</p>
+
+  <p>The "<dfn enum-value for="ImageDataPixelFormat"><code
+  data-x="dom-ImageDataPixelFormat-rgba-unorm8">rgba-unorm8</code></dfn>" value indicates that the
+  <code data-x="dom-imagedata-data">data</code> attribute of an <code>ImageData</code> must be of type
+  <span data-x="idl-Uint8ClampedArray">Uint8ClampedArray</span>.
+  The color components of each pixel must be stored in four sequential elements in the order of red, green, blue, and then alpha.
+  Each element represents the 8-bit unsigned normalized value for that component.</p>
+
+  <p>The "<dfn enum-value for="ImageDataPixelFormat"><code
+  data-x="dom-ImageDataPixelFormat-rgba-float16">rgba-float16</code></dfn>" value indicates that the
+  <code data-x="dom-imagedata-data">data</code> attribute of an <code>ImageData</code> must be of type
+  <span data-x="idl-Float16Array">Float16Array</span>.
+  The color components of each pixel must be stored in four sequential elements in the order of red, green, blue, and then alpha.
+  Each element represents the value for that component.</p>
+
+  <p>An <code>ImageData</code> object <dfn data-x="concept-imagedata-bitmap-representation">represents a rectanglar bitmap</dfn>
+  with width equal to the <code data-x="dom-imagedata-width">width</code> attribute
+  and height equal to the <code data-x="dom-imagedata-height">height</code> attribute.
+  The pixel values of this bitmap are stored in the <code data-x="dom-imagedata-data">data</code> attribute
+  in left-to-right order, row by row from top to bottom, starting with 0 for the top left pixel,
+  with the order and numerical representation of the color components of each
+  pixel determined by the <code data-x="dom-imagedata-pixelFormat">pixelFormat</code> attribute.
+  The color space of the pixel values of the bitmap is determined by the
+  <code data-x="dom-imagedata-colorSpace">colorSpace</code> attribute.</p>


The wrapping here seems wrong. I'm also a bit uncomfortable with postponing the cleanup as all of this will have to be redone and re-reviewed. Does it seem untenable to do the cleanup first and then land this?

The wrapping here seems wrong.

I re-wrapped this block to be 100 chars limit (which seems to be mostly adhered to).

I'm also a bit uncomfortable with postponing the cleanup as all of this will have to be redone and re-reviewed. Does it seem untenable to do the cleanup first and then land this?

The cleanup's changes to this section will be fairly small (remove the word attribute and change the xref), so it will be easy to land that on top of this. The other way around will basically require redoing this from scratch, because the cleanup will have moved all of the blocks around, and the cleanup will probably also take a long time in review.

I hope that the work I've done so far (e.g, the cleanup of getContextAttributes) shows that I'm serious about the stewardship of this area, and that I indeed prefer to prepare the ground before making changes, when it is beneficial. But in this the benefit appears marginal, and I do see that it would be extremely disruptive to the effort that we and the other reviewers have put in to this PR.

ccameron-chromium

Thanks, updated the PR.

ccameron-chromium · 2025-03-31T13:14:48Z

source

+   If <var>settings</var> was given and
+   <var>settings</var>["<code data-x="dom-ImageDataSettings-pixelFormat">pixelFormat</code>"]
+   equals "<span data-x="dom-ImageDataPixelFormat-rgba-float16">rgba-float16</span>" then
+   <var>bytesPerPixel</var> is equal to 8.</p></li>


Indeed, settings was conditional in the constructor, not here.

That said, it's easier to re-work this so that settings is not optional, and have the one function that didn't specify settings create its own ImageDataSettings. I've done that, and non-optional-ized settings. The diff turned out to be fairly small.

source

ccameron-chromium · 2025-03-31T13:22:55Z

source

-    Pixel <code data-x="idl-ArrayBuffer">ArrayBuffer</code></span> must have the correct size to
-    store <var>rows</var> × <var>pixelsPerRow</var> pixels.</p>
+    <p>If <var>source</var> was given:</p>
+    <ol>


source

ccameron-chromium · 2025-03-31T13:25:58Z

source

+  <p>The <code>ImageDataPixelFormat</code> enumeration is used to specify type of the
+  <code data-x="dom-imagedata-data">data</code> attribute of an <code>ImageData</code>
+  and the arrangement and numerical representation of the color components for each pixel.</p>
+
+  <p>The "<dfn enum-value for="ImageDataPixelFormat"><code
+  data-x="dom-ImageDataPixelFormat-rgba-unorm8">rgba-unorm8</code></dfn>" value indicates that the
+  <code data-x="dom-imagedata-data">data</code> attribute of an <code>ImageData</code> must be of type
+  <span data-x="idl-Uint8ClampedArray">Uint8ClampedArray</span>.
+  The color components of each pixel must be stored in four sequential elements in the order of red, green, blue, and then alpha.
+  Each element represents the 8-bit unsigned normalized value for that component.</p>
+
+  <p>The "<dfn enum-value for="ImageDataPixelFormat"><code
+  data-x="dom-ImageDataPixelFormat-rgba-float16">rgba-float16</code></dfn>" value indicates that the
+  <code data-x="dom-imagedata-data">data</code> attribute of an <code>ImageData</code> must be of type
+  <span data-x="idl-Float16Array">Float16Array</span>.
+  The color components of each pixel must be stored in four sequential elements in the order of red, green, blue, and then alpha.
+  Each element represents the value for that component.</p>
+
+  <p>An <code>ImageData</code> object <dfn data-x="concept-imagedata-bitmap-representation">represents a rectanglar bitmap</dfn>
+  with width equal to the <code data-x="dom-imagedata-width">width</code> attribute
+  and height equal to the <code data-x="dom-imagedata-height">height</code> attribute.
+  The pixel values of this bitmap are stored in the <code data-x="dom-imagedata-data">data</code> attribute
+  in left-to-right order, row by row from top to bottom, starting with 0 for the top left pixel,
+  with the order and numerical representation of the color components of each
+  pixel determined by the <code data-x="dom-imagedata-pixelFormat">pixelFormat</code> attribute.
+  The color space of the pixel values of the bitmap is determined by the
+  <code data-x="dom-imagedata-colorSpace">colorSpace</code> attribute.</p>


The wrapping here seems wrong.

I re-wrapped this block to be 100 chars limit (which seems to be mostly adhered to).

I'm also a bit uncomfortable with postponing the cleanup as all of this will have to be redone and re-reviewed. Does it seem untenable to do the cleanup first and then land this?

The cleanup's changes to this section will be fairly small (remove the word attribute and change the xref), so it will be easy to land that on top of this. The other way around will basically require redoing this from scratch, because the cleanup will have moved all of the blocks around, and the cleanup will probably also take a long time in review.

I hope that the work I've done so far (e.g, the cleanup of getContextAttributes) shows that I'm serious about the stewardship of this area, and that I indeed prefer to prepare the ground before making changes, when it is beneficial. But in this the benefit appears marginal, and I do see that it would be extremely disruptive to the effort that we and the other reviewers have put in to this PR.

ccameron-chromium · 2025-04-03T15:25:18Z

Let me know if there's anything further (once this lands, I'm going to do the patch to move ImageData over to be a sibling of ImageBitmap).

source

annevk

Please complete the checkboxes in OP (and leave a comment once you are finished so I get another notification).

Thanks for tackling this! Exciting to see Float16Array adoption on the web platform side.

ccameron-chromium · 2025-04-07T12:18:27Z

Thanks, updated the OP with the newly filed bugs. Will be landing the linked WPT tests (and a bunch more) shortly!

source

Kaiido · 2025-04-09T23:34:50Z

source

   data-x="dom-imagedata-colorSpace">colorSpace</code></dfn> attribute of <var>imageData</var> to
   <var>settings</var>["<dfn dict-member for="ImageDataSettings"><code
   data-x="dom-ImageDataSettings-colorSpace">colorSpace</code></dfn>"].</p></li>

-   <li><p>Otherwise, if <var>defaultColorSpace</var> was given, then initialize the <code


Why remove this step? Now createImageData() doesn't produce the proper ImageData by default.

Sorry. Fixed now!

ccameron-chromium force-pushed the image_data_float16 branch from 5adbe23 to 08076f0 Compare March 19, 2025 13:00

Kaiido reviewed Mar 21, 2025

View reviewed changes

Incorporate feedback from kaiido review

cb43615

ccameron-chromium mentioned this pull request Mar 21, 2025

ImageData: Add Float16Array backing support #10856

Closed

annevk reviewed Mar 24, 2025

View reviewed changes

ccameron-chromium mentioned this pull request Mar 26, 2025

Move ImageData interface documentation out of CanvasImageData (and cleanup) #11163

Open

Incorporate annevk feedback

02d9f4b

ccameron-chromium mentioned this pull request Mar 27, 2025

[HTML] 2D canvas floating point support w3ctag/design-reviews#1030

Closed

1 task

annevk reviewed Mar 31, 2025

View reviewed changes

Incorporate annevk review feedback

75eeed6

ccameron-chromium commented Mar 31, 2025

View reviewed changes

Incorporate annevk feedback

ebbcc00

annevk reviewed Apr 4, 2025

View reviewed changes

source Show resolved Hide resolved

source Show resolved Hide resolved

source Show resolved Hide resolved

source Show resolved Hide resolved

source Show resolved Hide resolved

source Show resolved Hide resolved

source Show resolved Hide resolved

ccameron-chromium added 2 commits April 7, 2025 11:43

Incorporate annevk feedback

333985f

Fix lint error!

469bc2a

annevk approved these changes Apr 7, 2025

View reviewed changes

ccameron-chromium mentioned this pull request Apr 7, 2025

Add Float16Array-backed ImageData support mdn/content#39023

Open

squelart reviewed Apr 8, 2025

View reviewed changes

source Outdated Show resolved Hide resolved

Incorporate squelart feedback.

086364d

annevk reviewed Apr 8, 2025

View reviewed changes

source Show resolved Hide resolved

source Show resolved Hide resolved

ccameron-chromium added 2 commits April 8, 2025 08:26

Incorporate annevk feedback

a2339d2

Incorporate annevk wrapping feedback

d5aec17

annevk merged commit 2e87c0f into whatwg:main Apr 8, 2025
2 checks passed

AtkinsSJ mentioned this pull request Apr 9, 2025

LibWeb: Add support for 16-bit float colors in canvas ImageData LadybirdBrowser/ladybird#4295

Open

Kaiido reviewed Apr 9, 2025

View reviewed changes

annevk mentioned this pull request Apr 10, 2025

createImageData() and getImageData() regressed #11206

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ImageData Float16Array support #11143

Add ImageData Float16Array support #11143

ccameron-chromium commented Mar 19, 2025 •

edited by pr-preview bot

Loading

Kaiido Mar 21, 2025

ccameron-chromium Mar 21, 2025

domfarolino commented Mar 22, 2025

ccameron-chromium commented Mar 23, 2025

domfarolino commented Mar 24, 2025

annevk Mar 24, 2025

ccameron-chromium Mar 26, 2025

ccameron-chromium commented Mar 30, 2025

annevk Mar 31, 2025

ccameron-chromium Mar 31, 2025

annevk Mar 31, 2025

ccameron-chromium Mar 31, 2025

annevk Apr 4, 2025

ccameron-chromium Apr 7, 2025

annevk Mar 31, 2025

ccameron-chromium Mar 31, 2025

ccameron-chromium left a comment

ccameron-chromium Mar 31, 2025

ccameron-chromium Mar 31, 2025

ccameron-chromium Mar 31, 2025

ccameron-chromium commented Apr 3, 2025

annevk left a comment

ccameron-chromium commented Apr 7, 2025

Kaiido Apr 9, 2025 •

edited

Loading

annevk Apr 10, 2025

ccameron-chromium Apr 10, 2025

		@@ -70609,30 +70614,61 @@ try {
		<p>To <dfn>initialize an <code>ImageData</code> object</dfn> <var>imageData</var>, given a
		positive integer number of rows <var>rows</var>, a positive integer number of pixels per row

Add ImageData Float16Array support #11143

Add ImageData Float16Array support #11143

Conversation

ccameron-chromium commented Mar 19, 2025 • edited by pr-preview bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

domfarolino commented Mar 22, 2025

ccameron-chromium commented Mar 23, 2025

domfarolino commented Mar 24, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ccameron-chromium commented Mar 30, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ccameron-chromium left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ccameron-chromium commented Apr 3, 2025

annevk left a comment

Choose a reason for hiding this comment

ccameron-chromium commented Apr 7, 2025

Kaiido Apr 9, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ccameron-chromium commented Mar 19, 2025 •

edited by pr-preview bot

Loading

Kaiido Apr 9, 2025 •

edited

Loading