Use case
This feature request is about the Canvas class in engine/painting.dart.
As extensively discussed in #31598, using regular canvas operations to draw pixel data is extremely slow.
This makes the only option we have decoding an image from our raw pixel data, e.g. using decodeImageFromPixels. This has another problem, pointed out by @andrewackerman in the mentioned issue, which is that creating an Image in Dart is an async operations.
This means that we will have to postpone drawing our raw pixel data to the next frame if we wish to use real-time operations.
The async Image problem
Images in Flutter currently have the inherent problem that rasterizing / decoding them is always an asynchronous operation. This is why using them for drawing pixel data does not work synchronously, but it also creates a whole nother class of problems:
Any other canvas operation in Flutter that involves Images is also asynchronous. So not only can you not draw pixel data synchronously, but you cannot rasterize any Picture synchronously, which is necessary for achieving various effects (e.g. this). Also Canvas.drawAtlas suffers from this problem.
drawPixels proposal
Here is an example proposal (written by @andrewackerman in #31598) for directly drawing pixel data.
Note that this might not cover all use cases because often we do need to have a decoded image (dart:ui.Image) available to us, e.g. when using Canvas.drawAtlas. It should, however, give an idea as to what we are looking for.
[...] it should have a similar signature to decodImageFromPixels [sic], so something like:
class Canvas {
...
void drawPixels(
Offset c,
Uint8List bytes,
int width,
int height,
PixelFormat pixelFormat, {
int rowBytes,
int targetWidth,
int targetHeight,
}) {
assert(c != null && bytes != null && width != null && height != null && pixelFormat != null);
assert(bytes.length == width * height * 4);
if (rowBytes == null) rowBytes = width * 4;
if (targetWidth == null && targetHeight == null) {
targetWidth = width;
targetHeight = height;
} else if (targetWidth == null) {
targetWidth = (targetHeight * (width / height)).toInt();
} else if (targetHeight == null) {
targetHeight = (targetWidth * (height / width)).toInt();
}
_drawPixels(c, pixelData, width, height, pixelFormat, rowBytes, targetWidth, targetHeight);
}
}Click to expand details
c is the position that the image will be drawn at.
bytes is the pixel data that will be drawn to the canvas, represented as byte groups of 4 bytes per pixel. The color channels will be extrapolated based on the value of pixelFormat.
width and height are the dimensions of the image represented by the pixel data. If width * height * 4 is not equal to pixelData.length, an error will be thrown.
rowBytes is the number of bytes consumed by each row of pixels in the data buffer. If unspecified, it defaults to width multiplied by the number of bytes per pixel in the provided format.
The targetWidth and targetHeight` arguments specify the size of the output image, in image pixels. If they are not equal to the intrinsic dimensions of the image, then the image will be draw. If exactly one of these two arguments is specified, then the aspect ratio will be maintained while forcing the image to match the other given dimension. If neither is specified, then the image maintains its real size. (Debatable as to whether these parameters should be supported as they don't really fall into the whole "just draw these pixels" mentality of this method.)**
@andrewackerman assumed that the byte data will always have 32 bits per pixel, which makes sense because the only PixelFormat values that currently exist are bgra8888 and rgba8888, which use 32 bits per pixel.
The decodeImageFromPixels function is open to any PixelFormat that might be supported in the future:
rowBytes is the number of bytes consumed by each row of pixels in the data buffer. If unspecified, it defaults to width multiplied by the number of bytes per pixel in the provided format.
I think that this approach would make sense to also be used for the Canvas method (if it will use PixelFormat). Consequently, the assertion for width * height * 4 should probably be width * height * bytesPerPixel and similarly, bytes should be pixel data, represented as byte groups of bytesPerPixel bytes per pixel.
Use case
This feature request is about the
Canvasclass inengine/painting.dart.As extensively discussed in #31598, using regular canvas operations to draw pixel data is extremely slow.
This makes the only option we have decoding an image from our raw pixel data, e.g. using
decodeImageFromPixels. This has another problem, pointed out by @andrewackerman in the mentioned issue, which is that creating anImagein Dart is anasyncoperations.This means that we will have to postpone drawing our raw pixel data to the next frame if we wish to use real-time operations.
The
asyncImageproblemImages in Flutter currently have the inherent problem that rasterizing / decoding them is always an asynchronous operation. This is why using them for drawing pixel data does not work synchronously, but it also creates a whole nother class of problems:Any other canvas operation in Flutter that involves
Images is also asynchronous. So not only can you not draw pixel data synchronously, but you cannot rasterize anyPicturesynchronously, which is necessary for achieving various effects (e.g. this). Also Canvas.drawAtlas suffers from this problem.drawPixelsproposalHere is an example proposal (written by @andrewackerman in #31598) for directly drawing pixel data.
Note that this might not cover all use cases because often we do need to have a decoded image (
dart:ui.Image) available to us, e.g. when using Canvas.drawAtlas. It should, however, give an idea as to what we are looking for.Click to expand details
@andrewackerman assumed that the byte data will always have 32 bits per pixel, which makes sense because the only
PixelFormatvalues that currently exist arebgra8888andrgba8888, which use 32 bits per pixel.The
decodeImageFromPixelsfunction is open to anyPixelFormatthat might be supported in the future:I think that this approach would make sense to also be used for the
Canvasmethod (if it will usePixelFormat). Consequently, the assertion forwidth * height * 4should probably bewidth * height * bytesPerPixeland similarly,bytesshould be pixel data, represented as byte groups ofbytesPerPixelbytes per pixel.