release mode is AOT, so its likely not due to JIT. Instead it could be a mix of memory locality, scheduling (due to the async nature of asset loading), and garbage collection. Its reasonable that the cost varies per count.

If you wanted to rule out cost of loading the asset from memory, you could instead cache the raw bytes and measure the cost of utf8 conversion followed by JSON parsing.

Also, you won't really get very real results running this on an emulator. I would investigate what the performance is like on a real device too

release mode is AOT, so its likely not due to JIT. Instead it could be a mix of memory locality, scheduling (due to the async nature of asset loading), and garbage collection. Its reasonable that the cost varies per count.

Ooh I failed to consider memory locality. I would love to learn how scheduling and GC would affect this though 👀

If you wanted to rule out cost of loading the asset from memory, you could instead cache the raw bytes and measure the cost of utf8 conversion followed by JSON parsing.

This could work. However, one thing that irks me is that moves further away from testing the actual implementation/interface (AssetImage.obtainKey). Perhaps this is unavoidable in this situation without moving to some more complicated E2E test.

Also, you won't really get very real results running this on an emulator. I would investigate what the performance is like on a real device too

Gee, you sure weren't kidding!

(edit: typos)

If you wanted to rule out cost of loading the asset from memory, you could instead cache the raw bytes and measure the cost of utf8 conversion followed by JSON parsing.

I just made this change, and the more I think about it, the more it makes sense in that it most closely tests the decoding+parsing change we are looking to make. Still, one thing this neglects is that the binary encoding may result in faster load from storage since the file will be smaller. Do you think this might be worth putting back into the test loop? The thing I wouldn't like about this is, if we ever changed the iteration count in the future, it would invalidate* previous data due to storage locality--though I do wonder if changing the iteration count invalidates data regardless.

*By "invalidate", I mean that it would no longer be useful to compare metrics from before the change to after the change.

I think it depends on how much variation you see when running on a real device. If the numbers are fairly stable around ~30 iterations, I think its fine if it doesn't scale completely linearly. We only need to worry about the delta between these numbers and the updated numbers