Skip to content

Pyramidal zarr writer#1964

Merged
petebankhead merged 23 commits intoqupath:mainfrom
Rylern:pyramidal-zarr-writer
Oct 14, 2025
Merged

Pyramidal zarr writer#1964
petebankhead merged 23 commits intoqupath:mainfrom
Rylern:pyramidal-zarr-writer

Conversation

@Rylern
Copy link
Contributor

@Rylern Rylern commented Aug 7, 2025

Add a zarr writer that (in that order):

  • Write the full resolution level by looking at the input image.
  • Write the second level by looking at the first level pixels that were just written (the input image is not considered anymore).
  • Write the third level by looking at the second level pixels that were just written.

This provides less flexibility than the existing OMEZarrWriter but is less likely to throw out of memory errors.

This PR only adds this new writer (PyramidalOMEZarrWriter). Other changes to existing classes were made to reuse as much code as possible between the two zarr writers.

I create this PR now to discuss about the public API (is only one writeImage() function enough?), but it's a draft because tests need to be added.

@Rylern Rylern marked this pull request as ready for review August 29, 2025 16:10
@petebankhead petebankhead self-requested a review September 17, 2025 13:01
Copy link
Member

@petebankhead petebankhead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works for me, but I find it a bit confusing that the API is quite different for OMEZarrWriter and PyramidalOMEZarrWriter, even when it seems like they should be doing something similar.

For example,

  • OMEZarrWriter.Builder requires downsamples to be passed in a method, whereas PyramidalOMEZarrWriter requires them to be passed to the constructor; if they aren't needed at construction, then I think defaulting to a downsample of 1 would be fine.
  • OMEZarrWriter.writeImage takes no arguments, uses a background thread and returns immediately, while PyramidalOMEZarrWriter requires a path and is blocking.

Adding example scripts to the PR showing how the writers should be used would be good. I'd expect the scripts to look almost identical, so that it's easy to switch from one implementation to the other. If that's not possible, we'll also need to explain that clearly on ReadTheDocs.

I've added some comments on specific things.

@Rylern
Copy link
Contributor Author

Rylern commented Sep 23, 2025

I addressed all comments. The public APIs of OMEZarrWriter and PyramidalOMEZarrWriter are now similar (this required some API changes on OMEZarrWriter). Here is a script on how to use them:

import qupath.lib.images.writers.ome.zarr.OMEZarrWriter
import qupath.lib.images.writers.ome.zarr.PyramidalOMEZarrWriter

var path = "/path/to/img.ome.zarr"
var server = getCurrentServer()

new OMEZarrWriter.Builder(server)   // or new PyramidalOMEZarrWriter.Builder(server)
    .downsamples(1, 4, 16)
    .build(path)
    .writeImage()

@Rylern
Copy link
Contributor Author

Rylern commented Sep 24, 2025

It is now possible to provide downsamples without including 1.

@petebankhead
Copy link
Member

Thanks, I think there's a bug in PyramidalOMEZarrWriter (I couldn't replicate it with OMEZarrWriter) connected to tile caching.

I ran the following code for 2 different images (you'll recognize them...), deleting the zarr in between:

import qupath.lib.images.writers.ome.zarr.OMEZarrWriter
import qupath.lib.images.writers.ome.zarr.PyramidalOMEZarrWriter

var server = getCurrentServer()
var path = buildPathInProject("anything.ome.zarr")

new PyramidalOMEZarrWriter.Builder(server)
    .downsamples(16, 64)
    .build(path)
    .writeImage()

It produced an image like this:

image

I initially thought that it was using the full image width and height, but those do in fact seem correctly downsampled (7936 x 4608px) - and the image displays properly when zoomed in so that pixels are requested from the highest available resolution.

@petebankhead
Copy link
Member

See also #2012 - it's a longstanding bug, and perhaps fixing it would be enough.

@Rylern
Copy link
Contributor Author

Rylern commented Oct 14, 2025

I fixed a bug occurring when the first requested downsample isn't one.

That doesn't fix your issue though. When I run your workflow, I also get a bit a CMU-1 in the output image. However, if I clear the tile cache between the two writes, then the output image is correct. So, I think this is clearly linked with #2012.

How do you think we should fix that? Automatically clear the cache if two consecutive writes to the same location are detected?

@petebankhead
Copy link
Member

Please check out #2013 and see if that fixes the issue - I'm not sure how exactly the modified time behaves with directories.

If you're temporarily creating a BioFormatsImageServer, then I think it would make sense to clear the cache specifically for that when it's serves its purpose - although I'm not sure how easy it is to get access to the required objects to call that method.

@Rylern
Copy link
Contributor Author

Rylern commented Oct 14, 2025

#2013 fixes the issue.

@petebankhead
Copy link
Member

I'll merge both then, thanks!

@petebankhead petebankhead merged commit c87373f into qupath:main Oct 14, 2025
3 checks passed
@Rylern Rylern deleted the pyramidal-zarr-writer branch October 14, 2025 12:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants