Pete’s blog

Update on this blog

2020-08-21T00:00:00+00:00

Now that QuPath v0.2 has been released, the blog has served its purpose as a record of personal musings along the way.

The blog will linger around a bit longer so that links still work, but I don’t plan to add new posts.

To be clear, QuPath is still being very actively developed! It’s just this blog that’s being retired.

Please use the forum at https://forum.image.sc/tag/qupath for any questions and discussions – and stay up to date with QuPath-related happenings through

the homepage: https://qupath.github.io
the documentation: https://qupath.readthedocs.io
the YouTube channel: https://youtube.com/c/qupath
QuPath’s twitter: https://twitter.com/qupath
my twitter: https://twitter.com/petebankhead

QuPath v0.2.0-m8 available

2019-12-10T00:00:00+00:00

QuPath v0.2.0-m8 is now online!

So is v0.2.0-m7 but I didn’t get around to writing about it. Both emerged just before teaching workshops because I wanted to use the changes…

The updates are relatively minor (and therefore hopefully not too disruptive if you’re on v0.2.0-m6), but focus on

fixing niggly annoyances
laying the foundation for later work.

If the niggly annoyances were annoying you then I hope you’ll enjoy the benefits.

The changelogs are below:

Version 0.2.0-m8

Fixed repainting bug that could cause existing annotations to temporarily shift when drawing new annotations
Fixed Zoom to fit bug that meant command did not correctly resize and center the image in the viewer
Added Match viewer resolutions command to help synchronize when using multiple viewers
Improved tile export within a script
Improved interactive transformations
- More options for Interactive image alignment, including support to specify affine transform manually
- Log affine transform when applying ‘Rotate annotation’

Version 0.2.0-m7

Fixed bug that could cause QuPath to freeze when selecting objects with a mini-viewer active, see https://github.com/qupath/qupath/issues/377
Improved performance converting shapes to geometries, see https://github.com/qupath/qupath/issues/378
Improved robustness when drawing complex shapes, see https://github.com/qupath/qupath/issues/376
Improved stability when script directories cannot be found, see https://github.com/qupath/qupath/issues/373
Prompt to save each image when closing a project with multiple viewers active
Updated Rotate annotation command to use JTS

For full details on what has changed, see the commit log.

Changing the hierarchy

2019-11-17T00:00:00+00:00

QuPath’s object hierarchy has been an important part of the software since the start.

It’s where all the annotations, cells and other objects reside - arranged in a hierarchical way according to a few (ostensibly) simple rules. It has served the software well… mostly.

In v0.2.0 the behavior is changing. The purpose of this post is to summarize how, and expand on my recent tweeting about the subject. If you use QuPath, it’s probably important to know this.

Hmmm, so in a moment of wild abandon I deleted a couple of dozen lines of code to see if #QuPath would survive. Now suddenly images with large & complex annotations work *much* more smoothly... pic.twitter.com/13rLlsywtc
— Pete Bankhead (@petebankhead) November 13, 2019

Goals of the object hierarchy

When I first wrote the object hierarchy, I had two main intentions:

To be able to create a hierarchical representation of tissue structures
To be able to dynamically calculate measurements quickly (e.g. number of cells in a region)

Through QuPath you could see the relationships between objects under the ‘Hierarchy’ tab:

Conceptually, you might have something like this:

Measurements were made quickly by recursively ‘looking down the hierarchy’ to see what was there.

The goal was that it should feel intuitive… so you could use the software and it would do as you expect, without requiring you to think about it all the time.

In practice, it needed some defined rules that were sufficiently easy to understand and fast to apply. These rules were based on the Region of Interest (ROI) and object type, and the two main ones were:

A detection (e.g. a cell) was a child of an object if its centroid was inside that object’s ROI
An annotation was a child of an object if its entire ROI was inside that object’s ROI

There were a few subtleties (e.g. detections couldn’t parent annotations)… but that was basically it.

Problems with the object hierarchy

Those rules initially worked fine for my purposes: mostly tissue microarray analysis, sometimes analysis of larger tissue sections.

However as QuPath has become used more widely (> 60,000 downloads now…), more sophisticated commands were added (e.g. the pixel classifier), and the software was applied to more complicated applications, the limitations have become increasingly clear.

1. Performance

Firstly, automatically resolving the hierarchy can make things sloooow.

Detections aren’t really the problem: figuring out if a centroid is inside a ROI is fast. Doing this for a million cells may not be super-fast… but it is usually tolerable.

However figuring out if an annotation is fully inside another annotation can get very involved. Particularly as annotations become larger, more complex, and more numerous.

And now with the pixel classifier, it’s a lot easier to generate thousands of complex annotations.

QuPath’s performance with many annotations is a reoccurring topic on the forum, e.g. here and here.

To find the culprit when things are sloooow, VisualVM is fantastically helpful.

2. Ambiguity

If performance was the only problem, someone smarter than me might find a way to make it faster.

But that doesn’t solve a bigger issue: whenever objects overlap, the simple rules for resolving the hierarchy break down. It’s no longer clear what relationships objects should have with one another, and the results can be surprising.

In the image below, which ellipse is the ‘parent’ of the cells in the middle?

3. Confusion

In practice, the ambiguity wasn’t usually a problem. Very often, when using QuPath the hierarchy is basically ignored. Objects are requested like they all exist in a big list, and how they are represented internally doesn’t really matter.

But sometimes it could cause confusion.

For example, in the image below from v0.1.2, one would expect that the selected rectangle contains cells. But because it is a ‘child’ of the ellipse, and the ellipse is not completely contained within the outer rectangle, it doesn’t - at least as far as the hierarchy is concerned.

Changes in v0.2.0-m1

My first attempt to address this was to make the hierarchy less important.

This was achieved by always counting detections inside annotations based on their spatial location, not where they happened to fall in the hierarchy.

The change was introduced in v0.2.0-m1 and depended on Java Topology Suite (JTS). Applying it to the example above, we get a result that I think is considerably more intuitive.

JTS brings all manner of very useful operations that can be performed on ROIs - and here the key feature is a spatial cache that makes it much faster to find objects based on location. This has made the performance aspect of using a hierarchy less critical.

Putting this into practice has involved being able to convert QuPath ROIs to a corresponding JTS representation (a ‘Geometry’). This has proven deceptively difficult, but after some teething problems in m6 I think this now works robustly. If you find ROIs that fail, please let me know…

This change helped reduce (not eliminate) the confusion of the hierarchy, but didn’t help performance.

In fact, performance is probably worse, because finding cells to count using the spatial cache is still slower than simply looking down the hierarchy.

Best of both worlds?

You might ask: if the hierarchy is so troublesome, why keep it at all?

Well, I still think the first of my original intentions makes sense: sometimes it is really helpful to be able to represent hierarchical structures within an image. Having a flat list of objects (e.g. like ImageJ’s ROI Manager) can be very limiting. So it would be nice not to dispense with it entirely.

The proposed solution in v0.2.0-m6 is to give the user control.

The idea is this:

Keep QuPath’s hierarchy the same internally. Objects do still have parent/child relationships, and are stored in a hierarchical way.
When adding detections, continue to add them as children of the annotation/TMA core that was selected whenever they were first detected. This hasn’t changed.
When adding an annotation, make the default behavior to add it directly below the image: don’t try to resolve where it should fit in based on the old (and broken) rules about ‘completely contained’.
Make it optional to request that the object is inserted to the ‘right’ place in the hierarchy.

The last point is achieved by a new Insert in hierarchy command, which you can access in three ways:

Objects → Annotations… → Insert into hierarchy
Right-clicking with one or more annotations selected
Ctrl + Shift + I

The rule for insertion is almost the same as before but takes into consideration the area of the annotations that might be parents. In short, the annotation will be assigned to the smallest potential parent that completely contains its ROI.

Here, you can see it in action. Notice that by default, annotations are in a ‘flat’ list - but the old hierarchical arrangement can be achieved individually or all together either with the menu options or using shortcut keys.

You can also use Objects → Annotations… → Resolve hierarchy to try to resolve all objects in the image in this way. This is scriptable with resolveHierarchy(), but best avoid using it if you have a lot of objects.

Together, I hope these changes will retain the advantages of representing objects hierarchically without the previous problems, and thereby make it possible to use QuPath for increasingly complex tasks.

Outlook

I don’t know if the approach in v0.2.0-m6 is the best one, although it’s the best I’ve come up with so far.

I suspect refinements will be needed as people try it out and find limitations.

But I hope that by including it in a milestone version that can be easily tested, and explaining the reasons here, it can start a discussion that helps iron out any bugs - or identifies a better approach altogether.

Please let me know what you think!

QuPath v0.2.0-m6 available

2019-11-17T00:00:00+00:00

QuPath v0.2.0-m6 is now online!

This milestone contains two important bug fixes, one big change and several smaller improvements.

Warning! It’s generally not a good idea to switch between different QuPath versions. You should be able to read older data files within v0.2.0-m6, but the reverse might not be true. Backup often!

If you find bugs, please report them on GitHub or image.sc - don’t just stick with an older milestone that seemed to work!

For all other questions or discussions, please use image.sc.

Two important bug fixes

1) In v0.2.0-m5, the Positive per mm^2 measurement could be wrong. It would divide by the image area, not the annotation area - so to get the correct measurement you would need to calculate the value yourself based on Num positive and area. This bug only affected v0.2.0-m5, not other versions.

2) Converting ROIs with ‘extra pieces nested within holes’ into Java Topology Suite Geometries could lose the nested bits. Most ROIs aren’t like this, not all commands involve JTS, and not all JTS-related commands depend upon the conversion being perfect… so the impact of any error should be quite limited - but it has now been addressed.

An example of a troublesome ROI with pieces inside holes is shown below.

One big change: The object hierarchy

The change to the way objects are handled in QuPath is important enough to have its own blog post. I recommend reading it to understand what is different.

To summarize the outcome:

The user now has more control over how and when the object hierarchy is used - the default behavior is different (and simpler) than it had been.
Things that worked before should still work (if they don’t, please report the bug!), but you should be aware that the precise way in which they work may not be identical.
Some operations, especially involving many objects/annotations, should be a lot faster than they were previously.

Everything else

Most other changes are concerned with stability and performance.

New ROI type (impacts data files!)

As part of addressing the JTS bug above, AreaROI has been replaced with GeometryROI.

What does that mean? Well, complex ROIs created in v0.2.0-m6 may not be readable in earlier QuPath versions. So please avoid modifying files in m6 if you plan to continue with an earlier QuPath version.

But it also means that some processing should be much faster, since there is less conversion between different ROI representions required.

To avoid losing work from the past, it should still be possible to read older data files into m6. But you may find the performance isn’t as good when doing this, because QuPath will occasionally need to convert old-style ROIs to new ones - which can be slow if they are very complex.

Smoother drawing

In addition to changing how the hierarchy operates, the way annotations are painted on screen has been updated. This should hopefully mean the drawing tools (e.g. brush) work more smoothly, even when many other objects are nearby.

If you see any weird visual effects it suggests I’ve got this wrong… but I’m not aware of any problems.

Reorganized menus

Some menus have been reorganised, and some old commands removed (OpenCV-based cell detection, local binary patterns) on the understanding they were rarely/never used and just added clutter.

If I’ve removed something you need we can discuss it. The reorganization shouldn’t really matter, because you use Ctrl + L to find commands - right?!

Create points from detections (and keep the detections)

The Convert detections to points button under the Counting tool now uses nucleus centroids (rather than cell centroids) where appropriate, and makes deleting the previous detections optional.

If you have a lot of point annotations, QuPath should handle them better and faster.

New Detection centroid distances 2D command

Building on the last option, there’s a new command that can be used to find distances to cells with different classifications.

Note there is a small rounding error… so that the distance from an object to itself is typically recorded as slightly greater than 0. This may be improved in a future release.

Other changes & fixes

The pixel classifier shows live area measurements with ‘Classification’ output (in m5 this worked only with ‘Probability’ output)
Extra shortcuts, including Ctrl+Alt+A to select annotations, Ctrl+Alt+D to select detections
Undo/Redo and tile cache size information added to Memory Monitor
Added support for ImageWriters to write to output streams
Updated build script to Gradle 6.0
Use bioformats_package.jar rather than separate dependences (easier to upgrade separately)

For full details on what has changed, see the commit log.

QuPath news

2019-11-16T00:00:00+00:00

As v0.2.0-m6 is near, it’s time for another update on what’s going on & what’s planned.

Firstly, the big news is that QuPath has been selected as one of 42 projects to be funded by the Chan Zuckerberg Initiative under the Essential Open Source Software for Science program.

Alongside recent ISSF3 funding (University of Edinburgh/Wellcome Trust), things are looking rather a lot better in terms of supporting and developing the software, at least over the next 12 months or so.

Look out for more announcements in this regard… and perhaps also a QuPath workshop in 2020.

Meanwhile, back in 2019 the milestones continue. I appreciate that semi-frequent-but-somewhat-sporadic not-called-stable updates aren’t ideal and can be disruptive, but it has been a necessary consequence of me scrabbling around trying to find spare hours to write the code whenever I could.

However, there would have been far fewer of those coding hours were it not for the irrepressible superuser Research Associate answering many of the questions on https://forum.image.sc/.

Along with Sara & Zbigniew from the La Jolla Institute for Immunology, RA has also been testing the latest milestones ahead of their release - helping to catch bugs almost as quickly as I have created them. So huge thanks to them!

I do hope that, despite disruption, the milestone approach has been useful. I have found it helpful (albeit stressful…) to get feedback as it went along, rather than coding in isolation for a year and then turning up suddenly with a whole new version no one wants.

It also seemed to me practically necessary, because supporting the old stuff while my head is full of the new stuff I’m currently working on is… hard.

Although supporting a plethora of milestone versions is also hard. If you are stuck on an older milestone, please do think of updating… at least for your next project, and do check out the changelog with each new release to see if there have been any bugs fixed that might affect you.

I hope to get off the milestone track over the next few months - and then start on the next version.

To demystify the process and help with planning, here’s my todo list for v0.2.0, describing the things that I think are essential before it can be considered stable:

1. Refining existing features

Polishing up the pixel classifier.
- Improving the options for scripting, converting to objects and making measurements.
Rewriting the object classifier
- Bringing it more in line with the pixel classifier.
- Improve support for more complex classification (e.g. with multiplexed images)

2. Documentation

User documentation
- An all-new website that is more comprehensive and more maintainable.
Developer documentation
- Comprehensive javadocs hosted… somewhere.

3. Performance & stability

Handling more objects
- Continuous improvements in how to work with & query large numbers of objects.
Better progress monitoring
- Is it working? Has it crashed? Ideally QuPath would give more feedback & not leave us guessing.
Fixing bugs

There is a lot more I would like to include. But I think these three main topics are the most urgent, and the most useful to the most people.

There will certainly be more to come later. I hope the brighter funding outlook will help QuPath transition into becoming a more wholeheartedly community project, and less subject to the whims (and mistakes) of a single developer.

To that end, I’d ask that if you are using the software and care about where it goes, please get involved!

The software really shouldn’t be seen as something that just exists and sometimes changes, but rather something you can actively help shape and influence.

One way is through participating in image.sc - not solely to ask questions, but also to answer them and take part in wider discussions.

I suggest the forum first because I prefer to keep things public where possible, so that the discussions might help others as well. However, if you want to talk projects and collaborations where the forum isn’t a good fit and I should wear my academic hat, you can also find me at the University of Edinburgh.

In the end, QuPath is just a tool that is hopefully useful. My interest is in trying to apply image analysis to further biomedical research - and hope that QuPath can help meet some of the needs in this area.

QuPath v0.2.0-m5 available

2019-11-02T00:00:00+00:00

QuPath v0.2.0-m5 is now online!

This milestone contains even more changes than m4. The main ones are summarized below.

If you are using an earlier milestone, it is strongly recommended to update (after backing up your work, of course).

Many bugs and issues have been fixed. If you find new ones, please report them on GitHub or image.sc - don’t just stick with an older version that seemed to work!

For all other questions or discussions (not bugs), please use image.sc only.

More functional pixel classifier!

The biggest visible change is that the pixel classifier has taken a large step towards being useful.

Currently, it can be applied to do three main things:

Measure areas (including relative proportions of different classified regions)
Create objects (both annotations & detections)
Classify detections (based on the pixel classification at the detection centroid)

With a little bit of scripting, you can also export the classification as an indexed image or probability map to use elsewhere.

Apply classifiers to different images

When you start building a pixel classifier, you can now switch to different images to immediately see the result of applying the classifier there.

When using a project, you can also save your classifier and then use Classify → Pixel classification → Load pixel classifier to get it back on a new image. The classifier itself is stored within the project directory.

Create a training image

Note that you can’t continue training the same classifier while skipping between images: a new classifier is always trained from the annotations on the current image only.

However, there is a way to work around this:

Put all the images that should be used to train the classifier in a single project
Open each image and create annotations for one or more representative (or challenging) regions, and assign the annotations all the same classification.
Run Classify → Create project training image

For Step 2. I suggest using Objects → Annotations… → Specify annotation, and choosing the classification Region*.

The end result should be a single image composed of the regions you selected, which can then be used to train a classifier - immediately seeing how it performs across a (parts of) multiple images.

Create project training image works by extracting regions dynamically from the original image files - it doesn’t need to write a new image file. Usually this should be ok, but if you have memory troubles you may want to write the image as an OME-TIFF for better performance.

New features & feature overlays

The performance of the pixel classifier depends a lot on the features used for training. First, you’ll need to select these with the Edit button.

The features are basically filtered copies of the image. The classification result is determined from the values of pixels in the feature images - so ideally these features should accent the kind of thing the classifier should detect.

When training a classifier, you can now visualize the features in context using the drop-down menu on the bottom right of the training window.

This helps with the decision about which features are meaningful: use too many and the classifier can perform worse rather than better.

The following table gives an informal summary of where each feature might come in useful:

Feature	Purpose
Gaussian filter	General-purpose (color & intensity)
Laplacian of Gaussian	Blob-like things, some edges
Weighted deviation	Textured vs. smooth areas
Gradient magnitude	Edges
Structure tensor eigenvalues	Long, stringy things
Structure tensor coherence	‘Oriented’ regions
Hessian determinant	Blob-like things (more specific than Laplacian)
Hessian eigenvalues	Long, stringy things

But now that you can visualize them, probably best do that and draw your own conclusions.

Generally, I stick with Gaussian and add others if necessary. I usually avoid structure tensor features as a) I haven’t found them terribly useful, and b) they take more time and memory to compute.

Advanced options

There are some additional advanced options available for the pixel classifier.

Perhaps the most interesting is the ability to set a boundary strategy. In short, this means that any area annotation (e.g. rectangle, polygon) with a (non-ignored) classification, QuPath can treat the boundary of the annotation as if it had a different classification from the rest of the shape.

The purpose of these methods are to make it possible to train a classifier that tries to learn the separation between dense structures.

Furthermore, when choosing a boundary strategy you also need to specify the line thickness - which also influences the thickness of other line annotations, regardless of boundary strategy.

A class whose name ends with an asterisk (e.g. Region* or Ignore*) is ignored in some circumstances, e.g. when calculating areas after pixel classification or when generating objects.

A multichannel fluorescence image (thanks to La Jolla Institute for Immunology)…

…with a pixel classifier using a boundary class with thin boundaries…

…or thicker boundaries.

Limitations & plans

There is a lot more to be done with the pixel classifier, including:

Make it (more easily) scriptable
Finalize the format for saving/loading classifiers (this might change!)
Create a way to make measurements ‘stick’ in annotations
Improve the local normalization option (you can try it already, but I don’t recommend it…)
Add extra options for what to do with the output (e.g. classify cells based on partial overlap, subcellular detection)
Add support for color transforms (not just image channels)
Make it possible to be more selective about which features are calculated for which channels
Show progress bars for things that take a long time
Document how all this relates to deep learning…

Simple thresholding

Not everything requires a fully-trained pixel classifier. Sometimes a simple threshold will do.

Classify → Pixel classification → Create simple thresholder is intended for such times.

This command should ultimately replace Simple tissue detection. The concept is similar but the classifier approach provides much more control and interactive feedback.

QuPath’s pixel classification framework is intended to be flexible (which is where deep learning will fit in soon). Setting a threshold is really creating a classifier… just a very simple one.

Wand trickery

The Wand tool has a new trick that is useful with the pixel classifier (including simple thresholding).

If you press the Ctrl key then all smoothing will be disabled, and the wand will expand for pixels with exactly the same color. This is helpful if you have an opaque overlay on top of the image, since the wand will then more closely expand to the edge of any region.

Separately, there is an additional wand option in the preferences to choose slightly different behavior by being applied to images with different color transforms.

Scripting improvements

As QuPath’s internals move around with each new milestone, scripts break. It shouldn’t always be this way, but for now maintaining backwards compatibility would just require too much time and be too restrictive when it comes to improving the software overall.

A few changes have been made to help mitigate the trouble I cause.

Automatic imports

Previously, QuPath would import one special class: QPEx. This contains all the common scripting methods like getAnnotationObjects() and runPlugin().

But going beyond this requires using parts of QuPath strewn across hundreds more source code files. These need to be imported before they can be used.

To help with this, the script editor can now import a list of useful classes. The following example prints them out:

This means that you can use any methods that you like from these classes without a need to import.

The line static qupath.lib.gui.scripting.QPEx* means that the available methods are imported directly - that’s why you can type getSelectedObject() rather than QPEx.getSelectedObject().

This doesn’t happen for all classes because it would be hard to separate a mess of hundreds of methods: using the class name (e.g. ROIs, PathObjects) is useful to group them by theme.

Here’s an example to programmatically create annotations:

And here’s one showing an error message:

Previously, you’d have to explicitly import the class before being able to do this (and curse me if I’ve moved the class since you last needed it).

Now, you can leave it up to the default import statements instead, provided that Run → Include default imports is selected.

(Some) more informative errors

To ease the transition further, I have added some more informative error messages for common (Pete-induced) problems.

Finding methods

The default imports are a start, but you still need to know what the classes are and what they can do.

One important trick is Ctrl + space. Press this in the script editor and it will try a (very rudimentary) auto-complete. Press space repeatedly to cycle through the options.

Another is the new describe method, used as shown here:

It is, admittedly, a poor substitute for a full software development environment. But there’s always IntelliJ for serious scripting.

New methods

Several new built-in methods have also been added to QPEx make life easier:

// Set the pixel width & height to 0.5 microns
setPixelSizeMicrons(0.5, 0.5)

// Set the names of channels (in order)
// Since v0.2.0-m4 channel names are more important for commands
// such as cell detection
setChannelNames('First', 'Second', 'Third')

// Replace the classifications for all objects
replaceClassification('Tumor', 'Stroma')

// Export summary measurements for the entire image
def path = buildFilePath(PROJECT_BASE_DIR, 'export.txt')
saveImageMeasurements(path)

Syntax highlighting

A final change in the script editor is in the colors used for syntax highlighting: methods no longer turn orange (although keywords do), and some other colors are different.

The reason for this is…

Dark mode

As the eagle-eyed will have spotted, some of the screenshots have been rather darker than before.

QuPath has always had a kind of ‘Dark mode’, but it wasn’t terribly consistent; the script editor, for example, remained a glaring white, and now and again black-on-almost-black text wasn’t very legible.

v0.2.0-m5 now has a dark mode that should be better behaved.

A ‘Cancel’ button that cancels

Continuing the theme of ‘things that were in QuPath but didn’t really work’, when running commands (e.g. cell detection) over a large area there was always a large ‘Cancel’ button.

This was more aspirational than functional: it made a request that the command should stop running, but in practice the command would almost invariably ignore this and continue on its merry way to completion… however long it took.

Now, cancel actually can cancel.

And, even better, QuPath will attempt to restore the state to how it was before the command was started - rather than just aborting the mission awkwardly and leaving a lot of lingering tiles around.

As part of making this work, progress when running a command is displayed differently - with colors used to indicate tiles that are pending, processing or complete.

Better cell detection in large regions

Alongside the improved ‘Cancel’ button, work has been done to improve the consistency of how QuPath resolves overlaps whenever a large region is split into smaller tiled regions for parallel processing.

This is most evident with Cell detection (including Positive cell detection), where the size of the overlaps has also been increased.

Note that this does make some difference to the cells that are detected. Absolute counts when running cell detection in v0.2.0-m5 and earlier versions are likely to differ.

However, it addresses an issue whereby running cell detection with the same parameters on the same (large) region could sometimes give slightly different results because of differences in how the tile overlaps were resolved. Now, the results should be the same each time.

Major ROI revisions

ROIs have undergone major changes… with much effort expended in making it look like they have not changed at all.

The primary difference is that QuPath makes much more use of Java Topology Suite (JTS). You can convert any QuPath ROI into a JTS Geometry simply by calling ROI.getGeometry(), opening up a world of topological possibilities.

One of the visible outcomes of this is that some commands can now be much faster, such as converting pixel classifications to objects and splitting objects/filling holes. The Objects → Annotations… → Remove fragments command (previously called Refine annotations) gives one way to do this.

JTS is also used in the (renamed) Analyze → Calculate features → Distance to annotations 2D command, shown below with the new Svidro2 colormap (added due to popular¹ demand).

¹ From one person, but a significant one person.

Improvements when annotating in detail

When using the pixel classifier, I often find I want to be really careful when annotating close to boundaries. Two changes help with this:

The viewer permits zooming in to a higher magnification
The (area) drawing tools can now ‘snap’ to pixel coordinates

The snapping is a work-in-progress; in particular, when moving a snapped object it can end up ‘between’ pixel coordinates. But even in this imperfect form I personally find it to be very useful.

It is possible to turn off Use pixel snapping in the preferences.

Using the brush tool to create a ROI with snapping on:

Using the brush tool to create a ROI with snapping off:

In each case a binary mask is created and shown in ImageJ using the following script:

def roi = getSelectedROI()
def img = BufferedImageTools.createROIMask(roi, 1.0)
def imp = new ij.ImagePlus("Roi", img)
def request = RegionRequest.createInstance("Anything", 1, roi)
def roiIJ = IJTools.convertToIJRoi(roi, -request.getX(), -request.getY(), 1.0)
imp.setRoi(roiIJ)
imp.show()

While sub-pixel annotations can look pleasingly smooth, one of the benefits of snapping is that the binary images created are more predictable. Another is that they (usually) contain fewer vertices, so QuPath may run a bit faster if you have a lot of annotations.

Writing images

QuPath’s ability to export images has improved in two main ways.

From the user interface

File → Export images… has been revised to give three options:

Original pixels Export image regions selecting from a range of file formats, including JPEG, PNG, ImageJ TIFF, ImageJ ZIP and OME-TIFF.
Rendered RGB Export an RGB version of the image as it appears with the current viewer settings. This is somewhat like ImageJ’s Flatten command.
OME TIFF Similar to original pixels exclusively for OME-TIFF format, but with more fine-grained options to control the output (e.g. with compression type, pyramidal levels).

From scripts

Writing images in scripts used to be extremely cumbersome. Well no more!

In the event that your image is a manageable size to export in one go you can use

def server = getCurrentServer()
def path = buildFilePath(PROJECT_BASE_DIR, 'output.ome.tif')
writeImageRegion(server, path)

Often, you’ll want to extract a region. Here, I export the selected ROI downsampled by a factor of 4:

def server = getCurrentServer()
def path = buildFilePath(PROJECT_BASE_DIR, 'output.ome.tif')
def roi = getSelectedROI()
double downsample = 4
def region = RegionRequest.createInstance(server.getPath(), downsample, roi)
writeImageRegion(server, region, path)

Or to export the full image downsampled by a factor of 40:

def server = getCurrentServer()
def path = buildFilePath(PROJECT_BASE_DIR, 'output.ome.tif')
double downsample = 40
def region = RegionRequest.createInstance(server, downsample)
writeImageRegion(server, region, path)

You can change the file extension depending upon what file type you want, but beware that some file types don’t support all possible images (e.g. .jpg cannot be used successfully with a multiplexed, non-RGB fluorescence image).

Perhaps more excitingly, because there is not yet a proper way to export the pixel classification results as an image, you can use this script (which may also help with batch processing using Run for project).

This should be thought of as a workaround rather than the official way… but better than nothing.

// We do need to import this...
import qupath.lib.classifiers.pixel.PixelClassificationImageServer

// Load the classifier
def project = getProject()
def classifier = project.getPixelClassifiers().get('to test')

// Get the current image & create the (dynamically calculated) classified image
def imageData = getCurrentImageData()
def server = new PixelClassificationImageServer(imageData, classifier)

// Define an output path & file extension (may also choose .png, .ome.tif or something else)
def path = buildFilePath(PROJECT_BASE_DIR, 'classification')
mkdirs(path)
def name = getProjectEntry().getImageName()
path = buildFilePath(path, name + '.tif')

// Write the image
writeImage(server, path)

Memory monitor & show input

A while back I wrote a Memory Monitor script.

It’s now a built-in command under View → Show memory monitor.

For good measure, it’s accompanied by View → Show input on screen to make shortcuts visible.

Many more improvements and bug fixes

There are lots of other changes and fixes, including:

Fixed size estimate for large images (previously caused some images not to open)
Fixed bug that meant the file chooser forgot the last directory
Distance to annotations command supports line and point ROIs (not just areas)
Converting tile classifications to annotations #359
Calculating intensity features for RGB fluorescence #365
Setting stroke thickness, thanks to @jballanc #362

For full details on what has changed, see the commit log.

Building QuPath

2019-10-01T00:00:00+00:00

Building software can be tricky, but hopefully this won’t be - thanks to Gradle.

The following instructions are for the latest version of QuPath (v0.2.0-m4 at the time of writing). Things won’t necessarily work quite the same way with older versions.

The following instructions assume:

You’re starting from scratch
You’re not an expert in building software

If you are an expert, you’ll know which steps you can skip or amend.

Step 1: Installing prerequisites

Install a Java Development Kit (JDK)

QuPath requires OpenJDK 11 (or later*). You can get this through https://adoptopenjdk.net/

During installation you may be asked if you want to add the JDK to your PATH. To make things easier, I do.

If you can’t (e.g. because of some other Java software needing the PATH set to something else) I’m afraid I’ll leave resolving that up to you.

-OpenJDK 11 vs. OpenJDK 13

At the time of writing, Gradle does not yet support OpenJDK 13. Therefore you’ll need OpenJDK 11 to start Gradle, even if you ultimately use OpenJDK 13 when building QuPath itself.

Download the latest JPackage

To create the QuPath installer, you’ll need jpackage.

One day, jpackage is likely to be part of the JDK, and everything will be simpler.

But for now, you need to download an early access build separately from jdk.java.net.

Extract the contents (e.g. right-click and unzip) to a folder of your choosing and remember where it is because you’ll need the path to that folder later.

Step 2: Get the QuPath source code

You can get the QuPath source code from https://github.com/qupath/qupath

If you’re using either Mac or Windows, the following steps may help.

Install Atom (a text editor) from https://atom.io/
Install GitHub Desktop from https://desktop.github.com/
Press the Clone or download button at https://github.com/qupath/qupath and choose Open in Desktop

Step 3: Add a gradle.properties file

Go to GitHub Desktop, make sure that the ‘Current repository’ is ‘qupath’ and choose Repository → Open in Atom

Under the ‘Project’ tab on the left within Atom, right-click on the top ‘qupath’ and choose New File.

When prompted to Enter the path for the new file type gradle.propertes

Inside gradle.properties, enter the following text

packager=/path/to/jpackage

where you’ll need to replace /path/to/jpackage with actual path.

Note that jpackage will be somewhere inside the folder you extracted earlier. On Windows, it’s the path to jpackage.exe you are looking for and what you need to type should look something like this:
packager=C:/Users/myname/Documents/jpackage/jdk-14/bin/jpackage.exe

If you want, you can add extra information to gradle.properties to customize how QuPath is built (or how quickly it is built), e.g.

// Essential!
packager=/path/to/jpackage

// Optional
org.gradle.parallel=true // If you are impatient
org.gradle.java.home=/path/to/another/jdk // If you want to use a different JDK to build QuPath (e.g. OpenJDK 13)
request-git-tag=true // If you installed Git & want to know *exactly* what was the last commit prior to building the software

Step 4: Build QuPath with Gradle

Open the QuPath source directory in a command prompt

One way to do this is to go back to GitHub Desktop and choose Repository → Open in Command Prompt

You may be asked if you want to install Git.

You don’t have to (I think…), but if you do then you’ll be ask a lot of questions during the installation. One of them is to choose a text editor, where you can select Atom.

Run gradlew

At the command prompt, type the following:

gradlew createPackage

for Windows, or

./gradlew createPackage

for MacOS and Linux.

This will download Gradle and all its dependencies, so may take a bit of time (and an internet connection) the first time you run it.

If all goes well, you should see a triumphant message that the build was successful.

Afterwards, you should find QuPath inside the ./build/dist subdirectory. You may then drag it to a more convenient location.

Congratulations! You’ve now built QuPath, and can run it as normal from now on… at least until there is another update, when you can repeat the (hopefully painless) process.

Extras

Variations & troubleshooting

The code above should create everything you need to run QuPath.

If you want an installer instead, you can use

gradlew createPackage -Ptype=installer

Note that for this to work you’ll need to install WIX Toolset.

Inevitably, things will go wrong at some point. When this happens, it’s worth running

gradlew clean

to clean up old files that could be causing trouble.

You can also combine the options, e.g.

gradlew clean createPackage -Ptype=installer

will clean first, build QuPath & create an installer all in one go.

Getting the latest updates

Once you’ve built QuPath once, updating it to use the latest source code should be much easier.

In GitHub Desktop, see the right-most button on the main toolbar. This serves two purposes: to Fetch information about the latest changes (from GitHub) and to Pull the changes down to your computer.

If the option is Fetch origin, and you press the button then if there are any changes to pull the text on the button will switch to Pull origin with info about the number of changes available.

You can press it again to pull those changes, and then rebuild QuPath using gradlew if necessary.

You can also use the middle button in GitHub Desktop to switch ‘branches’. Branches basically make it possible to have different versions of the code in development in parallel while trying out new changes.

The following screenshot shows QuPath where I have checked out a specific branch called ‘pete-m5’.

If the changes prove worthwhile, the changes in ‘pete-m5’ will be merged back into the ‘master’ branch.

Running from an IDE

You should be able to import QuPath into any IDE (e.g. eclipse, IntelliJ) that supports Gradle.

I personally use eclipse for QuPath development, which allows me to run the software in debug mode - and even change the code while it is running.

To do this, I use Run → Debug As → QuPath.

To make this option available, you’ll first need to create a debug configuration with Run → Debug Configurations….

Within this dialog, I use the following options to control the available memory and set the working directory/Java library path.

Scripting in v0.2.0-m4

2019-08-21T00:00:00+00:00

There have now been four milestone releases of QuPath v0.2.0 over the past six months - where milestone means ‘not finished, potentially buggy, but usable enough to be worthwhile’.

All together they’ve been downloaded over 10,000 times so far. From the image.sc forum and GitHub issues a few things have become clear:

The new annotation tools in the milestone versions make it hard to go back to v0.1.2
Clear, reproducible bug reports are very useful
The pixel classifier is pretty popular… and really needs to be saveable and scriptable one day
People use scripting a lot. And the changes I’m making are breaking scripts a lot

This post will focus on the last point, and give a bit of a brain dump on what I’ve been doing.

Why all the breaking?
What’s up with the documentation?
And javadocs?
What do I need to know to write fantastic scripts?
What next?

Why all the breaking?

Basically, the QuPath code wasn’t (and isn’t) as neat and tidy as it ought to be

Until v0.1.2, I wrote it largely in isolation with myself being the only developer I had to satisfy. I was sometimes too forgiving towards me.

Then, immediately after v0.1.2 was available, I had an unfortunate enforced break from developing it.

Since returning to QuPath in 2018, I’ve been working through a long backlog of ideas and necessary improvements to make the software more powerful, robust and accessible to other developers… and (not yet entirely successfully) applying for funding to try to share the task a bit.

What’s up with the documentation?

I have mostly left the documentation at https://github.com/qupath/qupath/wiki untouched. It refers to v0.1.2, which is still considered the ‘stable’ version.

Lacking in time, I prioritize working on the code to try to stabilize v0.2.0 as soon as possible. But I do manage bursts of documentation in under 280 characters now and then.

Fortunately, the mysterious superuser Research_Associate has written a spectacular post on image.sc that serves as alternative documentation and contains some more up-to-date information.

And javadocs?

Javadocs are on my todo list… I’ve written thousands of them in the last few months, but there are many more to write. Since the API isn’t stable (and I haven’t got around to it) they aren’t hosted online yet.

However QuPath is a lot easier to build now than it was. The steps are described in the ReadMe.

If you get that working, you can also try

gradlew mergedJavadocs

to generate some local Javadocs for yourself.

What do I need to know to write fantastic scripts?

Here’s an overview of the general concepts in QuPath:

Your images may (and probably should) be organized in a Project
- Each image in the project is represented by a ProjectImageEntry
- When you open a ProjectImageEntry, you get an ImageData displayed in the viewer
  - The ImageData stores a few things, including:
    - The ImageType (e.g. Brightfield, Fluorescence)
    - Any ColorDeconvolutionStains required (if brightfield)
    - An ImageServer, for accessing pixels and metadata
    - A PathObjectHierarchy, containing PathObjects in a tree-like structure
      - Each PathObject contains a ROI and a MeasurementList

When you analyze an image in QuPath, you take your ImageData, access pixels from the ImageServer and try to represent what the image contains in the PathObjectHierarchy. Then you query the object hierarchy to extract some kind of summary measurements.

This is all fairly similar to v0.1.2, so fits with that documentation here.

The following sections describe how to use these concepts in practice. Some of this has changed a lot.

Default imports

In the Script Editor, there is an option Run → Include default bindings.

If this is selected, QuPath will add the following line to the top of your script:

import static qupath.lib.gui.scripting.QPEx.*

This means you’ve access to all the static methods in QPEx and QP directly.

All the examples below assume that QPEx is imported one way or another. If you don’t want to rely on the default import, just put that line at the top of your scripts explicitly.

Projects

The following simple script prints the names of all images in a project:

def project = getProject()
for (entry in project.getImageList()) {
    print entry.getImageName()
}

The script below is rather more interesting; it will

Open each image in turn
Extract the annotations from the hierarchy
Print the image name & annotation count per image

def project = getProject()
for (entry in project.getImageList()) {
    def imageData = entry.readImageData()
    def hierarchy = imageData.getHierarchy()
    def annotations = hierarchy.getAnnotationObjects()
    print entry.getImageName() + '\t' + annotations.size()
}

The extra logging messages generated when opening each image can be annoying, so you might want to print everything at the end instead. Creating a StringBuilder can help:

def sb = new StringBuilder()
def project = getProject()
for (entry in project.getImageList()) {
    def imageData = entry.readImageData()
    def hierarchy = imageData.getHierarchy()
    def annotations = hierarchy.getAnnotationObjects()
    sb << entry.getImageName() + '\t' + annotations.size() << '\n'
}
print sb.toString()

Both options are rather a lot slower than they need to be, because QuPath will go to the bother of constructing the full ImageData (including ImageServer) for every image - even though it never needs to actually access pixels.

You can avoid this as follows:

def project = getProject()
for (entry in project.getImageList()) {
    def hierarchy = entry.readHierarchy()
    def annotations = hierarchy.getAnnotationObjects()
    print entry.getImageName() + '\t' + annotations.size()
}

Note: These scripts won’t work in v0.1.2, where the process was much more awkward…

Accessing the current image

The above scripts can access images in a project, regardless of whether they are open in the GUI or not.

Often, you only need to access the image currently open. In that case, just use

def imageData = getCurrentImageData()
print imageData

This gets the image from the current viewer. It is equivalent to:

def viewer = getCurrentViewer()
def imageData = viewer.getImageData()
print imageData

In conjunction with Run → Run for project you often don’t need to loop through project images directly - just write a script for the current image, then run that script for all images with Run for project.

Accessing image metadata

To get image metadata, you’ll need the ImageServer:

def imageData = getCurrentImageData()
def server = imageData.getServer()
print server

In recent QuPath milestones, this is equivalent to:

def server = getCurrentServer()
print server

You can then query properties of the image. Simple ones can be accessed directly, e.g.

def server = getCurrentServer()
print server.getWidth() + ' x ' + server.getHeight()

All the key metadata exists in an ImageServerMetadata object:

def server = getCurrentServer()
print server.getMetadata()

Pixel sizes are in a PixelCalibrationObject (different from v0.1.2, where you got them directly from the server!):

def server = getCurrentServer()
def cal = server.getMetadata().getPixelCalibration()
print cal

As a shortcut, you can also use

def server = getCurrentServer()
def cal = server.getPixelCalibration()
print cal

In the past, pixels were either in microns or uncalibrated. In the future, QuPath might need to support other pixel units and so this assumption is a bit less critical than it was before. It is tempting to make pixel size requests more general and elaborate (always asking for units), but for now the need to request pixel sizes in microns is so common that there remain helper methods to do this:

def server = getCurrentServer()
def cal = server.getPixelCalibration()
print cal.getPixelWidthMicrons()
print cal.getPixelHeightMicrons()
print cal.getAveragedPixelSizeMicrons()

You can expect the result to be Double.NaN if the size information is not available. You can check for this using ‘standard’ Java/Groovy.

double myNaN = Double.NaN

// Two Java/Groovy-friendly ways to check values are 'usable'
print Double.isNaN(myNaN)
print Double.isFinite(myNaN)

// A bad way to check for NaN - confusing because Java & Groovy handle == differently
print (myNaN == Double.NaN) // Don't do this!

Accessing pixels

If you want pixels, you’ll get them as a Java BufferedImage.

To do so, you need to request them from a server with a RegionRequest. This includes the server path, a downsample factor and bounding box coordinates (defined in full resolution pixel units, with the origin at the top left of the image):

import qupath.lib.regions.*

def server = getCurrentServer()
def path = server.getPath()

double downsample = 4.0
int x = 100
int y = 200
int width = 1000
int height = 2000

def request = RegionRequest.createInstance(path, downsample, x, y, width, height)

def img = server.readBufferedImage(request)
print img

There are two reasons why QuPath uses RegionRequest objects:

You’d otherwise need to pass a lot of parameters to the readBufferedImage method
RegionRequests can be (and are) used as keys for an image cache

In any case, the above script assumes a single-plane image. If you may have a z-stack, you can define the z-slice and time point in your request:

import qupath.lib.regions.*

def server = getCurrentServer()
def path = server.getPath()

double downsample = 4.0
int x = 100
int y = 200
int width = 1000
int height = 2000
int z = 0
int t = 0

def request = RegionRequest.createInstance(path, downsample, x, y, width, height, z, t)

def img = server.readBufferedImage(request)
print img

If you have a selected object with a ROI in the image, you can also use that to create the request:

import qupath.lib.regions.*

def server = getCurrentServer()
def roi = getSelectedROI()
double downsample = 4.0
def request = RegionRequest.createInstance(server.getPath(), downsample, roi)
def img = server.readBufferedImage(request)
print img

The server path previously was an image path and it could be used to construct a new server… but this is no longer the case. Rather, the key thing now is that it must be unique for a server, since it is used for caching.

server.getPath() may be renamed to server.getID() or similar in the future to reflect this.

Creating ROIs

Previously, there were public constructors for ROIs. You shouldn’t use these now!

Rather, use the static methods in the ROIs class.

This will require specifying the z-slice and timepoint. To avoid passing lots of parameters (and getting the order mixed up), you should instead use an ImagePlane object:

import qupath.lib.roi.ROIs
import qupath.lib.regions.ImagePlane

int z = 0
int t = 0
def plane = ImagePlane.getPlane(z, t)
def roi = ROIs.createRectangleROI(0, 0, 100, 100, plane)
print roi

There are various different kinds of ROI that can be created, including with createEllipseROI, createPolygonROI, createLineROI.

Creating objects

To actually make a ROI visible, it needs to be part of an object.

The PathObjects class helps in a similar way to ROIs - again, you shouldn’t create objects using constructors directly.

This script creates a new annotation with an ellipse ROI, and adds it to the hierarchy for the current image (using the QPEx.addObject() method):

import qupath.lib.objects.PathObjects
import qupath.lib.roi.ROIs
import qupath.lib.regions.ImagePlane

int z = 0
int t = 0
def plane = ImagePlane.getPlane(z, t)
def roi = ROIs.createEllipseROI(0, 0, 100, 100, plane)
def annotation = PathObjects.createAnnotationObject(roi)
addObject(annotation)

To create a detection rather than an annotation, you’d use createDetectionObject.

Putting it together with previous sections, to create square tiles across an entire image for the current ImagePlane we could use:

import qupath.lib.objects.PathObjects
import qupath.lib.roi.ROIs
import qupath.lib.regions.ImagePlane

def imageData = getCurrentImageData()
def plane = getCurrentViewer().getImagePlane()
def server = imageData.getServer()
int tileSize = 1024

def tiles = []
for (int y = 0; y < server.getHeight() - tileSize; y += tileSize) {
    for (int x = 0; x < server.getWidth() - tileSize; x += tileSize) {
        def roi = ROIs.createRectangleROI(x, y, tileSize, tileSize, plane)
        tiles << PathObjects.createAnnotationObject(roi)
    }
}
addObjects(tiles)

Working with BufferedImages

Once you have a BufferedImage, you are already in Java-land and don’t need QuPath-specific documentation for most things.

Scripts like this one to create binary images can then help with one major change. Previously, you had to do some awkward gymnastics to convert a ROI into a java.awt.Shape object. That’s now easier:

def roi = getSelectedROI()
def shape = roi.getShape()
print shape

Here’s a script applying this to pull out a region from an RGB image for a selected ROI, and show that region in ImageJ along with a new binary mask:

import qupath.lib.regions.*
import ij.*
import java.awt.Color
import java.awt.image.BufferedImage

// Read RGB image & show in ImageJ (won't work for multichannel!)
def server = getCurrentServer()
def roi = getSelectedROI()
double downsample = 4.0
def request = RegionRequest.createInstance(server.getPath(), downsample, roi)
def img = server.readBufferedImage(request)
new ImagePlus("Image", img).show()

// Create a binary mask & show in ImageJ
def shape = roi.getShape()
def imgMask = new BufferedImage(img.getWidth(), img.getHeight(), BufferedImage.TYPE_BYTE_GRAY)
def g2d = imgMask.createGraphics()
g2d.scale(1.0/request.getDownsample(), 1.0/request.getDownsample())
g2d.translate(-request.getX(), -request.getY())
g2d.setColor(Color.WHITE)
g2d.fill(shape)
g2d.dispose()
new ImagePlus("Mask", imgMask).show()

The mask is generated using Java’s built-in rendering of Shapes.

Working with ImageJ

The above is fine for simple cases, but fails to make the most of ImageJ. It doesn’t set the image metadata, so there’s no way to relate back extracted regions to where they were originally in the whole slide image. It also doesn’t work in general for multichannel images.

If you want to apply ImageJ scripting in QuPath, it is best to let QuPath take care of the conversion. IJTools is the new class that helps with that (or IJExtension to interact directly with the GUI).

The following script is similar to that above, but works for multichannel images and sets ImageJ properties. It also doesn’t create a mask directly, but rather converts the QuPath ROI so that further processing (e.g. to generate the mask) can be performed in ImageJ.

import qupath.lib.regions.*
import qupath.imagej.tools.IJTools
import qupath.imagej.gui.IJExtension
import ij.*

// Request an ImageJ instance - this will open the GUI if necessary
// This isn't essential, but makes it it possible to interact with any image that is shown
IJExtension.getImageJInstance()

// Read image & show in ImageJ
def server = getCurrentServer()
def roi = getSelectedROI()
double downsample = 4.0
def request = RegionRequest.createInstance(server.getPath(), downsample, roi)
def pathImage = IJTools.convertToImagePlus(server, request)
def imp = pathImage.getImage()
imp.show()

// Convert QuPath ROI to ImageJ Roi & add to open image
def roiIJ = IJTools.convertToIJRoi(roi, pathImage)
imp.setRoi(roiIJ)

This introduces another class: PathImage.

This is basically a wrapper for an image of some kind (here, an ImageJ ImagePlus) along with some calibration information.

Often we don’t need the PathImage wrapper, but here we keep it so that we can pass it to IJTools.convertToIJRoi(roi, pathImage) later.

Working with OpenCV

Rather than BufferedImage or ImagePlus objects, perhaps you prefer to write your processing code using OpenCV.

In v0.1.2, QuPath used the default OpenCV Java bindings - which were troublesome in multiple ways. Now, it uses JavaCPP.

However, although OpenCV can be nice to code with it can also be hard to code with interactively. Therefore in QuPath there are helper functions to help convert from OpenCV to ImageJ when necessary. The following shows this in action:

import qupath.lib.regions.*
import qupath.imagej.tools.IJTools
import qupath.opencv.tools.OpenCVTools
import org.bytedeco.opencv.opencv_core.Size
import static org.bytedeco.opencv.global.opencv_core.*
import static org.bytedeco.opencv.global.opencv_imgproc.*
import ij.*

// Read BufferedImage region
def server = getCurrentServer()
def roi = getSelectedROI()
double downsample = 4.0
def request = RegionRequest.createInstance(server.getPath(), downsample, roi)
def img = server.readBufferedImage(request)

// Convert to an OpenCV Mat, then apply a difference of Gaussians filter
def mat = OpenCVTools.imageToMat(img)
mat2 = mat.clone()
GaussianBlur(mat, mat2, new Size(15, 15), 2.0)
GaussianBlur(mat, mat, new Size(15, 15), 1.0)
subtract(mat, mat2, mat)
mat2.close()

// Convert Mat to an ImagePlus, setting pixel calibration info & then show it
def imp = OpenCVTools.matToImagePlus(mat, "My image")
IJTools.calibrateImagePlus(imp, request, server)
imp.show()

Manipulating ROIs

Having met IJTools and OpenCVTools, it may be nice to know there are also RoiTools and PathObjectTools classes. In all cases, these contain static methods that may be useful.

Here, we see how to create and merge two ROIs:

import qupath.lib.roi.ROIs
import qupath.lib.roi.RoiTools
import qupath.lib.objects.PathObjects
import qupath.lib.regions.ImagePlane

def plane = ImagePlane.getDefaultPlane()
def roi1 = ROIs.createRectangleROI(0, 0, 100, 100, plane)
def roi2 = ROIs.createEllipseROI(80, 0, 100, 100, plane)

def roi3 = RoiTools.combineROIs(roi1, roi2, RoiTools.CombineOp.ADD)
def annotation = PathObjects.createAnnotationObject(roi3)
addObject(annotation)

Working with Java Topology Suite

It’s quite possible that your ROI manipulation wishes extend beyond what QuPath ROIs support directly.

Fortunately, you can shift to the fabulous Java Topology Suite - rather easily. Here’s an example that will convert a QuPath ROI to a JTS Geometry, expand it, and then create a new annotation from the result:

import qupath.lib.objects.PathObjects
import qupath.lib.roi.jts.ConverterJTS;

def roi = getSelectedROI()

def geometry = roi.getGeometry()
geometry = geometry.buffer(100)
def roi2 = ConverterJTS.convertGeometryToROI(geometry, roi.getImagePlane())

def annotation = PathObjects.createAnnotationObject(roi2)
addObject(annotation)

Serialization & JSON

QuPath v0.1.2 uses Java’s built-in serialization quite a lot for saving/reloading things.

This is quite compact and easy to use, but horrendous to maintain and impractical for sharing data with anything written in another programming language. Still, it lives on in .qpdata files… for now.

JSON, by contrast, is text-based and readable. v0.1.2 already used JSON for project files (.qpproj), but now uses it increasingly where possible.

JSON is not always appropriate (e.g. attempting to represent a full object hierarchy containing a million objects as JSON would be horribly slow, complex and memory-hungry) but it is generally more maintainable and portable compared to Java serialization.

The library QuPath uses to help with JSON is Gson. Gson makes it beautifully straightforward to turn almost anything into a JSON representation and back… if you know exactly what Java class is involved.

Here’s a Groovy example that doesn’t rely on anything QuPath-specific:

import com.google.gson.GsonBuilder

def gson = new GsonBuilder()
        .setPrettyPrinting()
        .create()

def myMap = ['Hello': 1, 'I am a map': 2]
print myMap

def json = gson.toJson(myMap)
print json

def myMap2 = gson.fromJson(json, Map.class)
print myMap2

You may notice that the map you get back doesn’t look exactly the same when printed… what looked like an integer may now look like a floating point value. But otherwise they match.

In practice, when working with generic classes and subclasses things can rapidly become a lot more complex, bringing in the world of type hierarchy adapters and the like.

I have spent a long time battling with these things in the hope that you won’t have to. Rather than creating your own Gson object, you can request one from QuPath that is pre-initialized to work with a lot of the kind of structures you’ll meet in QuPath.

import qupath.lib.io.GsonTools

boolean prettyPrint = true
def gson = GsonTools.getInstance(prettyPrint)

println 'My ROI'
println gson.toJson(getSelectedROI())

println()
println 'My object'
println gson.toJson(getSelectedObject())

println()
println 'My server'
println gson.toJson(getCurrentServer())

To convert back, you’ll need to supply the appropriate QuPath class. Because of the magic GsonTools does for you, this doesn’t need to be the exact class - you can use PathObject and get a detection or annotation as appropriate.

import qupath.lib.objects.PathObject
import qupath.lib.io.GsonTools

boolean prettyPrint = true
def gson = GsonTools.getInstance(prettyPrint)

// Get the selected object & convert to JSON
def pathObject = getSelectedObject()
def json = gson.toJson(pathObject)
print json

// Create a NEW object from the JSON representation
def pathObject2 = gson.fromJson(json, PathObject)
print pathObject2

// Confirm that we really do have a *different* object
if (pathObject == pathObject2)
    print 'Objects are the same'
else
    print 'Objects are NOT the same'

// Add the object to the hierarchy to check it matches, with a name to help
pathObject2.setName('The one from JSON')
addObject(pathObject2)

This should also work for an ImageServer:

import qupath.lib.images.servers.ImageServer
import qupath.lib.io.GsonTools

boolean prettyPrint = true
def gson = GsonTools.getInstance(prettyPrint)

def server = getCurrentServer()
def json = gson.toJson(getCurrentServer())
def server2 = gson.fromJson(json, ImageServer)

print server
print server.class

print server2
print server2.class

if (server == server2)
    print 'Servers ARE the same'
else
    print 'Servers ARE NOT the same'

if (server.getMetadata() == server2.getMetadata())
    print 'Metadata IS the same'
else
    print 'Metadata IS NOT the same'

Figuring out a JSON way to represent ImageServers has taken up rather a lot of my time recently… but so far this seems to be working.

Note however that not everything can be converted to JSON. For example, you can’t do this with an object hierarchy or an ImageData. You also probably don’t/shouldn’t want to, given the efficiency issues mentioned above.

Nevertheless, where possible QuPath tries to use a representation that may be used elsewhere.

For example, for ROIs and objects, QuPath follows the GeoJSON specification. This should (although I haven’t tried…) make it possible to exchange regions with other software, e.g. get them into Python via Shapely.

Using GeoJSON does impose some limitations; notably, ellipses become polygons.

GsonTools also aims to wrap around OpenCV’s JSON serialization, e.g.

import org.bytedeco.opencv.global.opencv_core
import org.bytedeco.opencv.opencv_core.Mat
import qupath.lib.io.GsonTools

boolean prettyPrint = true
def gson = GsonTools.getInstance(prettyPrint)

def mat = Mat.eye(3, 3, opencv_core.CV_32FC1).asMat()
print gson.toJson(mat)

Eventually this will make OpenCV classifiers JSON-serializable within QuPath and finally avoid needing to retrain existing classifiers when reloading them.

What next?

This post gives an overview of QuPath scripting for v0.2.0-m4. The API has changed considerably before… albeit with quite a lot of resemblance.

The goal is to make everything more logical and easier to extend. Scripting should be intuitive, and allow you do interact with the data in whatever way you like. Admittedly there is more work to do to achieve this… but it’s a start.

For the main classes you’ll need, it should be possible to at least guess their names. For example, you should avoid using direct constructors for PathObjects and ROIs and use the static classes instead. Similarly, if you see Tools and the end of a classname you can be fairly sure it contains more static methods useful for manipulating objects of whatever type the classname begins with.

Here’s a list of some classes you might want to import, and their current locations:

import qupath.imagej.gui.IJExtension
import qupath.imagej.tools.IJTools
import qupath.lib.gui.scripting.QPEx
import qupath.lib.images.servers.ImageServer
import qupath.lib.io.GsonTools
import qupath.lib.objects.PathObjects
import qupath.lib.objects.classes.PathClassFactory
import qupath.lib.objects.classes.PathClassTools
import qupath.lib.regions.ImagePlane
import qupath.lib.regions.RegionRequest
import qupath.lib.roi.ROIs
import qupath.lib.roi.jts.ConverterJTS
import qupath.opencv.tools.OpenCVTools

Lest they move again or you need others, you can find them by searching on GitHub or (much easier) setting up QuPath with an IDE like IntelliJ.

In writing this post, I already see things in the API that I don’t like and want to refactor soon… and probably will. When I do, I’ll try to remember to update these scripts.

All of this remains a work-in-progress, but at least now there is some documentation for anyone who wants to script in the meantime.

QuPath v0.2.0-m4 now available

2019-08-20T00:00:00+00:00

QuPath v0.2.0-m4 is now online!

As usual, many changes while moving towards the next release. Here’s an overview of the main ones.

Please report bugs on GitHub - and for all other questions or discussions please use image.sc.

Cell detection & intensity measurements

The Positive cell detection command is no longer just for hematoxylin & DAB staining. It can now be used for any brightfield or multiplexed image where ‘positive’ cells can be identified with a single feature.

Original TIFF image if LuCa-7color_[13860,52919]_1x1component_data.tif from Perkin Elmer, part of the Bio-Formats sample data available under CC-BY 4.0.

As part of these changes, both Positive cell detection and Add intensity measurements now use channel names rather than numbers*.

This should make it rather a lot easier to decipher measurements later. The intensity command has also been adjusted to allow measurements (including textures) to be calculated within the cell nucleus only.

*-Be especially cautious when scripting - Add intensity measurements still uses channel order in scripts.

(Also, please avoid using scripts recorded using old versions of QuPath because of these changes.) -*

Generate image pyramids from separate fields of view

Perhaps the most exiting feature for anyone working with multispectral images…

A new QuPath script is able to merge separate spectrally-unmixed fields of view from a Vectra® Polaris™ system, and output a multichannel, pyramidal OME-TIFF.

So starting from a single field of view…

…it is possible to merge multiple fields of view…

…even resulting in a full whole slide image.

For this to work it is not necessary for all fields of view to be present. If part of the image is not scanned the pixel values in all channels will simply be 0.

Original images shown above thanks to Dr Sandrine Prost, University of Edinburgh.

The script is here. If it proves useful enough, it might become a built-in feature in a future release.

More options when importing images to projects

v0.2.0-m3 introduced a new image import dialog when working with projects, and the ability to drag & drop multiple images for import.

One of the options is to rotate the image. The option existed in m3… in m4 it actually works.

This differs from View → Rotate image, which is really only for visualization. Rotation on import treats the image as if it had been saved in a rotated way (i.e. with coordinates changed accordingly). It is designed to resolve cases, especially with Tissue Microarrays, where the image is oriented unexpectedly.

In general, you really need to set the Image type in QuPath before doing any analysis, since some commands (e.g. cell detection) change behavior based on the type. This can be auto-estimated (you can turn this on/off in the preferences), but it’s wise to take control since the estimate might be wrong.

You could always set the type later by double-clicking on the entry under the ‘Image’ tab:

Now you can - and probably should - specify the type in the import dialog.

You can also now specify the ‘server type’ or ‘image provider’ (same thing). Basically, this is the library that reads the pixels and image metadata, e.g. OpenSlide or Bio-Formats.

Previously, QuPath would make the decision for you but this wouldn’t always be optimal because some formats are supported by multiple readers in different ways. For example, an .ndpi file might open faster using OpenSlide - but if it’s a z-stack then you’d need to use Bio-Formats to access all the slices.

Now, if you know what you want you can take control by specifying the image reading library during import. (Choose wrong and the image might not open… but choose appropriately and it may open more quickly and more reliably. Or just let QuPath choose as before.)

Finally, QuPath is designed to handle very large, ‘pyramidal’ images. Such images contain data at with multiple resolutions stored in the same file.

QuPath is also designed for smaller microscopy images (mostly 2D).

But until now, it hasn’t worked particularly will with large images that aren’t pyramidal. To mitigate this, there’s now an option for QuPath to generate a ‘pseudo-pyramid’ if needed.

QuPath does still need to access all the full-resolution pixels at least once. Still, you can use this to substantially improve performance for moderately large images, e.g. 10,000 x 10,000 pixels. For even bigger non-pyramidal images conversion to a pyramidal OME-TIFF is a better idea.

Repair projects by finding missing/moved images

QuPath’s projects contain the following file paths (as URIs):

Absolute paths to all images
Absolute path to the project itself

That should all work fine so long as your images stay in exactly the same place.

The trouble is if images are moved or deleted, or you want to share a project with someone else who can’t access the images in the same way. Previously, this meant that in QuPath you needed to manually edit the project.qpproj file to make things work again.

This started to improve in v0.2.0-m3. Now whenever a project is being opened, QuPath will:

Check if all the absolute image paths match real files
If a file is missing
- Check if the project has moved, and if so try relative paths to the missing images
- Show a dialog for the user to fix any remaining broken paths / accept suggested changes

When the dialog appears you can fix a path by double-clicking on it and locating the missing file.

In practice, this only actually worked on Windows. In v0.2.0-m4 it should work on Linux and Mac as well - but it is still rather awkward if you have a lot of images.

So now there’s a new button ‘Search’. This is for cases where multiple files are missing (perhaps all of them?) and you need to update many paths at once.

You can click on ‘Search’ and choose a directory containing the missing images. QuPath will then try to find all the files in the directory with names that correspond to those that are missing.

But actually it can be a little more useful than that.

You don’t really need to give it the directory that contains the images; you can also give QuPath a ‘parent’ directory. QuPath will then search recursively through sub-directories (up to a limit, currently set to 8 levels) for missing images.

This should ease the pain of working with QuPath projects across multiple computers, or when files move around.

New ways to expand lines (including tumor margins!)

An earlier post on this blog discussed how to Create annotations around the tumor.

Basically, QuPath gives options to grow, shrink, add and subtract areas, and by combining these it is possible to help analysis around tumor margins etc.

This worked fine for areas, but wasn’t really designed to work with lines.

The new Polyline tool introduced in previous milestone releases helps:

However, applying the Expand annotations to line/polyline annotations would expand in all directions - and ‘remove interior’ had no effect (since the line had no thickness):

This meant that if you want a ‘thick’ boundary, you would need to carefully draw a line along the middle of where you want the boundary to be - rather than at the edge of the structure being annotated.

Now, in v0.2.0-m4 two things have changed:

There is an option to set the ‘line cap’ to be round (as before), flat or square
If selected, ‘Remove interior’ will remove an approximately 1-pixel thick line

Changing the line cap to Flat gives a quite different expansion:

Combined with ‘Remove interior’, this is enough to make a separation.

You can then use Objects → Annotations → Split annotations…

…which allows you to separate the inner/outer expansion regions.

Then you can delete one if you don’t want it, and refine any weird bits that might have emerged (like at the left side on this image).

More control of Measurement Maps

The perceptually uniform colormaps added in v0.2.0-m3 can now be accessed from a drop-down menu.

Your preferred colormap is also remembered across measurements & when QuPath is restarted.

You can also now double click on the labels that show the min/max display range to specify exact values. This is particularly useful if you want to create figures with the same display range for comparison.

Improved memory management & image caching

Under the hood, QuPath’s method of caching tiles has changed substantially (now using a Guava cache).

Also, the amount of memory given to tile caching is now adjustable in the preferences.

Previously, this was limited to 4 GB. This was plenty for most RGB images, but with multiplexed images it could fill up quickly. Now if you have a lot of RAM available, it’s possible to improve performance by having a much larger tile cache… which also helps when generating the OME-TIFFs as described above.

Many more improvements and bug fixes

There are lots of other changes and fixes, including:

You can access scripts from a project directly through the Automate menu again
Send region to ImageJ with overlay now only sends objects in the cropped field of view being sent (rather than lots of objects from anywhere in the image)
The font size can be adjusted in the preferences for the cursor location label
QuPath allows smaller regions to be extracted from images (down to a single pixel) - which helps with Send region to Image or adding intensity features
Code formatting in the script editor is asynchronous… which can be considered better or worse. It crashes less anyway.
A bug preventing SLIC superpixels from being generated as been fixed

For full details on what has changed, see the commit log.

QuPath v0.2.0-m3 now available

2019-07-25T00:00:00+00:00

Originally posted at forum.image.sc and even more originally tweeted here

QuPath v0.2.0-m3 is now online!

There are many changes, mostly below the surface - but quite a few above the surface too.

The biggest differences involve projects, which have been completely revised. It’s now easier to import images (just drag & drop multiple files) & QuPath tries harder to help resolve broken paths (e.g. when your images move).

You can finally set pixel sizes by double-clicking the values under the ‘Image’ tab - optionally with a line or area drawn to help figure out the calibration

Sidenote: you’ll need to be using a project to store the changes…

A major reason to update projects was to add ‘fancy’ images in the future, which aren’t stored directly in a single file but might need to be calculated dynamically (e.g. by applying some kind of transform). This could be to crop regions, concatenate channels, normalize colors…

Create sparse image from project is a first example. It generates a new image consisting of regions extracted from other images in a project (all the ones with an annotation classified as ‘Region*’). I expect this to be useful one day… maybe to train a pixel classifier.

Speaking of which, the pixel classifier remains a work-in-progress. Not yet save-able or scriptable, it has been extensively revised & can calculate some new features (including some in 3D!) on its path to being useful.

New, perceptually uniform colormaps have also arrived (thanks go here):

There’s more control when you send images to ImageJ - including z-stacks

Bio-Formats v6.2.0 is included - which means improved WSI support thanks to the the Bio-Formats team & Glencoe Software. QuPath’s ability to write pyramidal OME-TIFF images is also much improved (albeit still mostly accessible by scripting).

The brush & wand are better. Previously, exuberant brushing would result in lots of disconnected circles - now things are smoother. Also, with a compatible graphics tablet the Wand is pressure sensitive - and both tools behave a better with touchscreens.

v0.2.0-m2

v0.2.0-m3

And lastly, especially for coders, there’s a first implementation of a GsonTools class to help convert various QuPath-related things to & from a JSON representation. Here, it’s applied to convert a QuPath object into a GeoJSON feature.

Pete’s blog

Update on this blog

QuPath v0.2.0-m8 available

QuPath v0.2.0-m8 is now online!

Version 0.2.0-m8

Version 0.2.0-m7

Changing the hierarchy

Goals of the object hierarchy

Problems with the object hierarchy

1. Performance

2. Ambiguity

3. Confusion

Changes in v0.2.0-m1

Best of both worlds?

Outlook

QuPath v0.2.0-m6 available

QuPath v0.2.0-m6 is now online!

Two important bug fixes

One big change: The object hierarchy

Everything else

New ROI type (impacts data files!)

Smoother drawing

Reorganized menus

Create points from detections (and keep the detections)

New Detection centroid distances 2D command

Other changes & fixes

QuPath news

1. Refining existing features

2. Documentation

3. Performance & stability

QuPath v0.2.0-m5 available

QuPath v0.2.0-m5 is now online!

More functional pixel classifier!

Apply classifiers to different images

Create a training image

New features & feature overlays

Advanced options

Limitations & plans

Simple thresholding

Wand trickery

Scripting improvements

Automatic imports

(Some) more informative errors

Finding methods

New methods

Syntax highlighting

Dark mode

A ‘Cancel’ button that cancels

Better cell detection in large regions

Major ROI revisions

Improvements when annotating in detail

Writing images

From the user interface

From scripts

Memory monitor & show input

Many more improvements and bug fixes

Building QuPath

Step 1: Installing prerequisites

Install a Java Development Kit (JDK)

Download the latest JPackage

Step 2: Get the QuPath source code

Step 3: Add a gradle.properties file

Step 4: Build QuPath with Gradle

Open the QuPath source directory in a command prompt

Run gradlew

Extras

Variations & troubleshooting

Getting the latest updates

Running from an IDE

Scripting in v0.2.0-m4

Contents

Why all the breaking?

What’s up with the documentation?

And javadocs?

What do I need to know to write fantastic scripts?

Default imports

Projects

Accessing the current image

Accessing image metadata

Accessing pixels