benchmarking qupath-wsinfer

[kaczmarj]

i am using this issue to provide info on the data i use when benchmarking wsinfer in qupath.

image: TCGA-3C-AALI-01Z-00-DX1.F6E9A5DF-D8FB-45CF-B4BD-C6B76294C291.svs (link to image for download on Genomic Data Commons)

geojson representation of the square region. the region has an area of 100 millimeters squared. (the geojson coordinates are in pixels, and it is the equivalent of 100 mm2 in the whole slide image).

i will update this issue with my running times for a 12th gen i5 cpu and an nvidia 2080ti gpu, both in windows 11.

here are running times.... on an i5-12600K, it took 6 minutes 37 seconds. on an NVIDIA RTX 2080Ti, it took 40 seconds. please see other environment details below. this was using the WSI and ROI in this issue.

• os: windows 11
• qupath v0.4.4
• wsinfer extension v0.2.1
• model "breast-tumor-resnet34.tcga-brca"
• ROI of 100 mm^2 (see geojson below)
• openslide loader (i think -- i'm not sure how to change this to bioformats)

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "id": "9724d4e3-cd6d-4630-a474-4d71e35db065",
      "geometry": {
        "type": "Polygon",
        "coordinates": [
          [
            [
              25471,
              25869
            ],
            [
              65471,
              25869
            ],
            [
              65471,
              65869
            ],
            [
              25471,
              65869
            ],
            [
              25471,
              25869
            ]
          ]
        ]
      },
      "properties": {
        "objectType": "annotation"
      }
    }
  ]
}

here is a groovy script i was using to measure the runtime:

import groovy.time.TimeCategory

// Set device (cpu, gpu, or mps)
qupath.ext.wsinfer.ui.WSInferPrefs.deviceProperty().setValue("gpu");
// Set number of parallel workers
qupath.ext.wsinfer.ui.WSInferPrefs.numWorkersProperty().setValue(8);

// ---
selectAnnotations()
def timeStart = new Date()
qupath.ext.wsinfer.WSInfer.runInference("kaczmarj/breast-tumor-resnet34.tcga-brca")
def timeStop = new Date()
duration = TimeCategory.minus(timeStop, timeStart)
println duration

[petebankhead]

Thanks @kaczmarj !

openslide loader (i think -- i'm not sure how to change this to bioformats)

It's easiest if you create a project - then the docs describe how to specify the library to read the image here.

// Set number of parallel workers
qupath.ext.wsinfer.ui.WSInferPrefs.numWorkersProperty().setValue(8);

Is there any particular reason for choosing 8 - is it based on the number of processors, or testing different values?

[kaczmarj]

Is there any particular reason for choosing 8 - is it based on the number of processors, or testing different values?

the choice of 8 was arbitrary. i have 8 physical cores on my machine, but the main reason is i wanted the value to be set explicitly, because i assume it will have an effect on runtimes.

i get a strange issue when opening the svs file with bioformats. i commented on an existing forum post that describes the same behavior: https://forum.image.sc/t/problem-about-opening-some-svs-slides-in-qupath-v0-3-1-bio-formats-6-8-0/61404/28?u=kaczmarj

[alanocallaghan]

Is there any particular reason for choosing 8 - is it based on the number of processors, or testing different values?

the choice of 8 was arbitrary. i have 8 physical cores on my machine, but the main reason is i wanted the value to be set explicitly, because i assume it will have an effect on runtimes.

i get a strange issue when opening the svs file with bioformats. i commented on an existing forum post that describes the same behavior: https://forum.image.sc/t/problem-about-opening-some-svs-slides-in-qupath-v0-3-1-bio-formats-6-8-0/61404/28?u=kaczmarj

I don't think you should parallelize code by default, if somebody has a 2 core CPU and the scheduler isn't playing nice then that could freeze their system. Or, if you're going to set it by default, I prefer using the number of available cores/threads minus one or two

[petebankhead]

i get a strange issue when opening the svs file with bioformats. i commented on an existing forum post that describes the same behavior: https://forum.image.sc/t/problem-about-opening-some-svs-slides-in-qupath-v0-3-1-bio-formats-6-8-0/61404/28?u=kaczmarj

I saw the thread - can you check https://github.com/qupath/qupath/releases/tag/v0.5.0-rc2 to see if it has already been addressed?

I don't see this issue on my Mac on v0.4.4 or v0.5.0-rc2. The pyramid levels are different from OpenSlide (including 98.8) but the appearance is fine at all levels.

Spoke too soon! I see it, but it's new... it happens with the file I've just downloaded, but not with the older one I had on my computer...

I don't think you should parallelize code by default, if somebody has a 2 core CPU and the scheduler isn't playing nice then that could freeze their system. Or, if you're going to set it by default, I prefer using the number of available cores/threads minus one or two

I currently have the default set to min(4, num_processors) here.

Previously, the default was 1 but I found that increasing it would improve performance quite dramatically... up to a limit.

Probably the default should be updated to subtract 1 from the number of processors.

[MITEL-UNIUD]

Hi, I did the same test on MacBook Pro 14" 2021 (M1 Pro: 10 CPU cores, 16 GPU cores), 32GB unified memory, MacOS Sonoma 14.5, QuPath 0.5.1, WSInfer 0.30. I suppose openslide loader since Bio-formats is not yet supporting Apple Silicon.
I also played with batch size by adding this line:
qupath.ext.wsinfer.ui.WSInferPrefs.batchSizeProperty().setValue(16);.
Running times:

• mps (GPU): 68 seconds (the best was 59, but I was not able to reproduce it). This can be obtained with different workers and batch sizes (up to 24-32), +/- some second.
• CPU: it seems to accept much larger batch sizes (tested up to 256). Best time 7'40" (10 workers, 256 batch size). 8 workers and default batch size: 12'07".

[petebankhead]

Thanks!

I suppose openslide loader since Bio-formats is not yet supporting Apple Silicon.

This only affects a couple of file formats (at least .ndpi and .czi). The rest should work on Apple Silicon. With the image open in QuPath, you can check the 'Server type' under the 'Image' tab in QuPath to convert if Bio-Formats or OpenSlide was used.