CUDARelativeDifferencePrior by Imraj-Singh · Pull Request #1408 · UCL/STIR

Imraj-Singh · 2024-04-25T00:36:44Z

First attempt at integrating CudaRelativeDifferencePrior class into STIR. The idea was create a child of RelativeDifferencePrior to override the compute_value/compute_gradient methods with the CUDA-accelerated counterpart, and then add an set_up_cuda method. This is in a first-draft state with work left to do:

There are a lot of warnings with regards to -- virtual function override intended?.
Only a 3x3x3 neighbourhood size is implemented.
Threads and blocks sizes are not set well.
The set_up_cuda is a bit clunky (I did have it override the set-up previously but got a bit confused with CRTP things)

At present it does run with a test comparing outputs of CPU/GPU versions. I could include it in the pull-request but I need to spend sometime making the ctest I guess.

KrisThielemans

Thanks!

There are a lot of warnings with regards to -- virtual function override intended?.

please give an example

The set_up_cuda is a bit clunky (I did have it override the set-up previously but got a bit confused with CRTP things)

This won't help, as STIR will never call set_up_cuda. There's too much CRTP going on here (if any). Do you mean calling the baseclass set_up()?

I think it'd make more sense to use recon_buildblock/CUDA than CUDA_stir (why have stir twice?)

KrisThielemans · 2024-04-25T07:08:32Z

+  find_package(CUDAToolkit)
+  if (CUDAToolkit_FOUND)
+    set(CMAKE_CUDA_ARCHITECTURES "all")
+    find_package(CUDA REQUIRED)
+    include_directories("${CUDA_INCLUDE_DIRS}")
+    set(STIR_WITH_STIR_CUDA ON)
+    if(STIR_WITH_STIR_CUDA)
+      enable_language(CUDA)
+      message(STATUS "STIR CUDA support enabled. Using CUDA version ${CUDAToolkit_VERSION}.")
+    endif()
+  endif()


I think this is the other way around. I'd probably use https://cmake.org/cmake/help/latest/module/CheckLanguage.html#module:CheckLanguage first, if present, then enable_language, then find_package(CUDAToolkit) which probably needs a REQUIRED.

Before all this, I'd do a check on CMAKE_VERSION. Certainly at least 3.18, but 3.23 if we want to use "all" for [CMAKE_CUDA_ARCHITECTURES](https://cmake.org/cmake/help/latest/prop_tgt/CUDA_ARCHITECTURES.html#prop_tgt:CUDA_ARCHITECTURES) (which we do). If the CMake version is too old, generate FATAL_ERROR with the message to set DISABLE_STIR_CUDA=ON`.

Also, don't use include_directories but target_include_directories

done here: d3f3732

It's this line:

if (STIR_WITH_CUDA) target_include_directories(stir_registries PRIVATE ${CUDA_INCLUDE_DIRS}) endif()

KrisThielemans · 2024-04-25T07:25:29Z

+
+START_NAMESPACE_STIR
+
+static void


let's not repeat those. We can make them static members of RelativeDifferencePrior or wherever they need to be

I guess this is with regards to the compute weights @KrisThielemans ?

Imraj-Singh · 2024-04-25T11:40:17Z

There are a lot of warnings with regards to -- virtual function override intended?.

please give an example
Looks like this:

/home/user/sirf/STIR/src/include/stir/Array.h(177): warning #997-D: overloaded function "stir::VectorWithOffset<T>::resize [with T=stir::Array<1, float>]" is hidden by "stir::Array<num_dimensions, elemT>::resize [with num_dimensions=2, elemT=float]" -- virtual function override intended?
          detected during:
            instantiation of class "stir::Array<num_dimensions, elemT> [with num_dimensions=2, elemT=float]" 
(109): here
            instantiation of class "stir::Array<num_dimensions, elemT> [with num_dimensions=3, elemT=float]" 
/home/user/sirf/STIR/src/include/stir/recon_buildblock/RelativeDifferencePrior.h(177): here

/home/user/sirf/STIR/src/include/stir/Array.h(180): warning #997-D: overloaded function "stir::VectorWithOffset<T>::grow [with T=stir::Array<2, float>]" is hidden by "stir::Array<num_dimensions, elemT>::grow [with num_dimensions=3, elemT=float]" -- virtual function override intended?
          detected during instantiation of class "stir::Array<num_dimensions, elemT> [with num_dimensions=3, elemT=float]" 
/home/user/sirf/STIR/src/include/stir/recon_buildblock/RelativeDifferencePrior.h(177): here

/home/user/sirf/STIR/src/include/stir/Array.h(177): warning #997-D: overloaded function "stir::VectorWithOffset<T>::resize [with T=stir::Array<2, float>]" is hidden by "stir::Array<num_dimensions, elemT>::resize [with num_dimensions=3, elemT=float]" -- virtual function override intended?
          detected during instantiation of class "stir::Array<num_dimensions, elemT> [with num_dimensions=3, elemT=float]" 
/home/user/sirf/STIR/src/include/stir/recon_buildblock/RelativeDifferencePrior.h(177): here

/home/user/sirf/STIR/src/include/stir/recon_buildblock/RelativeDifferencePrior.h(152): warning #997-D: function "stir::GeneralisedPrior<DataT>::set_up(const std::shared_ptr<const DataT> &) [with DataT=stir::DiscretisedDensity<3, float>]" is hidden by "stir::RelativeDifferencePrior<elemT>::set_up [with elemT=float]" -- virtual function override intended?
          detected during:
            instantiation of class "stir::RelativeDifferencePrior<elemT> [with elemT=float]" 
/home/user/sirf/STIR/src/include/stir/recon_buildblock/CUDA/CudaRelativeDifferencePrior.h(12): here
            instantiation of class "stir::CudaRelativeDifferencePrior<elemT> [with elemT=float]" 
/home/user/sirf/STIR/src/recon_buildblock/CUDA/CudaRelativeDifferencePrior.cu(376): here

KrisThielemans · 2024-04-25T11:59:53Z

ok. Nothing to do with this PR. Would you mind creating a separate issue for it? Please add which compiler you're using. (It's a false positive, but would be nice to have a clean way to prevent the warning)

Imraj-Singh · 2024-04-25T15:06:12Z

ok. Nothing to do with this PR. Would you mind creating a separate issue for it? Please add which compiler you're using. (It's a false positive, but would be nice to have a clean way to prevent the warning)

I am having some issues getting it to register correctly in the stir registries. Perhaps it's related to that? I am not sure I am correctly making CudaRelativeDifferencePrior a child of RelativeDifferencePrior. I am a bit hesistant to open a issue as I think something is a bit wrong at the moment.

KrisThielemans · 2024-04-25T15:10:34Z

 target_include_directories(stir_registries PRIVATE ${STIR_INCLUDE_DIR})
 target_include_directories(stir_registries PRIVATE ${Boost_INCLUDE_DIR})
+if (STIR_WITH_CUDA)
+  target_include_directories(stir_registries PRIVATE ${CUDA_INCLUDE_DIRS})


I really hope we don't need CUDA_INCLUDE_DIRS for the registries. STIR .h files ideally have no CUDA includes at all. It's going to create trouble.

I thought we'd want the target to be the registeries as we need the cuda_runtime.h when building cuda files that will be in the registry... Also I couldn't quite figure out where else to put the target_include_directories, where would you suggest?

As far as I can see,cuda_runtime.h should NOT be included in src/include/stir/recon_buildblock/CUDA/CudaRelativeDifferencePrior.h, but in the .cxx/.cu. That means that the registries wouldn't need to know about the CUDA include dir, therefore you don't need the target_include_directories` at all. (The .cu files need it, but they get it by depending on CUDA::cudart target).

KrisThielemans · 2024-04-25T15:11:05Z

+
+#include "stir/recon_buildblock/RelativeDifferencePrior.h"
+#include "stir/DiscretisedDensity.h"
+#include <cuda_runtime.h>


move to cxx

this is done but dim3 is cuda specific, I changed it for a native struct and removed the cuda_runtime include, aren't the other two needed?

KrisThielemans · 2024-04-25T15:13:14Z

ok. Nothing to do with this PR. Would you mind creating a separate issue for it? Please add which compiler you're using. (It's a false positive, but would be nice to have a clean way to prevent the warning)

I am having some issues getting it to register correctly in the stir registries. Perhaps it's related to that?

no

I am not sure I am correctly making CudaRelativeDifferencePrior a child of RelativeDifferencePrior. I am a bit hesistant to open a issue as I think something is a bit wrong at the moment.

Please show errors/problems. I have no time to try this myself. sorry.

Imraj-Singh · 2024-04-25T15:25:37Z

ok. Nothing to do with this PR. Would you mind creating a separate issue for it? Please add which compiler you're using. (It's a false positive, but would be nice to have a clean way to prevent the warning)

I am having some issues getting it to register correctly in the stir registries. Perhaps it's related to that?

no

I am not sure I am correctly making CudaRelativeDifferencePrior a child of RelativeDifferencePrior. I am a bit hesistant to open a issue as I think something is a bit wrong at the moment.

Please show errors/problems. I have no time to try this myself. sorry.

Thanks @KrisThielemans

I was running ./STIR/build/src/utilities/stir_list_registries and did not see the prior under GeneralisedPrior<DiscretisedDensity<3,float>>. The output looks like this:

WARNING: FactoryRegistry:: overwriting previous value of key in registry.
     key: None
------------ ProjectorByBinPair --------------
Matrix
None
Parallelproj
Separate Projectors
------------ ForwardProjectorByBin --------------
Matrix
None
Parallelpr etc etc

Is the overwriting causing the issue, or I am not sure if I have updated all the neccessary files to register it correctly. I guess part of my concern is whether I updated all the neccessary files so that STIR can include the new class in the registeries. This is a quite new and pretty confusing

KrisThielemans · 2024-04-25T15:35:04Z

Do you have the equivalent of https://github.com/UCL/STIR/blob/8ced2d73933420457e0bc76074964b7d4ff00f0c/src/recon_buildblock/RelativeDifferencePrior.cxx#L140C1-L141C97
Maybe these days this can be done in the .h file (which would be cleanest)

Imraj-Singh · 2024-04-25T15:50:08Z

+// Explicit template instantiations
+template class stir::CudaRelativeDifferencePrior<float>;
+template <typename elemT>
+const char* const CudaRelativeDifferencePrior<elemT>::registered_name = "Cuda Relative Difference Prior";


@KrisThielemans yes I added an overwrite it

please use exact same syntax as in the lines quoted above. C++ is a bit tricky there. (or try and see what happens if you put the initialiser in the .h file). If that doesn't work, comment out the one in RelativeDifferencePrior, or remove that from the registry completely. I don't think we've ever tried deriving from an existing class in the registry.

@KrisThielemans is the reason for not deriving from an existing class in the registry, because it's just a bad idea? I couldn't think of a better idea...

Not sure what you mean. Ideally we can derive from an existing class, but if it doesn't work, we'll have to find another way...

So first thing to do is to fix your syntax and see what happens.

KrisThielemans · 2024-05-13T22:03:41Z

@Imraj-Singh any chance you can pick this up? I think my suggestions might resolve your trouble. Also, it'll need a merge with master of course.

I'd like to you the main CUDA support stuff elsewhere...

Imraj-Singh · 2024-05-15T16:27:00Z

@Imraj-Singh any chance you can pick this up? I think my suggestions might resolve your trouble. Also, it'll need a merge with master of course.

I'd like to you the main CUDA support stuff elsewhere...

Just back I will spend some time the next couple of days.

Imraj-Singh · 2024-05-17T13:15:08Z

@KrisThielemans I am still having troubles with the registering the class derived from an already registered class... The error is below.

/usr/bin/ld: ../recon_buildblock/librecon_buildblock.a(CudaRelativeDifferencePrior.cu.o):(.data.rel.ro.local+0x0): multiple definition of stir::CudaRelativeDifferencePrior::registered_name'; ../CMakeFiles/stir_registries.dir/recon_buildblock/recon_buildblock_registries.cxx.o:(.data.rel.ro.local+0x0): first defined here
collect2: error: ld returned 1 exit status
gmake[3]: *** [src/utilities/CMakeFiles/stir_list_registries.dir/build.make:220: src/utilities/stir_list_registries] Error 1
gmake[2]: *** [CMakeFiles/Makefile2:3108: src/utilities/CMakeFiles/stir_list_registries.dir/all] Error 2
gmake[1]: *** [CMakeFiles/Makefile2:3115: src/utilities/CMakeFiles/stir_list_registries.dir/rule] Error 2
gmake: *** [Makefile:894: stir_list_registries] Error 2`

I'll try remove the registering in the RelativeDifferencePrior

Imraj-Singh · 2024-05-17T13:21:08Z

@KrisThielemans I tried removing the registering of RelativeDifferencePrior but still having errors. FYI I took all the content out of the CUDA files to try and diagnose the issue.

KrisThielemans · 2024-06-02T01:02:21Z

/usr/bin/ld: ../recon_buildblock/librecon_buildblock.a(CudaRelativeDifferencePrior.cu.o):(.data.rel.ro.local+0x0): multiple definition of stir::CudaRelativeDifferencePrior::registered_name;

This is because your initialisation of registered_name is in the .h file. That means it will exist in every .cxx/.cu file that includes, so there are multiple definitions.

Possible solutions:

move it to the .cu
move it to a new .cxx

try to initialise inside the class definition, not outside. This apparently needs C++-17 https://en.cppreference.com/w/cpp/language/static, but we've moved to that anyway (you might have to merge master)

class CudaRelativeDifferencePrior : public RelativeDifferencePrior<elemT> {
  public:
      using RelativeDifferencePrior<elemT>::RelativeDifferencePrior;
      // Name which will be used when parsing a GeneralisedPrior object
      inline static const char* const registered_name = "CudaRelativeDifferencePrior;
     ...

If the latter works, it's my preferred solution (by far clearest for the user), and we could do it for the rest of STIR now as well.

KrisThielemans · 2024-06-02T01:07:58Z

Also, please add check_language(CUDA) and if it isn't there, set DISABLE_STIR_CUDA=ON, possibly with a warning. At present, if you don't have CUDA, it just fails without any help what to do.

…ks to Kris

Problems with numerical gradients. I had to revert the change to GeneralisedPrior::set_penalisation_factor which set _already_set_up to false, as that made test_priors fail on all priors (as the test changes the penalisation factor)

KrisThielemans · 2024-07-07T01:24:01Z

This should be nearly done now. For a 256x251x211 image with 2 cylinders, cuda_rdp_tests gives

norms: diff of gradients: 5.09262e-07 org: 64.8277 rel: 7.85563e-09
The Prior Value (CUDA) = 3385.94
The Prior Value (CPU) = 3385.94
Difference = 3.06435e-05
Value CUDA time = 1.35823s
Value CPU time = 8.6639s
Gradient CUDA time = 0.555103s
Gradient CPU time = 12.3153s

It also works for a smaller image with much more structure.

Sadly, I have problems in test_priors.

----- test Cuda Relative Difference Prior_no_kappa_with_eps  --> Gradient

INFO: Computing gradient

INFO: Computing objective function at target

INFO: Computing gradient of objective function by numerical differences (this will take a while)
Error : unequal values are 1.71268 and 1.61074. gradient
Numerical gradient test failed with for Cuda Relative Difference Prior_no_kappa_with_eps
Writing diagnostic files Cuda Relative Difference Prior_no_kappa_with_eps_target.hv, *gradient.hv (and *numerical_gradient.hv if full gradient test is used)
----- test Cuda Relative Difference Prior_no_kappa_with_eps  --> Hessian-vector product for convexity
----- test Cuda Relative Difference Prior_no_kappa_with_eps  --> Hessian against numerical
Error : unequal values are 1.19587 and 1.21844. Hessian
Numerical-Hessian test failed with for Cuda Relative Difference Prior_no_kappa_with_eps prior
----- testing Hessian-vector product (accumulate_Hessian_times_input)

INFO: Comparing Hessian*dx with difference of gradients

INFO: Computing gradient

ERROR: CUDA error in compute_gradient kernel execution: invalid argument
terminate called after throwing an instance of 'std::runtime_error'
  what():
ERROR: CUDA error in compute_gradient kernel execution: invalid argument

Aborted

Doing a visual check on the analytic vs numerical gradient indicates they are very similar, but numerically the diff is rather large.

(left images have range ~[-15.5,5.6] while the diff on the right has range ~[-.5,5])

The kernel invalid argument is in

compute-sanitizer --tool memcheck src/recon_test/test_priors
...
========= Program hit cudaErrorInvalidValue (error 1) due to "invalid argument" on CUDA API call to cudaMemcpy.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x34d163]
=========                in /usr/lib/wsl/drivers/nvdd.inf_amd64_fa2520b946613838/libcuda.so.1.1
=========     Host Frame:cudaMemcpy [0x6b8aa]
=========                in /usr/local/cuda-12.2/targets/x86_64-linux/lib/libcudart.so.12
=========     Host Frame:void stir::array_to_device<3, float>(float*, stir::Array<3, float> const&) [0x28b8a4]
=========                in /home/kris/devel/buildCondaGT/builds/STIR/build/src/recon_test/test_priors
=========     Host Frame:stir::CudaRelativeDifferencePrior<float>::compute_gradient(stir::DiscretisedDensity<3, float>&, stir::DiscretisedDensity<3, float> const&) [0x28bab3]

I don't know why.

KrisThielemans · 2024-07-07T01:25:57Z

Note that the test image are very small (8x9x10). Not sure if that matters or not. @Imraj-Singh ?

- there was a missing epsilon in the gradient kernel - abs was used, which is potentially dangerous, so now using std::abs - switch value back to double, as otherwise test_priors still failed

The Hessian() functions all incorrectly threw an error if kappa was alright. I've now moved this test into the check() functions, cleaning code a little bit.

- tests were never run with kappa - RDP limit tests with kappa were wrong Still numerical problems for the Hessian test for RDP

KrisThielemans · 2024-07-07T10:28:49Z

The above problems have now been resolved. 2 things remaining

the numerical Hessian test for the RDP tails with kappa (independent of the CUDA stuff)
GHA cannot run CUDA, so we have to disable the tests for CudaRDP in test_priors if there's no CUDA run-time

RDP test was failing with kappa as eps was too large. Now it is set relative to the iamge max.

CUDA run-time is not available on GitHub actions, so we need to able to disable them.

KrisThielemans · 2024-07-08T06:04:51Z

Everything fine now, except that there's no CUDA run-time on GHA, so we have to disable that test somehow.

It was only coded for float before. I haven't tested with double though.

KrisThielemans · 2024-07-08T07:10:01Z

All good now! (except a Zenodo time-out)

removed obsolete cuda_rdp_tests accordingly

KrisThielemans · 2024-07-08T12:21:46Z

Should be done now. I've added the timings to stir_timings. With default mMR image size, I get on my desktop (AMD Ryzen 9 5900 12-Core, RTX 3070, default num_threads=21) (timings in ms, first is CPU-time, second is wall-clock time)

        RDP_value                                                637.000                         524.449
        RDP_grad                                                4607.667                         879.874
        Cuda_RDP_value                                          1732.778                          86.533
        Cuda_RDP_grad                                            919.778                          46.042

compared to parallelproj forward proj 225ms and back proj 573ms.

Thanks @Imraj-Singh!

KrisThielemans

All done

Imraj-Singh added 3 commits April 25, 2024 00:02

first attempt at integration with stir

5745b8c

attempting to add to registry

2c4fa32

registry add and cmake cuda package/include dirs

52c4943

KrisThielemans requested changes Apr 25, 2024

View reviewed changes

Imraj-Singh added 4 commits April 25, 2024 14:46

changes to cudarelativedifference prior const etc

f41728e

updated Cmakelists

d3f3732

attempting to get the registries to work...

063633e

update cuda error

483cc13

KrisThielemans reviewed Apr 25, 2024

View reviewed changes

Comment thread CMakeLists.txt Outdated

KrisThielemans requested changes Apr 25, 2024

View reviewed changes

Imraj-Singh commented Apr 25, 2024

View reviewed changes

KrisThielemans mentioned this pull request Apr 26, 2024

RelativeDifferencePrior gives wrong results due to use of abs() #1409

Closed

Imraj-Singh added 2 commits May 16, 2024 19:58

Merge branch 'UCL:master' into cuda_stir

7e26bd4

Register errors

0249e93

Imraj-Singh added 4 commits June 4, 2024 09:39

Merge branch 'UCL:master' into cuda_stir

843cd13

error handling in cmake

5eec80b

Merge branch 'UCL:master' into cuda_stir

2e90b17

Not working, but added the functionality and the registery works than…

13def4f

…ks to Kris

KrisThielemans added 5 commits July 6, 2024 23:59

don't run CUDA tests on GHA

3e04184

CudaRDP: minor clean-up

a5ccc4c

correct check for kappa

d2f6d6d

CudaRDP: more checks and document 3x3x3 limitation

c31953e

CudaRDP: put cppdim3 type in the class definition

a0114ab

KrisThielemans force-pushed the cuda_stir branch from ff12c8a to a0114ab Compare July 7, 2024 00:18

add CudaRDP to test_priors (but fails)

ef66303

Problems with numerical gradients. I had to revert the change to GeneralisedPrior::set_penalisation_factor which set _already_set_up to false, as that made test_priors fail on all priors (as the test changes the penalisation factor)

KrisThielemans added 5 commits July 7, 2024 09:45

Cuda RDP fixes

c634a0d

- there was a missing epsilon in the gradient kernel - abs was used, which is potentially dangerous, so now using std::abs - switch value back to double, as otherwise test_priors still failed

decrease verbosity of SeperableGaussianFilter

165da28

correct array_to_device for non-contiguous case

7ac03fd

correct test on kappa characteristics in all priors

fa4d7fe

The Hessian() functions all incorrectly threw an error if kappa was alright. I've now moved this test into the check() functions, cleaning code a little bit.

add and correct tests for priors with kappa (still fails on RDP)

ccc962a

- tests were never run with kappa - RDP limit tests with kappa were wrong Still numerical problems for the Hessian test for RDP

KrisThielemans added 2 commits July 7, 2024 22:34

reduce eps for numerical Hessian test

dcd3b86

RDP test was failing with kappa as eps was too large. Now it is set relative to the iamge max.

make running of CUDA tests optional

3e43ca7

CUDA run-time is not available on GitHub actions, so we need to able to disable them.

KrisThielemans added 3 commits July 8, 2024 07:30

skip CUDA test if nvidia-smi not found

66c082b

CudaRDP: honour elemT template argument

7a2cf76

It was only coded for float before. I haven't tested with double though.

update release notes on the CUDA RDP [ci skip]

44dad5a

KrisThielemans added 3 commits July 8, 2024 09:28

added ctest for CUDA vs non-CUDA RDP

d4f2c9b

added RDP timings to stir_timings

05cdfaf

removed obsolete cuda_rdp_tests accordingly

disable CUDA RDP tests if no CUDA found

1f0d692

KrisThielemans mentioned this pull request Jul 8, 2024

add STIR's new CudaRelativeDifferencePrior SyneRBI/SIRF#1277

Merged

KrisThielemans approved these changes Jul 8, 2024

View reviewed changes

KrisThielemans merged commit b153fda into UCL:master Jul 8, 2024

Imraj-Singh deleted the cuda_stir branch December 31, 2024 10:40

Conversation

Imraj-Singh commented Apr 25, 2024

Uh oh!

KrisThielemans left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Imraj-Singh commented Apr 25, 2024

Uh oh!

KrisThielemans commented Apr 25, 2024

Uh oh!

Imraj-Singh commented Apr 25, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KrisThielemans May 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KrisThielemans commented Apr 25, 2024

Uh oh!

Imraj-Singh commented Apr 25, 2024

Uh oh!

KrisThielemans commented Apr 25, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KrisThielemans commented May 13, 2024

Uh oh!

Imraj-Singh commented May 15, 2024

Uh oh!

Imraj-Singh commented May 17, 2024

Uh oh!

Imraj-Singh commented May 17, 2024

Uh oh!

KrisThielemans commented Jun 2, 2024

Uh oh!

KrisThielemans commented Jun 2, 2024

Uh oh!

KrisThielemans commented Jul 7, 2024

Uh oh!

KrisThielemans commented Jul 7, 2024

Uh oh!

KrisThielemans commented Jul 7, 2024

Uh oh!

KrisThielemans commented Jul 8, 2024

Uh oh!

KrisThielemans May 13, 2024 •

edited

Loading