I figured it out, your first comment made me think about it more and after some digging I figured out what to try.

I ended up installing xdg-desktop-portal-gtk. And changed "~/.config/xdg-desktop-portal/portals.conf" from this:

[preferred]
org.freedesktop.impl.portal.Settings=darkman

To this:

[preferred]
default = hyprland;darkman;gtk

Then it worked properly. I guess the gtk programs where all listening for settings from darkman, but with the change I made then it would get them in order of what each portal exposed (or something like that).

Thank you for helping point me in the right direction.

Disabling darkman's portal support does stop the icons from changing, it also prevents darkmode for working obviously as well.

Is this maybe happening because I don't have my portals setup correctly? I have both xdg-desktop-portal-hyprland.service and xdg-desktop-portal.service running, but from my reading I may need another portal which implements more interfaces like the one for gtk.