Posted patch upstream to linux-fsdevel list.
https://lore.kernel.org/linux-fsdevel/[email protected]/
Will respin it to add some documentation about it in virtiofs.rst.
I think tag file is not even available at the time of KERNEL ADD event. It will become available soon after as soon as virtiofs driver binds to the device.
Ok, I tried above it seems to solve the problem.
The reason we were using ACTION==bind was because were were concerned about race conditions. At the time of device ADD event tag will not be available and it will become available once virtiofs driver binds to the device and publishes the tag through sysfs.
When I hot-plugged the virtiofs device I noticed following two events which seem to be of interest.
UDEV [1121.563699] add /devices/pci0000:00/0000:00:03.0/0000:01:00.0/virtio5 (virtio) ACTION=add DEVPATH=/devices/pci0000:00/0000:00:03.0/0000:01:00.0/virtio5 SUBSYSTEM=virtio MODALIAS=virtio:d0000001Av00001AF4 SEQNUM=3337 USEC_INITIALIZED=1121551842 DRIVER=virtiofs SYSTEMD_WANTS=mnt-virtiofs.mount TAGS=:systemd: CURRENT_TAGS=:systemd: UDEV [1121.565316] bind /devices/pci0000:00/0000:00:03.0/0000:01:00.0/virtio5 (virtio) ACTION=bind DEVPATH=/devices/pci0000:00/0000:00:03.0/0000:01:00.0/virtio5 SUBSYSTEM=virtio DRIVER=virtiofs MODALIAS=virtio:d0000001Av00001AF4 SEQNUM=3338 USEC_INITIALIZED=1121551842 SYSTEMD_WANTS=mnt-virtiofs.mount TAGS=:systemd: CURRENT_TAGS=:systemd:
Looks like udev generated both Add and bind action events. And looks like SYSTEMD_WANTS property was set for both of these. That probably means ATTR{tag}==""myfs" was true at the time of event generation. If it was not, then SYSTEMD_WANTS=mnt-virtiofs.mount property would not have been there.
I don't understand all the magic udev is doing. But looks like above is working. So I will leave it at that. If somebody runs into any races, we will need to find a solution for that.
@poettering about missing the event above, how do we take care of this in general. I mean for some of the devices in general we will like to receive event only after rootfs is mounted. How does systemd/udev handle that in general.
Ok, I am playing a bit udev rules and .mount unit file so that it is activated when udev rule triggers. Following seems to work for me when I hot plug the virtiofs device. udev rule gets executed and virtiofs is mounted. But when I reboot VM I don't see virtiofs being mounted. My guess is that event happened before rootfs was mounted. I had to replay udev events and then virtiofs was mounted.
Used following udev rule.
$ cat /etc/udev/rules.d/80-local.rules
ACTION=="bind", SUBSYSTEM=="virtio", DRIVER=="virtiofs", ATTR{tag}=="myfs", TAG+="systemd", ENV{SYSTEMD_WANTS}+="mnt-virtiofs.mount"
$ cat /etc/systemd/system/mnt-virtiofs.mount
[Unit]
Description=Virtiofs mount myfs
DefaultDependencies=no
ConditionPathExists=/mnt/virtiofs
ConditionCapability=CAP_SYS_ADMIN
Before=sysinit.target
[Mount]
What=myfs
Where=/mnt/virtiofs
Type=virtiofs
Above works for me when I hotplug a virtiofs device with "myfs" tag.
$systemctl status mnt-virtiofs.mount
● mnt-virtiofs.mount - Virtiofs mount myfs
Loaded: loaded (/etc/systemd/system/mnt-virtiofs.mount; static)
Active: active (mounted) since Tue 2023-10-03 13:44:19 EDT; 7min ago
Where: /mnt/virtiofs
What: myfs
Tasks: 0 (limit: 76966)
Memory: 12.0K
CPU: 16ms
CGroup: /system.slice/mnt-virtiofs.mount
For a fresh boot, I had to replay udev events for this to work.
$ udevadm trigger -s virtio -c bind
So I guess we need to find some solution so that udev event is not lost and udev rule can trigger after rootfs is mounted.
I think all the device discovery and driver taking ownership of device and publishing attrs in /sys is going to be all asynchronous (even if virtiofs was built in and not a module). So we probably will have to be ready for being able to deal with async. That will also be helpful when virtiofs deivce is hot-plugged or hot-removed and ideally in response one can mount/unmount a filesystem instance.
Ok, here is another attempt. This time patch looks even simpler. I have yet to test the uevent stuff. virtiofs driver is not controlling creating on kobj and uevent generation but it does create the attr "tag". So whenever "tag" is added/removed, we might have to send another uevent with CHANGE or something like that.
https://github.com/rhvgoyal/linux/commit/6cba9a0560d01418eb94eed359a69d8ddd0488dc
@poettering @stefanha @germag PTAL.
Thanks all for the feedback. I agree with both the points. Given there is one tag associated with the device, it makes sense to add it under device. Also don't export tag directly as file name. Instead it should be file data which should be read by the application.
Now tricky part is to figure out how I can add a tag as part of device directory. Will need to read more code and see if there are any examples of subsystems which are already doing something similar.