asset.dart's needsBuild(..) {..} returns whether an AssetBundle needs to be (re)built given a corresponding manifest.
Its approach is to check the existence and modified timestamp on the manifest and its directories.
However, when checking the directories (_wildcardDirectories) it checks if modified returns null, in which case it skips the directory.
For MemoryFileSystem this unfortunately means that if a "wildcard dir" is deleted, needsBuild will simply ignore it and will likely return false.

I believe this is a bug as any major changes to the filesystem should likely indicate that the AssetBundle needs rebuilding?