On a complex VPC with many network tiers, adding a private gateway appears to succeed, but doesn't work (e.g. can't actually ping).
Then attempting to restart the VPC results in a failure of the VPC coming online with this error:
What is more interesting is when adding to a VPC with only a single network tier, everything works!
I know there are limitations to the number of tiers you can add to a VPC (which from my testing appears to be in the low 20s) due to PCI bus limitations with the vNICs, but I am well below that number at 11. I even removed one of my network tiers and retested to confirm.
I'm not sure how to get more information from within the systemvm.
The errors in the management server are below, but don't appear too helpful to me:
2025-11-03 14:00:54,178 WARN [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-23059846]) (logid:f541c307) alertType=[21] dataCenterId=[1] podId=[null] clusterId=[null] message=[System Alert: Low Available Backup Storage in availability zone us-west-1].
","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.PlugNicAnswer":{"result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.152.21, file: ip_associations.json.4063441f-18a3-4edf-8b9e-2fc6a6da7580","null - success: Error: ipv4: FIB table does not exist.
"],"result":"true","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"true","details":"Nothing to do","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.PlugNicAnswer":{"result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.152.21, file: ip_associations.json.ce884acb-2ab5-4039-a21d-31a561203125","null - success: Error: ipv4: FIB table does not exist.
"],"result":"true","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.152.21, file: network_acl.json.f1edc32c-ed5a-453d-9311-1145b49bc3ec","null - success: # Warning: table ip nat is managed by iptables-nft, do not touch!
# Warning: table ip filter is managed by iptables-nft, do not touch!
# Warning: table ip mangle is managed by iptables-nft, do not touch!
"],"result":"true","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.PlugNicAnswer":{"result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.152.21, file: guest_network.json.4782c8f6-8bd5-4ec8-914e-b69810e5a6f6","null - success: Error: ipv4: FIB table does not exist.
# Warning: table ip nat is managed by iptables-nft, do not touch!
# Warning: table ip filter is managed by iptables-nft, do not touch!
# Warning: table ip mangle is managed by iptables-nft, do not touch!
"],"result":"true","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.PlugNicAnswer":{"result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.152.21, file: guest_network.json.7e9302e7-d0cd-4b21-ac07-685cea2c9a3d","null - success: Error: ipv4: FIB table does not exist.
# Warning: table ip nat is managed by iptables-nft, do not touch!
# Warning: table ip filter is managed by iptables-nft, do not touch!
# Warning: table ip mangle is managed by iptables-nft, do not touch!
"],"result":"true","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.PlugNicAnswer":{"result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.152.21, file: guest_network.json.6834844e-25c1-4b7f-abeb-69666aabd9cb","null - success: Error: ipv4: FIB table does not exist.
# Warning: table ip nat is managed by iptables-nft, do not touch!
# Warning: table ip filter is managed by iptables-nft, do not touch!
# Warning: table ip mangle is managed by iptables-nft, do not touch!
"],"result":"true","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.PlugNicAnswer":{"result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.152.21, file: guest_network.json.2f87ffd7-05e5-4ba1-8713-ddb3112d0111","null - success: Error: ipv4: FIB table does not exist.
# Warning: table ip nat is managed by iptables-nft, do not touch!
# Warning: table ip filter is managed by iptables-nft, do not touch!
# Warning: table ip mangle is managed by iptables-nft, do not touch!
"],"result":"true","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.PlugNicAnswer":{"result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.152.21, file: guest_network.json.fb5d1ee7-2b96-4a24-91dd-d07bdfe0f509","null - success: Error: ipv4: FIB table does not exist.
# Warning: table ip nat is managed by iptables-nft, do not touch!
# Warning: table ip filter is managed by iptables-nft, do not touch!
# Warning: table ip mangle is managed by iptables-nft, do not touch!
"],"result":"true","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.PlugNicAnswer":{"result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.152.21, file: guest_network.json.8e985b32-3a91-4fec-8c42-f9efbe5a88da","null - success: Error: ipv4: FIB table does not exist.
# Warning: table ip nat is managed by iptables-nft, do not touch!
# Warning: table ip filter is managed by iptables-nft, do not touch!
# Warning: table ip mangle is managed by iptables-nft, do not touch!
"],"result":"true","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.PlugNicAnswer":{"result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.152.21, file: guest_network.json.268f4edd-2e8a-41ae-85ce-d5e5fc03c161","null - success: Error: ipv4: FIB table does not exist.
# Warning: table ip nat is managed by iptables-nft, do not touch!
# Warning: table ip filter is managed by iptables-nft, do not touch!
# Warning: table ip mangle is managed by iptables-nft, do not touch!
"],"result":"true","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.PlugNicAnswer":{"result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.152.21, file: guest_network.json.6eda11dc-0d52-4749-8133-d2985efdfdca","null - success: Error: ipv4: FIB table does not exist.
# Warning: table ip nat is managed by iptables-nft, do not touch!
# Warning: table ip filter is managed by iptables-nft, do not touch!
# Warning: table ip mangle is managed by iptables-nft, do not touch!
"],"result":"true","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.PlugNicAnswer":{"result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.152.21, file: guest_network.json.d85b543b-8c4f-4551-afe3-43cfc1c55909","null - success: Error: ipv4: FIB table does not exist.
# Warning: table ip nat is managed by iptables-nft, do not touch!
# Warning: table ip filter is managed by iptables-nft, do not touch!
# Warning: table ip mangle is managed by iptables-nft, do not touch!
"],"result":"true","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.PlugNicAnswer":{"result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.152.21, file: guest_network.json.9edb27a9-03c9-4a54-8853-cfd2c50ec4f0","null - success: Error: ipv4: FIB table does not exist.
# Warning: table ip nat is managed by iptables-nft, do not touch!
# Warning: table ip filter is managed by iptables-nft, do not touch!
# Warning: table ip mangle is managed by iptables-nft, do not touch!
"],"result":"true","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.PlugNicAnswer":{"result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.152.21, file: guest_network.json.c99d55ed-2c84-49c5-84e1-ea00810debcb","null - success: Error: ipv4: FIB table does not exist.
# Warning: table ip nat is managed by iptables-nft, do not touch!
# Warning: table ip filter is managed by iptables-nft, do not touch!
# Warning: table ip mangle is managed by iptables-nft, do not touch!
"],"result":"true","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.152.21, file: static_routes.json.10ab581a-a46f-406d-910e-e824eeccfbe0","null - failed: java.io.IOException: Stream closed
at com.cloud.utils.script.OutputInterpreter.processError(OutputInterpreter.java:41)
"],"result":"false","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}}] }
Removing the Private Gateway allows the VPC to start again.
I'm assuming this is due to the number of network tiers since a VPC with only a single tier is fine.
problem
On a complex VPC with many network tiers, adding a private gateway appears to succeed, but doesn't work (e.g. can't actually ping).
Example:
Then attempting to restart the VPC results in a failure of the VPC coming online with this error:
What is more interesting is when adding to a VPC with only a single network tier, everything works!
I know there are limitations to the number of tiers you can add to a VPC (which from my testing appears to be in the low 20s) due to PCI bus limitations with the vNICs, but I am well below that number at 11. I even removed one of my network tiers and retested to confirm.
Here is the error in the agent log:
I'm not sure how to get more information from within the systemvm.
The errors in the management server are below, but don't appear too helpful to me:
Removing the Private Gateway allows the VPC to start again.
versions
Cloudstack 4.21.0
The steps to reproduce the bug
I'm assuming this is due to the number of network tiers since a VPC with only a single tier is fine.
What to do about it?
No response