engine/orchestration: fix missing vm powerstate update vm state by shwstppr · Pull Request #10407 · apache/cloudstack

shwstppr · 2025-02-17T12:27:10Z

Description

Fixes #10406
VMs were not moving to Stopped state when PowerReportMissing is processed.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Fixes apache#10406 VMs were not moving to Stopped state when PowerReportMissing is processed. Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr · 2025-02-17T12:29:42Z

@blueorangutan package

blueorangutan · 2025-02-17T12:32:04Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2025-02-17T12:34:52Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 15.99%. Comparing base (789f94b) to head (00a13ca).
Report is 15 commits behind head on 4.20.

Additional details and impacted files

@@            Coverage Diff            @@
##               4.20   #10407   +/-   ##
=========================================
  Coverage     15.98%   15.99%           
- Complexity    13077    13083    +6     
=========================================
  Files          5649     5649           
  Lines        495624   495625    +1     
  Branches      60003    60003           
=========================================
+ Hits          79250    79268   +18     
+ Misses       407526   407506   -20     
- Partials       8848     8851    +3

Flag	Coverage Δ
uitests	`4.01% <ø> (ø)`
unittests	`16.83% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

DaanHoogland

clgtm, but no idea if this has side effects.

blueorangutan · 2025-02-17T13:47:00Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12487

weizhouapache

code lgtm

2025-02-17T14:14:55,772 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) VM state report is updated. Host {"id":2,"name":"ref-trl-6851-k-Mu24-wei-zhou-kvm2","type":"Routing","uuid":"eb8661f9-6a36-4449-8923-cfe3fb849500"}, VM instance {"id":1159,"instanceName":"i-2-1159-VM","type":"User","uuid":"9e1de80e-ddcf-4c8b-87bc-1c1412292a46"}, power state: PowerReportMissing
2025-02-17T14:14:55,783 INFO  [c.c.v.ClusteredVirtualMachineManagerImpl] (AgentManager-Handler-1:[ctx-d817875a]) (logid:) VM VM instance {"id":1159,"instanceName":"i-2-1159-VM","type":"User","uuid":"9e1de80e-ddcf-4c8b-87bc-1c1412292a46"} is at Running and we received a PowerReportMissing report while there is no pending jobs on it
...
2025-02-17T14:14:55,843 INFO  [c.c.v.ClusteredVirtualMachineManagerImpl] (AgentManager-Handler-1:[ctx-d817875a]) (logid:) VM VM instance {"id":1159,"instanceName":"i-2-1159-VM","type":"User","uuid":"9e1de80e-ddcf-4c8b-87bc-1c1412292a46"} is sync-ed to at Stopped state according to power-off report from hypervisor.

sureshanaparti

clgtm

harikrishna-patnala

code LGTM

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr · 2025-02-18T09:15:53Z

@blueorangutan package

blueorangutan · 2025-02-18T09:18:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-02-18T09:41:50Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 12498

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr · 2025-02-18T10:21:05Z

@blueorangutan package

blueorangutan · 2025-02-18T10:22:04Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-02-18T11:32:36Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12499

DaanHoogland · 2025-02-18T12:53:53Z

@blueorangutan test

blueorangutan · 2025-02-18T12:56:03Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-02-19T05:14:58Z

[SF] Trillian test result (tid-12461)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 56397 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10407-t12461-kvm-ol8.zip
Smoke tests completed. 139 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_11_isolated_network_with_dynamic_routed_mode	`Error`	2.27	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	2.35	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	2.35	test_ipv4_routing.py
test_06_purge_expunged_vm_background_task	`Failure`	392.82	test_purge_expunged_vms.py

kiranchavala

LGTM

Steps to reproduce the issue

Deploy a vm from cloudstack
Login to the kvm host and destory the vm using virsh (virsh destory domain id)
The vm should get stopped in cloudstack after some time

Before the fix (the vm continued to be in running state)

2025-02-25` 06:59:13,941 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-16:null) (logid:) Run missing VM report. current time: 1740466753941
2025-02-25 06:59:13,942 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-16:null) (logid:) Detected missing VM. Host {"id":1,"name":"ol8.localdomain","type":"Routing","uuid":"23ad85e8-e338-4b1a-958c-9a65b2532e75"}, VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"6ae77659-2f5f-492c-b616-6d78ef134dcb"}, power state: PowerReportMissing, last state update: 2025-02-25T06:58:13+0000
2025-02-25 06:59:13,942 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-16:null) (logid:) VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"6ae77659-2f5f-492c-b616-6d78ef134dcb"} - time since last state update(60941 ms) has not passed graceful period yet
2025-02-25 06:59:13,942 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-16:null) (logid:) Done with process of VM state report. Host {"id":1,"name":"ol8.localdomain","type":"Routing","uuid":"23ad85e8-e338-4b1a-958c-9a65b2532e75"}


2025-02-25 07:00:13,989 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:null) (logid:) Run missing VM report. current time: 1740466813989
2025-02-25 07:00:13,990 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:null) (logid:) Detected missing VM. Host {"id":1,"name":"ol8.localdomain","type":"Routing","uuid":"23ad85e8-e338-4b1a-958c-9a65b2532e75"}, VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"6ae77659-2f5f-492c-b616-6d78ef134dcb"}, power state: PowerReportMissing, last state update: 2025-02-25T06:58:13+0000
2025-02-25 07:00:13,990 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:null) (logid:) VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"6ae77659-2f5f-492c-b616-6d78ef134dcb"} - time since last state update(120989 ms) has passed graceful period
2025-02-25 07:00:13,994 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:null) (logid:) Done with process of VM state report. Host {"id":1,"name":"ol8.localdomain","type":"Routing","uuid":"23ad85e8-e338-4b1a-958c-9a65b2532e75"}

After the fix (the vm goes to stop state)

2025-02-25 06:39:20,361 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) Run missing VM report. current time: 1740465560361

2025-02-25 06:39:20,362 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) Detected missing VM. Host {"id":1,"name":"ol8.localdomain","type":"Routing","uuid":"22e66fdc-690b-4ae1-bfd6-a5677315cea0"}, VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"9368adc3-6b06-48db-8fa4-4ad8dbb20a81"}, power state: PowerReportMissing, last state update: 2025-02-25T06:38:20+0000

2025-02-25 06:39:20,362 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"9368adc3-6b06-48db-8fa4-4ad8dbb20a81"} - time since last state update(60361 ms) has not passed graceful period yet

2025-02-25 06:39:20,362 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) Done with process of VM state report. Host {"id":1,"name":"ol8.localdomain","type":"Routing","uuid":"22e66fdc-690b-4ae1-bfd6-a5677315cea0"


2025-02-25 06:40:20,331 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:null) (logid:) VM state report is updated. Host {"id":1,"name":"ol8.localdomain","type":"Routing","uuid":"22e66fdc-690b-4ae1-bfd6-a5677315cea0"}, VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"9368adc3-6b06-48db-8fa4-4ad8dbb20a81"}, power state: PowerReportMissing

2025-02-25 06:40:20,341 INFO  [c.c.v.VirtualMachineManagerImpl] (AgentManager-Handler-1:null) (logid:) VM i-2-3-VM is at Running and we received a PowerReportMissing report while there is no pending jobs on it

2025-02-25 06:40:20,430 INFO  [c.c.v.VirtualMachineManagerImpl] (AgentManager-Handler-1:null) (logid:) VM i-2-3-VM is sync-ed to at Stopped state according to power-off report from hypervisor

2025-02-25 06:40:20,431 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:null) (logid:) Done with process of VM state report. Host {"id":1,"name":"ol8.localdomain","type":"Routing","uuid":"22e66fdc-690b-4ae1-bfd6-a5677315cea0"}

…he#10407) * engine/orchestration: fix missing vm powerstate update vm state Fixes apache#10406 VMs were not moving to Stopped state when PowerReportMissing is processed. Signed-off-by: Abhishek Kumar <[email protected]> * add unit tests Signed-off-by: Abhishek Kumar <[email protected]> * add license Signed-off-by: Abhishek Kumar <[email protected]> * add lenient Signed-off-by: Abhishek Kumar <[email protected]> --------- Signed-off-by: Abhishek Kumar <[email protected]>

engine/orchestration: fix missing vm powerstate update vm state

c4657a9

Fixes apache#10406 VMs were not moving to Stopped state when PowerReportMissing is processed. Signed-off-by: Abhishek Kumar <[email protected]>

boring-cyborg bot added component:compute component:orchestration labels Feb 17, 2025

shwstppr linked an issue Feb 17, 2025 that may be closed by this pull request

VM state is not updated when it is stopped unexpectedly on hypervisor #10406

Closed

DaanHoogland reviewed Feb 17, 2025

View reviewed changes

weizhouapache approved these changes Feb 17, 2025

View reviewed changes

DaanHoogland added this to the 4.20.1 milestone Feb 17, 2025

sureshanaparti approved these changes Feb 18, 2025

View reviewed changes

harikrishna-patnala approved these changes Feb 18, 2025

View reviewed changes

shwstppr added 2 commits February 18, 2025 14:35

add unit tests

aba797c

Signed-off-by: Abhishek Kumar <[email protected]>

add license

bbca682

Signed-off-by: Abhishek Kumar <[email protected]>

add lenient

00a13ca

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr marked this pull request as ready for review February 20, 2025 05:29

yadvr assigned kiranchavala and vladimirpetrov Feb 20, 2025

Pearl1594 added the status:needs-testing label Feb 20, 2025

kiranchavala approved these changes Feb 25, 2025

View reviewed changes

kiranchavala removed the status:needs-testing label Feb 25, 2025

yadvr merged commit e8ac477 into apache:4.20 Feb 25, 2025
26 checks passed

yadvr deleted the fix-missingvmpowersync-4.20 branch February 25, 2025 10:20

yadvr mentioned this pull request Feb 25, 2025

VM state is not updated when it is stopped unexpectedly on hypervisor #10406

Closed

Pearl1594 added this to ACS 4.20.1 Mar 17, 2025

Pearl1594 moved this to Done in ACS 4.20.1 Mar 17, 2025

kiranchavala mentioned this pull request Jan 28, 2026

Power Report Missing || CloudStack 4.20.1.0 #12526

Open

TimServers mentioned this pull request Mar 31, 2026

KVM HA: VM remains Running after virsh destroy; PowerReportMissing schedules HA but KVMInvestigator reports alive=true and restart is cancelled #12920

Open

Conversation

shwstppr commented Feb 17, 2025

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

shwstppr commented Feb 17, 2025

Uh oh!

blueorangutan commented Feb 17, 2025

Uh oh!

codecov bot commented Feb 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Feb 17, 2025

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

sureshanaparti left a comment

Choose a reason for hiding this comment

Uh oh!

harikrishna-patnala left a comment

Choose a reason for hiding this comment

Uh oh!

shwstppr commented Feb 18, 2025

Uh oh!

blueorangutan commented Feb 18, 2025

Uh oh!

blueorangutan commented Feb 18, 2025

Uh oh!

shwstppr commented Feb 18, 2025

Uh oh!

blueorangutan commented Feb 18, 2025

Uh oh!

blueorangutan commented Feb 18, 2025

Uh oh!

DaanHoogland commented Feb 18, 2025

Uh oh!

blueorangutan commented Feb 18, 2025

Uh oh!

blueorangutan commented Feb 19, 2025

Uh oh!

kiranchavala left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

codecov bot commented Feb 17, 2025 •

edited

Loading