DAOS-18395 test: recovery/pool_list_consolidation.py test_lost_majori…#17965

Draft

shimizukko wants to merge 21 commits intomasterfrom

makito/DAOS-18395

Contributor

shimizukko commented Apr 10, 2026

…ty_ps_replicas MD-on-SSD support

In MD-on-SSD, we need to load the pool dir to a new tmpfs because the pool dir will disappear when we
stop the servers, so load with prov_mem.

We need to use ddb prov_mem to remove rdb-pool, so add support in ddb_utils.py and the logic to determine the parameters for ddb prov_mem.

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).


          DAOS-18395 test: recovery/pool_list_consolidation.py test_lost_majori…

cf1e101

…ty_ps_replicas MD-on-SSD support

In MD-on-SSD, we need to load the pool dir to a new
tmpfs because the pool dir will disappear when we
stop the servers, so load with prov_mem.

We need to use ddb prov_mem to remove rdb-pool, so
add support in ddb_utils.py and the logic to determine
the parameters for ddb prov_mem.

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest
Signed-off-by: Makito Kano <[email protected]>

github-actions Bot commented Apr 10, 2026

Ticket title is 'CR Test Update - recovery/pool_list_consolidation.py test_lost_majority_ps_replicas MD-on-SSD Support'
Status is 'In Progress'
Labels: 'catastrophic_recovery'
https://daosio.atlassian.net/browse/DAOS-18395

shimizukko added 11 commits

April 10, 2026 00:41


          DAOS-18395 test: Fix pylint

0454e20

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest
Signed-off-by: Makito Kano <[email protected]>


          DAOS-18395 test: Add mount cleanup code

52ef33a

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest
Signed-off-by: Makito Kano <[email protected]>


          DAOS-18395 test: Fix pylint

60d15ac

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest
Signed-off-by: Makito Kano <[email protected]>


          DAOS-18395 test: Fix pylint

490a6ea

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest
Signed-off-by: Makito Kano <[email protected]>


          DAOS-18395 test: Fix pylint

419fad9

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest
Signed-off-by: Makito Kano <[email protected]>


          DAOS-18395 test: Fix pylint

a9128f1

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest
Signed-off-by: Makito Kano <[email protected]>


          DAOS-18395 test: Fix pylint

b3d2d3d

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest
Signed-off-by: Makito Kano <[email protected]>


          Merge branch 'master' into makito/DAOS-18395

34622ef

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest


          DAOS-18395 test: Add MD-on-SSD check before clean_mounts

33688d3

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest
Signed-off-by: Makito Kano <[email protected]>


          DAOS-18395 test: Fix pylint

992d747

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest
Signed-off-by: Makito Kano <[email protected]>


          Merge branch 'master' into makito/DAOS-18395

a2ac583

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest

Collaborator

daosbuild3 commented Apr 16, 2026

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17965/11/execution/node/728/log

Collaborator

daosbuild3 commented Apr 16, 2026

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17965/11/execution/node/813/log


          Merge branch 'master' into makito/DAOS-18395

998e1e2

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest

Collaborator

daosbuild3 commented Apr 18, 2026

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17965/12/execution/node/769/log

shimizukko added 4 commits

April 20, 2026 05:12


          DAOS-18395 test: Adjust ddb command preparation by removing the setti…

f870062

…ng of --db_path

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest
Signed-off-by: Makito Kano <[email protected]>


          Merge branch 'master' into makito/DAOS-18395

e3ab7a8

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest


          DAOS-18395 test: Update comment

9773dc8

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest
Signed-off-by: Makito Kano <[email protected]>


          Merge branch 'master' into makito/DAOS-18395

f7f98b4

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest

Collaborator

daosbuild3 commented Apr 20, 2026

Test stage Functional Hardware Medium completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17965/15/testReport/

Collaborator

daosbuild3 commented Apr 20, 2026

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17965/15/execution/node/705/log


          Merge branch 'master' into makito/DAOS-18395

b99fcca

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest

Collaborator

daosbuild3 commented Apr 21, 2026

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17965/16/execution/node/705/log


          Merge branch 'master' into makito/DAOS-18395

2e062c3

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest

Collaborator

daosbuild3 commented Apr 22, 2026

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17965/17/execution/node/715/log

shimizukko added 2 commits

April 23, 2026 08:49


          Merge branch 'master' into makito/DAOS-18395

621c2d5

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest


          Merge branch 'master' into makito/DAOS-18395

fe6150c

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_lost_majority_ps_replicas DdbTest

Collaborator

daosbuild3 commented Apr 24, 2026

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17965/18/execution/node/750/log

Contributor Author

shimizukko commented Apr 25, 2026

Engines aren't starting after dmg storage format for ddb.py tests. Possibly due to https://daosio.atlassian.net/browse/DAOS-18860
For some reason, pool_list_consolidation.py test is working. ddb.py fails in either PMEM or MD-on-SSD cluster. Same results after rerunning the tests in CI for a few times.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet