Skip to content

Fix test_disks_app_func#69449

Merged
nickitat merged 1 commit intomasterfrom
fix_test2
Sep 11, 2024
Merged

Fix test_disks_app_func#69449
nickitat merged 1 commit intomasterfrom
fix_test2

Conversation

@nickitat
Copy link
Member

@nickitat nickitat commented Sep 10, 2024

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

I have one of this tests constantly failing in my PR: https://s3.amazonaws.com/clickhouse-test-reports/68424/8cdc10cf656791ab88093cddb0cb07bb4209492a/integration_tests__aarch64__[5_6].html

The logs:

------------------------------ Captured log call -------------------------------
2024-09-09 20:06:05 [ 519 ] DEBUG : run container_id:roottestdisksappfunc_gw3_disks_app_test_1 detach:False nothrow:False cmd: ['bash', '-c', "echo 'tester' |/usr/bin/clickhouse disks --save-logs --disk test1 --query 'write 5.txt'"] (cluster.py:2165, exec_in_container)
2024-09-09 20:06:05 [ 519 ] DEBUG : Command:['docker', 'exec', 'roottestdisksappfunc_gw3_disks_app_test_1', 'bash', '-c', "echo 'tester' |/usr/bin/clickhouse disks --save-logs --disk test1 --query 'write 5.txt'"] (cluster.py:109, run_and_check)
2024-09-09 20:06:05 [ 519 ] DEBUG : Stderr:Cannot open file /var/lib/clickhouse/path1/5.txt: , errno: 2, strerror: No such file or directory (cluster.py:119, run_and_check)
2024-09-09 20:06:05 [ 519 ] DEBUG : run container_id:roottestdisksappfunc_gw3_disks_app_test_1 detach:False nothrow:False cmd: ['/usr/bin/clickhouse', 'disks', '--save-logs', '--disk', 'test1', '--query', 'read 5.txt'] (cluster.py:2165, exec_in_container)
2024-09-09 20:06:05 [ 519 ] DEBUG : Command:['docker', 'exec', 'roottestdisksappfunc_gw3_disks_app_test_1', '/usr/bin/clickhouse', 'disks', '--save-logs', '--disk', 'test1', '--query', 'read 5.txt'] (cluster.py:109, run_and_check)
2024-09-09 20:06:06 [ 519 ] DEBUG : Stderr:Cannot open file /var/lib/clickhouse/path1/5.txt: , errno: 2, strerror: No such file or directory (cluster.py:119, run_and_check)

And this is contained in clickhouse-disks.log:

❯ cat _instances_0_gw3/disks_app_test/logs/clickhouse-disks.log
Cannot gain read access of the disk directory: /var/lib/clickhouse/path1/
Disk test1 is marked as broken during startup: Code: 481. DB::ErrnoException: Cannot check read access to file: /var/lib/clickhouse/path1/: , errno: 2, strerror: No such file or directory. (PATH_ACCESS_DENIED), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c352d84
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000765cb5c
2. DB::ErrnoException::ErrnoException<String const&>(int, int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000c399300
3. void DB::ErrnoException::throwFromPath<String const&>(int, String const&, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000c398f14
4. DB::DiskLocal::startupImpl(std::shared_ptr<DB::Context const>) @ 0x000000000f59d370
5. DB::IDisk::startup(std::shared_ptr<DB::Context const>, bool) @ 0x000000000f5ab9c8
6. std::shared_ptr<DB::IDisk> std::__function::__policy_invoker<std::shared_ptr<DB::IDisk> (String const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr<DB::Context const>, std::map<String, std::shared_ptr<DB::IDisk>, std::less<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::IDisk>>>> const&, bool, bool)>::__call_impl<std::__function::__default_alloc_func<DB::registerDiskLocal(DB::DiskFactory&, bool)::$_0, std::shared_ptr<DB::IDisk> (String const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr<DB::Context const>, std::map<String, std::shared_ptr<DB::IDisk>, std::less<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::IDisk>>>> const&, bool, bool)>>(std::__function::__policy_storage const*, String const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr<DB::Context const>&&, std::map<String, std::shared_ptr<DB::IDisk>, std::less<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::IDisk>>>> const&, bool, bool) (.llvm.6409596264801681832) @ 0x000000000f5a1bf0
7. DB::DiskFactory::create(String const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr<DB::Context const>, std::map<String, std::shared_ptr<DB::IDisk>, std::less<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::IDisk>>>> const&, bool, bool, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>> const&) const @ 0x000000000f5a2b14
8. DB::DiskSelector::initialize(Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr<DB::Context const>, std::function<bool (Poco::Util::AbstractConfiguration const&, String const&, String const&)>) @ 0x000000000f5a5c30
9. DB::DisksApp::main(std::vector<String, std::allocator<String>> const&) @ 0x000000000c5caddc
10. Poco::Util::Application::run() @ 0x0000000013575d7c
11. mainEntryClickHouseDisks(int, char**) @ 0x000000000c5ced80
12. main @ 0x0000000007659504
13. ? @ 0x00000000000273fc
14. ? @ 0x00000000000274cc
 (version 24.9.1.2243)

Apparently the reason why it started to fail in my PR is because I accidentally change tests distribution / order in a way that this directory is no longer created by some other test case. Here is when the corresponding storage policy was initialised:

./_instances_0_gw3/disks_app_test/logs/clickhouse-server.log
382:2024.09.09 20:06:06.276138 [ 11 ] {68130d65-bb47-4e31-bb6b-7e0aa2f91df3} <Debug> executeQuery: (from 172.16.7.1:55848) CREATE TABLE test_table(word String, value UInt64) ENGINE=MergeTree() ORDER BY word SETTINGS storage_policy = 'test1' (stage: Complete)
383:2024.09.09 20:06:06.277207 [ 11 ] {68130d65-bb47-4e31-bb6b-7e0aa2f91df3} <Trace> StoragePolicy (test1): Storage policy test1 created, total volumes 1
384:2024.09.09 20:06:06.277236 [ 11 ] {68130d65-bb47-4e31-bb6b-7e0aa2f91df3} <Information> StoragePolicySelector: Storage policy `test1` loaded
412:2024.09.09 20:06:06.340978 [ 11 ] {ec6d7be8-00fd-4750-ac89-f25bbab1d632} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `test1`, having unreserved 82.14 GiB.
668:2024.09.09 20:06:10.718846 [ 1 ] {} <Information> Context: Shutdown disk test1

I'm not quite sure how all this works, but I don't see any problem if we just add an explicit mkdir for this directories.

@robot-clickhouse-ci-2 robot-clickhouse-ci-2 added the pr-not-for-changelog This PR should not be mentioned in the changelog label Sep 10, 2024
@nickitat nickitat mentioned this pull request Sep 10, 2024
Merged
20 tasks
@robot-ch-test-poll4
Copy link
Contributor

robot-ch-test-poll4 commented Sep 10, 2024 •
edited
Loading

This is an automated comment for commit acbeaa6 with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Check nameDescriptionStatus
Flaky testsChecks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integration tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc❌ failure
Successful checks
Check nameDescriptionStatus
BuildsThere's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS✅ success
Docs checkBuilds and tests the documentation✅ success
Fast testNormally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here✅ success
Integration testsThe integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests✅ success
Style checkRuns a set of checks to keep the code style clean. If some of tests failed, see the related log from the report✅ success
Unit testsRuns the unit tests for different release types✅ success
Upgrade checkRuns stress tests on server version from last release and then tries to upgrade it to the version from the PR. It checks if the new server can successfully startup without any errors, crashes or sanitizer asserts✅ success

@Avogar Avogar self-assigned this Sep 10, 2024
@nickitat
Copy link
Member Author

nickitat commented Sep 11, 2024

Integration tests flaky check (asan) - irrelevant to this PR, repeated invocation never worked

@nickitat nickitat added this pull request to the merge queue Sep 11, 2024
Merged via the queue into master with commit bdddab6 Sep 11, 2024
@nickitat nickitat deleted the fix_test2 branch September 11, 2024 11:20
@robot-clickhouse-ci-2 robot-clickhouse-ci-2 added the pr-synced-to-cloud The PR is synced to the cloud repo label Sep 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Avogar Avogar Avogar approved these changes

Assignees

@Avogar Avogar

Labels

pr-not-for-changelog This PR should not be mentioned in the changelog pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@nickitat @robot-ch-test-poll4 @Avogar @robot-clickhouse-ci-2