Skip to content

jemalloc: set muzzy_decay_ms = 10s#11993

Merged
akuzm merged 2 commits intomasterfrom
aku/more-jemalloc-tweaks
Jun 30, 2020
Merged

jemalloc: set muzzy_decay_ms = 10s#11993
akuzm merged 2 commits intomasterfrom
aku/more-jemalloc-tweaks

Conversation

@akuzm
Copy link
Contributor

@akuzm akuzm commented Jun 27, 2020 •
edited
Loading

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

As a reference for other tests we've made, see #11401

@blinkov blinkov added the pr-not-for-changelog This PR should not be mentioned in the changelog label Jun 27, 2020
@robot-clickhouse robot-clickhouse added the submodule changed At least one submodule changed in this PR. label Jun 27, 2020
@akuzm
Copy link
Contributor Author

akuzm commented Jun 29, 2020

Some more stats made with this script: https://gist.github.com/akuzm/d1677a2810c2a6f7cbd056de1384c7ec
They show which metric from ProfileEvents changed (with statistical significance), for how many queries, by how much.

 num_queries      min      med      max      avg   metric_name                         top_changed_tests                                                               

          85   -0.059    0.037    0.235    0.034   PerfBusCycles                       [('math',13),('arithmetic',10),('general_purpose_hashes',10)]                    
         105   -0.064    0.037    0.224    0.026   PerfTaskClock                       [('arithmetic',12),('math',12),('general_purpose_hashes',9)]                     
          91   -0.055    0.037    0.233    0.031   PerfRefCpuCycles                    [('math',12),('general_purpose_hashes',11),('arithmetic',10)]                    
         105   -0.064    0.037    0.224    0.026   PerfCpuClock                        [('arithmetic',12),('math',12),('general_purpose_hashes',9)]                     
         139   -0.085    0.037    0.202    0.009   UserTimeMicroseconds                [('uniq',21),('math',16),('website',14)]                                         
          78   -0.063    0.037    0.227    0.023   RealTimeMicroseconds                [('math',12),('joins_in_memory',10),('json_extract_simdjson',7)]                 
          94   -0.061    0.037    0.238     0.03   OSCPUVirtualTimeMicroseconds        [('arithmetic',12),('math',12),('general_purpose_hashes',8)]                     
          72   -0.059    0.034    0.226    0.023   client_time                         [('math',12),('joins_in_memory',10),('json_extract_simdjson',6)]                 
         122   -0.053    0.027    0.232    0.019   PerfCpuCycles                       [('math',14),('arithmetic',12),('general_purpose_hashes',11)]                    
          31   -0.065    0.023     3.56    0.172   PerfCacheMisses                     [('uniq',10),('agg_functions_min_max_any',6),('general_purpose_hashes',4)]       
         194   -0.084    0.012    0.495    0.011   PerfBranchMisses                    [('string_sort',28),('general_purpose_hashes',17),('constant_column_search',16)] 
         117    -0.06   -0.002     0.18    0.012   OSWriteChars                        [('select_format',21),('math',11),('arithmetic',8)]                              
          18   -0.183   -0.015    0.052   -0.056   PerfCacheReferences                 [('general_purpose_hashes',5),('synthetic_hardware_benchmark',3),('uniq',3)]     
          22   -0.071   -0.021     0.03    -0.09   PerfInstructions                    [('uniq',4),('general_purpose_hashes',3),('general_purpose_hashes_on_UUID',3)]   
         114   -0.087   -0.027    0.156   -0.016   server_time                         [('math',14),('joins_in_memory',11),('uniq',6)]                                  
          27   -0.223    -0.04    0.158   -0.047   PerfContextSwitches                 [('general_purpose_hashes',6),('joins_in_memory',6),('uniq',3)]                  
         159   -0.216   -0.091     0.06   -0.118   NetworkReceiveElapsedMicroseconds   [('codecs_int_insert',7),('website',6),('logical_functions_small',6)]            
          15   -0.245   -0.102    0.563   -0.075   DiskReadElapsedMicroseconds         [('parallel_final',10),('parse_engine_file',5)]                                  
          14   -0.655    -0.18    1.185   -0.183   NetworkSendElapsedMicroseconds      [('agg_functions_min_max_any',2),('if_to_multiif',2),('base64',1)]               
          95   -0.466   -0.325    0.065   -0.297   SystemTimeMicroseconds              [('uniq',28),('website',14),('agg_functions_min_max_any',13)]                    
         503   -0.824    -0.51   -0.002   -0.508   SoftPageFaults                      [('string_sort',105),('agg_functions_min_max_any',73),('uniq',45)]               
         491    -0.82   -0.515   -0.002   -0.512   PerfPageFaultsMinor                 [('string_sort',105),('agg_functions_min_max_any',73),('uniq',45)]               
          12   -0.767   -0.628    0.819     -0.5   OSCPUWaitMicroseconds               [('math',4),('logical_functions_medium',2),('cryptographic_hashes',1)]

A cumulative graph for relative metric changes: for value of change, plots the number of queries for which the change is less.
cumulative-custom

What we can see is a significant reduction in page faults and system time, with no changes in query time.

The query stability metrics didn't change much though (these are quantiles(0, 0.25, 0.5, 0.75, 0.95, 0.99, 1) of stddev of query run time, queries that are not "short" (>0.02 s)). There is some change for higher quantiles, but this is likely to be noise.

$ clickhouse-local --query "select count(*), arrayMap(x->floor(x, 3), quantiles(0., 0.25, 0.5, 0.75, 0.95, 0.99, 1.)(s/m)) q, version from (select version, median(time) m, stddevPop(time) s, test, query_index from file('analyze/query-runs.tsv', TSV, 'test text, query_index int, query_id text, version int, time float') group by test, query_index, version having m > 0.02) group by version format PrettySpace"
 count()   q                                             version

    1522   [0.005,0.019,0.029,0.047,0.116,0.29,3.062]          0 
    1524   [0.004,0.019,0.028,0.047,0.109,0.278,0.983]         1

I intend to merge this, hopefully it will make the testing more predictable, if only because we won't have to look at crazy SoftPageFaults fluctuations and wonder what they mean. @alexey-milovidov @azat any comments?

@akuzm akuzm changed the title [wip] more jemalloc tests jemalloc: set muzzy_decay_ms = 10s Jun 29, 2020
@akuzm
Copy link
Contributor Author

akuzm commented Jun 29, 2020

Ah, forgot the most important thing. Here's a graph for MemoryResident (it's already in test results archive, 'metrics/MemoryResident.png'):
MemoryResident

You can see that the memory usage is somewhat higher, but for long test that use constant amount of memory, it stabilizes around the same value. Looking at summary metrics, median of MemoryResident is 20% higher: https://clickhouse-test-reports.s3.yandex.net/11993/8b82ff215fe4b42216aa41adf9907aba37a3817d/performance_comparison/report.html#8 . This is to be expected -- muzzy pages is memory that is freed lazily and does not immediately influence the RSS. But it may have some influence on our query limiting behavior, when we limit server RSS to a percentage of total available memory (max_server_memory_usage_to_ram_ratio).

@akuzm akuzm marked this pull request as ready for review June 29, 2020 07:25
azat
azat reviewed Jun 29, 2020
View reviewed changes
contrib/jemalloc-cmake/CMakeLists.txt
# avoid spurious latencies and additional work associated with
# MADV_DONTNEED. See
# https://github.com/ClickHouse/ClickHouse/issues/11121 for motivation.
set (JEMALLOC_CONFIG_MALLOC_CONF "percpu_arena:percpu,oversize_threshold:0,muzzy_decay_ms:10000")
Copy link
Member

@azat azat Jun 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe worth enable it for other OSes too, just in case? (although not sure that it has analogs)

Copy link
Contributor Author

@akuzm akuzm Jun 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think MacOS has some similar madvise call for this, don't know about other systems. Let's enable it.

akuzm
akuzm commented Jun 29, 2020
View reviewed changes
@akuzm akuzm merged commit baec409 into master Jun 30, 2020
@akuzm akuzm deleted the aku/more-jemalloc-tweaks branch June 30, 2020 19:51
@akuzm akuzm mentioned this pull request Jul 13, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@azat azat azat left review comments

Assignees

No one assigned

Labels

pr-not-for-changelog This PR should not be mentioned in the changelog submodule changed At least one submodule changed in this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@akuzm @azat @blinkov @robot-clickhouse