Add `on_force` option to `shutdown_debug` by joshuay03 · Pull Request #3671 · puma/puma

joshuay03 · 2025-06-29T20:34:59Z

Description

Your checklist for this pull request

I have reviewed the guidelines for contributing to this repository.
I have added (or updated) appropriate tests if this PR fixes a bug or adds a feature.
My pull request is 100 lines added/removed or less so that it can be easily reviewed.
If this PR doesn't need tests (docs change), I added [ci skip] to the title of the PR.
If this closes any issues, I have added "Closes #issue" to the PR description or my commit messages.
I have updated the documentation accordingly.
All new and existing tests passed, including Rubocop.

joshuay03 · 2025-07-09T02:44:44Z

      environment: 'development'.freeze,
      # Number of seconds to wait until we get the first data for the request.
      first_data_timeout: 30,
+      force_shutdown_after: -1,


This is the default but it lived in the method definition. IMO it should live here because it's a configurable option.

joshuay03 · 2025-07-11T19:52:54Z

-      if @thread_pool
-        if timeout = options[:force_shutdown_after]
-          @thread_pool.shutdown timeout.to_f
-        else
-          @thread_pool.shutdown
-        end
-      end
+      @thread_pool.shutdown options[:force_shutdown_after]


Reading the code I'm quite sure that @thread_pool will always be truthy at this point, so we don't need the first if.

I've moved the default :force_shutdown_after value (-1) to the config so we no longer need to worry about passing in nil and overriding the default in the method definition i.e., we don't need the second if (and else).

joshuay03 · 2025-07-11T19:53:38Z

+        if @shutdown_debug == :on_force && !threads.empty?
+          shutdown_debug("Shutdown timeout exceeded")


Unlike the graceful shutdown case where we print before we join on the threads (which makes sense because the join will wait for as long as needed), here we print after joining - once if and before forcing a shutdown, and then again if and before forcefully killing threads.

joshuay03 · 2025-11-25T11:21:25Z

@dentarg Any further opinions on this? If not I'm going to go ahead and merge.

dentarg · 2025-11-25T22:47:50Z

Nice

How I tested this branch

I did kill $(cat /tmp/puma) while the request was running

$ echo 'force_shutdown_after 5; shutdown_debug(on_force: true); app { |env| sleep 120; [200, {}, ["OK"]] }' | ruby -rlogger -Ilib ./bin/puma --config /dev/stdin --port 8081 --log-requests --pidfile /tmp/puma -t 4:4
Puma starting in single mode...
* Puma version: 7.1.0 ("Neon Witch")
* Ruby version: ruby 3.4.7 (2025-10-08 revision 7a5688e2a2) +PRISM [arm64-darwin24]
*  Min threads: 4
*  Max threads: 4
*  Environment: development
*          PID: 64633
* Listening on http://0.0.0.0:8081
Use Ctrl-C to stop
- Gracefully stopping, waiting for requests to finish
64633: Shutdown timeout exceeded
64633: === Begin thread backtrace dump ===
64633: Thread 1/3: #<Thread:0x000000010431af60 sleep_forever>
64633: /Users/dentarg/src/puma/lib/puma/server.rb:647:in 'Thread#join'
64633: /Users/dentarg/src/puma/lib/puma/server.rb:647:in 'Puma::Server#stop'
64633: /Users/dentarg/src/puma/lib/puma/single.rb:38:in 'Puma::Single#stop_blocked'
64633: /Users/dentarg/src/puma/lib/puma/launcher.rb:286:in 'Puma::Launcher#do_graceful_stop'
64633: /Users/dentarg/src/puma/lib/puma/launcher.rb:450:in 'block in Puma::Launcher#setup_signals'
64633: /Users/dentarg/src/puma/lib/puma/single.rb:66:in 'Thread#join'
64633: /Users/dentarg/src/puma/lib/puma/single.rb:66:in 'Puma::Single#run'
64633: /Users/dentarg/src/puma/lib/puma/launcher.rb:208:in 'Puma::Launcher#run'
64633: /Users/dentarg/src/puma/lib/puma/cli.rb:73:in 'Puma::CLI#run'
64633: ./bin/puma:10:in '<main>'

64633: Thread 2/3: #<Thread:0x000000010483ca20@puma srv tp 001 /Users/dentarg/src/puma/lib/puma/thread_pool.rb:142 sleep>
64633: /dev/stdin:1:in 'Kernel#sleep'
64633: /dev/stdin:1:in 'block in Puma::DSL#_load_from'
64633: /Users/dentarg/src/puma/lib/puma/commonlogger.rb:47:in 'Puma::CommonLogger#call'
64633: /Users/dentarg/src/puma/lib/puma/configuration.rb:301:in 'Puma::Configuration::ConfigMiddleware#call'
64633: /Users/dentarg/src/puma/lib/puma/request.rb:103:in 'block in Puma::Request#handle_request'
64633: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:356:in 'Puma::ThreadPool#with_force_shutdown'
64633: /Users/dentarg/src/puma/lib/puma/request.rb:102:in 'Puma::Request#handle_request'
64633: /Users/dentarg/src/puma/lib/puma/server.rb:503:in 'Puma::Server#process_client'
64633: /Users/dentarg/src/puma/lib/puma/server.rb:262:in 'block in Puma::Server#run'
64633: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:183:in 'block in Puma::ThreadPool#spawn_thread'

64633: Thread 3/3: #<Thread:0x0000000104831b48@puma srv /Users/dentarg/src/puma/lib/puma/server.rb:281 run>
64633: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:433:in 'Thread#backtrace'
64633: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:433:in 'block in Puma::ThreadPool#shutdown_debug'
64633: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:431:in 'Array#each'
64633: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:431:in 'Enumerable#each_with_index'
64633: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:431:in 'Puma::ThreadPool#shutdown_debug'
64633: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:398:in 'Puma::ThreadPool#shutdown'
64633: /Users/dentarg/src/puma/lib/puma/server.rb:625:in 'Puma::Server#graceful_shutdown'
64633: /Users/dentarg/src/puma/lib/puma/server.rb:425:in 'Puma::Server#handle_servers'
64633: /Users/dentarg/src/puma/lib/puma/server.rb:283:in 'block in Puma::Server#run'

64633: === End thread backtrace dump ===
2025-11-25 23:42:36 +0100 Rack app ("GET /" - (127.0.0.1)): #<Puma::ThreadPool::ForceShutdown: Puma::ThreadPool::ForceShutdown>
Detected force shutdown of a thread
zsh: done        echo  |
zsh: terminated  ruby -rlogger -Ilib ./bin/puma --config /dev/stdin --port 8081 --log-requests

In the main branch, threads backtraces are printed but then Puma waits for force_shutdown_after and then exits

$ echo 'force_shutdown_after 5; shutdown_debug; app { |env| sleep 120; [200, {}, ["OK"]] }' | ruby -rlogger -Ilib ./bin/puma --config /dev/stdin --port 8081 --log-requests --pidfile /tmp/puma -t 4:4
Puma starting in single mode...
* Puma version: 7.1.0 ("Neon Witch")
* Ruby version: ruby 3.4.7 (2025-10-08 revision 7a5688e2a2) +PRISM [arm64-darwin24]
*  Min threads: 4
*  Max threads: 4
*  Environment: development
*          PID: 65459
* Listening on http://0.0.0.0:8081
Use Ctrl-C to stop
- Gracefully stopping, waiting for requests to finish
65459: === Begin thread backtrace dump ===
65459: Thread 1/7: #<Thread:0x0000000100dbaf78 sleep_forever>
65459: /Users/dentarg/src/puma/lib/puma/server.rb:668:in 'Thread#join'
65459: /Users/dentarg/src/puma/lib/puma/server.rb:668:in 'Puma::Server#stop'
65459: /Users/dentarg/src/puma/lib/puma/single.rb:38:in 'Puma::Single#stop_blocked'
65459: /Users/dentarg/src/puma/lib/puma/launcher.rb:285:in 'Puma::Launcher#do_graceful_stop'
65459: /Users/dentarg/src/puma/lib/puma/launcher.rb:449:in 'block in Puma::Launcher#setup_signals'
65459: /Users/dentarg/src/puma/lib/puma/single.rb:66:in 'Thread#join'
65459: /Users/dentarg/src/puma/lib/puma/single.rb:66:in 'Puma::Single#run'
65459: /Users/dentarg/src/puma/lib/puma/launcher.rb:207:in 'Puma::Launcher#run'
65459: /Users/dentarg/src/puma/lib/puma/cli.rb:73:in 'Puma::CLI#run'
65459: ./bin/puma:10:in '<main>'

65459: Thread 2/7: #<Thread:0x00000001013ccc40@puma srv tp 001 /Users/dentarg/src/puma/lib/puma/thread_pool.rb:141 sleep>
65459: /dev/stdin:1:in 'Kernel#sleep'
65459: /dev/stdin:1:in 'block in Puma::DSL#_load_from'
65459: /Users/dentarg/src/puma/lib/puma/commonlogger.rb:47:in 'Puma::CommonLogger#call'
65459: /Users/dentarg/src/puma/lib/puma/configuration.rb:300:in 'Puma::Configuration::ConfigMiddleware#call'
65459: /Users/dentarg/src/puma/lib/puma/request.rb:103:in 'block in Puma::Request#handle_request'
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:355:in 'Puma::ThreadPool#with_force_shutdown'
65459: /Users/dentarg/src/puma/lib/puma/request.rb:102:in 'Puma::Request#handle_request'
65459: /Users/dentarg/src/puma/lib/puma/server.rb:503:in 'Puma::Server#process_client'
65459: /Users/dentarg/src/puma/lib/puma/server.rb:262:in 'block in Puma::Server#run'
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:182:in 'block in Puma::ThreadPool#spawn_thread'

65459: Thread 3/7: #<Thread:0x00000001013cc8f8@puma srv tp 002 /Users/dentarg/src/puma/lib/puma/thread_pool.rb:141 sleep_forever>
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:172:in 'Thread::Mutex#sleep'
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:172:in 'Thread::ConditionVariable#wait'
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:172:in 'block (2 levels) in Puma::ThreadPool#spawn_thread'
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:155:in 'Thread::Mutex#synchronize'
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:155:in 'block in Puma::ThreadPool#spawn_thread'

65459: Thread 4/7: #<Thread:0x00000001013cc790@puma srv tp 003 /Users/dentarg/src/puma/lib/puma/thread_pool.rb:141 sleep_forever>
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:172:in 'Thread::Mutex#sleep'
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:172:in 'Thread::ConditionVariable#wait'
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:172:in 'block (2 levels) in Puma::ThreadPool#spawn_thread'
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:155:in 'Thread::Mutex#synchronize'
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:155:in 'block in Puma::ThreadPool#spawn_thread'

65459: Thread 5/7: #<Thread:0x00000001013cc628@puma srv tp 004 /Users/dentarg/src/puma/lib/puma/thread_pool.rb:141 sleep_forever>
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:172:in 'Thread::Mutex#sleep'
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:172:in 'Thread::ConditionVariable#wait'
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:172:in 'block (2 levels) in Puma::ThreadPool#spawn_thread'
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:155:in 'Thread::Mutex#synchronize'
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:155:in 'block in Puma::ThreadPool#spawn_thread'

65459: Thread 6/7: #<Thread:0x00000001013c2100@puma srv tp reap /Users/dentarg/src/puma/lib/puma/thread_pool.rb:322 sleep>
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:326:in 'Kernel#sleep'
65459: /Users/dentarg/src/puma/lib/puma/thread_pool.rb:326:in 'block in Puma::ThreadPool::Automaton#start!'

65459: Thread 7/7: #<Thread:0x00000001013c1cf0@puma srv /Users/dentarg/src/puma/lib/puma/server.rb:281 run>
65459: /Users/dentarg/src/puma/lib/puma/server.rb:631:in 'Thread#backtrace'
65459: /Users/dentarg/src/puma/lib/puma/server.rb:631:in 'block in Puma::Server#graceful_shutdown'
65459: /Users/dentarg/src/puma/lib/puma/server.rb:629:in 'Array#each'
65459: /Users/dentarg/src/puma/lib/puma/server.rb:629:in 'Enumerable#each_with_index'
65459: /Users/dentarg/src/puma/lib/puma/server.rb:629:in 'Puma::Server#graceful_shutdown'
65459: /Users/dentarg/src/puma/lib/puma/server.rb:425:in 'Puma::Server#handle_servers'
65459: /Users/dentarg/src/puma/lib/puma/server.rb:283:in 'block in Puma::Server#run'

65459: === End thread backtrace dump ===




2025-11-25 23:43:38 +0100 Rack app ("GET /" - (127.0.0.1)): #<Puma::ThreadPool::ForceShutdown: Puma::ThreadPool::ForceShutdown>
Detected force shutdown of a thread
zsh: done        echo  |
zsh: terminated  ruby -rlogger -Ilib ./bin/puma --config /dev/stdin --port 8081 --log-requests

dentarg

I think we should try to avoid capture_io

dentarg · 2026-02-20T13:02:06Z

I rebased this. I'll save a link to the commit before rebase from Nov 28, 2025, for anyone curious. be6e624

joshuay03 · 2026-02-24T00:38:58Z

@dentarg Thanks! I completely forgot about this. I'll merge soon.

dentarg · 2026-02-26T22:25:01Z

I started to look at the failing tests after I rebased, but I got distracted. What's up with JRuby/TruffleRuby? A bit telling that all of the JVM based rubies fail?

Requested changes have been addressed.

joshuay03 · 2026-03-08T22:50:54Z

I started to look at the failing tests after I rebased, but I got distracted. What's up with JRuby/TruffleRuby? A bit telling that all of the JVM based rubies fail?

Resolved. TL;DR: Thread.backtrace returning nil for dead threads.

dentarg · 2026-03-11T08:19:22Z

Thank you for solving this @joshuay03! If not sooner, I'll take this for spin when we have it in a release.

joshuay03 mentioned this pull request Jun 29, 2025

Make shutdown_debug more interesting #3666

Closed

joshuay03 moved this to In Progress / Pending Review in Open Source Jun 29, 2025

joshuay03 added this to Open Source Jun 29, 2025

joshuay03 force-pushed the fix-3666 branch from 9dc0dfc to 55736c4 Compare July 9, 2025 02:40

joshuay03 commented Jul 9, 2025

View reviewed changes

joshuay03 force-pushed the fix-3666 branch from 55736c4 to 55a1b5a Compare July 9, 2025 02:50

joshuay03 commented Jul 9, 2025

View reviewed changes

Comment thread lib/puma/server.rb Outdated

joshuay03 force-pushed the fix-3666 branch 5 times, most recently from c851fb3 to d1d8839 Compare July 9, 2025 04:31

joshuay03 marked this pull request as ready for review July 9, 2025 04:56

github-actions Bot added the waiting-for-review Waiting on review from anyone label Jul 9, 2025

joshuay03 force-pushed the fix-3666 branch 2 times, most recently from e659047 to c2797ae Compare July 11, 2025 19:50

joshuay03 commented Jul 11, 2025

View reviewed changes

joshuay03 force-pushed the fix-3666 branch 2 times, most recently from 06cd4dc to 74fa8dc Compare November 22, 2025 12:18

joshuay03 requested a review from dentarg November 22, 2025 12:19

dentarg approved these changes Nov 25, 2025

View reviewed changes

dentarg mentioned this pull request Nov 25, 2025

Fix Ctrl-C, set force_shutdown_after and worker_shutdown_timeout to small values #3763

Open

7 tasks

dentarg reviewed Nov 25, 2025

View reviewed changes

Comment thread test/test_puma_server.rb Outdated

dentarg previously requested changes Nov 25, 2025

View reviewed changes

github-actions Bot added waiting-for-changes Waiting on changes from the requestor and removed waiting-for-review Waiting on review from anyone waiting-for-changes Waiting on changes from the requestor labels Nov 25, 2025

joshuay03 force-pushed the fix-3666 branch from 544e108 to be6e624 Compare November 28, 2025 10:25

github-actions Bot added waiting-for-changes Waiting on changes from the requestor and removed waiting-for-review Waiting on review from anyone labels Jan 20, 2026

github-actions Bot added waiting-for-review Waiting on review from anyone and removed waiting-for-changes Waiting on changes from the requestor labels Feb 12, 2026

dentarg force-pushed the fix-3666 branch from be6e624 to 58ce445 Compare February 20, 2026 13:01

dentarg added the feature label Feb 20, 2026

joshuay03 force-pushed the fix-3666 branch 7 times, most recently from f60582b to 36256ac Compare March 1, 2026 20:23

joshuay03 force-pushed the fix-3666 branch 4 times, most recently from 1518355 to 4c2e3a3 Compare March 8, 2026 22:26

Add on_force option to shutdown_debug

eb06f3c

joshuay03 force-pushed the fix-3666 branch from 4c2e3a3 to eb06f3c Compare March 8, 2026 22:33

joshuay03 merged commit 138ba14 into puma:main Mar 8, 2026
80 checks passed

joshuay03 moved this from In Progress / Pending Review to Done in Open Source Mar 8, 2026

This was referenced Apr 10, 2026

[ruby-on-rails] Bump Puma to 8.0.0 hayat01sh1da/botpress-accuracy-checkers#90

Merged

[ruby-on-rails] Update Gem Dependencies hayat01sh1da/tutorials#223

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `on_force` option to `shutdown_debug`#3671

Add `on_force` option to `shutdown_debug`#3671
joshuay03 merged 1 commit intopuma:mainfrom
joshuay03:fix-3666

joshuay03 commented Jun 29, 2025 •

edited

Loading

Uh oh!

joshuay03 Jul 9, 2025

Uh oh!

Uh oh!

joshuay03 Jul 11, 2025

Uh oh!

joshuay03 Jul 11, 2025

Uh oh!

joshuay03 commented Nov 25, 2025

Uh oh!

dentarg commented Nov 25, 2025

Uh oh!

Uh oh!

dentarg left a comment

Uh oh!

dentarg commented Feb 20, 2026

Uh oh!

joshuay03 commented Feb 24, 2026

Uh oh!

dentarg commented Feb 26, 2026

Uh oh!

joshuay03 commented Mar 8, 2026

Uh oh!

Uh oh!

dentarg commented Mar 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		if @shutdown_debug == :on_force && !threads.empty?
		shutdown_debug("Shutdown timeout exceeded")

Conversation

joshuay03 commented Jun 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Your checklist for this pull request

Uh oh!

joshuay03 Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joshuay03 Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

joshuay03 Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

joshuay03 commented Nov 25, 2025

Uh oh!

dentarg commented Nov 25, 2025

Uh oh!

Uh oh!

dentarg left a comment

Choose a reason for hiding this comment

Uh oh!

dentarg commented Feb 20, 2026

Uh oh!

joshuay03 commented Feb 24, 2026

Uh oh!

dentarg commented Feb 26, 2026

Uh oh!

joshuay03 commented Mar 8, 2026

Uh oh!

Uh oh!

dentarg commented Mar 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

joshuay03 commented Jun 29, 2025 •

edited

Loading