Root Cause Analysis: "Too many open files" on macOS Docker (VirtioFS)

Issue: #14307

Executive Summary

Environment: 12 CPU cores, macOS + OrbStack/VirtioFS, ~24,000 PHP files in ~9,200 dirs + ~16,892 vendor files in ~4,603 dirs. Docker container ulimits explicitly set to nofile: soft=1048576, hard=1048576.

The core cause is 8 worker processes each independently performing redundant filesystem scanning and file hashing during initialization. This is the only variable that changes between failing cold runs (8 workers) and succeeding warm runs (1-2 workers).

With 12 CPU cores, the Scheduler computes 12 ideal processes but maximumNumberOfProcesses caps it at 8 (default in
conf/config.neon:95). Each worker repeats ~219,000 filesystem operations that the main process already completed. With 8 concurrent workers, this produces ~1,750,000 redundant FUSE operations — all funneled through VirtioFS's FUSE layer, which has inherently higher latency and limited concurrency compared to native filesystems.

Identified exhaustion mechanism: The EMFILE error originates from the macOS HOST side, not the container. Apple's VirtioFS implementation (via Virtualization.framework) holds an O_PATH file descriptor for every accessed inode and does not proactively close them — they are released only on FUSE_FORGET (guest kernel dentry cache eviction). With ~54,700 unique inodes (files + dirs) accessed by 8 concurrent workers, this can exceed the macOS per-process limit
kern.maxfilesperproc (24,576–122,880 depending on hardware/RAM). The container's own ulimit (1,048,576) is far above per-process FD usage (~45 peak), confirming the bottleneck is not inside the container.

Proven Facts (from code analysis)

1. Worker Initialization Call Chain

Each worker process follows this path:

WorkerCommand::execute()
  -> CommandHelper::begin()
    -> ContainerFactory::create()
      -> postInitializeContainer()
        -> getService('betterReflectionSourceLocator')
          -> BetterReflectionSourceLocatorFactory::create()
            -> for each directory:
              OptimizedDirectorySourceLocatorRepository::getOrCreate()
                -> OptimizedDirectorySourceLocatorFactory::createByDirectory()
                  1. FileFinder::findFiles()
                  2. hash_file() on ALL files
                  3. createCachedDirectorySourceLocator()
                     -> cache->load()                        [FileCacheStorage - disk]
                     -> symbolFinderInFiles->findSymbols()   [only on cache miss]
                     -> cache->save()

2. PHP Internal FD Behavior at Each Step

Symfony Finder (FileFinder.php:32-59):
Uses RecursiveDirectoryIterator + RecursiveIteratorIterator internally. Holds one directory handle per recursion depth level (stack-based). After foreach: iterator destroyed, 0 directory FDs retained. followLinks() uses lstat()/stat() — no additional FDs. Peak: O(depth) = ~5-10 dir FDs.

hash_file('sha256', $file) — PHP C code (ext/hash/hash.c): php_stream_open_wrapper_ex() → php_stream_read() (1024B chunks) → php_stream_close(). All synchronous within one C function call. FD opened and closed BEFORE returning to userspace. Peak: 1 FD.

php_strip_whitespace($file) — PHP C code (ext/tokenizer): zend_stream_open() → tokenize → close. Peak: 1 FD, synchronous.

file_get_contents() (FileReader.php:18): php_stream_open_wrapper() → read → php_stream_close(). Peak: 1 FD, synchronous.

include $filePath (FileCacheStorage.php:53): Opens file, reads, compiles to opcodes, closes FD. Opcodes stored in memory, not as open FD. Peak: 1 FD.

is_file(), file_exists(), is_dir(): Use stat(2) syscall. No FD opened.

3. FD Inheritance via proc_open()

PHP's tmpfile() calls libc tmpfile() which does NOT set O_CLOEXEC. Similarly, stream_socket_server() (TCP server) does not set SOCK_CLOEXEC. These FDs survive fork()+exec() in proc_open().

However, PHP's proc_open() (in ext/standard/proc_open.c) DOES set FD_CLOEXEC on parent-end pipe FDs via make_descriptor_cloexec() → fcntl(fd, F_SETFD, FD_CLOEXEC). This means pipe FDs are closed on exec and do NOT accumulate across workers.

The spawning loop in ParallelAnalyser.php:170 is synchronous — all 8 workers are spawned before the event loop runs. Each new worker inherits all previously created tmpfile FDs (but NOT pipe FDs):

Before loop:     parent has ~5 FDs (base + TCP server)
Spawn worker 1:  parent +2 tmpfiles = 7 FDs   → child 1 inherits ~7
Spawn worker 2:  parent +2 = 9                 → child 2 inherits ~9
Spawn worker 3:  parent +2 = 11                → child 3 inherits ~11
...
Spawn worker 8:  parent +2 = 19-20             → child 8 inherits ~20

Inherited FDs: 7 × 2 tmpfile FDs (14) + 1 TCP server socket + ~5 base FDs = ~20 inherited FDs for worker 8. These inherited FDs are useless to the child (it never uses them) but consume FD table entries.

4. Actual Peak FD Count Per Process

Worker 8 (worst case):

Worker 1 (best case): ~15-20 FDs (minimal inheritance).

Main process (after all workers spawned): ~30 FDs.

With a container ulimit of 1,048,576, even the worst-case ~43 FDs per process is negligible. EMFILE cannot originate from inside the container.

5. Source Locator Cache IS Shared via Filesystem

Cache → FileCacheStorage writes PHP files to a shared cache directory. Cache keys are deterministic (sprintf('odsl-%s', $directory) + enum-support variant key). The main process populates this cache BEFORE workers are spawned:

1. Main process: CommandHelper::begin() → full scan + hash + parse → cache->save() to disk
2. Workers spawned (via ParallelAnalyser::analyse())
3. Each worker: scan + hash → cache->load() → HIT → hashes match → skips php_strip_whitespace

Even on a cold run, workers skip php_strip_whitespace. But they STILL perform Finder scan + hash_file.

6. Worker Count Depends on File Count

// Scheduler.php:48-53
$numberOfProcesses = min(
    max((int) floor(count($jobs) / $this->minimumNumberOfJobsPerProcess), 1),
    $cpuCores,
);
$usedNumberOfProcesses = min($numberOfProcesses, $this->maximumNumberOfProcesses);
// maximumNumberOfProcesses default = 8  (conf/config.neon:95)

Source links: Scheduler.php:48-53 — conf/config.neon:95

With 12 CPU cores:

• Cold run, 24K files: 24000/20 = 1200 jobs → min(floor(1200/2), 12) = 12 → capped to 8 workers
• Warm run, 50 changed files: 50/20 = 3 jobs → min(floor(3/2), 12) = 1 → 1 worker
• Warm run, 200 changed files: 200/20 = 10 jobs → min(floor(10/2), 12) = 5 → 5 workers

Per-Worker Filesystem Operations (Redundant)

Each worker, regardless of cache state:

With 8 workers: ~1,750,000 redundant FUSE operations during initialization.

Main process performs the same ~219,000 ops independently, so total is ~1,969,000 FUSE ops — of which ~1,750,000 (89%) are redundant.

Identified FD Exhaustion Mechanism

Apple VirtioFS inode caching on the macOS host

The EMFILE error originates from the macOS HOST, not from inside the Docker container. The mechanism:

1. Apple's VirtioFS implementation (via Virtualization.framework, used by OrbStack on macOS) runs a host process (com.apple.Virtualization.VirtualMachine) that services all FUSE requests from the guest VM.

2. O_PATH FD per inode: For each filesystem inode accessed by the guest, the host process opens and holds an O_PATH file descriptor. These FDs are used for efficient caching — allowing the host to service subsequent requests for the same inode without path resolution.

3. FDs are NOT proactively closed: These inode-tracking FDs are released only when the guest kernel sends FUSE_FORGET (triggered by dentry cache eviction in the Linux VM). Under heavy load with large working sets, the guest kernel retains dentry cache entries, delaying FUSE_FORGET messages.

4. Scale of the problem: With ~24,000 PHP files + ~16,892 vendor files in ~13,800 directories = ~54,700 unique inodes. When 8 workers concurrently scan all these files, the host process accumulates FDs for all accessed inodes.

5. macOS per-process limit: kern.maxfilesperproc on Apple Silicon ranges from 24,576 (lower-end configs, e.g. 8GB RAM) to 122,880 (higher-end configs). On lower-end machines, ~54,700 inode FDs can approach or exceed the limit — especially when combined with the host process's own operational FDs.

6. EMFILE propagation: When the host process hits its FD limit, the resulting EMFILE errno propagates through the FUSE protocol to the guest. The FUSE response header (fuse_out_header.error) carries the negated errno value directly. The guest kernel surfaces this to the PHP process as "Too many open files".

7. macOS 14 (Sonoma) improvement: Apple improved FD recycling in Sonoma, but 8 concurrent workers accessing ~54,700 inodes can still overwhelm the recycling mechanism.

Why container-side limits are ruled out

The reporter's Docker configuration explicitly sets nofile: soft=1048576, hard=1048576. With peak per-process FD usage of ~43 (worker 8 worst case), the container's own ulimit is 4+ orders of magnitude above actual usage. This definitively eliminates:

• Container ulimit -n as a bottleneck
• Docker VM kernel file-max (set proportionally high by OrbStack)

Why FUSE queue limits are ruled out

The Linux FUSE client has max_background (default 12) and congestion_threshold (default 9). However:

• max_background only limits background requests (readahead, writeback, FUSE_RELEASE). The requests PHPStan generates — OPEN, STAT, READDIR — are foreground requests and are NOT subject to this limit.
• Exceeding max_background causes blocking (the requesting task sleeps on blocked_waitq), NOT EMFILE or request rejection.
• FUSE_RELEASE (file close notification) is sent asynchronously as a background request, which means close notifications to the host can be delayed — contributing to the host-side FD accumulation described above.

Correlation with observed behavior

The 8 concurrent workers not only multiply FUSE operations but also delay FUSE_FORGET messages (the guest kernel retains dentry entries under memory pressure from 8 processes), causing the host to hold FDs longer.

Why Redundant Scanning Is Definitively the Core Cause

This is a proof by elimination:

The ONLY difference between cold and warm runs is the number of workers (determined by files-to-analyze count via the Scheduler). Each worker performs the same initialization filesystem operations regardless — the Finder scan and hash_file are NOT proportional to how many files the worker will analyze. The 2-8x difference in concurrent FUSE load is what pushes the host-side VirtioFS process past its FD limit.

This proves:

• It's not about php_strip_whitespace (skipped in both cases via cache)
• It's not about tmpfile leaks (those are secondary, during error handling)
• It's not about any specific PHP function's FD behavior (all are sequential, 1 FD)
• It's not about the container's own ulimit (1,048,576 — far above usage)
• It IS about 8 workers redundantly repeating the main process's filesystem work, causing host-side VirtioFS FD exhaustion

The Fix

This issue only manifests in virtualized environments on macOS (Docker via OrbStack/Docker Desktop using VirtioFS). On native filesystems there is no FUSE intermediary holding FDs. The bottleneck is Apple's VirtioFS holding an O_PATH FD per accessed inode (released only on FUSE_FORGET) combined with macOS kern.maxfilesperproc (24,576–122,880) applying to the single host-side VirtioFS process. With 8 workers accessing ~54,700 inodes concurrently, this limit is exceeded. Container-side limits (1M ulimit, ~33-43 FDs peak) are not the issue.

PHPStan cannot control host OS settings. The only lever it controls is how many FUSE operations it generates — and 89% are redundant. Eliminating them keeps the host-side VirtioFS process below its FD limit, resolving the "Too many open files" error without requiring any environment changes.

What workers currently do (redundantly)

Each worker runs OptimizedDirectorySourceLocatorFactory::createByDirectory() which unconditionally:

1. Scans all directories via FileFinder::findFiles() (~13,800 dirs → ~55,000 FUSE ops)
2. Computes hash_file('sha256') on every file (~40,900 files → ~122,700 FUSE ops)
3. Only THEN checks the filesystem cache (Cache::load())

Even when the cache contains valid data (cache HIT on warm runs), steps 1 and 2 are always performed for cache validation. The scan and hash are the expensive operations — they account for ~219,000 FUSE ops per worker. The symbol discovery (SymbolFinderInFiles::findSymbols()) IS skipped on cache hit, but it's not the bottleneck.

What workers need

Workers need three mapping arrays from OptimizedDirectorySourceLocator:

• classToFile: array<string, string> — lowercase FQCN → file path
• functionToFiles: array<string, array<int, string>> — lowercase function name → file paths
• constantToFile: array<string, string> — constant name → file path

These are pure data — arrays of strings, fully serializable. The OptimizedDirectorySourceLocator also holds three service references (FileNodesFetcher, Cache, PhpVersion), but these are NOT part of the shared data — each worker creates its own instances via its DI container.

Approach: share pre-computed mapping data via temp file

Step 1: Main process collects mapping data (already computed)

The main process already computes all ODSL mappings during CommandHelper::begin() → ContainerFactory::create() → postInitializeContainer() → BetterReflectionSourceLocatorFactory::create(). After begin() returns, all mapping data exists in memory in the OptimizedDirectorySourceLocatorRepository (per-directory locators) and the single createByFiles() instance (vendor files).

Step 2: Serialize to temp file

Before spawning workers in ParallelAnalyser::analyse(), serialize all mapping data to a temp file in $tmpDir. Structure:

[
    'directories' => [
        '/path/to/src' => [$classToFile, $functionToFiles, $constantToFile],
        '/path/to/vendor/pkg' => [$classToFile, $functionToFiles, $constantToFile],
        // ...
    ],
    'byFiles' => [
        'odsl-installed-files' => [$classToFile, $functionToFiles, $constantToFile],
    ],
]

Estimated size: ~3-5 MB for ~40,900 files. Format: PHP serialize() (fastest) or var_export() (consistent with FileCacheStorage pattern).

Step 3: Workers load pre-computed data instead of scanning

Pass the temp file path as a new --odsl-data CLI argument to workers. This follows the existing pattern used by --tmp-file/--instead-of. The path flows through CommandHelper::begin() → ContainerFactory::create() as a DI container dynamic parameter.

Critical timing detail: postInitializeContainer() calls $container->getService('betterReflectionSourceLocator') which triggers BetterReflectionSourceLocatorFactory::create() → OptimizedDirectorySourceLocatorRepository::getOrCreate() for every directory. The pre-computed data must be available as a container parameter BEFORE this service creation occurs. The DI dynamic parameter mechanism satisfies this constraint — parameters are set before createContainer() at ContainerFactory::create().

When the parameter is set, OptimizedDirectorySourceLocatorRepository::getOrCreate() constructs OptimizedDirectorySourceLocator directly from the pre-computed arrays + worker-local services (FileNodesFetcher, Cache, PhpVersion), skipping FileFinder::findFiles() and hash_file() entirely.

Two entry points that must be covered

ODSL instances are created through two independent code paths:

1. Repository path: OptimizedDirectorySourceLocatorRepository::getOrCreate($directory) — called from BetterReflectionSourceLocatorFactory::create() and ComposerJsonAndInstalledJsonSourceLocatorMaker::create() for scanned directories.

2. Direct factory path: OptimizedDirectorySourceLocatorFactory::createByFiles($files, 'odsl-installed-files') — called from ComposerJsonAndInstalledJsonSourceLocatorMaker::create() for individual vendor files from Composer classmap/files autoloading.

Both paths perform FileFinder scan + hash_file() on all files. Both must check for pre-computed data and skip scanning when available.

Cleanup

Delete the temp file after ParallelAnalyser::analyse() completes (in a finally block). If PHPStan crashes before cleanup, stale files accumulate in $tmpDir — consistent with existing cache behavior.

Impact

This eliminates ~1,750,000 redundant FUSE operations (89% of total) during worker initialization. Each worker's init reduces from ~219,000 FUSE ops to ~1 (reading the temp file). No changes to analysis behavior, no backward compatibility concerns (all affected classes are final, not @api), no memory increase (same data, different delivery mechanism).

Secondary improvement: close-on-exec for tmpfiles

PHP's proc_open() already sets FD_CLOEXEC on parent-end pipe FDs (via make_descriptor_cloexec() in ext/standard/proc_open.c), so pipe FDs do not accumulate across workers. However, tmpfile() FDs (from libc) and the TCP server socket lack O_CLOEXEC and DO accumulate: worker 8 inherits ~20 FDs (14 tmpfile + 1 TCP + ~5 base) that are useless to it.

This is a hygiene improvement, not a fix for the reported issue. The ~20 inherited FDs are 4+ orders of magnitude below the container's 1M ulimit, and the bottleneck is host-side VirtioFS FD exhaustion from ~1,750,000 FUSE operations — not container-side FD count. PHP does not expose fcntl() natively, so setting O_CLOEXEC requires either FFI or restructuring tmpfile creation timing.

Appendix: Investigated and Ruled Out as Root Cause

During analysis, several other code paths were examined as potential sources of the EMFILE error. They are documented here to explain why they were ruled out — and in one case, to flag a real but secondary bug.

FileCacheStorage::clearUnusedFiles() (lines 118-185)

Why investigated: This method iterates over cached files on disk, which could theoretically open many FDs if it ran during worker initialization.

Why ruled out: It only runs once per cache format upgrade (guarded by the CACHED_CLEARED_VERSION constant). It is not triggered during normal worker startup. Even when it does run, it iterates directories sequentially with O(depth) + 1 open FDs at any point — far below any system limit.

tmpfile leaks in Process.php quit()

Why investigated: When workers crash, cleanup code in quit() may fail to call fclose() on tmpfile handles (specifically when the exit callback does not fire). Leaked FDs could accumulate and contribute to EMFILE.

Why ruled out as root cause: These leaks are real bugs, but they are a consequence of the failure, not its trigger. The causal chain is: redundant FUSE operations → host-side VirtioFS FD exhaustion → EMFILE → worker crash →
quitAll() → tmpfile FDs not closed → additional (but secondary) FD pressure. The tmpfile leaks cannot cause the initial EMFILE because they only occur after workers have already started failing. On a healthy run where no workers crash, these code paths are never reached.

Sources

• Apple Developer Forums #741572 — VirtioFS host-side FD behavior, inode caching
• Linux kernel fs/fuse/inode.c — FUSE_DEFAULT_MAX_BACKGROUND = 12 definition
• Linux kernel fs/fuse/dev.c — Background request blocking on blocked_waitq (not EMFILE)
• Linux kernel fs/fuse/file.c — FUSE_RELEASE sent asynchronously as background request
• virtiofsd GitLab issue #15 — Panic when reaching max opened file descriptor
• OrbStack issue #2255 — "too many open files" with bind mounts on macOS
• PHP ext/standard/proc_open.c — make_descriptor_cloexec() sets FD_CLOEXEC on parent-end pipe FDs
• moby/moby PR #45534 — Normalize RLIMIT_NOFILE to sensible defaults