The preprocessor is a text-to-text transformation pass that processes C preprocessing directives before lexing. It consumes raw source text and produces expanded source text annotated with GCC-compatible line markers, ready for the lexer/parser pipeline.

The preprocessor implements the ISO C11 translation phases 2-4:

1. Line splicing -- backslash-newline sequences are joined.
2. Comment replacement -- /* ... */ block comments are replaced with a single space (per C11 5.1.1.2 phase 3); // ... line comments are stripped.
3. Directive processing and macro expansion -- #define, #include, #if/#ifdef/#elif/#else/#endif, #pragma, #error, #warning, #line, and #undef are handled. All macro references in non-directive lines are expanded.

The entire pass operates on raw text (not tokens). All scanning is done on byte slices (&[u8]) for performance, since C preprocessor tokens are ASCII. UTF-8 multi-byte sequences inside string and character literals are copied verbatim without interpretation.

                 ┌───────────────────────────────────────────┐
  raw source ──> │  join_continued_lines  (phase 2)         │
                 │  strip_block_comments  (phase 3)         │
                 │  ┌─────────────────────────────────────┐ │
                 │  │  per-line loop:                      │ │
                 │  │    directive?  → process_directive() │ │
                 │  │    active?    → expand macros        │ │
                 │  │    inactive?  → emit blank line      │ │
                 │  └─────────────────────────────────────┘ │
                 └──────────────┬────────────────────────────┘
                                │
                    expanded text + line markers
                                │
         ┌──────────────────────┼──────────────────────────┐
         │ side channels:       │                          │
         │  errors[]            │  warnings[]              │
         │  weak_pragmas[]      │  redefine_extname[]      │
         │  macro_expansion_info[]                         │
         └─────────────────────────────────────────────────┘

The output is a single String containing expanded C source with embedded line markers of the form # <line> "<file>" [flags] (following GCC conventions: flag 1 = entering an include, flag 2 = returning from an include). The downstream lexer uses these markers to maintain accurate source locations.

Every macro is stored in a MacroTable (a FxHashMap<String, MacroDef>) which also carries an asm_mode flag. When asm_mode is true, $ is not treated as an identifier character during macro expansion, so that AT&T assembly immediates like $MACRO_NAME correctly trigger expansion of MACRO_NAME. Each macro definition carries these fields:

pub struct MacroDef {
    pub name: String,
    pub is_function_like: bool,
    pub params: Vec<String>,       // empty for object-like macros
    pub is_variadic: bool,         // last param is `...`
    pub has_named_variadic: bool,  // e.g., `args...` (GNU extension)
    pub body: String,              // replacement text
}

#define VERSION 42

Expansion is straightforward: every occurrence of the identifier in non-directive text is replaced with the body. After substitution, the result is rescanned for further macro expansion.

#define MAX(a, b) ((a) > (b) ? (a) : (b))

Expansion follows the C11 algorithm (sections 6.10.3 -- 6.10.3.4):

1. Argument collection -- parse_macro_args matches the opening ( through the balanced closing ), splitting on commas at depth 0. String and character literals are skipped so that commas and parentheses inside them do not confuse the parser.

2. Argument prescan -- every argument is fully macro-expanded (expand_text) before substitution. Arguments adjacent to # or ## use the raw (unexpanded) form instead.

3. Stringification and token pasting -- handle_stringify_and_paste processes #param and a ## b operators (see below).

4. Parameter substitution -- substitute_params replaces parameter names in the body with (expanded) argument text.

5. Rescanning -- the substituted body is re-expanded with the macro's own name added to the "currently expanding" set, preventing self-referential recursion.

#param wraps the raw (unexpanded) argument text in double quotes, escaping embedded " and \ characters per C11 6.10.3.2.

a ## b concatenates the text of the left and right operands into a single token. When an operand is a parameter, the raw argument text is used (not the prescan-expanded text). The pasted result is wrapped in internal paste-protection markers (0x02/0x03) to prevent re-substitution during the same parameter replacement pass; these markers are stripped before the final output.

An anti-paste guard (would_paste_tokens) inserts protective spaces between adjacent tokens that would otherwise form unintended multi-character tokens (e.g., two / becoming a // comment, two + becoming ++).

#define LOG(fmt, ...) printf(fmt, __VA_ARGS__)

The variadic arguments (everything after the last named parameter) are collected as a single comma-separated string. Both the standard ... / __VA_ARGS__ form and the GNU named-variadic extension (args...) are supported.

The GNU , ## __VA_ARGS__ extension is implemented: when __VA_ARGS__ is empty and appears to the right of ## preceded by a comma, the comma is removed. This enables the common pattern:

#define DBG(fmt, ...) fprintf(stderr, fmt, ## __VA_ARGS__)
DBG("hello")  // expands to: fprintf(stderr, "hello")

A "currently expanding" set (FxHashSet<String>) tracks which macros are actively being expanded. When an identifier matches a macro in this set, it is blue-painted -- prefixed with a sentinel byte (0x01) that prevents it from being recognized as a macro during rescanning. All blue-paint markers are stripped from the final output.

This set is allocated once per preprocessing run and cleared/reused for each source line (expand_line_reuse), avoiding per-line allocation overhead.

When a source line has unbalanced parentheses (more ( than )), the preprocessor accumulates subsequent lines into a pending_line buffer until parentheses balance. This handles macro invocations whose arguments span multiple lines:

SOME_MACRO(
    arg1,
    arg2,
    arg3
)

A safety limit (MAX_PENDING_NEWLINES = 100,000) prevents runaway accumulation from genuinely unbalanced source. The preprocessor also detects when a line ends with a function-like macro name (even after expansion) and accumulates the next line in case it starts with (.

Conditional directives (#ifdef, #endif, etc.) that appear inside accumulated multi-line arguments are still processed to keep the conditional stack correct, but are not added to the pending text.

The ConditionalStack maintains a Vec<ConditionalState> where each entry tracks:

• Whether any branch in the #if/#elif/#else chain has been taken.
• Whether the current branch is active.
• Whether the parent context is active (for proper nesting).

Lines inside an inactive branch are replaced with empty lines to preserve line numbering. Nested #if blocks within inactive branches are tracked (push/pop) but their conditions are not evaluated.

#if and #elif expressions go through a multi-stage pipeline:

1. resolve_defined_in_expr -- Replace defined(X), defined X, __has_builtin(X), __has_attribute(X), __has_feature(X), __has_extension(X), __has_include(X), and __has_include_next(X) with 1 or 0.

2. Macro expansion -- The resolved expression is expanded through the standard macro expansion pipeline.

3. Second resolve_defined_in_expr pass -- Catches cases where macro expansion introduced new __has_*() calls.

4. replace_remaining_idents_with_zero -- Per the C standard, any remaining identifiers (except true/false) evaluate to 0.

5. evaluate_condition -- First expands simple object-like macros in the condition, then evaluates via eval_const_expr.

The expression evaluator (eval_const_expr) is a recursive-descent parser supporting:

• Integer literals (decimal, hex 0x, octal 0) with U/L/LL/ULL suffixes.
• Character literals with escape sequences.
• defined(X) / defined X.
• The ternary operator ? :.
• Logical: &&, ||, !.
• Bitwise: &, |, ^, ~.
• Shifts: <<, >>.
• Relational: ==, !=, <, >, <=, >=.
• Arithmetic: +, -, *, /, % (including unary - and +).
• Parenthesized sub-expressions.

Per C99 6.10.1, arithmetic uses intmax_t (signed) or uintmax_t (unsigned) depending on whether any operand has a U suffix. Hex and octal literals that exceed INT64_MAX are implicitly unsigned.

The search order is GCC-compatible and depends on the include style:

Quoted includes (#include "file.h"):

1. Directory of the currently-processed file.
2. Directory of the original source file.
3. -iquote paths.
4. -I paths.
5. -isystem paths.
6. Default system paths (bundled headers, /usr/include, GCC internal headers).
7. -idirafter paths.

System includes (#include <file.h>):

1. -I paths.
2. -isystem paths.
3. Default system paths.
4. -idirafter paths.

Path resolution results are cached in an include_resolve_cache (FxHashMap) keyed by (include_path, is_system, current_dir) to avoid repeated filesystem stat() calls.

Paths are made absolute without resolving symlinks (make_absolute instead of canonicalize), matching GCC behavior where #include "..." searches are relative to the symlink location, not the symlink target.

#include_next searches for the header starting from the next include path after the one that contained the currently-processed file. This is used by system headers to layer architecture-specific headers over generic ones. Falls back to regular #include resolution if the current file cannot be located in any search path.

When the #include argument does not start with " or <, it is macro-expanded first. All spaces in the resulting path are stripped by normalize_include_path to remove both anti-paste spaces (inserted by would_paste_tokens to prevent // comment formation) and inter-token spaces preserved from macro bodies. This normalization is only applied to macro-expanded paths; direct include paths are left unchanged.

• Maximum depth of 200 (matching GCC's default).
• Files without #pragma once or include guards may be re-included (intentional, to support patterns like TCC's conditional self-inclusion).
• Only excessive nesting of the same file path triggers the depth limit.

After preprocessing an included file, the raw source is scanned for the classic include guard pattern:

#ifndef GUARD_MACRO
#define GUARD_MACRO
  ...
#endif

Detection is intentionally conservative -- it returns None for files with #else/#elif at the outermost level, code before #ifndef, or content after #endif. When a guard is detected and the guard macro is still defined on subsequent #include of the same file, re-processing is skipped entirely (same optimization as GCC and Clang).

When well-known standard headers are included (e.g., stdarg.h, stdbool.h, complex.h), the preprocessor injects compiler-builtin macro definitions regardless of whether the real system header is found. For example, including stdarg.h defines va_start, va_end, va_copy, and va_arg as macros expanding to __builtin_* forms. The va_list, __builtin_va_list, and __gnuc_va_list types are handled natively by the parser/sema/lowerer pipeline (not injected as typedef text, which would break when stdarg.h is included from nested headers).

When a standard header file cannot be found on disk (no system headers installed or cross-compiling without a sysroot), the preprocessor injects minimal fallback declarations so that compilation can proceed. These are only injected when the real header is absent -- when a project provides its own headers (e.g., musl, dietlibc), the fallbacks are not used and cannot cause conflicts.

Pragmas are dispatched by handle_pragma and produce either side-channel data or synthetic tokens injected into the output for the parser.

Unrecognized pragmas (including #pragma GCC diagnostic ...) are silently ignored.

On construction, define_predefined_macros populates the macro table from a static table of (name, body) pairs. Categories include:

__LINE__ and __COUNTER__ are not stored in the macro table. They use Cell<usize> fields on MacroTable and are expanded by special-case code in expand_identifier, avoiding per-line MacroDef allocation:

__FILE__ and __BASE_FILE__ are stored in the macro table as regular object-like macros but have dedicated update APIs to avoid full MacroDef reallocation on every #include:

These are recognized by is_defined() for #ifdef and evaluated specially in #if expressions by resolve_defined_in_expr:

• __has_builtin(X) -- returns 1 for supported compiler builtins.
• __has_attribute(X) -- returns 1 for supported GCC attributes.
• __has_feature(X) / __has_extension(X) -- always 0.
• __has_include(X) / __has_include_next(X) -- probes the include path resolution machinery and returns 1 if the header exists.

Since the preprocessor can operate without system headers, builtin_macros.rs defines essential macros that would normally come from standard headers:

Additionally, define_type_traits_macros provides macros that supplement the standard headers or serve as fallbacks when system headers are unavailable:

The integer constant macros (INT64_C, UINT32_C, etc.) are function-like macros that use ## for suffix pasting (e.g., INT64_C(x) expands to x ## LL).

The returned String from preprocess() contains:

• An initial line marker: # 1 "filename".
• Expanded source lines with macros resolved.
• Blank lines preserving the original line numbering.
• Line markers at include boundaries (# 1 "header.h" 1 on entry, # N "parent.c" 2 on return).
• Synthetic pragma tokens (e.g., __ccc_pack_set_4 ;).

PreprocessorDiagnostic carries file, line, col, and message fields, enabling GCC-compatible file:line:col: error: / warning: output formatting.

The preprocessor operates on raw text strings rather than a token stream. This is a deliberate trade-off:

• Simplicity: no need for a tokenizer that handles both pre- and post-expansion token grammars.
• Performance: byte-slice scanning with &[u8] avoids allocation overhead. Hot paths use bytes_to_str (unchecked UTF-8 conversion for ASCII identifiers) and batch slice copies instead of per-character operations.
• Compatibility: the output is a flat string with line markers, which is the same interface GCC and Clang present to their parsers via -E.

The FxHashSet<String> used for tracking "currently expanding" macros is allocated once and reused across all lines via expand_line_reuse. A separate reusable set (directive_expanding) is kept for directive-level expansions (#if, #elif, #line, #error). This eliminates a measurable per-line allocation overhead when preprocessing large headers.

Self-referential macro names are marked with a 0x01 prefix byte during expansion. This is a single-byte sentinel chosen because it cannot appear in valid C source. The markers survive through rescanning (preventing re-expansion) and are stripped in one final pass over the output string. Paste-protection markers (0x02/0x03) serve a similar role for ## results.

Several expansion helpers return Cow<str> to avoid allocating when the common-case fast path applies (e.g., handle_stringify_and_paste returns Cow::Borrowed(&mac.body) when the body contains no # characters, strip_block_comments returns Cow::Borrowed(source) when there are no block comments).

The preprocessor claims __GNUC__ 14, __GNUC_MINOR__ 2 to satisfy version checks in the Linux kernel, glibc, QEMU, and other major projects. The predefined macro set is tuned for an LP64 Linux/ELF/x86-64 target by default, with set_target providing overrides for aarch64, riscv64, i686, and i386.

• __DATE__ and __TIME__ are static -- they expand to compile-time constants rather than the actual build timestamp.
• _Float128 is aliased to long double -- on x86-64, this loses the difference between 80-bit extended precision and IEEE binary128.
• __has_feature and __has_extension always return 0 -- no Clang feature set is modeled.
• _Pragma("...") is skipped -- the C99 _Pragma operator is consumed and discarded during macro expansion rather than being executed.
• No #embed support -- the C23 #embed directive is not implemented.
• No digraph/trigraph processing -- trigraphs (??=, ??/, etc.) and digraphs (<:, :>) are not translated in the preprocessor phase.
• Expression evaluator does not support floating-point -- #if expressions are evaluated using integer arithmetic only (standard-compliant, since preprocessor expressions are integer constant expressions).