Emacs Internal #02: Data First — Deconstructing Lisp_Object in C

2026-03-05T00:00:00+00:00

In the first part of this GNU Emacs series, I focused on the history and explains why there is a Lisp interpreter embedded inside a text editor. Before diving into this part, I recommend reading the previous post:

The Mathematical Foundation: McCarthy's Lisp</h2>
Before diving into the source code, I left a short reference on Lisp here. Feel free to skip it if you are familiar with its background.

Wiki - Lisp</a> (LISt Processing)</li>
The Roots of Lisp</a> - Paul Graham</li>
How Lisp Became God's Own Programming Language</a> - Two-Bit history</li> </ul>
First Principle: Data and Operations</h2>
This is how I personally approach reading source code: I start from how general computation works.

Given some data, and some operation, then we get a new piece of data </blockquote>
Starting with the very basic, `3 + 4 = 7</code>. The data is 3</code> and 4</code>. The operation is +</code>.`
`If we pile up the abstractions of basic math operations with data abstractions:`
`Complex numbers: $$ (a + bi)(c + di) = (ac - bd) + (ad + bc)i $$</li> Matrix multiplication: $$ C_{ij} = \sum_{k=1}^{n} A_{ik} B_{kj} $$</li> Convolution: $$ (f * g)(t) = \int_{-\infty}^{\infty} f(\tau)g(t - \tau), d\tau $$</li>`A step function: $$ H(x) = \begin{cases} 1 & \text{if } x \ge 0 \ 0 & \text{if } x < 0 \end{cases} $$</li> </ul> From mathematical computation to a Von Neumann machine, the computation can be lowered through IRs and eventually to assembly code. op rd, r1, r2 </code></pre> When I think about compilers here, I usually picture SSA form at this stage. At this level, the model is brutally clean: Data is a sequence of bits in the memory hierarchy, waiting to be fetched into a register.</li> Operations are high-level semantics that the compiler lowers — pass by pass, IR by IR — until they become the native instruction set the silicon actually understands.</li> </ul> This leads me to three things: First, modern compilers work from this first principle. LLVM's main challenge is to merge, traverse, and select a sequence of instructions so that we have the least computation time (usually on a single core). MLIR aims to unify lowering across heterogeneous hardware targets, especially when the hardware supports domain-specific operations like convolution, matrix multiplication, and precision conversion. (MLIR is the next planned series to dive in.) Second, the idea that code is data, and data is code keeps showing up for me. Data is just bits; instructions are also just bit patterns stored in memory. The instruction stream lives in the same memory hierarchy as the data it operates on, and the CPU treats some bits as "code" only because the program counter (PC) points to them. (Thanks for error correct by r/emacs community) Lisp machines of the day were not a different architectural alternative to "von Neumann machine". They had RAM, disk, cpu, some peripherals attached to them. Basically a high-end workstations that just happened to have an OS written in Lisp, and bunch of user's program also implemented in Lisp, a lisp compiler, and so on. Differences in computational environment, and how one worked with them existed, but from the hardware point of view, Lisp machines, were also von Neumann architecture. They do seem to had some stuff implemented in hardware that accelerated certain computations important for Lisp. But architecturally, this isn't different than say Intel shipping AES encryption in hardware and calling it a "crypto machine", or as we see nowadays, extensions on GPUs that benefit LLMs. — u/authurno1 on Reddit </blockquote> Third, when I read code, I tend to start from the data: in C/C++ terms, the struct</code> or private members of a class. Data is often more self-descriptive than operations. Once I understand the data model, the operations become transformations over that model. This is a personal bias, but it matches how I think about functional programming (FP) and data-oriented programming (DOP). It also explains why OOP doesn’t click with me as easily: it starts from behavior and encapsulation, while I prefer to anchor my understanding in data first. From this lens I could talk about side effects, mutability, and other concepts, but that would take us too far. Starting with the data... Lisp_Object: The Universal C Type</h2> Tagged Pointer Layout</h3> Back to the GNU Emacs source code</a>, the core data type used to represent Elisp values in C is called Lisp_Object</code>, defined in src/lisp.h</code>. For simplicity, using a 64-bit system to explain. Lisp_Object is a 64-bit machine word. For pointers, because heap allocations are 8-byte aligned, their lowest 3 bits are guaranteed to be 000</code>. Emacs simply embeds the 3-bit type tag directly into these "free" zero bits. For immediate integers (fixnums), the upper 62 bits hold the actual value. 64-bit Lisp_Object: ┌────────────────────────────────────────────────┬─────┐ │ pointer or value (61 bits) │ tag │ │ │ 3b │ └────────────────────────────────────────────────┴─────┘ </code></pre> Why the lowest 3 bits? Because all heap-allocated objects are 8-byte aligned (due to GCALIGNMENT</code>), their addresses always end in 000 in binary. These 3 bits are "free" — we can borrow them to store type information without losing any address precision. The tag is a enum, named Lisp_Type</code>. And the simplified source code is as below: enum Lisp_Type { Lisp_Symbol = 0, // 0b000 Lisp_Type_Unused0 = 1, // 0b001 Lisp_Int0 = 2, // 0b010 Lisp_Int1 = 6, // 0b110 <-- ! Lisp_String = 4, // 0b100 Lisp_Vectorlike = 5, // 0b101 Lisp_Cons = 3, // 0b010 <-- ! Lisp_Float = 7 // 0b111 }; </code></pre> PS. This tagged pointer technique is actually a universal pattern across systems programming. It solves two problems: First, in dynamically typed contexts, the execution engine must know a value's type before operating on it. Second, placing this metadata in an extra struct field wastes memory and causes cache-misses from pointer chasing. To survive memory bus bottlenecks, engineers cram metadata directly into the unused bits of pointers. We'll discuss in the next post. Stealing One More Bit</h3> Looking closely to the Lisp_Int0</code> and Lisp_Int1</code>, something looks weird... Lisp_Int0 = 0b010 Lisp_Int1 = 0b110 ^^ lowest 2 bits are the same! </code></pre> This design actually doubled the value that can be represented by a Lisp_Int</code> Normal 3-bit tag: ┌─────────────────────────────────────────────────────┬─────┐ │ value (61 bits) │ tag │ │ │ 3b │ └─────────────────────────────────────────────────────┴─────┘ Range: -2^60 to 2^60-1 Fixnum with 2-bit tag: ┌───────────────────────────────────────────────────────┬───┐ │ value (62 bits) │tag│ │ │2b │ └───────────────────────────────────────────────────────┴───┘ Range: -2^61 to 2^61-1 (doubled!) </code></pre> One important distinction: for a fixnum, the upper bits hold the integer value directly (an immediate). For all other types, those bits are a heap pointer to the underlying C struct. The Operation Conventions</h3> The macros (or in debug mode is inline function) that work on Lisp_Object</code> follow a naming convention: X</code> prefix — eXtract: strip the tag bits and get the underlying value or pointer</li> P</code> suffix — Predicate: check the type, returns bool</li> CHECK_</code> prefix — Assert: like a predicate, but signals a Lisp error if the type is wrong</li> </ul> For example, to check if an object is an integer and then read it: // Source: src/bignum.h if (FIXNUMP (obj)) // P: check type tag { EMACS_INT n = XFIXNUM (obj); // X: extract value } </code></pre> internally, XSTRING</code>, XCONS</code>, XFIXNUM</code> and all other X macros work by masking off the tag bits using XUNTAG, then casting to the appropriate C struct pointer. For XFIXNUM</code> the mask is 2 bits, so // Source: src/lisp.h — XFIXNUM_RAW return XLI(a) >> INTTYPEBITS; // INTTYPEBITS = 2 for fixnums </code></pre> PS. By performing a right shift (>></code>) on a signed integer, it forces the compiler to emit an arithmetic shift instruction. The hardware preserves the sign bit. and for other types using XUNTAG</code>: // Source: src/lisp.h — XUNTAG (for pointer types: XCONS, XSTRING, XSYMBOL, ...) #define XUNTAG(a, type, ctype) \ ((ctype ) ((uintptr_t) XLP(a) - (uintptr_t) LISP_WORD_TAG(type))) // e.g. XCONS expands to: return XUNTAG (a, Lisp_Cons, struct Lisp_Cons); // ^^^^^^ subtract the tag word from the raw pointer address </code></pre> Why clear the lower 3 bit tag using subtraction (-</code>) instead of a bitwise AND (& ~0x7</code>)? On architectures like x86, memory addressing supports Base - Offset</code>. A C compiler can fold the tag clearing and the subsequent struct access (like .car</code> or .cdr</code>) into a single instruction, saving a CPU register. Historical Update: The Evolution of LSB Tagging I originally joked that this optimization was because Emacs was built by the "same maniacs who built GCC." While true that early GNU hackers were intimately familiar with compilers, the specific evolution of this 3-bit LSB (Least Significant Bit) tagging scheme has a more precise history. According to the paper Evolution of Emacs Lisp</a>, early Emacs used 7-bit MSB (Most Significant Bit) tags. The modern LSB tagging scheme, which elegantly exploits the 8-byte alignment of heap allocations, was actually pioneered and implemented by Stefan Monnier during the development of Emacs 22 in 2007. (Huge thanks to Reddit user GuDzpoz for the rigorous fact-checking and for pointing me to this paper!) </blockquote> The Big Picture</h3> McCarthy's Lisp (1960) abstract math atom eq car cdr cons quote cond │ │ Emacs engineers bridge: │ "statically typed C must represent │ dynamically typed Lisp" ▼ Lisp_Object (src/lisp.h) C layer ┌──────────────────────┬────┐ │ pointer or value │tag │ ← one machine word │ 61 bits │ 3b │ └──────────────────────┴────┘ │ ├─ tag = Cons → CONSP() → XCONS() → struct Lisp_Cons ├─ tag = String → STRINGP() → XSTRING() → struct Lisp_String ├─ tag = Int0/1 → FIXNUMP() → XFIXNUM() → EMACS_INT (immediate) └─ tag = Symbol → SYMBOLP() → XSYMBOL() → struct Lisp_Symbol machine bits </code></pre> With the data representation in place, we can now map McCarthy's original 7 axioms directly onto these C macros. Mapping McCarthy's 7 Axioms to C</h2> If McCarthy's 7 axioms are the soul of Lisp, the Emacs source is its physical body — but that body is not confined to a single file. The axioms split across three files depending on whether they are about data representation, memory, or control flow: Axiom</th> Meaning</th> C struct / function</th> File</th></tr></thead> atom</code></td> is it NOT a pair?</td> !CONSP(obj)</code> (e.g., EMACS_INT</code>, struct Lisp_String</code>, etc.)</td> lisp.h</code></td></tr> eq</code></td> are two refs identical?</td> Lisp_Object</code> (raw 64-bit word compare)</td> lisp.h</code></td></tr> car</code></td> first element of pair</td> struct Lisp_Cons</code> - .car</code> field</td> lisp.h</code></td></tr> cdr</code></td> rest of pair</td> struct Lisp_Cons</code> - .cdr</code> field</td> lisp.h</code></td></tr> cons</code></td> construct a new pair</td> struct Lisp_Cons</code> allocated by Fcons()</code></td> alloc.c</code></td></tr> quote</code></td> return without evaluating</td> Fquote()</code> — special form</td> eval.c</code></td></tr> cond</code></td> branch on predicate</td> Fcond()</code> — special form</td> eval.c</code></td></tr> </tbody></table> Notice the split: the first four axioms — atom</code>, eq</code>, car</code>, cdr</code> — are pure data operations, living entirely in lisp.h</code>. cons</code> crosses into memory management. Only quote</code> and cond</code> require the evaluator — they are the boundary where data becomes behavior. PS. Other important files written in C. lread.c</code> — tokenizing and reading Lisp source into Lisp_Object</code> trees</li> eval.c</code> — evaluating those trees</li> alloc.c</code> — allocating and garbage-collecting Lisp objects</li> xdisp.c</code> — redisplay engine</li> </ul> Emacs C sources, use a lot of Lisp idioms abstracted as preprocessor macros, masking C language as Lisp look alike. Observe that, when you use them, you are not writing Lisp, you are writing pure C that just happens to look like Lisp. Those preprocessor macros exist for use in C core only, they are not visible to Elisp, and they happen to be macros for practical reasons of C programming: to always get inlined, in both release and debug builds. Alternative would be of course to implement them as inlined functions and I think they have start to replace some of those preprocessor macros with inlined versions. I am not really watching the mailing list and commited patches, so don't take me for the word. If you want connection to the Lisp implementation, I think you should look into Fatom, Feq, Fcar and Fcdr in data.c, which are those "McCarthian operators" we are using in Elisp. Another remark about McCarthian Lisp, since you touch on it, is that Lisp is a theory of computation that also happens to be practical tool. As a computation theory, it stands at the same level as as lambda calculus and turing machine; a mathematical construct, that happens to be runnable, so to say. In McCarthy's papers it varies how many are needed, depending on which paper you read. Regardless, the idea is that those "axioms" is all you need to build computations on. A closed universe. A mathematical theory. Those axioms are like Euclidean axioms, something you can build other mathematical constructs, it is just that here we are talking about computing. However, only in theory, i.e. in McCarthy's papers! :). in practical terms, I don't think there is any practical Lisp, more than toy examples, based on only those first 5, or 9 or 7 or how many forms McCarthy thought at some point in time are "basics". Emacs does not have statements on which are "elementary forms". C core implements ~1700 of "primitive" forms. Graham came up with 17 for Arc. Common Lisp has 25. Now, I have never used Arc, but I am sure that none of Common Lisp implementations in Common Lisp are implemented directly on top of only those 25 so called "special operators", even though, in theory that might have been the idea. Of course I don't know for sure, I discovered Lisp after the Lisp, and am just learning this myself, but to be practical you have to talk to the outside world, and outside world is often a bit more complex than what those 25 forms cover. u/authurno1 on Reddit </blockquote> Next step</h2> The tagged pointer trick Emacs uses handles 8 fundamental types perfectly, but how does the editor fit the remaining dozens of complex types (like Buffers and Windows) within the exact same 3-bit restriction? In the next post, we will look at how GNU Emacs expands this layout using a technique called "Poor Man's Inheritance", compare it to Tagged Unions (like C++ std::variant</code>), and see how LLVM reinvents this exact memory philosophy in the modern era. Emacs Internal Series: #01: Emacs is a Lisp Runtime in C, Not an Editor</a></li> #02: Data First — Deconstructing Lisp_Object in C</li> #03: Tagged Union, Tagged Pointer, and Poor Man's Inheritance</a></li> </ul> Emacs Internal #01: is a Lisp Runtime in C, Not an Editor 2026-02-26T00:00:00+00:00 I tried to move to an LLM-friendly platform like VSCode or Cursor, but I kept returning to GNU Emacs. After reading other users' stories, I realized this is a common pattern. Very few tools survive 40 years and still feel hard to leave. I read parts of the source code and discovered that Emacs is not just a code editor. There is design philosophy, system software trade-offs, and code that still feels like a treasure today. I want to record some personal discoveries that might be worth sharing. Before we dive into the implementation, here is the "why" and the history I looked up. The Church: why people cannot leave GNU Emacs</h2> Starting with the long-standing joke: Emacs, "a great operating system, lacking only a decent editor" -- Editor War</a> </blockquote> Joshua Blias's Returning to the Church (of Emacs)</a> is a true story about switching to Neovim and coming back to GNU Emacs. Here are common reasons why, in this modern world, people still use GNU Emacs, from a Reddit post</a>. "Because I love being a pariah at the office."</li> "I learn one editor, once, and use it for my whole career."</li> "It fits like a tailored suit." -- customizable experience, and you can mostly customize "EVERYTHING".</li> Tools that cannot be replaced: OrgMode, Magit</li> Tsoding - The Annoying Usefulness of Emacs</a></li> The romantic Free Software Foundation spirit and unwillingness to be controlled by big tech</li> </ul> So here is my first big question: The history: Why Emacs Embeds an Elisp Interpreter in C</h2> In the 1970s, hackers at the MIT AI Lab used a text editor called TECO</a>. Unlike modern editors with a cursor, users had to input a sequence of password-like strings to cast the magic that edits text (TECO manual</a>). To reduce the pain, people started writing "macros" to speed up the process. As macros grew larger and more complicated, they needed variables, if-else</code> control flow, and loops. At that point, Richard Stallman</a> and Guy Steele</a> (the creator of Scheme) made a decision: "If the macro is complicated enough to act like a programming language, why not give it a real Turing-complete</a> programming language?" This is the birth of Emacs (Editor MACroS). An interpreter made the editor itself programmable, so users could extend and evolve it live without recompiling or waiting for upstream changes. In the earliest Emacs, the "interpreter" was just TECO's macro language. Later, GNU Emacs adopted Emacs Lisp. Lisp was a natural choice because its syntax is simple, its macros are powerful, and the interpreter is small and flexible, which makes live customization easy. Later, Lisp machines were a commercial failure, and C on von Neumann architecture dominated the industry. When Richard Stallman and the Free Software Foundation</a> wanted a free Emacs on Unix, there was no Lisp environment there. So he wrote a Lisp virtual machine and interpreter core in C, effectively reviving the spirit of Lisp machines in a Unix ecosystem, because it was the path of least resistance to a complete Unix toolchain. This helps explain why GNU Emacs' source code looks the way it does, and why jokes like "a great operating system" evolved. Things learned from the Story</h2> GNU Emacs source code directly</h3> After understanding more about Emacs' history, the code and directory layout feel more reasonable. In this series, we'll discuss how C implements the Lisp interpreter, memory allocation, dynamic binding, and more. Worse is better: human nature</h3> As to why C became dominant and not Lisp, a classic articulation of this "less elegant but more successful" outcome is Richard P. Gabriel's The Rise of Worse is Better</a>. Sometimes the real world works this way too... Greenspun's Tenth Rule</h3> Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp. -- Philip Greenspun</a> </blockquote> In more direct words: "Don't fool yourself: it starts as a simple config, but it won't stop there." Humans have unlimited desire, and long-lived software eventually evolves into a DSL and needs an embedded virtual machine. Vimscript is a perfect victim of this Greenspun's Rule. On the other side, Richard Stallman's vision already foresaw this curse. So Vim eventually needed a fork: Neovim using a Lua runtime (still an interpreter inside). They chose Lua because LuaJIT is a modern, fast runtime. Now Lua turns Neovim into a "Lua virtual machine text editor platform." Richard Stallman would probably laugh and say, "we did this 40 years ago." Although the idea looks similar, Neovim often feels faster than Emacs in practice because it leans on newer runtimes and techniques (e.g., LuaJIT, async jobs, and RPC). The following are some ways Neovim outperforms Emacs. </th> Emacs Lisp</th> Neovim</th></tr></thead> architecture</td> Monolithic C core + embedded Elisp interpreter</td> C core + embedded LuaJIT + Msgpack RPC host/guest plugins</td></tr> concurrency</td> single-threaded main loop</td> event loop; async jobs and RPC-based plugins</td></tr> union tagging</td> tagged pointers + cons cells/immediates</td> NaN-tagging</a> in LuaJIT</td></tr> </tbody></table> For modern editors like VSCode, an interpreter is still inside. VSCode is essentially a web browser for code, built on Electron (Chromium + V8). That means a VM is part of the editor's core. From a language-design viewpoint it may look less elegant (JavaScript is slow), but in practice it feels fast because of JIT, SIMD, and async IPC that keeps the UI responsive. Either way, the pattern repeats: a VM sits at the core of the editor. Appendix (PS): Other Greenspun's Tenth Rule victims Bash scripting</li> LaTeX</li> CMake</li> Printer/PDF scripting languages</li> eBPF in the Linux kernel</li> Tcl (Tool Command Language) in the EDA industry</li> LLVM TableGen (.td</code>)</li> </ul> This is also why my cache simulator Stratum</a> uses Racket to create a DSL for cache configuration. Next step:</h2> A. How to build a tiny Emacs Lisp interpreter in C with only seven elements quote</li> atom</li> eq</li> car</li> cdr</li> cons</li> cond</li> </ol> B. How Lisp_Object</code> works? Emacs Internal Series: #01: Emacs is a Lisp Runtime in C, Not an Editor</li> #02: Data First — Deconstructing Lisp_Object in C</a></li> #03: Tagged Union, Tagged Pointer, and Poor Man's Inheritance</a></li> </ul> Wishful Thinking from SICP, Practically 2026-02-04T00:00:00+00:00 This is not a pure technical blog post. It is an attempt to connect engineering principles to real life. I've been trying to understand my anxiety and self-doubt through engineering principles. Structure and Interpretation of Computer Programs (SICP)</a> wishful thinking gave me a framework to think about this. But I'm learning that understanding a pattern intellectually doesn't mean I've solved it emotionally. This post is about what I'm learning—not what I've figured out. What is Wishful Thinking?</h2> In Chapter 2 of SICP, the authors introduce a powerful strategy for building abstractions: wishful thinking. The idea is simple but profound: write code as if the pieces you need already exist. Define the interface you wish you had, then trust that the implementation details can be filled in later. When explaining this concept, the authors write: We are using here a powerful strategy of synthesis: wishful thinking. We haven’t yet said how a rational number is represented, or how the procedures numer</code>, denom</code>, and make-rat</code> should be implemented. Even so, if we did have these three procedures, we could then add</code>, subtract</code>, multiply</code>, divide</code>, and test equality</code> ... </blockquote> To make this concrete, here's the same concept in C++ without OOP. Notice how we use functions that don't exist yet, and we haven't defined how the Rational data type is represented—that's wishful thinking: int numer(const Rational& r); int denom(const Rational& r); Rational make_rat(int numerator, int denominator); // Now we can write high-level operations using our "wished" interface Rational add_rat(const Rational& x, const Rational& y) { return make_rat(numer(x) denom(y) + numer(y) denom(x), denom(x) denom(y)); } Rational sub_rat(const Rational& x, const Rational& y) { return make_rat(numer(x) denom(y) - numer(y) denom(x), denom(x) denom(y)); } Rational mul_rat(const Rational& x, const Rational& y) { return make_rat(numer(x) numer(y), denom(x) denom(y)); } Rational div_rat(const Rational& x, const Rational& y) { return make_rat(numer(x) denom(y), denom(x) numer(y)); } bool equal_rat(const Rational& x, const Rational& y) { return numer(x) denom(y) == numer(y) denom(x); } </code></pre> Now, the rest of the missing pieces tend to emerge naturally, almost effortlessly. This way of thinking gave me a way to stop reacting blindly to every signal, and instead design my next step deliberately. In that sense, wishful thinking wasn’t just a programming technique for me—it became a survival tool. How I Apply Wishful Thinking to My Life</h2> How do I actually use wishful thinking in my life? I follow these steps: Define the outcome Specify the mental state or concrete achievement you are aiming for. </li> Identify the procedures Define the procedures that would contribute to achieving that outcome, without worrying about their implementation yet. </li> Lazy evaluation Defer thinking and decision‑making until action is actually required. Avoid premature optimization, over‑planning, or the urge to make everything perfect upfront. </li> </ol> For example, I define my perfect day like this: (define (a-great-day) (let ([breakfast (with-hot-coffee)] [commute-to-work (listen-to-good-music)] [work (perform-meaningful-contribution)] [family (initiate-a-warm-hug)] [growth (learn-1-simple-thing-today)]))) </code></pre> Or my ideal life: (define (my-ideal-life) (let ([mind (inner-peace)] [body (healthy-body)] [family (cherish-everyday)] [interaction (spread-warmth-and-kindness)]))) </code></pre> The following are my own reminders: 01. Define the Goal</h3> A gentle reminder from Naval Ravikant: “Desire is a contract you make with yourself to be unhappy until you get what you want.” That quote forces me to define my “good day” and “ideal life” with extreme restraint. I often catch myself chasing things that society rewards—status, recognition, external validation. But pursuing those metrics usually leads to anxiety and dissatisfaction, which is the opposite of what I actually want. The irony is obvious: I was optimizing for the wrong objective function. Clarity is not motivation; it is a filter. </blockquote> When the end state is not clearly defined, every option feels equally important. Everything competes for attention, and motion feels like progress—but nothing converges. Once the end state is clear, most options fall away naturally. What looked like luck was just knowing where I was heading. Effortless doesn't mean easy. It means the system knows where it's going. 02. Find the Procedures</h3> Always work top-down, not bottom-up. Think recursive. </blockquote> I've been struggling at work for three years. I ship solid technical work—optimizations, stable regressions—but I struggle with communication and office dynamics. When my senior colleague consistently rejects my code reviews, I get defensive. I speak too directly. I miss social cues. The mismatch is real: this environment needs both technical skills and political awareness. I have the former, I'm learning the latter is not my strength. This hurts to admit. I'm trying to distinguish between useful feedback and noise. But I notice I sometimes filter out painful truths by labeling them 'not aligned with growth.' When someone points out I'm too aggressive in code reviews, I want to dismiss it. But maybe that's exactly the feedback I need. I'm still learning where the line is between protecting myself from toxic criticism and avoiding uncomfortable growth. Not all feedback is input; some of it is noise. </blockquote> 03. Lazy Evaluation</h3> This is the most important part. Before execution, don’t panic. Avoid catastrophizing the future. Trust that when execution starts, the next required step will naturally become clearer. Just like a lazy evaluator, clarity is produced only when demanded. </li> Execute lazily. Live at runtime. This idea maps surprisingly well to Zen thinking. Zen does not ask you to stop thinking—it asks you to stop precomputing. When I execute a task, I try to stay fully inside the current stack frame. I am only here, only now. In that sense, I am a single-threaded system. Humans are not compilers running with -O3</code>. We don't need to speculate every branch, unroll every loop, or pipeline every future scenario. Multitasking looks powerful, but for human brains it often results in cache stalls, context switching, and mental thrashing. </li> Meaning emerges during execution, not before it. Zen has a quiet reminder: The path appears where the feet land. </blockquote> In engineering terms, many insights are late-bound. They cannot be derived at design time—they emerge only when the system runs. Wishful thinking lets me defer unnecessary decisions. Zen teaches me to stay present enough to recognize the right one when it finally arrives. </li> </ol> Conclusion</h2> I found wishful thinking when I was looking for answers. I was struggling—with self-doubt, with not knowing my worth, with trying to prove something I couldn't name. Systematic thinking felt like solid ground in uncertain terrain. But here's what I'm learning: I use frameworks to feel in control when I'm actually scared. When life feels chaotic, I reach for structure. When I'm hurt, I intellectualize. This isn't wrong—it's how my mind works. But it has limits. Some wounds don't need better architecture. They need time, and rest, and the courage to admit I don't have it figured out. SICP taught me wishful thinking as engineering practice. Life is teaching me it's also about wishing for peace while not knowing how to get there—and learning that's okay. Some days I remember this. Most days I forget and try to optimize my way out of uncertainty. That's where I am: somewhere between understanding and living, between the wish and the trust. I'm still learning. Other Reference</h2> Applicative-Order vs Normal-Order Evaluation</a></li> Thunk (Haskell)</a></li> The Lazy Method</a>, by Josh Brindley.</li> </ul> Stratum: Architecting a Configurable Cache Simulator with C++ and Racket 2026-01-29T00:00:00+00:00 The Interview That Changed Everything</h2> During a recent interview, the conversation started well. The interviewer asked about my open-source contribution to rv32emu</a>—specifically, how I achieved 52% faster lookups and 35% faster insertions in the red-black tree implementation (commit 434c466</a>). My brain immediately started searching for the answer from memory—both the biological kind and the DRAM kind. Since I'd explained this optimization before, the answer was already cached: The optimization: Eliminate the parent pointer from each node, shrinking node size by 20%. Fewer bytes per node means better cache density. The technique: Instead of storing parent pointers in every node, maintain a path array on the stack during tree traversal. The path array stays hot in L1 cache, while the old approach scattered parent pointers across the heap, causing cache misses at every step. // Old: 3 pointers per node struct map_node { void key, data; unsigned long parent_color; struct map_node left, right; }; // New: 2 pointers per node struct map_node { void key, data; struct map_node left, right_red; }; // Path tracking on stack (stays in L1) rb_path_entry_t path[RB_MAX_DEPTH]; for (pathp = path; pathp->node; pathp++) { pathp->cmp = (rb->comparator)(node->key, pathp->node->key); } </code></pre> Simple, clean, and—I thought—demonstrated solid understanding of cache behavior. If there were a camera rolling, I probably had a pretty confident smile on my face. Then the interviewer asked the next question: "What is the main cause of cache-miss latency?" </blockquote> Well, I replied. (Also from my cache.) "If we can't find data in L1, we fall back to L2. And since L2 is usually bigger than L1, finding an address in L2 is slower." Seemed reasonable, right? "If L2 is the same size as L1, where does the latency difference come from?" </blockquote> "Physical distance," I replied. "The wire length from CPU to L1 versus L2." The interviewer paused, then continued: "What if you ran your benchmark on an ARM big.LITTLE SoC instead of x86?" > "What if the cache associativity changed?" > "What if the replacement policy was different?" </blockquote> My smile faded. If this were a cartoon, there'd be question marks floating above my head. That's when I realized: I could optimize for cache behavior, but I didn't actually understand how caches work. My rb-tree optimization succeeded on one machine with one cache configuration. I had no idea if it would work anywhere else—or why. Back to the basics: Study Cache in a Software Engineer's Perspective</h2> After that interview, I needed to understand caches from first principles. I started with CS:APP Chapter 6 (book</a>, lectures</a>)—the canonical resource for cache architecture. The Fundamentals That Matter</h3> Cache Structure: It's Just a 2D Array A cache is organized as S sets × E ways. Each cache line contains three fields: 1 valid bit t tag bits B = 2^b bytes per line per line per cache block +-------+------------------+---+---+-----+-----+ Set 0: | Valid | Tag | 0 | 1 | ... | B-1 | <- Way 0 +-------+------------------+---+---+-----+-----+ Set 1: | Valid | Tag | 0 | 1 | ... | B-1 | <- Way 1 +-------+------------------+---+---+-----+-----+ ... +-------+------------------+---+---+-----+-----+ Set S-1: | Valid | Tag | 0 | 1 | ... | B-1 | +-------+------------------+---+---+-----+-----+ Cache size: C = S × E × B data bytes </code></pre> Example: 8-way set-associative, 256 sets, 64-byte blocks -> 256 × 8 × 64 = 128 KB cache Associativity: The Trade-off I Missed This is what the interviewer was asking about. Associativity (E-way) determines how many cache lines in a set can hold data: Direct-mapped (E=1): Each address maps to exactly ONE location Fastest (no way selection)</li> Most conflict misses</li> </ul> </li> N-way set-associative (E=N): Each address can go into N locations within a set Hardware checks all N tags in parallel</li> Requires N-input multiplexer -> higher latency</li> </ul> </li> Fully-associative: Any address can go anywhere No conflict misses</li> Slowest (compare against ALL cache lines)</li> </ul> </li> </ol> Now the interviewer's question made sense: Same size, different latency -> different associativity -> different tag comparison circuitry. L2 isn't slower because it's "bigger"—it's slower because it's 16-way instead of 8-way. The Recursive Pattern</h3> Cache Read Process: Select the set using set_index</li> Check all ways in the set (parallel tag comparison)</li> Handle the result: Cache hit: Return data immediately</li> Cache miss: Fetch from next level (L2, L3, or DRAM)</li> Store in current cache (allocate a line, evict if needed)</li> Return data to CPU</li> </ol> </li> </ul> </li> </ol> Wait. Step 3 is interesting. When L1 misses, it asks L2. When L2 misses, it asks L3. When L3 misses, it asks DRAM. Each level follows the same pattern: check locally, delegate on miss. This is a recursive process—the same pattern SICP Chapter 1.2 describes as "deferred operations that build up." Each cache level is a function that either returns data or calls the next level. ; Cache lookup as recursive process (SICP perspective) ; This is pseudocode to illustrate the concept, not actual Stratum code (define (cache-lookup addr level) (let ((result (probe-cache level addr))) (if (hit? result) (extract-data result) (let ((data (cache-lookup addr (next-level level)))) (cache-fill level addr data) ; Write fetched data back to current level data)))) </code></pre> That's when I decided: implement cache hierarchy as recursive types in C++. Instead of a traditional OOP design with virtual methods and polymorphism, use template metaprogramming to bind the hierarchy at compile time—just like hardware does at synthesis time. Further reading: CS:APP Chapter 6</a> - Cache fundamentals</li> SICP Chapter 1.2</a> - Recursive processes</li> What Every Programmer Should Know About Memory</a> - Section 3.1</li> </ul> Building Stratum: Digging Through Memory Layers</h2> Here's how I think about cache hierarchy from a compiler backend perspective: The name "Stratum" comes from this mental model: memory hierarchy as geological layers. Above ground: CPU and registers—visible, fast, directly accessible. Below ground: Cache stratums—hidden, progressively deeper and slower. When the compiler generates a load instruction (lw</code> in RISC-V), it's essentially saying "dig down through the strata until you find this data." L1 cache miss? Dig deeper to L2.</li> L2 cache miss? Dig deeper to L3.</li> L3 cache miss? Excavate all the way down to DRAM.</li> </ul> From the grounds down. </blockquote> (Apologies to Canadian aviation students Not up--we're compiler people, we dig) From a compiler backend perspective, every load instruction digs down through memory layers to find data. Which stratum is your data in? L1? L2? DRAM? The depth determines the latency. That's why this project is named "Stratum"—because cache hierarchy is literally about drilling through geological layers of memory. Design Principles</h3> This geological metaphor shaped Stratum's architecture: 1. Discrete layers: Each cache level is independent 2. Miss delegation: On miss, ask the next layer down 3. Fixed topology: Like rock strata, layers don't rearrange at runtime This maps to the recursive pattern from SICP Chapter 1.2: each level is a function that either returns data or recursively calls the next level. using MemType = MainMemory<"MainMemory">; using L2Type = Cache<"L2", MemType, 512, 8, 64, LRUPolicy, 10>; using L1Type = Cache<"L1", L2Type, 64, 8, 64, LRUPolicy, 4>; // ^ ^ ^ ^ ^ ^ ^ // Name NextLayer Sets Ways BlockSize Policy HitLatency </code></pre> Each cache level is a type that statically binds to the next layer. The entire hierarchy resolves at compile time—just like how hardware interconnects are fixed when you synthesize a Verilog/VHDL design into gates. Why this design? Traditional OOP would use virtual methods and runtime dispatch. But cache topology is fixed at hardware synthesis—there's no runtime decision about "which next level to query." Think of it like wiring in Verilog (wire</code> keyword), signal connections in VHDL, or module connections in Chisel (:=</code> and <></code> operators): L1's miss port is hardwired to L2's input port. You don't change this at runtime; it's baked into the silicon. Template metaprogramming captures this constraint: compile-time binding eliminates virtual dispatch overhead, mirroring how HDL synthesis produces fixed interconnects. Disadvantages: Template error messages are cryptic Template instantiation errors are... educational. You will become very familiar with std::enable_if</code> and SFINAE, whether you want to or not. </li> Recompilation required for every configuration This is fine for one-off experiments but painful for systematic exploration. (Just like changing Verilog parameters requires resynthesis. This is why I added the Racket code generator.) </li> </ol> Advantages: Significantly faster than OOP design No vtable overhead, no runtime dispatch—like the difference between software function calls vs. hardwired logic gates. </li> Very simple configuration Just a few parameters to specify cache size, associativity, block size, etc. </li> </ol> Automating Configuration with Racket</h3> The C++ template approach was fast, but had a painful limitation: Every configuration change requires full recompilation. Want to compare 2-level vs 3-level cache? Recompile. Want to test LRU vs FIFO? Recompile. Want to sweep associativity from 4-way to 16-way? Recompile. For a single experiment, this is tolerable. For systematic exploration across dozens of configurations, it's a productivity killer. Solution: Generate the C++ code programmatically. Instead of manually editing templates, define configurations as S-expressions and generate all variants at one: ;; Compare 3-level vs 2-level hierarchy (define experiments (case_001 ;; name sets way latency policy nextLevel (L1 64 8 4 LRUPolicy L2) (L2 512 8 64 LRUPolicy L3) (L3 8192 16 64 LRUPolicy MainMemory)) (case_002 ;; name sets way latency policy nextLevel (L1 64 8 4 LRUPolicy L2) (L2 512 8 64 LRUPolicy MainMemory))) </code></pre> Run racket config.rkt</code> to generate case_001.cpp</code> and case_002.cpp</code>. Compile once, run both experiments. Then build using cmake, we can have all different reports of executable ready to compare. racket scripts/config.rkt # Regenerate C++ code cmake --build build # Build executable # Run experiments ./build/bin/case_001 > results_3level.txt ./build/bin/case_002 > results_2level.txt diff results_3level.txt results_2level.txt </code></pre> Output: ========================================================= Running Simulation: Sequential (/path/to/sequential.txt) ========================================================= === Simulation Results (Aggregated) === Level Hits Misses Avg Latency (cyc) L1 4375 625 4 L2 0 625 0 L3 0 625 0 MainMemory 625 0 232 </code></pre> Why Racket for Code Generation? TL;DR: I didn't want to write a parser. In Racket, the configuration file IS the program: ;; This is valid Racket code AND valid configuration data: (define experiments '((case_001 (L1 64 8 4 LRUPolicy L2) (L2 512 8 64 LRUPolicy L3)))) </code></pre> No JSON. No YAML. No configparser</code>. Just read</code> the file and you have nested lists ready to process. In Python, you'd need to pick a format (JSON? YAML? TOML?) and write parsing logic: import json with open("config.json") as f: experiments = json.load(f) # Now deal with dicts/lists name = config["name"] # String keys everywhere </code></pre> The cost: ~100 lines of Racket code vs ~150-200 lines of Python (after adding json.load</code>, error handling, and dict unpacking). The benefit: For someone who already knows Lisp, Racket is faster to write and harder to break (no missing commas in JSON, no YAML indentation errors). If you don't already know Racket, use Python. The productivity gain only exists if you're fluent in Lisp. Building Stratum: Answering the Questions I Couldn't</h2> After studying CS:APP and building Stratum</a>, I can now properly answer those interview questions. More importantly, I understand why my rb-tree optimization worked--and when it might not. Question 1: What Causes Cache-Miss Latency?</h3> My original answer was wrong. I said "L2 is bigger so slower." The main reason: L2/L3 are physically larger—longer wires and interconnects stretch the access pipeline. Size and wire delay dominate once you leave L1. But there's more to it. Latency also comes from tag comparison and way selection. When you access L2, hardware must: Compare the address tag against all ways in the set (parallel)</li> Select the matching way using a multiplexer</li> </ol> Higher associativity (more ways) = more parallel comparisons + bigger multiplexer = longer critical path. Why L2 is slower than L1: L1: 4-8 ways (faster tag compare, shorter wires)</li> L2: 16+ ways (more comparisons, longer wires)</li> Physical distance dominates, but associativity adds overhead</li> </ul> Real hardware nuance: Modern CPUs mitigate associativity costs with banking and pipelining. In practice, size/wire delay and staging dominate L2/L3 latency; associativity is an important but secondary knob. Question 2: Same Size, Different Latency—Why?</h3> The real answer: Different associativity. Same capacity (C = S E * B</code>), different organizations: Cache A: 512 sets × 1 way (direct-mapped)</li> Cache B: 64 sets × 8 ways (8-way associative)</li> </ul> Cache B is slower because: 8 parallel tag comparisons (vs 1 in Cache A)</li> 8-input multiplexer (vs direct wire in Cache A)</li> </ul> Trade-off: Direct-mapped: Fast but conflict misses</li> 8-way: Slower but fewer conflicts</li> </ul> Real hardware nuance: Modern CPUs narrow this gap using parallel tag+data access, way prediction, and banking. But higher associativity still tends to add latency while reducing conflict misses. Question 3: What If Associativity/Policy Changes?</h3> The hard truth: My rb-tree optimization might not work everywhere. On ARM big.LITTLE: Little cores: Smaller L1, lower associativity -> More conflict misses -> My path array advantage shrinks</li> Big cores: Larger L1, higher associativity -> My optimization still wins</li> </ul> Real hardware complexity: Cache-sensitive optimizations are microarchitecture-dependent. Beyond capacity and associativity, replacement policy, prefetchers, line size, VIPT/TLB behavior, and LLC organization can flip results. What I learned from Stratum: You can't just benchmark on one machine and claim victory. Cache-sensitive code needs profiling across architectures. That's why I added Valgrind trace support—capture real workload patterns, then test against different cache configurations in Stratum. Best practice: Always validate on multiple cores (ARM big.LITTLE, x86, RISC-V) with hardware counters and trace-driven simulation. Honest Evaluation of Stratum</h2> What Stratum Does Well: Pre-Silicon Validation: Test cache configurations before committing to RTL design Example: Compare 2-level vs 3-level hierarchy trade-offs</li> Evidence-based decision making for SoC development</li> </ul> </li> Research and Education: Compare replacement policies (LRU vs FIFO vs Random) with real traces</li> Teach cache concepts with observable, measurable behavior</li> Sensitivity analysis for associativity and block size</li> </ul> </li> Workload Analysis: Profile real applications via Valgrind traces Identify cache-unfriendly access patterns in production code</li> Validate optimizations (like my rb-tree path array approach)</li> </ul> </li> </ol> What Stratum Doesn't Do: Mesh/NoC topologies: Requires runtime routing, breaks compile-time binding</li> Non-inclusive/NUCA caches: Complex invalidation protocols not modeled</li> Prefetchers: Only demand-driven accesses supported</li> Multicore coherence: No MESI/MOESI protocol simulation</li> Power modeling: No energy consumption tracking</li> </ul> Why These Limitations Are Acceptable: Stratum optimizes for clarity and rapid experimentation, not production-grade simulation. For detailed performance modeling, use gem5 or ZSim. For hardware verification, use RTL simulators. Design Pattern Applicability: This template-based hierarchy pattern isn't limited to CPU caches. The same approach applies to other memory hierarchies: NPU/GPU memory: L1 texture cache -> L2 cache -> HBM (High Bandwidth Memory) -> Host DRAM</li> Distributed systems: Redis cache -> Local DB -> Remote DB -> Cold storage</li> CDN architecture: Edge cache -> Regional cache -> Origin server</li> DMA transfers: On-chip buffer -> L2 -> Main memory -> Peripheral device</li> </ul> The key insight: Any hierarchical lookup with fixed topology at "compile time" (or deployment time) can use this zero-overhead template approach. The abstraction of "check current level, on miss delegate to next level" is universal. Try It Yourself</h2> Quick Start (5 minutes): git clone https://github.com/TheCloudlet/Stratum.git cd Stratum cmake -B build -DCMAKE_BUILD_TYPE=Release cmake --build build ./build/bin/stratum # Run default experiments </code></pre> Dependencies: C++20 compiler (GCC 10+/Clang 11+/MSVC 2019+), CMake 3.20+ Want to test your own traces? Visit Stratum README</a> for instructions on: Generating Valgrind memory traces from your programs</li> Creating custom cache configurations</li> Using the Racket code generator (optional)</li> </ul> License: MIT - Fork it, modify it, break it, learn from it. Conclusion: From Interview Failure to First Principles</h2> That interview question—"What causes cache-miss latency?"—exposed a gap between optimizing for cache behavior and understanding how caches actually work. I could write cache-friendly code by intuition, but I couldn't explain why it worked or predict when it wouldn't. Building Stratum closed that gap. What this project taught me: Cache behavior is about access patterns, not just capacity My rb-tree optimization worked because the path array had sequential access (L1-friendly)</li> The old parent-pointer approach had random heap access (L1-hostile)</li> Conflict misses matter more than capacity misses (associativity isn't just a spec number)</li> </ul> </li> Template metaprogramming can mirror hardware constraints Zero-cost abstraction: cache hierarchy as types, not virtual dispatch</li> Compile-time binding mirrors hardware reality (topology is fixed at synthesis)</li> Design constraints become compiler guarantees</li> </ul> </li> Building tools teaches concepts better than reading alone Implementing LRU forced me to understand why timestamp arrays beat linked lists</li> Exposed why associativity directly impacts latency</li> Debugging Valgrind traces revealed access patterns I'd never noticed in profilers</li> </ul> </li> </ol> What I still don't understand: How replacement policies map to hardware counters (e.g., tree-PLRU in Intel)</li> Why real L1 caches use physical vs virtual indexing (TLB interactions)</li> How MESI/MOESI states work in multi-core scenarios (cache coherence protocols)</li> </ul> Next steps: Reading "A Primer on Memory Consistency and Cache Coherence" and implementing a MOESI simulator to close these gaps. Connect with me: GitHub: TheCloudlet/Stratum</a></li> Questions/feedback: Open an issue or PR</li> Interested in collaborating? Let's talk.</li> </ul> About: Yi-Ping Pan (Cloudlet) 2026-01-01T00:00:00+00:00 Yi-Ping Pan is a software engineer with a deep interest in compilers, systems programming, and functional programming. He has worked on projects spanning compiler infrastructure (LLVM, MLIR), low-level systems in C++, and exploring the elegance of Haskell and functional paradigms. He has contributed to open-source tooling and enjoys thinking about the intersection of language design and runtime performance — from how programs are parsed and optimized, to how they execute on real hardware. Outside of engineering, he writes, tinkers with Emacs configurations, and occasionally gets lost in a good book. He can be reached at [email protected]">[email protected]</a>, or found on GitHub</a> and LinkedIn</a>. Introducing Coogle: Bringing Haskell's Hoogle to C++ 2025-12-02T00:00:00+00:00 Why I Started This Project?</h2> The story started in 2024, when I decided to review basic algorithms and data structures. I started a repository called "From Zero to Leetcode Hero</a>." My intention was to learn and solve LeetCode problems using C++ and Haskell together. Playing with both languages was fun. I discovered that tail recursion and iteration share almost the same control flow. The main difference lies in how the compiler (like LLVM) handles the stack frame prologue and epilogue, but that is a story for another time. While exploring Haskell, I found an incredibly useful tool called Hoogle</a>, which allows you to search for functions by their type signatures. For example, if I want a function that takes a list of integers and returns a single integer (like a total or a count), I can search for [Int] -> Int</code>. Hoogle will suggest relevant functions like sum</code>, product</code>, or length</code>. add :: Int -> Int -> Int add = undefined </code></pre> add</code> is the function name.</li> ::</code> means "has type".</li> Int -> Int -> Int</code> means it takes two Int</code> arguments and returns an Int</code>.</li> add = undefined</code> means the implementation is still pending. We don't care about the body yet; the signature tells the whole story.</li> </ul> The last type in the chain is the return type. So, a signature like Int -> Char</code> means it takes an integer and returns a character. This strict enforcement ensures that Hoogle can accurately match tools to your needs, and guarantees no "surprise actions" (side effects) modify values outside the function scope. Honestly, when I first encountered Haskell types, I was completely confused by the syntax. It took me a while to fully translate my mindset, but then signature searching just clicked. Why is searching by type signature useful? It allows me to ignore all the operations and irrelevant details, and focus on how data flows through functions. From a compiler engineer's perspective, looking at code this way is a superpower. It forces us to separate the Plumbing (how we move data) from the Logic (what we do with data). This separation is the essence of Data Abstraction taught in foundational texts like SICP: you focus on the behavior of functions at the highest level (the interface) without concerning yourself with their messy implementation details. From my daily work experience, for example, when I need to handle getting a real name from an AST node, I often find myself searching for functions that convert ASTNode</code> to std::string</code>. This is especially true when dealing with unfamiliar third-party libraries or legacy codebases. So I am looking for the function signature like: foo :: ASTNode -> String foo = undefined </code></pre> or in C++ notation: std::string foo(const ASTNode& node); </code></pre> What if I could use this approach in my everyday VHDL compiler development? Searching for the input class and the output class would be incredibly useful. </blockquote> However, I couldn't find any similar tool for C/C++. That's why I created Coogle — a high-performance C++ command-line tool for searching C/C++ functions based on their type signatures, inspired by Hoogle from the Haskell ecosystem. The Semantic Gap: Why Text Matching Fails C++ Code</h2> Why not just use grep</code> or rg</code> to search for function names or patterns in header files? </blockquote> That is the first question everyone asks. The answer is simple: C++ is not a regular language, and source code is not just text—it is structure. Consider this scenario: You want a function that takes an ASTNode</code> and returns a std::string</code>. Regex approach: Writing std::string\s+\w+$\sconst\s+ASTNode&\s\w+\s$</code> </li> The problem: This will miss many valid matches: // The const is trailing std::string getName(ASTNode const& node); // There is a line break std::string getName(const ASTNode& node); </code></pre> </li> </ul> Sure, you might claim that you can write more complex regex to cover these cases. But do you really want to write that crazy regex, or would you rather use something like this? # Standard search ./coogle . "std::string(ASTNode const&)" # It understands references and whitespace automatically ./coogle . "std::string ( ASTNode & )" # Wildcards supported ./coogle . "std::string ()" </code></pre> "What if I wrap the regex in a Python script?" </blockquote> Fair point! However, here's the thing: writing a Python script to manage complex regexes is basically building your own fragile parser. Sure, it's more convenient than raw regex, but it doesn't actually solve the correctness problem. Even with a script wrapping text-matching logic, you'll still run into two fundamental C++ features that trip up regex—but Coogle handles them naturally: Type Aliases (The Semantic Gap) — Regex sees text; Coogle sees types. Code: using NodeID = uint64_t;</code></li> Search: You want a function returning uint64_t</code>.</li> Regex/Python: Will miss NodeID get_id()</code> because "NodeID" != "uint64_t"</code>.</li> Coogle: Understands the alias and finds the match.</li> </ul> </li> Template Nesting (The "Greedy" Match Problem) — Regex struggles with balanced brackets and recursive structures. Code: std::map<std::string, std::vector<int>></code></li> Regex: Writing a regex to correctly match nested <...></code> without accidentally matching the closing ></code> of a different template is a nightmare (and theoretically impossible for standard regex engines).</li> Coogle: Parses the AST structure correctly.</li> </ul> </li> </ol> Therefore, if you want correctness and reliability, building a proper parser is the only way—and that is exactly what Coogle does. Building Coogle</h2> I'd like to thank Professor Ching-Han Chen (陳慶瀚) at National Central University, Taiwan, for teaching me the MIAT methodology. One of the most valuable lessons I learned was this: the best way to start any project is to clearly define your inputs and outputs first. So that's exactly where I began with Coogle. Defining the Interface</h3> The tool needed to be simple and intuitive. I envisioned a command-line interface that takes two arguments: File or directory path — where to search</li> Type signature — what to find</li> </ol> The output should tell me exactly where the matching functions are: # Input: Search for a function that takes an int and returns an int ./coogle . "int(int)" # Output: Show the file, line number, and function name foo.cpp:42:addOne bar.cpp:108:increment </code></pre> Simple enough. With the interface defined, I could now think about the internal architecture. At the highest level, Coogle is just a function that transforms inputs into outputs: ┌──────────────────────────────┐ │ INPUT │ │ • C/C++ File Path │ │ • Search Signature String │ └──────────────┬───────────────┘ │ v ┌───────────────┐ │ Coogle │ └───────┬───────┘ │ v ┌──────────────┴───────────────┐ │ OUTPUT │ │ • file:line:functionName │ │ • file:line:functionName │ │ • ... │ └──────────────────────────────┘ </code></pre> I sketched out how data would flow through Coogle. Following the functional programming principle that everything is a function, I broke down the entire system into a pipeline where each stage has clearly defined inputs and outputs: ══════════════════════════════════════════════════════════ Coogle Processing Pipeline ══════════════════════════════════════════════════════════ [C/C++ File Path] [Search Signature String] | | v v findSourceFiles() parseFunctionSignature() (recursively discover) (custom parser) | | | Output: List<FilePath> | Output: Signature v v | | +-----------> foreach file <--------+ | v clang_parseTranslationUnit() (via libclang) | | Output: CXTranslationUnit (AST) v visitor() (CXCursor, Signature) -> CXChildVisitResult | | Extract function declarations | Build actual signature isSigntaureMatch() | Output: List<Match> | where Match = {file, line, name, sig} v fmt::print() (List<Match>) -> IO | v [Screen] </code></pre> This looked simple enough, and I thought this would be a one-week project. I was completely wrong. I jumped into several potholes and had to crawl my way out. Let me share the key challenges I faced. Pothole 1: Understanding libclang</code></h2> To be honest, I don't feel comfortable using tools or concepts that I don't understand well. My background is in Communication Engineering (Electrical Engineering), where we were trained in highly detailed mathematical modeling and derivation from first principles. We learned to question "black box" solutions and rigorously verify foundational theories before applying any abstraction—from Karnaugh maps to Gray codes, from probability theory to QPSK modulation. My first "panic moment" came during COVID-19, when I volunteered to help my company build a pandemic survey web form. I chose the Django framework, but felt completely blind while implementing the website. It was like baking without understanding the chemistry—tweaking the amount of salt, yeast, and dough resting time, with no idea what was actually happening under the hood. Nevertheless, it eventually worked. When working with libclang</code>, I felt the same way. I knew I only needed to focus on the visitor function, but the syntax looked so unfamiliar—this didn't look like the familiar C/C++ code I was used to. CXChildVisitResult visitor(CXCursor Cursor, [[maybe_unused]] CXCursor Parent, CXClientData ClientData) { auto Ctx = static_cast<VisitorContext >(ClientData); CXCursorKind Kind = clang_getCursorKind(Cursor); if (Kind == CXCursor_FunctionDecl || Kind == CXCursor_CXXMethod) { // Build actual signature from libclang coogle::SignatureStorage ActualStorage; // Get return type CXType RetType = clang_getCursorResultType(Cursor); assert(RetType.kind != CXType_Invalid && "Invalid return type obtained from libclang"); CXString RetSpelling = clang_getTypeSpelling(RetType); std::string_view RetTypeSV = clang_getCString(RetSpelling); std::string_view RetTypeInterned = ActualStorage.internString(RetTypeSV); std::string_view RetTypeNorm = coogle::normalizeType(ActualStorage.arena(), RetTypeInterned); clang_disposeString(RetSpelling); //... } } </code></pre> After some research, I realized that Clang might not be so different from the RTL compiler I'm used to. At minimum, I needed to get a grasp of the high-level code structure and how data flows—that would comfort me a bit. The full story of demystifying libclang</code>—including the visitor pattern, AST traversal strategies, and how different languages handle compilation—deserves its own dedicated post. For now, let's treat this as a black box with a well-defined interface: input a file, output an AST. Stay tuned for: "Inside libclang: From Visitor Pattern to AST Mastery" Pothole 2: The std::string</code> and Template Matching Issue</h2> At the beginning, I thought std::string</code> was a real type. (Yes, this is how naive I was—despite working with professional C++ programmers and being tortured by std::string</code> issues regularly, I didn't know this fundamental fact). So when I expected to find std::string</code> but actually got std::basic_string<...></code>, I was very confused. I documented my learning journey in detail here: Back to Basics: From C char to string_view (Notes from building Coogle)</a> The fix: Canonicalization (The Search for Truth) To solve this, I couldn't just store the function signature as it appears in the source code. I had to store the Canonical Type. Clang provides GetCanonicalType()</code>. This strips away all the "sugar"—typedefs, type aliases, and redundant qualifiers. This process mirrors the Lisp philosophy of Uniformity (Homogeneity). Just as Lisp treats code as data (S-expressions), canonicalization treats disparate C++ type aliases as a single, uniform data structure. By stripping away the syntactic sugar, we reveal the underlying mathematical truth of the function's type. Pothole 3: The Translation Unit Trap (The Flood of Headers)</h2> This was the funniest bug. When I searched for a simple signature like void(void)</code> inside main.cpp</code>, I expected to find my own utility functions. Instead, Coogle vomited thousands of results: internal functions from libc++</code>, std::vector</code> helpers, and obscure symbols from the system SDK. The Root Cause: I forgot how C++ compilers actually work. libclang</code> operates on a Translation Unit (TU). When reading #include <iostream></code>, the preprocessor copy-pastes the entire content of iostream</code> (and everything it includes) into the file. So, to the parser, those std::</code> functions are just as much "part of the file" as main()</code>. The Fix: Defense in Depth (Double Filtering) To address this, I realized I needed a rigorous filtering strategy to separate "my code" from "library code." I implemented a two-layer filter: Layer 1: The Safety Net (System Header Check) First, I ask Clang if the cursor is explicitly inside a system header. This handles cases where file paths might be ambiguous but the compiler knows it's a library. CXSourceLocation location = clang_getCursorLocation(cursor); if (clang_Location_isInSystemHeader(location)) { return CXChildVisit_Continue; // Layer 1: Drop system headers immediately } </code></pre> Layer 2: The Strict Scope (File Provenance Check) Second, even if it's not a system header, I don't want to see results from other user headers included in the TU unless I explicitly asked for them. I verify that the cursor physically resides in the target file. CXFile File; clang_getSpellingLocation(location, &File, &Line, &Column, nullptr); CXString FileName = clang_getFileName(File); const char FileNameStr = clang_getCString(FileName); // Layer 2: Only show results from the file we're explicitly parsing if (!FileNameStr || Ctx->CurrentFile != FileNameStr) { return CXChildVisit_Continue; } </code></pre> By combining System Header Filtering (Blacklist) with Explicit File Matching (Whitelist), I achieved zero noise. Coogle now respects the user's intent: "Search this file, and only this file." Pothole 4: Performance Issue When Dealing With Large Codebases</h2> Running Coogle on a small "Hello World" project was instant. But when I unleashed it on a massive codebase like LLVM itself? It choked. It got stuck for over 40 minutes, eating up CPU cycles like there was no tomorrow. Profiling revealed two things: Too many things are being parsed</li> std::string</code> memory allocating issue</li> </ol> Issue 1: The "Lazy Parser" Strategy</h3> By default, Clang behaves like a compiler—it wants to build a perfect, complete AST. It resolves every #include</code>, parses every template inside <vector></code>, and validates every function body. The Epiphany: Coogle is a Search Engine, not a Compiler. I don't need to generate binary code; I just need to read the Signatures. I realized I could aggressively turn off Clang's "heavy lifting" features to trade correctness for speed. I decided to strip down the parsing process to the bare minimum by injecting specific compiler flags and options. 1. Cutting the Cord to System Headers Since I already implemented the "File Filtering" logic in Pothole 3, parsing system headers was now purely wasted time. ArgsVec.push_back("-nostdinc"); // Stop searching standard system directories ArgsVec.push_back("-nostdinc++"); // Stop searching standard C++ directories </code></pre> The Logic: This tells Clang: "Ignore the standard library." The Gain: We skip parsing thousands of lines from <iostream></code>, <vector></code>, and <string></code>. The parser no longer wastes time building ASTs for the entire STL ecosystem. 2. Skipping the Implementation Details I don't care how a function is implemented; I only care what it takes and returns. unsigned Options = CXTranslationUnit_SkipFunctionBodies | ... </code></pre> The Logic: CXTranslationUnit_SkipFunctionBodies</code>. The Gain: libclang</code> completely ignores the code inside func { ... }</code>. This is huge. For a 1000-line function, we now parse only the first line. 3. Tolerating Imperfection Since I cut off the system headers, the code is technically "broken" (types like std::string</code> are now undefined symbols to the parser). unsigned Options = ... | CXTranslationUnit_Incomplete; </code></pre> The Logic: CXTranslationUnit_Incomplete</code>. The Gain: This tells Clang: "It's okay if you find missing symbols or headers. Don't error out; just give me what you have." This makes the parser resilient to my aggressive optimization strategy. Issue 2: String Allocation Overhead (Memory Pool Optimization)</h3> While fixing the parser flags solved the CPU bottleneck, I noticed the memory usage was still alarmingly high. This is where my previous deep dive into std::string</code> and SSO (from Pothole 2) came back to save me. The old Signature</code> structure I used looked like this: struct Signature { std::string RetType; // Original return type std::string RetTypeNorm; // Normalized return type std::vector<std::string> ArgType; // Original argument types std::vector<std::string> ArgTypeNorm; // Normalized argument types }; </code></pre> For every matched signature, I needed at least 4 heap allocations (void(void)</code>), which is obviously very expensive. To solve this, I implemented a String Interning mechanism backed by a linear memory arena: Storage: A central std::vector<char></code> (or similar deque) acts as a persistent string pool.</li> Reference: Instead of holding std::string</code> (which owns memory), my AST nodes now hold std::string_view</code> (which borrows memory).</li> Deduplication: Before storing a type name, I check if it exists in the pool. If yes, I return a view to the existing data.</li> </ul> struct Signature { std::string_view RetType; // Original return type std::string_view RetTypeNorm; // Normalized return type span<std::string_view> ArgTypes; // Original argument types span<std::string_view> ArgTypesNorm; // Normalized argument types } attribute((packed)); </code></pre> By using string_view</code> pointing to a stable memory arena, we are effectively enforcing Immutability. Once a string is interned, it never changes. This immutability eliminates the need for defensive copying and complex ownership management, much like how functional languages handle data structures. Initially, I considered caching all int</code> strings in a hash table to avoid duplicates, but I found that searching the table with string comparisons and checking whether the string is preallocated was actually slower than directly writing to the string pool. The Result: This shifted the architecture from "Object-Oriented Ownership" (everyone owns their strings) to "Data-Oriented Sharing". Memory footprint dropped significantly, and more importantly, cache locality improved because related string data was now packed tightly in the arena rather than scattered across the heap. In the end, I successfully reduced the whole AST parsing time for LLVM from being completely stuck (40+ minutes) down to just 6 minutes. While still not blazing fast, it's at least workable. If we want it even faster, we could integrate compile_commands.json</code> or customize JSON file dumps, so we parse once and can check different signatures in split seconds. Conclusion (The Takeaway)</h2> Building Coogle wasn't just about making a search tool—it was a journey of demystifying compilers and embracing the power of Plumbing vs. Logic separation. I started with fear: fear of libclang</code>, fear of the "black box," fear of diving into unfamiliar territory. But by applying the principles I learned from both SICP and practical engineering—Data Abstraction (separating interface from implementation), Canonicalization (normalizing types to their single source of truth), and Lazy Evaluation (only parsing what we need)—I managed to tame the beast. The four potholes I encountered taught me valuable lessons: Understanding libclang</code> — Sometimes you don't need to understand everything; treating components as black boxes with clear interfaces is okay.</li> Template Matching — std::string</code> isn't what I thought it was, and understanding the underlying type system is crucial.</li> Translation Unit Filtering — Defense in depth with layered filtering (system headers + file provenance) achieves zero noise.</li> Performance Optimization — Aggressive optimization strategies (lazy parsing + memory pools) can reduce 40+ minutes to 6 minutes.</li> </ol> Now, Coogle serves as my daily driver for navigating complex C++ codebases. It's not perfect, but it's built on a solid understanding of how C++ compilers actually work under the hood. What's Next? Integrate compile_commands.json</code> for persistent AST caching</li> Consider building a language server protocol (LSP) extension</li> </ul> If you're interested in trying Coogle, check out the repository: github.com/TheCloudlet/Coogle</a> Appendix: Architecture Diagram</h2> +---------------------+ +----------------------+ | C/C++ Source | | User Query | | (File / Project) | | "int(int)" | +----------+----------+ +-----------+----------+ | | v v +----------+------------------------------+----------+ | Coogle Frontend | | | | [ libclang Parser ] <---- (Translation Unit) | | | | | v | | (Clang AST) | | | | | v | | < RecursiveASTVisitor > --+ | | | | | | | (Visit) | (Filter) | | v v | | [ FunctionDecl ] [ System Header Check ] | +-----------+----------------------------------------+ | v +-----------+----------------------------------------+ | Coogle Backend | | | | 1. Type Extraction (Get Raw Signature) | | "NodeID get(int)" | | | | | v | | 2. Canonicalization & Matching | | | | | v | | 3. String Interning (Memory Pool) | | Store unique signature via string_view | +-----------+----------------------------------------+ | v +-------+-------+ | Output | +---------------+ </code></pre> Back to Basics: From C char to string_view (Notes from building Coogle) 2025-11-20T00:00:00+00:00 Background</h2> When I was implementing Coogle</a>, I discovered something bizarre about C++ strings. That is why I wrote this note to summarize what I learned about C++ strings. Coogle is a search engine that I implemented in C++. It is designed to quickly and efficiently find function signatures in large codebases—especially useful for 30-year-old C++ codebases that were migrated from C and mixed with C++11/14/17/20 features like my lovely everyday job. Since I like Haskell and pure functions, I tried to pay tribute to Hoogle, a Haskell function search engine. The idea is to be able to search for function signatures in C/C++ and get back a list of matching functions. First step, support simple types like: int</code>, char</code>. Okay that is not too hard. // This is my test file int add(int a, int b); int std::string(std::string s); </code></pre> I expect when I search "int(int, int)", Coogle should be able to find this signature quickly. So, implemented that, and it worked fine. But, when I try to search "std::string(std::string)" to find function signatures that take and return std::string</code>, I found that Coogle could not find any results, even though there are many such functions in my test codebase. This is really a WTF moment for me. I couldn't understand why and got stuck for a while, then I printed out the full AST tree using libclang. That's when I saw something confusing: std::basic_string<char, std::char_traits<char>, std::allocator<char>></code> showed up instead of std::string</code>. What the heck is a trait and allocator? I don't understand C++ strings at all. I thought it was supposed to be just std::string</code>! Well... So I dug deeper and read more about C++ strings. My initial thinking was to ask LLMs about C++ strings and designs. But since I work on legacy C and C++ code, and lots of my algorithms are implemented in C (for milking out best performance), I decided to first understand C strings and character types deeply. As Jserv always says in C Programming series</a>: "Be honest with yourself. You don't know C." — Jserv Huang </blockquote> So, before we tackle std::string, let's be honest with ourselves and look at the chaos of char. Everything below is based on my research using different resources, including language standards, articles. That may not be 100% accurate, but I tried my best to summarize what I learned. If you find any mistakes, please kindly point them out to me. PART 1: C99 chars and strings</h2> Starting with my conclusion. In C, char doesn't really mean 'character'. It just means the smallest addressable unit of memory (a byte). ASCII or other encodings just happen to fit into this unit. — The Cloudlet </blockquote> Or maybe through the lens of VHDL we can see more clearly: library IEEE; use IEEE.std_logic_1164.all; use IEEE.numeric_std.all; package C_Types is -- Defined in <limits.h> -- Most likely 8, but could be 16 or 32 on DSPs constant CHAR_BIT : integer := 8; -- "unsigned char" -> 0 to 255 subtype c_unsigned_char is unsigned(CHAR_BIT-1 downto 0); -- "signed char" -> -128 to 127 subtype c_signed_char is signed(CHAR_BIT-1 downto 0); -- Compiler defines "__CHAR_UNSIGNED__" on platforms like ARM. -- Set by Arch (x86=false, ARM=true) constant CHAR_IS_UNSIGNED : boolean := ???; alias c_char is case CHAR_IS_UNSIGNED generate when false => c_signed_char; -- x86 (Default) when true => c_unsigned_char; -- ARM end generate; end package C_Types; </code></pre> VHDL is chosen here specifically because of its strictness. It forces us to see the distinction between "Raw Bits" (std_logic_vector), "Unsigned Math" (unsigned), and "Signed Math" (signed). Quick C99 Char Summary</h3> Please refer to ISO C99 standard §6.2.5 for more details. The "Three Types" Rule: char, signed char, and unsigned char are three distinct types. The plain char is a wild card—it behaves like signed on x86 but unsigned on ARM. </li> Usage Rule: Use char</code> ONLY for text strings.</li> Use signed char</code> for small value calculation</li> Use unsigned char</code> (or uint8_t) for binary data.</li> </ul> </li> </ol> C Strings philosophy recap</h3> This is CS101 stuff, I don't think we need to talk too much deeply about it. Just doing a quick recap and why C strings are designed this way. The Core Mechanism \0</code> is the null terminator.</li> Operations like strlen</code>, strcpy</code>, strcmp</code> are linear scan O(n)</code>.</li> </ul> ┌─────┬─────┬─────┬─────┬─────┬─────┬─────┬─────┬─────┬─────┬─────┬─────┬─────┬─────┐ │ H │ e │ l │ l │ o │ , │ │ W │ o │ r │ l │ d │ ! │ \0 │ ├─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┤ │ 72 │ 101 │ 108 │ 108 │ 111 │ 44 │ 32 │ 87 │ 111 │ 114 │ 108 │ 100 │ 33 │ 0 │ ├─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┤ │0x48 │0x65 │0x6C │0x6C │0x6F │0x2C │0x20 │0x57 │0x6F │0x72 │0x6C │0x64 │0x21 │0x00 │ └─────┴─────┴─────┴─────┴─────┴─────┴─────┴─────┴─────┴─────┴─────┴─────┴─────┴─────┘ [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] ^ ^ | | str (pointer to first character) Null terminator ('\0') </code></pre> The Why of Null-Terminated Strings I have searched around for why C strings are designed this way. What I can find is at 1970, there are 2 typical ways to represent strings in memory: Pascal style: Length-prefixed strings (1 byte length + data)</li> C style: Null-terminated strings (data + '\0')</li> </ol> But I haven’t found a definitive answer to why C chose null-terminated strings over length-prefixed strings. Instead, I stumbled upon some interesting articles pointing out that C strings are, frankly, a pain. Check out Joel Spolsky’s classic post, Back to Basics</a>, where he describes the performance problems with strcat and how some developers resorted to “Fed-Up Strings” (Pascal-style strings in C) as a workaround for the shortcomings of null-terminated strings. People have different implementations for string handling in C, just like the Linux kernel has its qstr implementation under include/linux/dcache.h</code> when dealing with filesystem paths. Google's Abseil</a> library also introduced StringPiece</code> (which later evolved into string_view in C++17). The commonly accepted approach across these implementations is to keep a length field along with the char pointer to avoid repeated strlen calls. Therefore, C++'s std::string</code> is not the only solution to C string problems—but understanding its design can be helpful for compiler, library, and systems programmers. Update: Community Insights (2025-11-26) For the nul-terminated string design, some kind comments on Reddit pointed out that: Because in ye olden days a one byte length wasn't large enough to represent long strings, but using more bytes for the length field was inefficient for short strings. And I guess no one wanted the complexity of a variable length encoding for the length prefix. And so we live with the pain of null terminated strings today. — u/Kered13 </blockquote> Also, thanks to u/vip17 for pointing out these excellent resources for diving deeper into the history: There are lots of places that explain the history of C's null-terminated strings: \ Null-terminated strings on the PDP-7 (retrocomputing.stackexchange)</a> \</li> Why did C and Linux API use null-terminated strings? (reddit r/cpp_questions)</a> \</li> Why do strings in C need to be null terminated? (stack overflow)</a></li> </ul> </blockquote> It seems the struggle between efficiency and simplicity has been with us since the dawn of C. Update: The Memory's Perspective (2025-11-27) Although we say byte</code> is the smallest addressable unit of memory, I found out that in DRAM chips, the smallest unit that can be read/written is actually a word</code> (typically 4 or 8 bytes). The concept of byte</code> is more of a logical abstraction provided by the CPU architecture or ISA. When the CPU accesses memory, it reads/writes in chunks of words. The memory controller then breaks these words down into bytes for the CPU to process. This means that even though we work with bytes in our code, under the hood, the memory system operates on larger units. In the following reference, there is a detailed explanation of how memory systems work, including the concepts of cache lines, memory hierarchy, and how data is fetched from DRAM to CPU caches. Reference: Paper: What Every Programmer Should Know About Memory</a> So, the answer to "Why C char are designed this way?" Is to fit the basic unit of memory access in ISA/CPU architecture while providing a simple abstraction for programmers to work with text data. PART 2: std::string and the Template Monster</h2> Okay, back to Coogle's bug. I was searching for std::string(std::string)</code> in the AST, but I see zero results. I then printed out the full type name stored in the AST using libclang</code> and there it was, the monster type name: std::basic_string<char, std::char_traits<char>, std::allocator<char>></code> The Typedef Illusion</h3> std::string</code> is just a typedef (alias) for a specific instantiation of the std::basic_string</code> template class. </blockquote> The search failed because I was looking for a name that doesn't exist in the type system. std::string</code> is syntactic sugar for human programmers. The compiler sees only basic_string<char></code> and its full template instantiation with default arguments. basic_string</code> actually is a template with three parameters, two of which have defaults: template< class CharT, // Character type (char, wchar_t, etc.) class Traits = char_traits<CharT>, // Character operations (comparison, copying) class Allocator = allocator<CharT> // Memory management strategy > class basic_string; </code></pre> Knowing this aliasing can be enough to solve my issue, but maybe understanding each parameter will help improving our programming skills. Deconstructing the monster CharT: Defines what kind of character the string holds—char</code> for ASCII/UTF-8, wchar_t</code> for wide characters, char16_t</code> for UTF-16, etc. Simple enough.</li> </ul> </li> Traits: Hmm... is more interesting. Accoding to cppreference:char_traits</a>: The char_traits class is a traits class template that abstracts basic character and string operations for a given character type. The defined operation set is such that generic algorithms almost always can be implemented in terms of it. </blockquote> It means we can define custom behaviors for character operations like comparison, copying, length calculation, etc. For example, VHDL files are case-insensitive, so we could define a case_insensitive_char_traits</code> that overrides comparison functions to ignore case differences. Cool! </li> </ul> </li> Allocator: This is a powerful tool controls memory management. The default uses new</code>/delete</code>, but you can plug in custom allocators for specific memory strategies—arena allocation, pool allocation, debugging allocators that track leaks, etc. This is what C++17's PMR (Polymorphic Memory Resources) builds on, which we'll cover in PART 3.</li> </ul> </li> </ul> std::basic_string</code> provides a flexible, reusable string class that can adapt to different character types, behaviors, and memory strategies. </blockquote> Quick summary diagram: Template Class (defined once) ┌────────────────────────────────────────────────────────────┐ │ template<class CharT, class Traits, class Allocator> │ │ class basic_string { │ │ CharT* data_; │ │ size_t size_; │ │ size_t capacity_; │ │ // ... methods ... │ │ }; │ └────────────────────────────────────────────────────────────┘ │ │ Template Instantiation │ ┌────────────────────────┼────────────────────────┐ │ │ │ ▼ ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────────────┐ │ std::string │ │ std::wstring │ │ std::pmr::u16string │ ├─────────────────┤ ├─────────────────┤ ├──────────────────────┤ │ CharT: │ │ CharT: │ │ CharT: │ │ char │ │ wchar_t │ │ char16_t │ │ Traits: │ │ Traits: │ │ Traits: │ │ char_traits │ │ char_traits │ │ char_traits │ │ <char> │ │ <wchar_t> │ │ <char16_t> │ │ Allocator: │ │ Allocator: │ │ Allocator: │ │ allocator │ │ allocator │ │ polymorphic_alloc │ │ <char> │ │ <wchar_t> │ │ <char16_t> │ └─────────────────┘ └─────────────────┘ └──────────────────────┘ "Hello" L"Hello" u"Hello" 6 bytes 12/24 bytes 12 bytes PMR runtime alloc! All are typedefs of basic_string with different template arguments! std::pmr::u16string -> DIFFERENT allocator = DIFFERENT type! </code></pre> class basic_string</h3> After discussing the arguments of the template, let's look at how basic_string</code> is implemented under the hood. This is exactly the same concept as Linux kernel's qstr</code>. Using a struct</code> to hold the pointer and length of the string data. There are some more features like capacity management (just like the previous mentioned strcat</code> from Joel Spolsky's article), but the core idea is the same. The code is self explanatory, so I will just paste it here: template<class CharT, class Traits, class Allocator> class basic_string { private: CharT data_; // Pointer to character buffer size_t size_; // Current length (excluding null terminator) size_t capacity_; // Allocated capacity Allocator alloc_; // Often zero-size due to EBO // Many implementations use a union for SSO (Small String Optimization): union { struct { CharT ptr; size_t size; size_t capacity; } heap; // For long strings struct { CharT buffer[16]; // Size varies by implementation unsigned char size; } stack; // For short strings (SSO) } data_; // Methods, iterators, etc. }; </code></pre> For more detail like short string optimization, and pmr</code>/allocator, please refer to PART 3. PART 3: The Cost of Abstraction (SSO, Linux, and string_view)</h2> Small String Optimization (SSO)</h3> Looking at the above code snippet of basic_string</code>, I noticed the union</code> that holds either a heap-allocated buffer or a small stack buffer. This is called Small String Optimization (SSO). The concept of SSO is simple! Stack memory allocation is much faster than heap allocation. So preallocating a small buffer inside the string object can avoid small strings calling new</code>/delete</code> frequently. In brief, stack allocation in compiler or assembly is just moving the stack pointer (prolog/epilog insertion), while heap allocation involves complex bookkeeping, searching for free blocks, and updating metadata. I then researched a bit then find Linux kernel also uses this SSO technique under virtual filesystem path. The Linux kernel's dentry</code> structure uses a union</code> called shortname_store</code> to hold either a short string directly in the structure or a pointer to a longer string allocated on the heap. This is exactly the same idea as C++'s SSO. [source: include/linux/dcache.h</code>] / Try to keep struct dentry aligned on 64 byte cachelines (this will give reasonable cacheline footprint with larger lines without the large memory footprint increase). / #ifdef CONFIG_64BIT # define DNAME_INLINE_WORDS 5 / 192 bytes / #else # ifdef CONFIG_SMP # define DNAME_INLINE_WORDS 9 / 128 bytes / # else # define DNAME_INLINE_WORDS 11 / 128 bytes / # endif #endif union shortname_store { unsigned char string[DNAME_INLINE_LEN]; unsigned long words[DNAME_INLINE_WORDS]; }; #define DNAME_INLINE_LEN (DNAME_INLINE_WORDSsizeof(unsigned long)) struct dentry { / RCU lookup touched fields / unsigned int d_flags; / protected by d_lock / seqcount_spinlock_t d_seq; / per dentry seqlock / struct hlist_bl_node d_hash; / lookup hash list / struct dentry d_parent; / parent directory / union { struct qstr __d_name; / for use ONLY in fs/dcache.c / const struct qstr d_name; }; struct inode d_inode; / Where the name belongs to - NULL is negative / union shortname_store d_shortname; //<-- SSO buffer for short names // ... }; </code></pre> Why Linux's SSO is Genius While the concept is similar to C++, Linux implements it with a crucial twist for performance: Branchless Access. In C++, accessing std::string</code> often involves a check: "Is this short or long?" before deciding where to read data. Linux avoids this check completely. The d_name.name</code> pointer (inside struct qstr</code>) is set up during creation to point either to the internal d_shortname</code> buffer or the external heap. Consumer side: When you read a filename, you just follow the pointer. Zero branching. Zero checks.</li> Cache Line: Also, notice the comments about 64-byte alignment. The buffer size is calculated so the whole struct fits perfectly into a CPU cache line.</li> </ul> This shows that for System Programmers, Memory Layout isn't just about saving bytes—it's about saving CPU cycles. Update: Community Insights (2025-11-26) Question from Reddit: If you avoid a branch, but gain a pointer indirection, how big win is that in the end, considering the branch prediction that modern CPU are capable of? u/Supadoplex </blockquote> Answer or trying to answer: I don't know the exact answer, but here is my research. It depends on predictability vs cache locality: Branch misprediction cost: Modern x86 (Skylake/Zen): ~15-20 cycles (per Agner Fog's measurements)</li> Older architectures: can be 30+ cycles</li> </ul> Pointer indirection cost: (Numbers from agner.org</a>) L1 cache hit: ~4-5 cycles (Intel/AMD spec)</li> L2 hit: ~12 cycles</li> L3 hit: ~40 cycles</li> RAM miss: 200+ cycles</li> </ul> So... If your branch is unpredictable (<80% hit rate) and your pointer data is cache-resident, indirection usually wins. — Cloudlet </blockquote> Customize size of SSO buffer Well, sometimes I think I need to customize the size of SSO buffer for specific use cases. For example, I once need to deal with complex Verilog/VHDL mix language path names, which can be quite long. So maybe next time I will try to implement a customize string to support larger SSO buffer. The key is finding the right balance between stack size and heap allocation frequency. That's too far for this article, maybe next time. The Missing Piece: std::string_view (C++17)</h3> Look at the internal structure of std::string_view</code> (from GCC's libstdc++</code>). It is strikingly similar to Linux kernel's qstr</code>: template<typename _CharT, typename _Traits = std::char_traits<_CharT>> class basic_string_view { //... private: size_t _M_len; const _CharT* _M_str; } </code></pre> Just a pointer and a length. No allocator. No heap. No ownership. The owership is owned by the std::string</code> (the most common), char</code> array, or mmap</code>ed file. Of course, there is no free lunch. std::string_view</code> is a borrowed reference (like Rust's &str</code>). Must ensure the original data (the Owner) outlives the view. Conclusion: The "Swiss Army Knife" vs. The Scalpel</h2> std::string</code> tries to do too much. It handles ownership, resizing, traits, and allocation all in one class. It is a Swiss Army Knife—great for general applications where productivity comes first. (I have to admit, using std::string and RAII feels good and is incredibly brain-friendly.) But for a high-performance tool like Linux kernel or LLVM compiler, it is over-engineered. Linux Kernel's struct qstr</code> shows us the elegance of simplicity: just a pointer and a length (and a hash). The good news? Modern C++ (C++17) finally admitted this with std::string_view</code>. It strips away the allocator magic and memory ownership, giving us back the raw efficiency of a C-style struct</code>, but with type safety. So, for writing high-performance tools: Treat std::string</code> as a heavy container (like std::vector</code>).</li> Use std::string_view</code> as your primary interface.</li> And always remember: Simplicity is the ultimate sophistication.</li> </ul> Further Study</h2> What is PMR (Polymorphic Memory Resources) in C++17?</li> How absl::Cord</code> solves the contiguous memory problem for huge strings?</li> </ul>

Emacs Internal #01: is a Lisp Runtime in C, Not an Editor

Things learned from the Story</h2>

Wishful Thinking from SICP, Practically

Other Reference</h2>

Applicative-Order vs Normal-Order Evaluation</a></li>
Thunk (Haskell)</a></li>
The Lazy Method</a>, by Josh Brindley.</li> </ul>

Stratum: Architecting a Configurable Cache Simulator with C++ and Racket

Back to the basics: Study Cache in a Software Engineer's Perspective</h2>
After that interview, I needed to understand caches from first principles. I started with CS:APP Chapter 6 (book</a>, lectures</a>)—the canonical resource for cache architecture.</p>

Building Stratum: Answering the Questions I Couldn't</h2>
After studying CS:APP and building Stratum</a>, I can now properly answer those interview questions. More importantly, I understand why</em> my rb-tree optimization worked--and when it might not.</p>

About: Yi-Ping Pan (Cloudlet)

Introducing Coogle: Bringing Haskell's Hoogle to C++

Back to Basics: From C char to string_view (Notes from building Coogle)

PART 3: The Cost of Abstraction (SSO, Linux, and string_view)</h2>

Further Study</h2>

What is PMR (Polymorphic Memory Resources) in C++17?</li>
How `absl::Cord</code> solves the contiguous memory problem for huge strings?</li> </ul>`

The Cloudlet

Emacs Internal #03: Tagged Union, Tagged Pointer, and Poor Man's Inheritance

Emacs Internal #02: Data First — Deconstructing Lisp_Object in C

The Cloudlet

Emacs Internal #03: Tagged Union, Tagged Pointer, and Poor Man's Inheritance

Conclusion</h2> This article outlines three ways memory is structured to handle dynamic typing:</p>

Emacs Internal #02: Data First — Deconstructing Lisp_Object in C

Emacs Internal #01: is a Lisp Runtime in C, Not an Editor

Things learned from the Story</h2>

Wishful Thinking from SICP, Practically

How I Apply Wishful Thinking to My Life</h2> How do I actually use wishful thinking</em> in my life?</p> I follow these steps:</p>

Other Reference</h2> Applicative-Order vs Normal-Order Evaluation</a></li> Thunk (Haskell)</a></li> The Lazy Method</a>, by Josh Brindley.</li> </ul>

Stratum: Architecting a Configurable Cache Simulator with C++ and Racket

Building Stratum: Answering the Questions I Couldn't</h2> After studying CS:APP and building Stratum</a>, I can now properly answer those interview questions. More importantly, I understand why</em> my rb-tree optimization worked--and when it might not.</p>

About: Yi-Ping Pan (Cloudlet)

Introducing Coogle: Bringing Haskell's Hoogle to C++

Back to Basics: From C char to string_view (Notes from building Coogle)

PART 3: The Cost of Abstraction (SSO, Linux, and string_view)</h2>

Further Study</h2> What is PMR (Polymorphic Memory Resources) in C++17?</li> How absl::Cord</code> solves the contiguous memory problem for huge strings?</li> </ul>

Other Reference</h2>

Applicative-Order vs Normal-Order Evaluation</a></li>
Thunk (Haskell)</a></li>
The Lazy Method</a>, by Josh Brindley.</li> </ul>

Building Stratum: Answering the Questions I Couldn't</h2>
After studying CS:APP and building Stratum</a>, I can now properly answer those interview questions. More importantly, I understand why</em> my rb-tree optimization worked--and when it might not.</p>

Further Study</h2>

What is PMR (Polymorphic Memory Resources) in C++17?</li>
How `absl::Cord</code> solves the contiguous memory problem for huge strings?</li> </ul>`