DEV Community: Uthman Oladele The latest articles on DEV Community by Uthman Oladele (@uthman_dev). https://dev.to/uthman_dev https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3569065%2Fe3bfeb63-9218-4baf-8014-41dbbd3ad9b8.jpg DEV Community: Uthman Oladele https://dev.to/uthman_dev en Stop Writing Bash Scripts, There's a Better Way Uthman Oladele Tue, 03 Mar 2026 07:18:45 +0000 https://dev.to/uthman_dev/stop-writing-bash-scripts-theres-a-better-way-50en https://dev.to/uthman_dev/stop-writing-bash-scripts-theres-a-better-way-50en <p>If you've ever written a Bash script longer than 30 lines, you know the feeling.<br> You come back to it a week later and have no idea what it does. Error handling is<br> a mess of <code>$?</code> checks, variables are global by default, and debugging feels like<br> archaeology.</p> <p>I got tired of it. But I also didn't want to spin up a Python virtual environment<br> just to move some files around or hit an API. That felt like overkill.</p> <p>So I built Logos.</p> <h2> What is Logos? </h2> <p>Logos is a small scripting language I wrote in Go. It is designed for the kind of<br> work Bash handles poorly: readable logic, proper error handling, built-in HTTP and<br> file operations, and code you can actually come back to a week later.</p> <p>It is not trying to replace Python. It is not trying to replace Go. It is just a<br> tool for that middle ground where Bash becomes unreadable but pulling in a full<br> language feels like too much.</p> <p>Scripts use the <code>.lgs</code> extension and you run them with the <code>lgs</code> command.</p> <h2> What does it look like? </h2> <p>Here is a simple HTTP request in Logos:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>let res = httpGet("https://api.example.com/data") if res.ok { let data = parseJson(res.value.body) print(data["message"]) } else { print("Error: " + res.error) } </code></pre> </div> <p>No exceptions. No try/catch. Functions return a result table with <code>ok</code>, <code>value</code>,<br> and <code>error</code> fields. You check what you need and move on.</p> <p>Compare that to doing the same thing in Bash. You are manually checking exit<br> codes, parsing curl output with awk, and hoping nothing breaks silently.</p> <h2> Why not just use Python? </h2> <p>Python is great. But for quick scripts, you are dealing with virtual environments,<br> import management, and a language that was not really designed for CLI work. It<br> also needs to be installed and configured on every machine you run it on.</p> <p>Logos ships as a single binary. Install it and run your script. That is it.</p> <h2> Why not just use Go? </h2> <p>Go requires a compile step. If you are writing a quick script to back up some<br> files or hit an API, you do not want to set up a module and run <code>go build</code> every<br> time you make a change. Logos runs scripts directly, like a shell language should.</p> <p>Also, Logos can be embedded inside Go applications as a scripting engine. You can<br> register Go functions, set variables, and run Logos scripts from within your Go<br> code. That is something Go itself cannot do cleanly.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="n">vm</span> <span class="o">:=</span> <span class="n">logos</span><span class="o">.</span><span class="n">NewWithConfig</span><span class="p">(</span><span class="n">logos</span><span class="o">.</span><span class="n">SandboxConfig</span><span class="p">{</span> <span class="n">AllowFileIO</span><span class="o">:</span> <span class="no">false</span><span class="p">,</span> <span class="n">AllowNetwork</span><span class="o">:</span> <span class="no">false</span><span class="p">,</span> <span class="n">AllowShell</span><span class="o">:</span> <span class="no">false</span><span class="p">,</span> <span class="n">AllowExit</span><span class="o">:</span> <span class="no">false</span><span class="p">,</span> <span class="p">})</span> <span class="n">vm</span><span class="o">.</span><span class="n">Register</span><span class="p">(</span><span class="s">"greet"</span><span class="p">,</span> <span class="k">func</span><span class="p">(</span><span class="n">args</span> <span class="o">...</span><span class="n">logos</span><span class="o">.</span><span class="n">Object</span><span class="p">)</span> <span class="n">logos</span><span class="o">.</span><span class="n">Object</span> <span class="p">{</span> <span class="n">name</span> <span class="o">:=</span> <span class="n">args</span><span class="p">[</span><span class="m">0</span><span class="p">]</span><span class="o">.</span><span class="p">(</span><span class="o">*</span><span class="n">logos</span><span class="o">.</span><span class="n">String</span><span class="p">)</span><span class="o">.</span><span class="n">Value</span> <span class="k">return</span> <span class="o">&amp;</span><span class="n">logos</span><span class="o">.</span><span class="n">String</span><span class="p">{</span><span class="n">Value</span><span class="o">:</span> <span class="s">"hello "</span> <span class="o">+</span> <span class="n">name</span><span class="p">}</span> <span class="p">})</span> <span class="n">vm</span><span class="o">.</span><span class="n">Run</span><span class="p">(</span><span class="s">`print(greet("world"))`</span><span class="p">)</span> </code></pre> </div> <h2> What can it do? </h2> <ul> <li>File I/O: read, write, copy, move, delete, glob</li> <li>HTTP: GET, POST, PATCH, DELETE</li> <li>JSON: parse and stringify</li> <li>Concurrency: spawn blocks and spawn for-in loops</li> <li>A formatter: <code>lgs fmt yourfile.lgs</code> </li> <li>A compiler: <code>lgs build yourfile.lgs</code> compiles your script to a standalone binary</li> <li>A REPL: just run <code>lgs</code> with no arguments</li> <li>A standard library: math, string, array, path, time, logging, testing, and more</li> </ul> <h2> A real example </h2> <p>Here is a backup script written in Logos:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>let source = trim(prompt("Source directory: ")) if !fileExists(source) { print("Error: " + source + " does not exist") exit(1) } let dest = trim(prompt("Backup destination: ")) let timestamp = replace(dateTimeStr(), " ", "_") let backupPath = dest + "/backup_" + timestamp let result = fileMkdir(backupPath) if !result.ok { print("Failed to create backup directory") exit(1) } let files = fileReadDir(source) for entry in files.value { fileCopy(source + "/" + entry, backupPath + "/" + entry) } print("Backup complete: " + backupPath) </code></pre> </div> <p>Clean. Readable. Easy to come back to.</p> <h2> How to install </h2> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>curl <span class="nt">-fsSL</span> https://raw.githubusercontent.com/codetesla51/logos/main/install.sh | sh </code></pre> </div> <p>Works on Linux and macOS. Picks the right binary for your OS and architecture<br> automatically.</p> <h2> Try it </h2> <ul> <li>GitHub: <a href="proxy.php?url=https%3A%2F%2Fgithub.com%2Fcodetesla51%2Flogos" rel="noopener noreferrer">https://github.com/codetesla51/logos</a> </li> <li>Documentation: <a href="proxy.php?url=https%3A%2F%2Flogos-lang.vercel.app%2Fdocs" rel="noopener noreferrer">https://logos-lang.vercel.app/docs</a> </li> </ul> <p>If you have feedback, run into bugs, or want to contribute examples or standard<br> library utilities, open an issue or a PR. The project is new and I am actively<br> working on it.</p> <p>I built this because I wanted it to exist. Maybe you will find it useful too.</p> bash go programming From Zero to Programming Language: A Complete Implementation Guide Uthman Oladele Sun, 04 Jan 2026 15:19:21 +0000 https://dev.to/uthman_dev/from-zero-to-programming-language-a-complete-implementation-guide-1mn8 https://dev.to/uthman_dev/from-zero-to-programming-language-a-complete-implementation-guide-1mn8 <p>Ever wondered how Python, JavaScript, or Go actually work under the hood? I spent months researching and implementing different language designs, and compiled everything into a comprehensive guide that takes you from basic lexical analysis to JIT compilation.</p> <h2> What You'll Build </h2> <p>By following this guide, you'll create a complete programming language implementation, starting with a simple calculator and progressively adding:</p> <ul> <li> <strong>Lexer &amp; Parser</strong> - Transform source code into Abstract Syntax Trees</li> <li> <strong>Interpreters</strong> - Direct AST execution (simplest approach)</li> <li> <strong>Bytecode VMs</strong> - Stack-based virtual machines like Python's CPython</li> <li> <strong>LLVM Integration</strong> - Generate native machine code</li> <li> <strong>Garbage Collection</strong> - Automatic memory management strategies</li> </ul> <h2> Why This Guide is Different </h2> <p>Most compiler tutorials give you fragments. This gives you <strong>complete, runnable code</strong> in Go that you can actually execute and modify.</p> <h3> Real Performance Numbers </h3> <p>No hand-waving here. The guide includes actual benchmarks:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>Tree-Walking Interpreter: 10-100x slower than native Bytecode VM: 5-50x slower than native JIT Compiled: 1-5x slower (can match native) AOT Compiled: Baseline (native speed) </code></pre> </div> <p>Real-world example - Fibonacci(40):</p> <ul> <li>C (gcc -O3): 0.5s</li> <li>Python (CPython): 45s (90x slower)</li> <li>Python (PyPy JIT): 2.5s (5x slower)</li> </ul> <h3> Progressive Learning Path </h3> <p>The guide is structured for gradual complexity:</p> <p><strong>Week 1: Build an Interpreter</strong><br> Start with a tree-walking interpreter - the simplest execution model. You'll have a working language by the end of the weekend.</p> <p><strong>Week 2: Add a Bytecode VM</strong><br> Compile to bytecode and build a stack-based virtual machine. Understand how Python and Java work internally.</p> <p><strong>Week 3-4: Native Code Generation</strong><br> Use LLVM to generate optimized machine code. Learn what makes Rust and Swift fast.</p> <p><strong>Beyond: JIT Compilation</strong><br> Study how V8 and HotSpot achieve near-native performance through runtime optimization.</p> <h2> Complete Working Example </h2> <p>The guide includes a full calculator language implementation with:</p> <ul> <li>Lexer (tokenization)</li> <li>Recursive descent parser</li> <li>AST generation</li> <li>Tree-walking interpreter</li> <li>Variables and expressions </li> </ul> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="n">source</span> <span class="o">:=</span> <span class="s">` x = 10 y = 20 z = x + y * 2 `</span> <span class="n">lexer</span> <span class="o">:=</span> <span class="n">NewLexer</span><span class="p">(</span><span class="n">source</span><span class="p">)</span> <span class="n">parser</span> <span class="o">:=</span> <span class="n">NewParser</span><span class="p">(</span><span class="n">lexer</span><span class="p">)</span> <span class="n">ast</span> <span class="o">:=</span> <span class="n">parser</span><span class="o">.</span><span class="n">Parse</span><span class="p">()</span> <span class="n">interpreter</span> <span class="o">:=</span> <span class="n">NewInterpreter</span><span class="p">()</span> <span class="n">interpreter</span><span class="o">.</span><span class="n">Eval</span><span class="p">(</span><span class="n">ast</span><span class="p">)</span> <span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"z = %d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">interpreter</span><span class="o">.</span><span class="n">vars</span><span class="p">[</span><span class="s">"z"</span><span class="p">])</span> <span class="c">// z = 50</span> </code></pre> </div> <p>This isn't pseudocode - it's actual running Go code you can build on.</p> <h2> What's Covered </h2> <h3> The Compilation Pipeline </h3> <ul> <li> <strong>Lexical Analysis</strong>: Breaking source code into tokens</li> <li> <strong>Syntax Analysis</strong>: Building Abstract Syntax Trees</li> <li> <strong>Semantic Analysis</strong>: Type checking and symbol resolution</li> <li> <strong>Code Generation</strong>: Bytecode, LLVM IR, or direct interpretation</li> </ul> <h3> Execution Models Deep Dive </h3> <p><strong>Interpreters</strong></p> <ul> <li>Direct AST execution</li> <li>Simplest to implement</li> <li>Best for scripting and configuration languages</li> </ul> <p><strong>Virtual Machines</strong></p> <ul> <li>Stack-based vs register-based architectures</li> <li>Bytecode design and instruction sets</li> <li>Function calls and stack frames</li> <li>Control flow implementation</li> </ul> <p><strong>LLVM Integration</strong></p> <ul> <li>Generating LLVM IR</li> <li>Type system mapping</li> <li>Optimization passes</li> <li>Cross-platform native code generation</li> </ul> <p><strong>JIT Compilation</strong> (Advanced)</p> <ul> <li>Profiling and hot path detection</li> <li>Runtime code generation</li> <li>Deoptimization strategies</li> <li>Type specialization</li> </ul> <h3> Garbage Collection </h3> <p>Deep dive into automatic memory management:</p> <ol> <li> <strong>Reference Counting</strong> - Immediate reclamation, can't handle cycles</li> <li> <strong>Mark-and-Sweep</strong> - Handles cycles, stop-the-world pauses</li> <li> <strong>Copying/Generational</strong> - Best performance, most complex</li> </ol> <p>Each approach includes working implementations and trade-off analysis.</p> <h2> Real-World Insights </h2> <p>The guide doesn't just teach theory - it explains practical decisions:</p> <ul> <li>Why does Python use bytecode instead of direct interpretation?</li> <li>How does JavaScript achieve near-native performance?</li> <li>Why are Go compilation times so fast?</li> <li>What makes Rust's borrow checker possible?</li> </ul> <h3> Trade-offs Made Clear </h3> <p><strong>Development Complexity:</strong></p> <ul> <li>Interpreter: Weekend project</li> <li>Bytecode VM: 1-2 weeks</li> <li>JIT Compiler: Months</li> <li>AOT with LLVM: 2-4 weeks</li> </ul> <p><strong>Execution Speed:</strong></p> <ul> <li>Interpreter: 10-100x slower than native</li> <li>Bytecode VM: 5-50x slower</li> <li>JIT: 1-5x slower (can match native)</li> <li>AOT: Native speed</li> </ul> <p><strong>Startup Time:</strong></p> <ul> <li>Interpreter: Instant</li> <li>Bytecode VM: Very fast</li> <li>JIT: Slow (warmup period)</li> <li>AOT: Instant (pre-compiled)</li> </ul> <h2> Key Highlights </h2> <h3> Complete Implementations </h3> <p>Every major component includes full, working code:</p> <ul> <li>Lexer with position tracking and error handling</li> <li>Recursive descent parser with operator precedence</li> <li>Stack-based VM with complete instruction set</li> <li>LLVM IR generation with control flow</li> </ul> <h3> No Handwaving </h3> <p>The guide tackles the hard parts:</p> <ul> <li>How to make executable memory for JIT compilation</li> <li>Platform-specific calling conventions</li> <li>Why reference counting can't handle cycles</li> <li>Managing instruction pointer and call stacks</li> </ul> <h3> Practical Examples </h3> <p>See how to implement:</p> <ul> <li>Variables and assignments</li> <li>Arithmetic expressions with correct precedence</li> <li>Control flow (if/while) in bytecode</li> <li>Function calls with proper stack frames</li> <li>Type checking and semantic analysis</li> </ul> <h2> Who This Is For </h2> <p><strong>You should read this if you:</strong></p> <ul> <li>Want to understand how programming languages work</li> <li>Are building a DSL or configuration language</li> <li>Curious about compiler design but intimidated by Dragon Book</li> <li>Want to contribute to language projects (Rust, Go, Python)</li> <li>Need to implement a scripting system for your application</li> </ul> <p><strong>Prerequisites:</strong></p> <ul> <li>Comfortable with Go (or can read and adapt the code)</li> <li>Basic understanding of data structures (trees, stacks)</li> <li>Curiosity about how things work under the hood</li> </ul> <p>No CS degree required. No prior compiler knowledge assumed.</p> <h2> Learning Path Recommendation </h2> <ol> <li> <strong>Start with the complete calculator example</strong> - Get something working immediately</li> <li> <strong>Add control flow</strong> - Implement if statements and loops using the bytecode examples</li> <li> <strong>Add functions</strong> - Use the call frame implementation provided</li> <li> <strong>Explore LLVM</strong> - Generate native code when you're ready for more performance</li> <li> <strong>Study GC</strong> - Understand automatic memory management</li> </ol> <p>Each step builds on the previous, and you'll have a working language at each stage.</p> <h2> What You'll Gain </h2> <p>After working through this guide:</p> <ul> <li> <strong>Deep understanding</strong> of how interpreters, compilers, and VMs work</li> <li> <strong>Practical experience</strong> building complex systems from scratch</li> <li> <strong>Appreciation</strong> for language design trade-offs</li> <li> <strong>Foundation</strong> for contributing to real language projects</li> <li> <strong>Confidence</strong> to build domain-specific languages</li> </ul> <h2> Resources Included </h2> <p>The guide references essential learning materials:</p> <ul> <li>"Crafting Interpreters" by Bob Nystrom</li> <li>LLVM tutorials and documentation</li> <li>Real-world language implementations to study</li> <li>Performance benchmarking techniques</li> </ul> <h2> Get Started </h2> <p>The complete guide with all code examples is available on GitHub:</p> <p><strong><a href="proxy.php?url=https%3A%2F%2Fgithub.com%2Fcodetesla51%2Fhow-to-build-a-programming-language" rel="noopener noreferrer">github.com/codetesla51/how-to-build-a-programming-language</a></strong></p> <p>Clone the repo, run the examples, and start building your own language today.</p> <h2> Feedback Welcome </h2> <p>This is a living guide. If you find issues, have questions, or want to contribute improvements, please open an issue or PR on GitHub.</p> <p>Building a programming language is one of the most rewarding projects in computer science. It demystifies the entire software stack and gives you superpowers for understanding any codebase.</p> <p>Start small. Build a calculator. Add features incrementally. Break things. Fix them. That's how you learn.</p> <p>Happy language building!</p> programming go computerscience How Buffer Pooling Doubled My HTTP Server's Throughput (4,000 7,721 RPS) Uthman Oladele Wed, 12 Nov 2025 16:46:35 +0000 https://dev.to/uthman_dev/how-buffer-pooling-doubled-my-http-servers-throughput-4000-7721-rps-3i0g https://dev.to/uthman_dev/how-buffer-pooling-doubled-my-http-servers-throughput-4000-7721-rps-3i0g <p>Last week, I shared my journey building an HTTP server from scratch in Go using raw TCP sockets. The performance was decent—around 4,000 requests per second at peak—but I knew there was room for improvement.</p> <p>Then I learned about <strong>buffer pooling</strong>, and everything changed.</p> <h2> The Problem: Death by a Thousand Allocations </h2> <p>My original server had a hidden performance killer. For every single HTTP request, the server was doing this:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="k">func</span> <span class="n">handleConnection</span><span class="p">(</span><span class="n">conn</span> <span class="n">net</span><span class="o">.</span><span class="n">Conn</span><span class="p">)</span> <span class="p">{</span> <span class="n">buffer</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span> <span class="m">8192</span><span class="p">)</span> <span class="c">// New allocation</span> <span class="n">conn</span><span class="o">.</span><span class="n">Read</span><span class="p">(</span><span class="n">buffer</span><span class="p">)</span> <span class="c">// ... process request ...</span> <span class="c">// Buffer gets garbage collected</span> <span class="p">}</span> </code></pre> </div> <p>Seems innocent, right? But when you're handling thousands of requests per second, this becomes:</p> <ul> <li> <strong>Constant memory allocation</strong> - Creating new 8KB buffers constantly</li> <li> <strong>Garbage collector pressure</strong> - GC running frequently to clean up discarded buffers</li> <li> <strong>CPU cycles wasted</strong> - Allocation and deallocation overhead instead of serving requests</li> </ul> <p>At 4,000 RPS, my server was allocating and throwing away <strong>32 MB of memory every single second</strong>.</p> <h2> The Solution: Buffer Pooling with sync.Pool </h2> <p>The core idea is brilliantly simple: <strong>reuse buffers instead of creating new ones</strong>.</p> <p>Go's standard library provides <code>sync.Pool</code> for exactly this purpose. Here's how I implemented it:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="k">var</span> <span class="n">requestBufferPool</span> <span class="o">=</span> <span class="n">sync</span><span class="o">.</span><span class="n">Pool</span><span class="p">{</span> <span class="n">New</span><span class="o">:</span> <span class="k">func</span><span class="p">()</span> <span class="k">interface</span><span class="p">{}</span> <span class="p">{</span> <span class="n">buf</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span> <span class="m">8192</span><span class="p">)</span> <span class="k">return</span> <span class="n">buf</span> <span class="p">},</span> <span class="p">}</span> <span class="k">func</span> <span class="n">handleConnection</span><span class="p">(</span><span class="n">conn</span> <span class="n">net</span><span class="o">.</span><span class="n">Conn</span><span class="p">)</span> <span class="p">{</span> <span class="c">// Get a buffer from the pool</span> <span class="n">buffer</span> <span class="o">:=</span> <span class="n">requestBufferPool</span><span class="o">.</span><span class="n">Get</span><span class="p">()</span><span class="o">.</span><span class="p">([]</span><span class="kt">byte</span><span class="p">)</span> <span class="k">defer</span> <span class="n">requestBufferPool</span><span class="o">.</span><span class="n">Put</span><span class="p">(</span><span class="n">buffer</span><span class="p">)</span> <span class="c">// Return it when done</span> <span class="n">conn</span><span class="o">.</span><span class="n">Read</span><span class="p">(</span><span class="n">buffer</span><span class="p">)</span> <span class="c">// ... process request ...</span> <span class="p">}</span> </code></pre> </div> <h3> How It Works </h3> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>Request 1: Get buffer from pool → Use it → Return to pool Request 2: Get SAME buffer → Reset &amp; use → Return to pool Request 3: Get SAME buffer → Reset &amp; use → Return to pool ↓ (Zero new allocations!) </code></pre> </div> <p>Instead of the old wasteful cycle:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>Request 1: Allocate → Use → GC cleanup Request 2: Allocate → Use → GC cleanup Request 3: Allocate → Use → GC cleanup ↓ (Constant allocation churn) </code></pre> </div> <h2> Implementation: Three Strategic Pools </h2> <p>I identified three hot paths where buffers were being repeatedly allocated:</p> <h3> 1. Request Buffer Pool (8KB) </h3> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="k">var</span> <span class="n">requestBufferPool</span> <span class="o">=</span> <span class="n">sync</span><span class="o">.</span><span class="n">Pool</span><span class="p">{</span> <span class="n">New</span><span class="o">:</span> <span class="k">func</span><span class="p">()</span> <span class="k">interface</span><span class="p">{}</span> <span class="p">{</span> <span class="n">buf</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span> <span class="m">8192</span><span class="p">)</span> <span class="k">return</span> <span class="n">buf</span> <span class="p">},</span> <span class="p">}</span> </code></pre> </div> <p>Used for reading incoming HTTP requests from TCP connections.</p> <h3> 2. Chunk Buffer Pool (256 bytes) </h3> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="k">var</span> <span class="n">chunkBufferPool</span> <span class="o">=</span> <span class="n">sync</span><span class="o">.</span><span class="n">Pool</span><span class="p">{</span> <span class="n">New</span><span class="o">:</span> <span class="k">func</span><span class="p">()</span> <span class="k">interface</span><span class="p">{}</span> <span class="p">{</span> <span class="n">buf</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span> <span class="m">256</span><span class="p">)</span> <span class="k">return</span> <span class="n">buf</span> <span class="p">},</span> <span class="p">}</span> </code></pre> </div> <p>Used for smaller, chunked reads during request parsing.</p> <h3> 3. Response Buffer Pool </h3> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="k">var</span> <span class="n">responseBufferPool</span> <span class="o">=</span> <span class="n">sync</span><span class="o">.</span><span class="n">Pool</span><span class="p">{</span> <span class="n">New</span><span class="o">:</span> <span class="k">func</span><span class="p">()</span> <span class="k">interface</span><span class="p">{}</span> <span class="p">{</span> <span class="k">return</span> <span class="nb">new</span><span class="p">(</span><span class="n">bytes</span><span class="o">.</span><span class="n">Buffer</span><span class="p">)</span> <span class="p">},</span> <span class="p">}</span> </code></pre> </div> <p>Used for building HTTP response strings before sending.</p> <h2> The Results: Nearly 2x Performance Gain </h2> <p>I ran the same benchmark suite before and after implementing buffer pooling:</p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>Metric</th> <th>Before Buffer Pools</th> <th>After Buffer Pools</th> <th>Improvement</th> </tr> </thead> <tbody> <tr> <td><strong>Peak RPS</strong></td> <td>4,000</td> <td>7,721</td> <td> <strong>+93%</strong> 🔥</td> </tr> <tr> <td><strong>Response Time</strong></td> <td>0.250ms</td> <td>0.130ms</td> <td><strong>48% faster</strong></td> </tr> <tr> <td><strong>Memory Allocations</strong></td> <td>Constant</td> <td>Minimal</td> <td><strong>~95% reduction</strong></td> </tr> <tr> <td><strong>GC Pressure</strong></td> <td>High</td> <td>Low</td> <td><strong>Significantly reduced</strong></td> </tr> </tbody> </table></div> <h3> Before vs After Comparison </h3> <p><strong>Before (4,000 RPS):</strong><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>ab <span class="nt">-n</span> 10000 <span class="nt">-c</span> 10 <span class="nt">-k</span> http://localhost:8080/ping Requests per second: 4000.12 <span class="o">[</span><span class="c">#/sec]</span> Time per request: 0.250 <span class="o">[</span>ms] <span class="o">(</span>mean<span class="o">)</span> </code></pre> </div> <p><strong>After (7,721 RPS):</strong><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>ab <span class="nt">-n</span> 10000 <span class="nt">-c</span> 10 <span class="nt">-k</span> http://localhost:8080/ping Requests per second: 7721.45 <span class="o">[</span><span class="c">#/sec]</span> Time per request: 0.130 <span class="o">[</span>ms] <span class="o">(</span>mean<span class="o">)</span> </code></pre> </div> <h2> Key Takeaways </h2> <h3> 1. Memory Management Matters—A Lot </h3> <p>Even in a garbage-collected language like Go, being mindful of allocations can dramatically impact performance. The GC is smart, but avoiding work is always faster than doing work efficiently.</p> <h3> 2. Profile Before Optimizing </h3> <p>I initially thought my bottleneck was request parsing or routing logic. A quick profiling session revealed that memory allocation was the real culprit.</p> <h3> 3. sync.Pool Is Your Friend </h3> <p>For any frequently allocated/deallocated objects (buffers, temporary structs, builders), <code>sync.Pool</code> is a simple way to get significant performance gains.</p> <h3> 4. The 80/20 Rule Applies </h3> <p>Three strategic buffer pools gave me a 93% performance improvement. This took maybe 30 minutes to implement once I understood the concept.</p> <h2> Important Caveats </h2> <h3> This Is Still a Learning Project </h3> <p>While 7,721 RPS sounds impressive, this benchmark tests a simple <code>/ping</code> endpoint that returns "pong". Real-world applications with:</p> <ul> <li>Database queries</li> <li>Business logic</li> <li>File I/O</li> <li>External API calls</li> </ul> <p>...will see significantly lower RPS (typically 100-500 for most web apps). Buffer pooling helps with the networking layer, but your application logic will likely be the bottleneck in production.</p> <h3> Not a Silver Bullet </h3> <p>Buffer pooling helps when:</p> <ul> <li>You're allocating the same size/type repeatedly</li> <li>The objects are expensive to create</li> <li>Allocation shows up in profiling</li> </ul> <p>It won't help if your bottleneck is CPU-bound computation, database queries, or external API calls.</p> <h2> The Journey So Far </h2> <p>Here's the complete performance evolution of this project:</p> <ol> <li> <strong>Initial buggy version:</strong> ~250 RPS (Connection handling bugs)</li> <li> <strong>Bug fix:</strong> 1,389 RPS (Fixed request handling)</li> <li> <strong>Keep-alive optimization:</strong> 1,710 RPS (Connection reuse)</li> <li> <strong>Concurrency optimization:</strong> 4,000 RPS (Found optimal load)</li> <li> <strong>Buffer pooling:</strong> 7,721 RPS (Memory optimization) 🚀</li> </ol> <p><strong>Total improvement: ~31x from the first version!</strong></p> <h2> Try It Yourself </h2> <p>The full source code is available on GitHub:<br> <a href="proxy.php?url=https%3A%2F%2Fgithub.com%2Fcodetesla51%2Fraw-http" rel="noopener noreferrer">https://github.com/codetesla51/raw-http</a></p> <p>Clone it, run benchmarks before and after commenting out the buffer pools, and see the difference yourself:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>git clone https://github.com/codetesla51/raw-http <span class="nb">cd </span>raw-http go run main.go <span class="c"># In another terminal</span> ab <span class="nt">-n</span> 10000 <span class="nt">-c</span> 10 <span class="nt">-k</span> http://localhost:8080/ping </code></pre> </div> <h2> What's Next? </h2> <p>Now that I've squeezed significant performance out of the memory layer, I'm curious about:</p> <ul> <li> <strong>HTTP/2 support</strong> - Binary framing instead of text parsing</li> <li> <strong>Connection pooling strategies</strong> - How do production servers handle thousands of concurrent connections?</li> <li> <strong>Zero-copy techniques</strong> - Can I avoid copying data between buffers entirely?</li> </ul> <p><strong>Have you used buffer pooling in your projects? What performance gains did you see?</strong> Drop a comment below—I'd love to hear about your optimization stories!</p> <p><em>Building from first principles teaches you things frameworks hide. This project continues to surprise me with how much performance can be unlocked by understanding what's really happening under the hood.</em></p> <p><strong>Project Stats:</strong></p> <ul> <li>🚀 7,721 RPS peak performance</li> <li>🔧 Built with raw TCP sockets in Go</li> <li>📚 Learning-focused, not production-ready</li> <li>⚡ +93% performance from buffer pooling alone</li> </ul> <p><em>Check out the <a href="proxy.php?url=https%3A%2F%2Fgithub.com%2Fcodetesla51%2Fraw-http" rel="noopener noreferrer">full project on GitHub</a></em></p> go backend Rebuilding Grep in Go: What I Learned About Unix Text Processing Uthman Oladele Tue, 11 Nov 2025 12:04:59 +0000 https://dev.to/uthman_dev/rebuilding-grep-in-go-what-i-learned-about-unix-text-processing-1oc https://dev.to/uthman_dev/rebuilding-grep-in-go-what-i-learned-about-unix-text-processing-1oc <p>I use grep every day but had no idea how it works. So I built a basic version in Go.</p> <p>Not to replace grep. Just to stop being the guy who pipes to grep without understanding what's actually happening.</p> <h2> What I Built </h2> <p>A stripped-down grep that does pattern matching with these flags:</p> <ul> <li> <code>-i</code> - Case-insensitive</li> <li> <code>[](url)-n</code> - Line numbers</li> <li> <code>-c</code> - Count matches</li> <li> <code>-v</code> - Invert (show non-matches)</li> <li> <code>-r</code> - Recursive search</li> <li> <code>-l</code> - Just list filenames</li> </ul> <p>Plus binary file detection and color output. That's it. No context lines, no fancy regex modes, no performance optimizations.</p> <h2> The Interesting Parts </h2> <h3> Binary Files Don't Print Garbage </h3> <p>When grep hits a binary file (executable, image, whatever), it says "Binary file matches" instead of filling your terminal with nonsense.</p> <p>How does it know?<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="k">func</span> <span class="n">IsBinary</span><span class="p">(</span><span class="n">filePath</span> <span class="kt">string</span><span class="p">)</span> <span class="p">(</span><span class="kt">bool</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span> <span class="n">f</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">os</span><span class="o">.</span><span class="n">Open</span><span class="p">(</span><span class="n">filePath</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="k">return</span> <span class="no">false</span><span class="p">,</span> <span class="n">err</span> <span class="p">}</span> <span class="k">defer</span> <span class="n">f</span><span class="o">.</span><span class="n">Close</span><span class="p">()</span> <span class="n">buffer</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span> <span class="m">1024</span><span class="p">)</span> <span class="n">n</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">f</span><span class="o">.</span><span class="n">Read</span><span class="p">(</span><span class="n">buffer</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="o">&amp;&amp;</span> <span class="n">err</span> <span class="o">!=</span> <span class="n">io</span><span class="o">.</span><span class="n">EOF</span> <span class="p">{</span> <span class="k">return</span> <span class="no">false</span><span class="p">,</span> <span class="n">err</span> <span class="p">}</span> <span class="k">if</span> <span class="n">bytes</span><span class="o">.</span><span class="n">IndexByte</span><span class="p">(</span><span class="n">buffer</span><span class="p">[</span><span class="o">:</span><span class="n">n</span><span class="p">],</span> <span class="m">0</span><span class="p">)</span> <span class="o">!=</span> <span class="o">-</span><span class="m">1</span> <span class="p">{</span> <span class="k">return</span> <span class="no">true</span><span class="p">,</span> <span class="no">nil</span> <span class="p">}</span> <span class="k">return</span> <span class="no">false</span><span class="p">,</span> <span class="no">nil</span> <span class="p">}</span> </code></pre> </div> <p>Read 1KB. If there's a null byte (<code>\0</code>), it's binary. Text files don't have null bytes.</p> <p>Simple check. Works.</p> <h3> Recursive Search Without Exploding </h3> <p>The <code>-r</code> flag searches directories. This means reading entries, checking if they're files or directories, and recursing when needed.</p> <p>The trick: Don't stop on errors. One locked file shouldn't kill your entire search.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="k">if</span> <span class="n">info</span><span class="o">.</span><span class="n">IsDir</span><span class="p">()</span> <span class="p">{</span> <span class="k">if</span> <span class="o">!</span><span class="n">opts</span><span class="o">.</span><span class="n">Recursive</span> <span class="p">{</span> <span class="k">return</span> <span class="m">0</span><span class="p">,</span> <span class="n">fmt</span><span class="o">.</span><span class="n">Errorf</span><span class="p">(</span><span class="s">"%s is a directory"</span><span class="p">,</span> <span class="n">fileName</span><span class="p">)</span> <span class="p">}</span> <span class="n">entries</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">os</span><span class="o">.</span><span class="n">ReadDir</span><span class="p">(</span><span class="n">fileName</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="k">return</span> <span class="m">0</span><span class="p">,</span> <span class="n">fmt</span><span class="o">.</span><span class="n">Errorf</span><span class="p">(</span><span class="s">"error reading directory: %v"</span><span class="p">,</span> <span class="n">err</span><span class="p">)</span> <span class="p">}</span> <span class="n">total</span> <span class="o">:=</span> <span class="m">0</span> <span class="k">for</span> <span class="n">_</span><span class="p">,</span> <span class="n">entry</span> <span class="o">:=</span> <span class="k">range</span> <span class="n">entries</span> <span class="p">{</span> <span class="n">path</span> <span class="o">:=</span> <span class="n">filepath</span><span class="o">.</span><span class="n">Join</span><span class="p">(</span><span class="n">fileName</span><span class="p">,</span> <span class="n">entry</span><span class="o">.</span><span class="n">Name</span><span class="p">())</span> <span class="n">subCount</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">grepFile</span><span class="p">(</span><span class="n">pattern</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="n">fmt</span><span class="o">.</span><span class="n">Fprintf</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">Stderr</span><span class="p">,</span> <span class="s">"warning: %v</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">err</span><span class="p">)</span> <span class="k">continue</span> <span class="c">// Keep going</span> <span class="p">}</span> <span class="n">total</span> <span class="o">+=</span> <span class="n">subCount</span> <span class="p">}</span> <span class="k">return</span> <span class="n">total</span><span class="p">,</span> <span class="no">nil</span> <span class="p">}</span> </code></pre> </div> <p>Log it, move on. Real grep does this. So should you.</p> <h3> Flags Interact in Weird Ways </h3> <p><strong>Case-insensitive search:</strong> Modify the pattern before compiling the regex.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="k">if</span> <span class="n">opts</span><span class="o">.</span><span class="n">CaseInsensitive</span> <span class="p">{</span> <span class="n">pattern</span> <span class="o">=</span> <span class="s">"(?i)"</span> <span class="o">+</span> <span class="n">pattern</span> <span class="p">}</span> <span class="n">re</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">regexp</span><span class="o">.</span><span class="n">Compile</span><span class="p">(</span><span class="n">pattern</span><span class="p">)</span> </code></pre> </div> <p><strong>List files only:</strong> Stop after the first match.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="k">if</span> <span class="n">matched</span> <span class="p">{</span> <span class="n">count</span><span class="o">++</span> <span class="k">if</span> <span class="n">opts</span><span class="o">.</span><span class="n">ListFilesOnly</span> <span class="p">{</span> <span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"%s:</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">fileName</span><span class="p">)</span> <span class="k">return</span> <span class="n">count</span><span class="p">,</span> <span class="no">nil</span> <span class="c">// Done</span> <span class="p">}</span> <span class="p">}</span> </code></pre> </div> <p><strong>Invert match:</strong> Flip the boolean.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="n">matched</span> <span class="o">:=</span> <span class="n">re</span><span class="o">.</span><span class="n">MatchString</span><span class="p">(</span><span class="n">line</span><span class="p">)</span> <span class="k">if</span> <span class="n">opts</span><span class="o">.</span><span class="n">Invert</span> <span class="p">{</span> <span class="n">matched</span> <span class="o">=</span> <span class="o">!</span><span class="n">matched</span> <span class="p">}</span> </code></pre> </div> <p>Getting these combinations right took a few tries. <code>-r -l</code> should recurse and list filenames. <code>-v -i</code> should invert case-insensitive matches. Test all the combinations.</p> <h3> Color Output </h3> <p>Grep highlights matches in red. To do this:</p> <ol> <li>Find all regex matches in the line</li> <li>Replace each match with a colored version</li> <li>Print the result </li> </ol> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="n">matchColor</span> <span class="o">:=</span> <span class="n">color</span><span class="o">.</span><span class="n">New</span><span class="p">(</span><span class="n">color</span><span class="o">.</span><span class="n">FgRed</span><span class="p">)</span><span class="o">.</span><span class="n">SprintFunc</span><span class="p">()</span> <span class="n">coloredLine</span> <span class="o">:=</span> <span class="n">re</span><span class="o">.</span><span class="n">ReplaceAllStringFunc</span><span class="p">(</span><span class="n">line</span><span class="p">,</span> <span class="k">func</span><span class="p">(</span><span class="n">m</span> <span class="kt">string</span><span class="p">)</span> <span class="kt">string</span> <span class="p">{</span> <span class="k">return</span> <span class="n">matchColor</span><span class="p">(</span><span class="n">m</span><span class="p">)</span> <span class="p">})</span> </code></pre> </div> <p><code>ReplaceAllStringFunc</code> walks through matches and applies your function. Easy.</p> <h2> What I Actually Learned </h2> <p><strong>Null bytes are the binary file signal.</strong> One check, problem solved.</p> <p><strong>Compile regex once, use it everywhere.</strong> Compiling per-line is stupid slow. Compile once at the start.</p> <p><strong>Scanner vs ReadFile depends on the file type.</strong> Text files get <code>bufio.Scanner</code> for line-by-line reading. Binary files get <code>ReadFile</code> to check the whole thing. Different tools for different jobs.</p> <p><strong>Recursive operations need error tolerance.</strong> One bad file can't crash everything. Log it, continue.</p> <p><strong>Manual flag parsing sucks.</strong> Next time I'm using a library. Checking string prefixes gets old fast.</p> <h2> What's Missing </h2> <p>Real grep has a lot more:</p> <ul> <li>Context lines (<code>-A</code>, <code>-B</code>, <code>-C</code>)</li> <li>Extended regex (<code>-E</code>)</li> <li>Fixed string search (<code>-F</code>)</li> <li>Parallel search</li> <li>Memory-mapped files for huge files</li> <li>Proper CLI argument parsing</li> </ul> <p>Mine doesn't. It does basic pattern matching. That's the point - understand the core, skip the extras.</p> <h2> Try It </h2> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>git clone https://github.com/codetesla51/go-coreutils.git <span class="nb">cd </span>go-coreutils/grep go build <span class="nt">-o</span> <span class="nb">grep </span>grep.go ./grep <span class="s2">"pattern"</span> file.txt </code></pre> </div> <p>Basic usage:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>./grep <span class="s2">"error"</span> logs.txt ./grep <span class="nt">-i</span> <span class="nt">-n</span> <span class="s2">"warning"</span> logs.txt ./grep <span class="nt">-r</span> <span class="s2">"TODO"</span> ./src ./grep <span class="nt">-c</span> <span class="s2">"func"</span> main.go </code></pre> </div> <h2> Why This Matters </h2> <p>Grep is everywhere. Understanding how it works changes how you use it. You stop thinking "magic search command" and start thinking "regex matcher with file handling."</p> <p>Plus, now when someone asks "how does grep detect binary files?" you actually know.</p> <p><strong>Source:</strong> <a href="proxy.php?url=https%3A%2F%2Fgithub.com%2Fcodetesla51%2Fgo-coreutils" rel="noopener noreferrer">github.com/codetesla51/go-coreutils</a></p> <p>More at <a href="proxy.php?url=https%3A%2F%2Fdevuthman.vercel.app" rel="noopener noreferrer">devuthman.vercel.app</a></p> linux go Building a Terminal Calculator That Actually Does Logic: Axion Uthman Oladele Sun, 09 Nov 2025 12:45:05 +0000 https://dev.to/uthman_dev/building-a-terminal-calculator-that-actually-does-logic-axion-1p0m https://dev.to/uthman_dev/building-a-terminal-calculator-that-actually-does-logic-axion-1p0m <p>I built a calculator that does more than just add numbers. It's a full mathematical computing environment in your terminal with logical operations, comparison operators, unit conversions, and persistent memory.</p> <p>This is an active project - I'm constantly working on it, adding features, fixing bugs, and improving performance.</p> <h2> Why Build Another Calculator? </h2> <p>Because most CLI calculators are just arithmetic. They add, subtract, maybe handle some trig functions, and that's it.</p> <p>I wanted something that could handle real computational work:</p> <ul> <li>Compare values and return boolean results</li> <li>Chain logical operations with proper precedence</li> <li>Convert between units without leaving the terminal</li> <li>Store variables and reuse them across sessions</li> <li>Handle scientific notation properly</li> <li>Remember your calculation history</li> </ul> <p>Basically, a calculator that works the way you think, not just the way computers add numbers.</p> <h2> What It Does </h2> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>» 2 + 3 <span class="k">*</span> 4 Result: 14 » sin<span class="o">(</span>30<span class="o">)</span> + cos<span class="o">(</span>60<span class="o">)</span> Result: 1 » 5 <span class="o">&gt;</span> 3 Result: 1 » <span class="o">(</span>5 <span class="o">&gt;</span> 3<span class="o">)</span> <span class="o">&amp;&amp;</span> <span class="o">(</span>2 &lt; 4<span class="o">)</span> Result: 1 » x <span class="o">=</span> sqrt<span class="o">(</span>16<span class="o">)</span> Result: 4 » convert 100 cm to m 100 cm <span class="o">=</span> 1 m </code></pre> </div> <p>It's not just a calculator - it's a mathematical environment.</p> <h2> The Core Features </h2> <h3> 1. Logical and Comparison Operations </h3> <p>This is what sets Axion apart. You can compare values and chain logical operations:</p> <p><strong>Comparison operators:</strong></p> <ul> <li> <code>&gt;</code>, <code>&lt;</code>, <code>&gt;=</code>, <code>&lt;=</code>, <code>==</code>, <code>!=</code> </li> <li>Returns <code>1</code> for true, <code>0</code> for false</li> </ul> <p><strong>Logical operators:</strong></p> <ul> <li> <code>&amp;&amp;</code> (AND) - returns 1 if both operands are non-zero</li> <li> <code>||</code> (OR) - returns 1 if at least one operand is non-zero</li> </ul> <p><strong>Proper precedence:</strong><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>» 0 <span class="o">||</span> 1 <span class="o">&amp;&amp;</span> 0 Result: 0 <span class="c"># evaluated as: 0 || (1 &amp;&amp; 0)</span> » <span class="o">(</span>5 <span class="o">&gt;</span> 3<span class="o">)</span> <span class="o">&amp;&amp;</span> <span class="o">(</span>2 &lt; 4<span class="o">)</span> Result: 1 » temp <span class="o">=</span> 25 » isComfortable <span class="o">=</span> <span class="o">(</span>temp <span class="o">&gt;</span> 18<span class="o">)</span> <span class="o">&amp;&amp;</span> <span class="o">(</span>temp &lt; 28<span class="o">)</span> Result: 1 </code></pre> </div> <p>The precedence order matters:</p> <ol> <li> <code>||</code> (lowest)</li> <li><code>&amp;&amp;</code></li> <li>Comparison operators</li> <li>Arithmetic operators</li> <li>Parentheses (highest)</li> </ol> <h3> 2. Variable System with Persistence </h3> <p>Variables persist across sessions:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>» radius <span class="o">=</span> 5 Result: 5 » area <span class="o">=</span> pi <span class="k">*</span> radius^2 Result: 78.5398 » isLarge <span class="o">=</span> radius <span class="o">&gt;=</span> 10 Result: 0 </code></pre> </div> <p>Close the calculator, open it again, and your variables are still there. Stored in JSON, loaded automatically.</p> <h3> 3. Unit Conversion System </h3> <p>Built-in conversions across three categories:</p> <p><strong>Length:</strong> m, cm, mm, km, in, ft, yd, mi<br> <strong>Weight:</strong> kg, g, mg, lb, oz, ton<br> <strong>Time:</strong> s, ms, min, h, d<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>» convert 5280 ft to mi 5280 ft <span class="o">=</span> 1 mi » convert 2.5 kg to lb 2.5 kg <span class="o">=</span> 5.51156 lb </code></pre> </div> <p>The system prevents invalid conversions (like trying to convert kilograms to meters).</p> <h3> 4. Scientific Computing </h3> <p>Full scientific notation support and mathematical functions:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>» 2e-10 + 3.5E+12 Result: 3500000000000 » log<span class="o">(</span>100<span class="o">)</span> + <span class="nb">ln</span><span class="o">(</span>e<span class="o">)</span> + log2<span class="o">(</span>16<span class="o">)</span> Result: 9.60517 » mean<span class="o">(</span>10, 20, 30, 40, 50<span class="o">)</span> Result: 30 </code></pre> </div> <p><strong>Available functions:</strong></p> <ul> <li>Trig: <code>sin()</code>, <code>cos()</code>, <code>tan()</code>, <code>asin()</code>, <code>acos()</code>, <code>atan()</code> </li> <li>Log: <code>ln()</code>, <code>log()</code>, <code>log10()</code>, <code>log2()</code>, custom base with <code>log(x, base)</code> </li> <li>Stats: <code>mean()</code>, <code>median()</code>, <code>mode()</code>, <code>sum()</code>, <code>product()</code> </li> <li>Utility: <code>abs()</code>, <code>ceil()</code>, <code>floor()</code>, <code>round()</code>, <code>sqrt()</code>, <code>exp()</code> </li> <li>Special: factorial with <code>!</code>, <code>mod()</code>, <code>print()</code> </li> </ul> <h3> 5. Calculation History </h3> <p>Everything you calculate gets stored:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>» <span class="nb">history</span> ┌─ Calculation History ────────────────────────────┐ │ 2 + 3 <span class="k">*</span> 4 <span class="o">=</span> 14 │ sin<span class="o">(</span>30<span class="o">)</span> + cos<span class="o">(</span>60<span class="o">)</span> <span class="o">=</span> 1 │ x <span class="o">=</span> sqrt<span class="o">(</span>16<span class="o">)</span> <span class="o">=</span> 4 │ convert 100 cm to m <span class="o">=</span> 1 m └──────────────────────────────────────────────────┘ </code></pre> </div> <p>Stored in JSON, persists across sessions.</p> <h2> How It Works Under the Hood </h2> <p>Axion uses a proper compiler-style pipeline:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>User Input ↓ Tokenizer → Breaks input into tokens (numbers, operators, functions) ↓ Parser → Builds Abstract Syntax Tree (AST) with proper precedence ↓ Evaluator → Walks the AST and computes the result ↓ Output → Color-coded terminal display </code></pre> </div> <h3> The Parser </h3> <p>This was the hardest part. Getting operator precedence right took multiple attempts.</p> <p>The parser uses recursive descent with precedence climbing. Each precedence level gets its own parsing function:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="c">// Simplified version</span> <span class="k">func</span> <span class="p">(</span><span class="n">p</span> <span class="o">*</span><span class="n">Parser</span><span class="p">)</span> <span class="n">parseExpression</span><span class="p">()</span> <span class="o">*</span><span class="n">Node</span> <span class="p">{</span> <span class="k">return</span> <span class="n">p</span><span class="o">.</span><span class="n">parseLogicalOr</span><span class="p">()</span> <span class="c">// Lowest precedence</span> <span class="p">}</span> <span class="k">func</span> <span class="p">(</span><span class="n">p</span> <span class="o">*</span><span class="n">Parser</span><span class="p">)</span> <span class="n">parseLogicalOr</span><span class="p">()</span> <span class="o">*</span><span class="n">Node</span> <span class="p">{</span> <span class="n">left</span> <span class="o">:=</span> <span class="n">p</span><span class="o">.</span><span class="n">parseLogicalAnd</span><span class="p">()</span> <span class="c">// Handle || operators</span> <span class="k">return</span> <span class="n">left</span> <span class="p">}</span> <span class="k">func</span> <span class="p">(</span><span class="n">p</span> <span class="o">*</span><span class="n">Parser</span><span class="p">)</span> <span class="n">parseLogicalAnd</span><span class="p">()</span> <span class="o">*</span><span class="n">Node</span> <span class="p">{</span> <span class="n">left</span> <span class="o">:=</span> <span class="n">p</span><span class="o">.</span><span class="n">parseComparison</span><span class="p">()</span> <span class="c">// Handle &amp;&amp; operators</span> <span class="k">return</span> <span class="n">left</span> <span class="p">}</span> <span class="k">func</span> <span class="p">(</span><span class="n">p</span> <span class="o">*</span><span class="n">Parser</span><span class="p">)</span> <span class="n">parseComparison</span><span class="p">()</span> <span class="o">*</span><span class="n">Node</span> <span class="p">{</span> <span class="n">left</span> <span class="o">:=</span> <span class="n">p</span><span class="o">.</span><span class="n">parseAddition</span><span class="p">()</span> <span class="c">// Handle &gt;, &lt;, ==, etc.</span> <span class="k">return</span> <span class="n">left</span> <span class="p">}</span> <span class="c">// And so on...</span> </code></pre> </div> <p>This structure ensures <code>2 + 3 &gt; 4 || 0</code> parses correctly:</p> <ol> <li>Parse <code>2 + 3</code> (addition has higher precedence)</li> <li>Parse <code>&gt; 4</code> (comparison)</li> <li>Parse <code>|| 0</code> (logical OR has lowest precedence)</li> </ol> <h3> Token Types </h3> <p>The tokenizer recognizes:</p> <ul> <li> <code>NUMBER</code> - integers, decimals, scientific notation</li> <li> <code>OPERATOR</code> - <code>+</code>, <code>-</code>, <code>*</code>, <code>/</code>, <code>^</code>, <code>!</code> </li> <li> <code>COMPARISON</code> - <code>&gt;</code>, <code>&lt;</code>, <code>&gt;=</code>, <code>&lt;=</code>, <code>==</code>, <code>!=</code> </li> <li> <code>LOGICAL</code> - <code>&amp;&amp;</code>, <code>||</code> </li> <li> <code>FUNCTION</code> - <code>sin</code>, <code>cos</code>, <code>log</code>, etc.</li> <li> <code>LPAREN</code>/<code>RPAREN</code> - parentheses for grouping</li> <li> <code>COMMA</code> - function argument separator</li> </ul> <h3> The Evaluator </h3> <p>Walks the AST recursively. Each node type has its own evaluation logic:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="k">switch</span> <span class="n">node</span><span class="o">.</span><span class="n">Type</span> <span class="p">{</span> <span class="k">case</span> <span class="n">NODE_NUMBER</span><span class="o">:</span> <span class="k">return</span> <span class="n">parseFloat</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">Value</span><span class="p">)</span> <span class="k">case</span> <span class="n">NODE_OPERATOR</span><span class="o">:</span> <span class="n">left</span> <span class="o">:=</span> <span class="n">Eval</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">Left</span><span class="p">)</span> <span class="n">right</span> <span class="o">:=</span> <span class="n">Eval</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">Right</span><span class="p">)</span> <span class="k">return</span> <span class="n">applyOperator</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">Value</span><span class="p">,</span> <span class="n">left</span><span class="p">,</span> <span class="n">right</span><span class="p">)</span> <span class="k">case</span> <span class="n">NODE_COMPARISON</span><span class="o">:</span> <span class="n">left</span> <span class="o">:=</span> <span class="n">Eval</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">Left</span><span class="p">)</span> <span class="n">right</span> <span class="o">:=</span> <span class="n">Eval</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">Right</span><span class="p">)</span> <span class="k">if</span> <span class="n">compareValues</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">Value</span><span class="p">,</span> <span class="n">left</span><span class="p">,</span> <span class="n">right</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="m">1.0</span> <span class="c">// true</span> <span class="p">}</span> <span class="k">return</span> <span class="m">0.0</span> <span class="c">// false</span> <span class="k">case</span> <span class="n">NODE_AND</span><span class="o">:</span> <span class="n">left</span> <span class="o">:=</span> <span class="n">Eval</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">Left</span><span class="p">)</span> <span class="n">right</span> <span class="o">:=</span> <span class="n">Eval</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">Right</span><span class="p">)</span> <span class="k">if</span> <span class="n">left</span> <span class="o">!=</span> <span class="m">0</span> <span class="o">&amp;&amp;</span> <span class="n">right</span> <span class="o">!=</span> <span class="m">0</span> <span class="p">{</span> <span class="k">return</span> <span class="m">1.0</span> <span class="p">}</span> <span class="k">return</span> <span class="m">0.0</span> <span class="p">}</span> </code></pre> </div> <p>Comparisons and logical operations return <code>1.0</code> (true) or <code>0.0</code> (false), which lets you use them in further calculations.</p> <h2> Real-World Use Cases </h2> <h3> Physics Calculations </h3> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>» F <span class="o">=</span> 9.8 <span class="k">*</span> 75 <span class="c"># Force = mass * acceleration</span> Result: 735 » isValidForce <span class="o">=</span> F <span class="o">&gt;</span> 0 <span class="o">&amp;&amp;</span> F &lt; 1000 Result: 1 » E <span class="o">=</span> F <span class="k">*</span> 10 <span class="c"># Energy = force * distance</span> Result: 7350 </code></pre> </div> <h3> Financial Math </h3> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>» principal <span class="o">=</span> 1000 » rate <span class="o">=</span> 0.05 » compoundInterest <span class="o">=</span> principal <span class="k">*</span> <span class="o">(</span>1 + rate<span class="o">)</span>^10 Result: 1628.89 » isProfit <span class="o">=</span> compoundInterest <span class="o">&gt;</span> principal Result: 1 </code></pre> </div> <h3> Engineering </h3> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>» voltage <span class="o">=</span> 12 » current <span class="o">=</span> 2.5 » power <span class="o">=</span> voltage <span class="k">*</span> current Result: 30 » resistance <span class="o">=</span> voltage / current Result: 4.8 » isSafeVoltage <span class="o">=</span> voltage &lt; 50 Result: 1 </code></pre> </div> <h2> Project Structure </h2> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>Axion/ ├── main.go # Entry point ├── cmd/ # Cobra CLI commands │ └── cmd.go # REPL implementation ├── tokenizer/ # Lexical analysis │ └── tokenizer.go ├── parser/ # Syntax analysis │ └── parser.go ├── evaluator/ # Expression evaluation │ └── evaluator.go ├── units/ # Unit conversion │ └── units.go ├── history/ # History management │ └── history.go └── constants/ # Constants loading └── constants.go </code></pre> </div> <p>Each module has a single responsibility. The Cobra framework handles CLI commands and the interactive REPL.</p> <h2> Test Coverage </h2> <p>Core computational modules have solid test coverage:</p> <ul> <li> <strong>Units:</strong> 100% (unit conversion system)</li> <li> <strong>Tokenizer:</strong> 94% (lexical analysis)</li> <li> <strong>Parser:</strong> 76.4% (AST construction)</li> <li> <strong>Evaluator:</strong> 74.5% (mathematical computation)</li> </ul> <p>The utility modules (constants, history, settings) and CLI handlers don't have tests yet. They're next on the list.</p> <h2> Installation </h2> <p><strong>Quick install (Linux/macOS):</strong><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>git clone https://github.com/codetesla51/Axion.git <span class="nb">cd </span>Axion <span class="nb">chmod</span> +x install.sh ./install.sh <span class="nb">source</span> ~/.bashrc </code></pre> </div> <p>The script builds the binary, creates a symlink in <code>~/.local/bin</code>, and adds it to your PATH.</p> <p><strong>Manual install:</strong><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>git clone https://github.com/codetesla51/Axion.git <span class="nb">cd </span>Axion go build <span class="nt">-o</span> axion ./axion </code></pre> </div> <h2> What I Learned </h2> <p><strong>Operator precedence is harder than it looks.</strong> Getting <code>&amp;&amp;</code> to bind tighter than <code>||</code> while both bind looser than comparisons required multiple parsing passes.</p> <p><strong>Recursive descent parsing is elegant.</strong> Once you understand the pattern (each precedence level calls the next higher level), it's straightforward to extend.</p> <p><strong>The Cobra framework is powerful.</strong> Building a CLI with subcommands, flags, and help text is trivial with Cobra. Would have taken way longer without it.</p> <p><strong>Testing matters.</strong> The bugs I caught with unit tests would have been nightmares to debug in production. Writing tests as you go saves time.</p> <p><strong>JSON is good enough for most storage needs.</strong> No need for a database when you're just storing variables and history. JSON files work fine and are human-readable.</p> <h2> What's Next </h2> <p>Potential additions:</p> <ul> <li>Matrix operations</li> <li>Complex numbers</li> <li>Custom function definitions</li> <li>Graphing capabilities</li> <li>More unit categories (temperature, pressure, etc.)</li> <li>Export calculations to different formats</li> </ul> <p>But the core is solid. It does what it needs to do.</p> <h2> Try It </h2> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>git clone https://github.com/codetesla51/Axion.git <span class="nb">cd </span>Axion ./install.sh axion </code></pre> </div> <p>Then try some calculations:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>» x <span class="o">=</span> 10 » y <span class="o">=</span> 20 » <span class="o">(</span>x <span class="o">&gt;</span> 5<span class="o">)</span> <span class="o">&amp;&amp;</span> <span class="o">(</span>y &lt; 25<span class="o">)</span> Result: 1 » convert 5 km to mi 5 km <span class="o">=</span> 3.10686 mi » print<span class="o">(</span>sin<span class="o">(</span>45<span class="o">))</span> 0.707107 </code></pre> </div> <p><strong>Built with Go and Cobra.</strong> Full source on <a href="proxy.php?url=https%3A%2F%2Fgithub.com%2Fcodetesla51%2FAxion" rel="noopener noreferrer">GitHub</a>.</p> <p>Check out more at <a href="proxy.php?url=https%3A%2F%2Fdevuthman.vercel.app" rel="noopener noreferrer">devuthman.vercel.app</a></p> go programming Building Git from Scratch in Go: What I Learned About Version Control Internals Uthman Oladele Sun, 09 Nov 2025 12:36:05 +0000 https://dev.to/uthman_dev/building-git-from-scratch-in-go-what-i-learned-about-version-control-internals-4dih https://dev.to/uthman_dev/building-git-from-scratch-in-go-what-i-learned-about-version-control-internals-4dih <p>I didn't understand Git until I broke it open and rebuilt it from scratch. No libraries, no shortcuts - just SHA-256 hashing, tree structures, and commit graphs.</p> <p>I built a Git implementation in Go without using any Git libraries. No magic, just content-addressable storage and the object model. It works, it taught me more about Git in a week than years of using it, and here's what I learned.</p> <h2> How I Learned This </h2> <p>This wasn't from a single source. I pieced it together from:</p> <ul> <li> <strong>CodeCrafters</strong> - Their "Build Your Own Git" challenge gave me the structure and pushed me to actually implement things</li> <li> <strong>Random YouTube videos</strong> - Honestly, just searching "how git works internally" and watching whatever came up. Some were helpful, most weren't</li> <li> <strong>"Building Git" book</strong> - Wasn't really helpful for what I needed, but it did clarify some object format details</li> </ul> <p>Most of the learning came from trial and error. Breaking things, reading error messages, and debugging for hours.</p> <h2> Why Build This? </h2> <p>I use Git every day but had no idea how it actually works. Just <code>git add</code>, <code>git commit</code>, and hope nothing breaks. I wanted to understand what's really happening under the hood.</p> <p>The goal wasn't to make something production-ready. I just wanted answers:</p> <ul> <li>Why are commits so cheap?</li> <li>How does Git deduplicate files automatically?</li> <li>What the hell is a "tree object"?</li> <li>Why is branching fast?</li> </ul> <p>Turns out the best way to understand something is to build it yourself.</p> <h2> What It Does </h2> <p>Go-Git implements the core stuff:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>go-git init <span class="c"># Initialize repository</span> go-git config <span class="c"># Set user identity</span> go-git add &lt;files...&gt; <span class="c"># Stage files</span> go-git commit <span class="nt">-m</span> <span class="s2">"message"</span> <span class="c"># Create commit</span> go-git log <span class="c"># View history</span> </code></pre> </div> <p>It handles content-addressable storage, the staging area, tree objects, commit history, and zlib compression. Basically everything Git does to manage your code, minus branches, merges, and remotes.</p> <h2> The Three Main Ideas </h2> <h3> 1. Content-Addressable Storage </h3> <p>Every object (file, directory, commit) gets stored by its SHA-256 hash:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>.git/objects/ab/c123def456... ↑↑ ↑↑↑↑↑↑↑ │ └─ Rest of hash (filename) └───── First 2 chars (subdirectory) </code></pre> </div> <p>This is actually genius. Same content = same hash = automatic deduplication. You could store the same <code>README.md</code> across 100 commits and it only takes up disk space once.</p> <p>Your file's hash IS its address. No need for a separate indexing system.</p> <h3> 2. The Three Trees </h3> <p>Git tracks files through three layers:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>Working Directory → Staging Area → Repository (your files) (.git/index) (.git/objects) </code></pre> </div> <ul> <li> <code>go-git add</code> moves files from working directory → staging area</li> <li> <code>go-git commit</code> snapshots staging area → repository</li> </ul> <p>The staging area is literally just a text file mapping paths to hashes:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>100644 abc123... README.md 100644 def456... src/main.go </code></pre> </div> <p>When you commit, Git hashes this whole thing into a tree object.</p> <h3> 3. Tree Objects (The Hard Part) </h3> <p>Files get stored as <strong>blobs</strong>. Directories get stored as <strong>trees</strong>.</p> <p>Simple project structure:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>project/ README.md src/ main.go lib/ helper.go </code></pre> </div> <p>Git stores it like this:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>Commit (abc123) ↓ Root Tree (def456) ├─ blob: README.md (hash: abc123) └─ tree: src/ (hash: def456) ├─ blob: main.go (hash: ghi789) └─ tree: lib/ (hash: jkl012) └─ blob: helper.go (hash: mno345) </code></pre> </div> <p>Here's what finally clicked for me: <strong>Trees don't contain their children - they just reference them by hash.</strong> That's why Git is fast. If a directory doesn't change, same hash, just reuse the tree. No need to re-store anything.</p> <p><strong>The tricky part:</strong> You have to build trees bottom-up (deepest first) because parent trees need their children's hashes.</p> <p>You can't hash <code>src/</code> until you know the hash of <code>src/lib/</code>. Can't hash the root tree until you know the hash of <code>src/</code>. The order matters, period.</p> <p>This took me over 5 hours to get right.</p> <h2> What Was Hard </h2> <h3> Tree Building Order </h3> <p>First try: Build trees top-down, starting from root. Immediately failed - you don't have the child tree hashes yet.</p> <p>Fix: Sort directories by depth, build the deepest ones first:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="n">sort</span><span class="o">.</span><span class="n">Slice</span><span class="p">(</span><span class="n">dirs</span><span class="p">,</span> <span class="k">func</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span> <span class="kt">int</span><span class="p">)</span> <span class="kt">bool</span> <span class="p">{</span> <span class="k">return</span> <span class="n">strings</span><span class="o">.</span><span class="n">Count</span><span class="p">(</span><span class="n">dirs</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="s">"/"</span><span class="p">)</span> <span class="o">&gt;</span> <span class="n">strings</span><span class="o">.</span><span class="n">Count</span><span class="p">(</span><span class="n">dirs</span><span class="p">[</span><span class="n">j</span><span class="p">],</span> <span class="s">"/"</span><span class="p">)</span> <span class="p">})</span> </code></pre> </div> <p>Then for each directory, check if any trees you've already built are its children and add them.</p> <h3> Binary vs Hex Encoding </h3> <p>Tree objects store hashes as 32 binary bytes, not 64-character hex strings.</p> <p>My bug:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="n">content</span> <span class="o">+=</span> <span class="n">entry</span><span class="o">.</span><span class="n">BlobHash</span> <span class="c">// Wrong! This is a 64-char hex string</span> </code></pre> </div> <p>The fix:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="n">hashBytes</span><span class="p">,</span> <span class="n">_</span> <span class="o">:=</span> <span class="n">hex</span><span class="o">.</span><span class="n">DecodeString</span><span class="p">(</span><span class="n">entry</span><span class="o">.</span><span class="n">BlobHash</span><span class="p">)</span> <span class="n">content</span> <span class="o">=</span> <span class="nb">append</span><span class="p">(</span><span class="n">content</span><span class="p">,</span> <span class="n">hashBytes</span><span class="o">...</span><span class="p">)</span> <span class="c">// 32 binary bytes</span> </code></pre> </div> <p>This made my trees twice as big as they should've been. Spent an hour debugging because it was subtle - objects were still readable but tree traversal was completely broken.</p> <h3> Excluding .git/ When Staging </h3> <p>When you do <code>go-git add .</code>, you don't want to accidentally add <code>.git/objects/</code> to the index.</p> <p>First attempt: Check if the path contains <code>.git</code>. Problem: this also excluded files like <code>my.git.file</code>.</p> <p>The right way:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="k">if</span> <span class="n">d</span><span class="o">.</span><span class="n">Name</span><span class="p">()</span> <span class="o">==</span> <span class="s">".git"</span> <span class="o">&amp;&amp;</span> <span class="n">d</span><span class="o">.</span><span class="n">IsDir</span><span class="p">()</span> <span class="p">{</span> <span class="k">return</span> <span class="n">filepath</span><span class="o">.</span><span class="n">SkipDir</span> <span class="p">}</span> </code></pre> </div> <p><code>filepath.SkipDir</code> during directory traversal is the way to go.</p> <h2> How Objects Work </h2> <p>Every object follows this format:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>&lt;type&gt; &lt;size&gt;\0&lt;content&gt; </code></pre> </div> <p>Gets compressed with zlib, stored at <code>.git/objects/&lt;hash[:2]&gt;/&lt;hash[2:]&gt;</code>.</p> <p><strong>Blob:</strong><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>blob 13\0Hello, World! </code></pre> </div> <p>Hash it → <code>a0b1c2d3...</code> → Store at <code>.git/objects/a0/b1c2d3...</code></p> <p><strong>Tree:</strong><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>tree 74\0100644 README.md\0&lt;32-byte-hash&gt;040000 src\0&lt;32-byte-hash&gt; </code></pre> </div> <p>Modes:</p> <ul> <li> <code>100644</code> = regular file</li> <li> <code>040000</code> = directory (tree object)</li> </ul> <p><strong>Commit:</strong><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>commit &lt;size&gt;\0 tree abc123... parent 789xyz... author Uthman &lt;email&gt; timestamp committer Uthman &lt;email&gt; timestamp Initial commit </code></pre> </div> <p>Commits form a directed acyclic graph. <code>go-git log</code> just walks backwards through this chain from HEAD.</p> <h2> What I Actually Learned </h2> <p><strong>Content-addressable storage is elegant.</strong> Hash = address. This one idea enables deduplication, integrity checking, and efficient storage.</p> <p><strong>Trees are graphs, not nested structures.</strong> A tree object doesn't "contain" subtrees - it references them by hash. This indirection is what makes Git efficient. Multiple commits can share the same tree if a directory didn't change.</p> <p><strong>Building bottom-up is necessary.</strong> You can't hash a parent without knowing its children's hashes. The order matters fundamentally.</p> <p><strong>Compression matters.</strong> Without zlib, <code>.git/objects/</code> would be 3-4x larger. Git uses compression everywhere for a reason.</p> <p><strong>Binary formats are tricky.</strong> Working with <code>\0</code> null bytes and binary hash data requires careful handling. Text formats would be easier but less efficient.</p> <h2> What's Missing </h2> <p>This is a learning project, not production software:</p> <ul> <li>No branches (only <code>main</code> exists)</li> <li>No merge operations</li> <li>No diff/status commands</li> <li>No remote operations (push/pull/fetch)</li> <li>No <code>.gitignore</code> support</li> <li>No packed objects (each object is a separate file)</li> <li>Plain text index (real Git uses binary format)</li> </ul> <p>These limitations exist because this project focuses on Git's core: the object model, staging, and commits. Adding branches/merging/remotes would be another 2-3x the code and shift focus from fundamentals to features.</p> <h2> Try It Yourself </h2> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>git clone https://github.com/codetesla51/go-git.git <span class="nb">cd </span>go-git ./install.sh <span class="c"># Or build manually:</span> go build <span class="nt">-buildvcs</span><span class="o">=</span><span class="nb">false</span> <span class="nt">-o</span> go-git <span class="nb">ln</span> <span class="nt">-s</span> <span class="si">$(</span><span class="nb">pwd</span><span class="si">)</span>/go-git ~/.local/bin/go-git </code></pre> </div> <p>Then:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code><span class="nb">mkdir </span>my-project <span class="nb">cd </span>my-project go-git init go-git config <span class="nb">echo</span> <span class="s2">"Hello World"</span> <span class="o">&gt;</span> README.md go-git add README.md go-git commit <span class="nt">-m</span> <span class="s2">"Initial commit"</span> go-git log </code></pre> </div> <p><strong>Want to really learn?</strong> Clone it, break something on purpose (change the hash function to MD5, remove zlib compression), and see what fails. Watch how Git's assumptions about immutability and content-addressing cascade through the system. That's how you really understand it.</p> <h2> Final Thoughts </h2> <p>Building this taught me more about Git in a week than years of using it did. I now understand why commits are cheap (just pointers to trees), how deduplication works (content-addressable storage), and why branching is fast (just moving a pointer).</p> <p>If you want to truly understand a tool, build it yourself.</p> <p><strong>Built with Go, no Git libraries used.</strong> All hashing, compression, and object storage is custom implementation.</p> <p>Check out more of my work at <a href="proxy.php?url=https%3A%2F%2Fdevuthman.vercel.app" rel="noopener noreferrer">devuthman.vercel.app</a> | <a href="proxy.php?url=https%3A%2F%2Fgithub.com%2Fcodetesla51%2Fgo-git" rel="noopener noreferrer">GitHub</a></p> cli git go programming Building an HTTP Server from TCP Sockets: 250 4,000 RPS Uthman Oladele Fri, 07 Nov 2025 12:59:48 +0000 https://dev.to/uthman_dev/building-an-http-server-from-tcp-sockets-250-4000-rps-2m93 https://dev.to/uthman_dev/building-an-http-server-from-tcp-sockets-250-4000-rps-2m93 <p>Most developers use frameworks like Express or Flask without understanding what happens underneath. I built an HTTP/HTTPS server from raw TCP sockets in Go to learn how web servers actually work - no frameworks, just socket programming and protocol implementation.</p> <p>The result? A journey from 250 RPS with buggy connection handling to 4,000 RPS at peak performance. Here's how I did it and what I learned.</p> <h2> Why Build This? </h2> <p>I wanted to understand HTTP at the protocol level - not just use it through abstractions. What does "parsing a request" actually mean? How does keep-alive work? What makes one server faster than another?</p> <p>Building from TCP sockets up forced me to learn:</p> <ul> <li>How HTTP requests are structured byte-by-byte</li> <li>Connection lifecycle and reuse patterns</li> <li>TLS/SSL encryption from first principles</li> <li>Why implementation details matter for performance</li> </ul> <h2> What I Built </h2> <p>A lightweight HTTP/HTTPS server that:</p> <ul> <li>Parses HTTP requests directly from TCP socket connections</li> <li>Routes requests with custom method and path matching</li> <li>Serves static files with proper MIME type detection</li> <li>Supports HTTP/1.1 keep-alive for connection reuse</li> <li>Handles form data and JSON request bodies</li> <li>Includes optional TLS encryption for HTTPS</li> <li>Achieves 4,000 requests per second at peak</li> </ul> <p><strong>Tech stack:</strong> Pure Go, no web frameworks. Just <code>net</code> package for TCP, <code>crypto/tls</code> for HTTPS, and <code>html/template</code> for rendering.</p> <h2> The Performance Journey </h2> <h3> Initial Version: 250 RPS (Buggy) </h3> <p>My first implementation had a critical bug in request processing combined with <code>Connection: close</code> on every response. The server worked but was crawling.</p> <h3> Bug Fix: 1,389 RPS </h3> <p>Fixed the request handling logic but still sent <code>Connection: close</code> on every response. This meant every request required a new TCP handshake - expensive.</p> <h3> Keep-Alive Implementation: 1,710 RPS </h3> <p>Added proper HTTP/1.1 keep-alive support. Connections stayed open for multiple requests. Immediate improvement but not optimal yet.</p> <h3> Peak Performance: 4,000 RPS </h3> <p>Discovered that concurrency level matters significantly. At 10 concurrent connections with connection reuse, the server hit 4,000 RPS with 0.252ms response times.</p> <p><strong>Total improvement: 16x from initial version</strong></p> <h2> How It Actually Works </h2> <h3> 1. TCP Connection Handling </h3> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="n">listener</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">net</span><span class="o">.</span><span class="n">Listen</span><span class="p">(</span><span class="s">"tcp"</span><span class="p">,</span> <span class="s">":8080"</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="p">}</span> <span class="k">for</span> <span class="p">{</span> <span class="n">conn</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">listener</span><span class="o">.</span><span class="n">Accept</span><span class="p">()</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="k">continue</span> <span class="p">}</span> <span class="k">go</span> <span class="n">handleConnection</span><span class="p">(</span><span class="n">conn</span><span class="p">)</span> <span class="c">// One goroutine per connection</span> <span class="p">}</span> </code></pre> </div> <p>Each connection gets its own goroutine. Go's scheduler handles thousands of these efficiently.</p> <h3> 2. HTTP Request Parsing </h3> <p>Reading from the socket gives you raw bytes. You need to parse:</p> <ul> <li>Request line: <code>GET /path HTTP/1.1</code> </li> <li>Headers: <code>Content-Type: application/json</code> </li> <li>Body: Form data or JSON payload </li> </ul> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="k">func</span> <span class="n">parseRequest</span><span class="p">(</span><span class="n">conn</span> <span class="n">net</span><span class="o">.</span><span class="n">Conn</span><span class="p">)</span> <span class="p">(</span><span class="o">*</span><span class="n">Request</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span> <span class="n">reader</span> <span class="o">:=</span> <span class="n">bufio</span><span class="o">.</span><span class="n">NewReader</span><span class="p">(</span><span class="n">conn</span><span class="p">)</span> <span class="c">// Read request line</span> <span class="n">requestLine</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">reader</span><span class="o">.</span><span class="n">ReadString</span><span class="p">(</span><span class="sc">'\n'</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="k">return</span> <span class="no">nil</span><span class="p">,</span> <span class="n">err</span> <span class="p">}</span> <span class="c">// Parse method, path, version</span> <span class="n">parts</span> <span class="o">:=</span> <span class="n">strings</span><span class="o">.</span><span class="n">Split</span><span class="p">(</span><span class="n">strings</span><span class="o">.</span><span class="n">TrimSpace</span><span class="p">(</span><span class="n">requestLine</span><span class="p">),</span> <span class="s">" "</span><span class="p">)</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">parts</span><span class="p">)</span> <span class="o">!=</span> <span class="m">3</span> <span class="p">{</span> <span class="k">return</span> <span class="no">nil</span><span class="p">,</span> <span class="n">errors</span><span class="o">.</span><span class="n">New</span><span class="p">(</span><span class="s">"invalid request line"</span><span class="p">)</span> <span class="p">}</span> <span class="n">method</span> <span class="o">:=</span> <span class="n">parts</span><span class="p">[</span><span class="m">0</span><span class="p">]</span> <span class="n">path</span> <span class="o">:=</span> <span class="n">parts</span><span class="p">[</span><span class="m">1</span><span class="p">]</span> <span class="c">// Read headers until empty line</span> <span class="n">headers</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="k">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kt">string</span><span class="p">)</span> <span class="k">for</span> <span class="p">{</span> <span class="n">line</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">reader</span><span class="o">.</span><span class="n">ReadString</span><span class="p">(</span><span class="sc">'\n'</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="o">||</span> <span class="n">line</span> <span class="o">==</span> <span class="s">"</span><span class="se">\r\n</span><span class="s">"</span> <span class="p">{</span> <span class="k">break</span> <span class="p">}</span> <span class="c">// Parse header: "Content-Type: application/json"</span> <span class="c">// ... header parsing logic</span> <span class="p">}</span> <span class="k">return</span> <span class="o">&amp;</span><span class="n">Request</span><span class="p">{</span><span class="n">Method</span><span class="o">:</span> <span class="n">method</span><span class="p">,</span> <span class="n">Path</span><span class="o">:</span> <span class="n">path</span><span class="p">,</span> <span class="n">Headers</span><span class="o">:</span> <span class="n">headers</span><span class="p">},</span> <span class="no">nil</span> <span class="p">}</span> </code></pre> </div> <h3> 3. Keep-Alive Implementation </h3> <p>The breakthrough came from proper connection reuse:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="k">func</span> <span class="n">handleConnection</span><span class="p">(</span><span class="n">conn</span> <span class="n">net</span><span class="o">.</span><span class="n">Conn</span><span class="p">)</span> <span class="p">{</span> <span class="k">defer</span> <span class="n">conn</span><span class="o">.</span><span class="n">Close</span><span class="p">()</span> <span class="k">for</span> <span class="p">{</span> <span class="n">req</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">parseRequest</span><span class="p">(</span><span class="n">conn</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="k">break</span> <span class="c">// Connection closed or invalid request</span> <span class="p">}</span> <span class="n">response</span> <span class="o">:=</span> <span class="n">router</span><span class="o">.</span><span class="n">Handle</span><span class="p">(</span><span class="n">req</span><span class="p">)</span> <span class="c">// Send response with keep-alive</span> <span class="n">conn</span><span class="o">.</span><span class="n">Write</span><span class="p">([]</span><span class="kt">byte</span><span class="p">(</span><span class="s">"HTTP/1.1 200 OK</span><span class="se">\r\n</span><span class="s">"</span><span class="p">))</span> <span class="n">conn</span><span class="o">.</span><span class="n">Write</span><span class="p">([]</span><span class="kt">byte</span><span class="p">(</span><span class="s">"Connection: keep-alive</span><span class="se">\r\n</span><span class="s">"</span><span class="p">))</span> <span class="n">conn</span><span class="o">.</span><span class="n">Write</span><span class="p">([]</span><span class="kt">byte</span><span class="p">(</span><span class="s">"Content-Length: "</span> <span class="o">+</span> <span class="n">strconv</span><span class="o">.</span><span class="n">Itoa</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">response</span><span class="p">))</span> <span class="o">+</span> <span class="s">"</span><span class="se">\r\n</span><span class="s">"</span><span class="p">))</span> <span class="n">conn</span><span class="o">.</span><span class="n">Write</span><span class="p">([]</span><span class="kt">byte</span><span class="p">(</span><span class="s">"</span><span class="se">\r\n</span><span class="s">"</span><span class="p">))</span> <span class="n">conn</span><span class="o">.</span><span class="n">Write</span><span class="p">([]</span><span class="kt">byte</span><span class="p">(</span><span class="n">response</span><span class="p">))</span> <span class="c">// Connection stays open for next request</span> <span class="k">if</span> <span class="n">req</span><span class="o">.</span><span class="n">Headers</span><span class="p">[</span><span class="s">"Connection"</span><span class="p">]</span> <span class="o">==</span> <span class="s">"close"</span> <span class="p">{</span> <span class="k">break</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> </code></pre> </div> <p>This simple change - keeping connections open - improved performance by 23% (1,389 → 1,710 RPS).</p> <h3> 4. Routing System </h3> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="k">type</span> <span class="n">Handler</span> <span class="k">func</span><span class="p">(</span><span class="o">*</span><span class="n">Request</span><span class="p">)</span> <span class="p">(</span><span class="kt">string</span><span class="p">,</span> <span class="kt">string</span><span class="p">)</span> <span class="k">type</span> <span class="n">Router</span> <span class="k">struct</span> <span class="p">{</span> <span class="n">routes</span> <span class="k">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="k">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="n">Handler</span> <span class="c">// method -&gt; path -&gt; handler</span> <span class="p">}</span> <span class="k">func</span> <span class="p">(</span><span class="n">r</span> <span class="o">*</span><span class="n">Router</span><span class="p">)</span> <span class="n">Register</span><span class="p">(</span><span class="n">method</span><span class="p">,</span> <span class="n">path</span> <span class="kt">string</span><span class="p">,</span> <span class="n">handler</span> <span class="n">Handler</span><span class="p">)</span> <span class="p">{</span> <span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">routes</span><span class="p">[</span><span class="n">method</span><span class="p">]</span> <span class="o">==</span> <span class="no">nil</span> <span class="p">{</span> <span class="n">r</span><span class="o">.</span><span class="n">routes</span><span class="p">[</span><span class="n">method</span><span class="p">]</span> <span class="o">=</span> <span class="nb">make</span><span class="p">(</span><span class="k">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="n">Handler</span><span class="p">)</span> <span class="p">}</span> <span class="n">r</span><span class="o">.</span><span class="n">routes</span><span class="p">[</span><span class="n">method</span><span class="p">][</span><span class="n">path</span><span class="p">]</span> <span class="o">=</span> <span class="n">handler</span> <span class="p">}</span> <span class="k">func</span> <span class="p">(</span><span class="n">r</span> <span class="o">*</span><span class="n">Router</span><span class="p">)</span> <span class="n">Handle</span><span class="p">(</span><span class="n">req</span> <span class="o">*</span><span class="n">Request</span><span class="p">)</span> <span class="kt">string</span> <span class="p">{</span> <span class="k">if</span> <span class="n">handler</span><span class="p">,</span> <span class="n">exists</span> <span class="o">:=</span> <span class="n">r</span><span class="o">.</span><span class="n">routes</span><span class="p">[</span><span class="n">req</span><span class="o">.</span><span class="n">Method</span><span class="p">][</span><span class="n">req</span><span class="o">.</span><span class="n">Path</span><span class="p">];</span> <span class="n">exists</span> <span class="p">{</span> <span class="n">statusCode</span><span class="p">,</span> <span class="n">body</span> <span class="o">:=</span> <span class="n">handler</span><span class="p">(</span><span class="n">req</span><span class="p">)</span> <span class="k">return</span> <span class="n">createResponse</span><span class="p">(</span><span class="n">statusCode</span><span class="p">,</span> <span class="n">body</span><span class="p">)</span> <span class="p">}</span> <span class="k">return</span> <span class="n">createResponse</span><span class="p">(</span><span class="s">"404"</span><span class="p">,</span> <span class="s">"Not Found"</span><span class="p">)</span> <span class="p">}</span> </code></pre> </div> <h3> 5. HTTPS/TLS Support </h3> <p>Adding TLS was surprisingly straightforward with Go's <code>crypto/tls</code>:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="n">cert</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">tls</span><span class="o">.</span><span class="n">LoadX509KeyPair</span><span class="p">(</span><span class="s">"server.crt"</span><span class="p">,</span> <span class="s">"server.key"</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="p">}</span> <span class="n">config</span> <span class="o">:=</span> <span class="o">&amp;</span><span class="n">tls</span><span class="o">.</span><span class="n">Config</span><span class="p">{</span><span class="n">Certificates</span><span class="o">:</span> <span class="p">[]</span><span class="n">tls</span><span class="o">.</span><span class="n">Certificate</span><span class="p">{</span><span class="n">cert</span><span class="p">}}</span> <span class="n">listener</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">tls</span><span class="o">.</span><span class="n">Listen</span><span class="p">(</span><span class="s">"tcp"</span><span class="p">,</span> <span class="s">":8443"</span><span class="p">,</span> <span class="n">config</span><span class="p">)</span> <span class="c">// Same connection handling as HTTP</span> <span class="k">for</span> <span class="p">{</span> <span class="n">conn</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">listener</span><span class="o">.</span><span class="n">Accept</span><span class="p">()</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="k">continue</span> <span class="p">}</span> <span class="k">go</span> <span class="n">handleConnection</span><span class="p">(</span><span class="n">conn</span><span class="p">)</span> <span class="p">}</span> </code></pre> </div> <p>The TLS layer handles encryption/decryption transparently. Your HTTP parsing code stays the same.</p> <h2> Performance Analysis </h2> <p>Testing with different concurrency levels revealed interesting patterns:</p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>Concurrency</th> <th>RPS</th> <th>Response Time</th> <th>Notes</th> </tr> </thead> <tbody> <tr> <td>10</td> <td>4,000</td> <td>0.252ms</td> <td><strong>Peak performance</strong></td> </tr> <tr> <td>50</td> <td>2,926</td> <td>0.342ms</td> <td>Excellent</td> </tr> <tr> <td>100</td> <td>2,067</td> <td>0.484ms</td> <td>Very good</td> </tr> <tr> <td>500</td> <td>2,286</td> <td>0.437ms</td> <td>Good</td> </tr> <tr> <td>1000</td> <td>1,463</td> <td>0.683ms</td> <td>Moderate load</td> </tr> </tbody> </table></div> <p><strong>Key insights:</strong></p> <ul> <li>Sweet spot: 10-200 concurrent connections</li> <li>Sub-millisecond response times at low concurrency</li> <li>0% failure rate across all tests</li> <li>Connection reuse was the single biggest optimization</li> </ul> <h2> Example Usage </h2> <p>Here's how you'd use this server:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight go"><code><span class="n">router</span> <span class="o">:=</span> <span class="n">server</span><span class="o">.</span><span class="n">NewRouter</span><span class="p">()</span> <span class="c">// Simple API endpoint</span> <span class="n">router</span><span class="o">.</span><span class="n">Register</span><span class="p">(</span><span class="s">"GET"</span><span class="p">,</span> <span class="s">"/ping"</span><span class="p">,</span> <span class="k">func</span><span class="p">(</span><span class="n">req</span> <span class="o">*</span><span class="n">server</span><span class="o">.</span><span class="n">Request</span><span class="p">)</span> <span class="p">(</span><span class="kt">string</span><span class="p">,</span> <span class="kt">string</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="n">server</span><span class="o">.</span><span class="n">CreateResponse</span><span class="p">(</span><span class="s">"200"</span><span class="p">,</span> <span class="s">"text/plain"</span><span class="p">,</span> <span class="s">"OK"</span><span class="p">,</span> <span class="s">"pong"</span><span class="p">)</span> <span class="p">})</span> <span class="c">// Form handling with browser detection</span> <span class="n">router</span><span class="o">.</span><span class="n">Register</span><span class="p">(</span><span class="s">"POST"</span><span class="p">,</span> <span class="s">"/login"</span><span class="p">,</span> <span class="k">func</span><span class="p">(</span><span class="n">req</span> <span class="o">*</span><span class="n">server</span><span class="o">.</span><span class="n">Request</span><span class="p">)</span> <span class="p">(</span><span class="kt">string</span><span class="p">,</span> <span class="kt">string</span><span class="p">)</span> <span class="p">{</span> <span class="n">username</span> <span class="o">:=</span> <span class="n">req</span><span class="o">.</span><span class="n">Body</span><span class="p">[</span><span class="s">"username"</span><span class="p">]</span> <span class="n">browser</span> <span class="o">:=</span> <span class="n">req</span><span class="o">.</span><span class="n">Browser</span> <span class="c">// Parsed from User-Agent</span> <span class="k">if</span> <span class="n">username</span> <span class="o">==</span> <span class="s">"admin"</span> <span class="p">{</span> <span class="k">return</span> <span class="n">server</span><span class="o">.</span><span class="n">CreateResponse</span><span class="p">(</span><span class="s">"200"</span><span class="p">,</span> <span class="s">"text/html"</span><span class="p">,</span> <span class="s">"OK"</span><span class="p">,</span> <span class="s">"&lt;h1&gt;Welcome "</span><span class="o">+</span><span class="n">username</span><span class="o">+</span><span class="s">"!&lt;/h1&gt;&lt;p&gt;Browser: "</span><span class="o">+</span><span class="n">browser</span><span class="o">+</span><span class="s">"&lt;/p&gt;"</span><span class="p">)</span> <span class="p">}</span> <span class="k">return</span> <span class="n">server</span><span class="o">.</span><span class="n">CreateResponse</span><span class="p">(</span><span class="s">"401"</span><span class="p">,</span> <span class="s">"text/html"</span><span class="p">,</span> <span class="s">"Unauthorized"</span><span class="p">,</span> <span class="s">"&lt;h1&gt;Login Failed&lt;/h1&gt;"</span><span class="p">)</span> <span class="p">})</span> </code></pre> </div> <p>Quick start:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>git clone https://github.com/codetesla51/raw-http <span class="nb">cd </span>raw-http go mod tidy go run main.go <span class="c"># Test it</span> curl http://localhost:8080/ping <span class="c"># Returns: pong</span> curl <span class="nt">-X</span> POST http://localhost:8080/login <span class="se">\</span> <span class="nt">-d</span> <span class="s2">"username=admin&amp;password=secret"</span> </code></pre> </div> <h2> What I Learned </h2> <h3> 1. HTTP Is Just Text Over TCP </h3> <p>Seeing the raw bytes demystified HTTP completely. It's not magic - just structured text following a protocol.</p> <h3> 2. Connection Reuse Matters </h3> <p>The jump from 1,389 to 1,710 RPS came purely from keeping connections open. TCP handshakes are expensive.</p> <h3> 3. Concurrency Level Impacts Performance </h3> <p>More concurrent connections doesn't always mean better performance. The server performed best at 10-200 concurrent requests, then degraded at higher levels.</p> <h3> 4. Go's Concurrency Model Shines </h3> <p>One goroutine per connection is simple and scales well. No need for complex event loops or worker pools.</p> <h3> 5. TLS/SSL Is Less Scary Than It Seems </h3> <p>With proper libraries, adding encryption is straightforward. The protocol complexity is handled for you.</p> <h3> 6. Security Requires Constant Attention </h3> <p>Path traversal protection, request size limits, and proper error handling are essential. Easy to miss when building from scratch.</p> <h2> Limitations </h2> <p>This is a learning project, not production software:</p> <ul> <li>Basic error handling</li> <li>Simple routing (no regex or path parameters)</li> <li>No middleware system</li> <li>Limited HTTP method support</li> <li>Self-signed certificates only</li> </ul> <p>But that's the point - understanding fundamentals before adding features.</p> <h2> Try It Yourself </h2> <p>The full source code is on GitHub: <a href="proxy.php?url=https%3A%2F%2Fgithub.com%2Fcodetesla51%2Fraw-http" rel="noopener noreferrer">codetesla51/raw-http</a></p> <p>Routes to try:</p> <ul> <li> <code>/</code> - Home page</li> <li> <code>/ping</code> - Simple API endpoint</li> <li> <code>/login</code> - Form handling demo</li> <li> <code>/welcome</code> - Template rendering</li> </ul> <p>For HTTPS (port 8443), the repo includes self-signed certificates for testing. For production, use Let's Encrypt or another CA.</p> <h2> Conclusion </h2> <p>Building an HTTP server from TCP sockets taught me more about web programming in a week than years of using frameworks. The 16x performance improvement wasn't just about optimization - it was about understanding what actually makes servers fast.</p> <p>If you're curious about how the tools you use every day actually work, I highly recommend building them yourself. Start small, measure everything, and don't be afraid to make mistakes. My first version was buggy and slow, but that's how you learn.</p> <p><strong>More projects:</strong> <a href="proxy.php?url=https%3A%2F%2Fdevuthman.vercel.app" rel="noopener noreferrer">devuthman.vercel.app</a><br><br> <strong>Source code:</strong> <a href="proxy.php?url=https%3A%2F%2Fgithub.com%2Fcodetesla51%2Fraw-http" rel="noopener noreferrer">github.com/codetesla51/raw-http</a></p> <p>Built with Go 1.21+ • Created by Uthman</p> programming go networking