How do you scale yourself?

Let’s start with a tangentially related preamble. I answered this Quora question in September 2012 with:

1. Optimize: automate. Replace (part of) yourself with very small shell scripts.

2. Scale vertically: become more efficient. Experience and training can help you perform the same tasks faster.

3. Scale horizontally for writes: shard. You can either delegate subtasks to others (divide and conquer / master-worker style) or just split your work between people at the same hierarchical level (my favorite solution, but remember collaboration is hard - graphs are inherently more complex than trees).

4. Scale horizontally for reads: add redundancy. Teach others how to do your job. This also increases availability and tolerance to failure by augmenting your bus factor.

Some of those things may or may not change with AI, but the point is: as long as I have been working, my worry has not been having nothing to do; it has always been dealing with too much.

Am I worried Claude will come for my job?

Several people have asked me that question recently. Some are programmers, some are not.

They ask this because they do not understand my job. At its core, it is about reducing workloads through optimization and automation.

Sure, I work on AI now, but that is not the important part. In that respect, AI is just the updated version of a very small shell script.

So let me answer that question. Yes, maybe someday robots will take my job. But if that happens, it means they will have taken yours too, because otherwise my job will be to make it happen.

Not everybody agrees, but personally, I am enthusiastic about the end state where nobody has to work. Sure, this is a turning point in the history of mankind and we will have to adapt to the state of abundance, but we will still be much better off than we are now.

So, do I worry about my job? Absolutely not. What worries me is the transition period when I have a job and you don’t.

P.S. — I am being provocative on purpose. There are things I hope we never automate. But ask yourself: if money was not a concern for you or anyone else, what would you do? This is what I want for everyone in the end. (Seriously, go read Manna now if you haven’t yet.)

Product substitution

The first thing a new versatile tech product does is almost completely replace more specialized products. Before smartphones, we had portable GPS devices for cars, MP3 players, PDAs, compact digital cameras, and of course non-smart phones. Some things are only partially replaced, for instance credit cards, loyalty cards, calculators, agendas, portable gaming consoles, and alarm clocks.

So what would go away if we had general-purpose robots? The first category I can think of is specialized robots such as cleaning robots, kitchen robots, and lawn mowers. But it would probably also directly replace less obvious things such as home alarms, and possibly partially pre-cooked food — see later regarding that.

Job substitution

Beyond products, new technology often replaces or diminishes the demand for certain jobs. AI is already doing that with many white-collar jobs, but giving it a bodily extension will go further.

It may not be the case for the first models, but I suspect after a while, a robot will be able to fix a simple leak, paint a wall or do simple electrical wiring jobs given the right tools. It should also be able to perform the yearly inspection of your boiler and pipes — although regulatory changes will probably take longer. Maybe eventually some simple medical acts too (intramuscular injections, changing a bandage…).

What was not done

But the most impactful part will probably not be the jobs that will be replaced; it will be the things most people did not do. To find those out, just look for what people wealthy enough to pay for full-time help get that the rest of us don’t. Bringing what used to be a luxury to everyone through automation is textbook disruption.

Why would I buy pre-cooked food if I have a robot that can make me home-cooked meals? It could go to the market to get fresh products daily, or have them delivered. It could clean the kitchenware and take out the trash.

Also, we all have those small imperfections in our homes that we don’t fix because they are not worth the time. That small defect that should be painted over, that slightly moldy sealant that should be replaced. A robot could vacuum and change your sheets daily if you want it to.

Oh, and of course I think robots could save lives too. One obvious way being calling 911 and performing resuscitation if you have a cardiac arrest. I also think having robots at home can help the elderly stay out of nursing care longer if they want to.

Enthusiasm

In general, I am extremely enthusiastic about home robotics. I want to “live in Star Wars” — well, the peaceful part of it. I know it is likely to make me underestimate the issues it could cause and the difficulty in making it happen.

Like AI, this is a field where predicting anything beyond a few years is very hard. There are many unsolved problems to achieve all this, and the order in which they will be solved is not obvious. However, I think it is worth trying to anticipate these changes.

Redis Patterns for Coding Agents

Agent-first documentation, with humans as an afterthought. But like the mobile-first web, it may actually be an improvement for humans as well in the end.

I made a code review tool - Shaygan Hooshyari

Towelie is a code review tool for code written by agents. I have been using it for large changesets. It gives you a code review UI and spits out a prompt to paste into the agent CLI. A simple tool that does one thing well.

The Factory’s Real Problem Isn’t on the Shop Floor - Renan Devillieres

Renan is the CEO of OSS Ventures, a startup studio for industrial B2B software.

He posits that what hinders reshoring is labor cost, but not that of blue-collar workers. Today the white-to-blue-collar ratio excluding R&D is above 1, and white-collar labor is more expensive. The way to make factories competitive is to lower that ratio significantly thanks to modern tools.

I don’t know if my job will still exist in ten years - Sean Goedecke

Nobody knows. If you think you do, you are delusional.

Personally I bet that I will still have a job, but it will be very different from the one I have now.

2026 is the year of AI infrastructure

A tweet by Z.ai, which is one of my favorite labs currently.

I think they are right, as recently demonstrated by Anthropic tightening its limits. Infrastructure is becoming crucial for resilience and cost-efficiency.

Lou of Z.ai has also posted her personal view that OpenAI is investing in infra whereas Anthropic outsources. I do not completely agree, Anthropic did order a million TPUs from Qualcomm, after all.

“Flow” is gone

Several people have posted similar things on that topic, including Stanislas Polu:

They’ve deprived us from the state of “flow” that many of us were thriving for. […] Coding with agents is an interruption management game. […] The excitement remains “building” shit, and more of it.

… and Julien Danjou:

I chose throughput over depth, and I keep choosing it every morning.

A generation of developers is about to start their careers with AI from day one. […] If you’ve never held the full system in your head, you don’t know which questions to ask, and you won’t catch the bugs that live in the gaps between modules. […] I chose the trade anyway, because the leverage is real, even if the loss is too.

A fighting retreat - Will Wilson

The CEO of Antithesis with one of the best posts about company culture, how all organizations lose what makes them different as they grow, and how to slow it down.

(No, despite the URL, this is not about Rust.)

I know this all too well, I have known it even before I started working because it happened to communities I was a member of as well. This is part of why I have only worked at small companies.

In any case if you have to grow, read that post, it is full of insight. But maybe you don’t have to? See next link.

The Org Chart Math Behind AI-Native Speed - Tomasz Tunguz

AI solves communication in large organizations… by removing large organizations.

This is one reason AI-native startups are pulling ahead, and why building AI companies feels fun. The advantage comes from organizational structure. Fewer humans, fewer channels, faster iteration, compounding speed.

Working on products people hate - Sean Goedecke

Sometimes a product is funded just well enough to exist, but not well enough to be loved (like many enterprise-grade box-ticking features) and there’s nothing the engineers involved can do about it.

Where the Horses Went - Chris Paxton

When a technology finally clicks, the changes spread faster than anyone expects and have far more serious implications for society. I want to lay out a case here for robotics finally “clicking” within the next few years, and what that could look like for everyone. […] The level of investment and growth in robotics and AI is reaching a fever pitch, well beyond what I expected 1-2 years ago. And, perhaps more importantly, most of what I have believed are substantial blockers to robotics deployments now seem solvable on a technical level.

Avery Pennarun on code reviews

Every layer of review [or process approval] makes you 10x slower. […] AI can’t fix this. […] The only way to sustainably go faster is fewer reviews. […] But you can’t just not review things!

Review pipelines — layers of QA — don’t work. Instead, they make you slower while hiding root causes. […] But, the call of AI coding is strong. That first, fast step in the pipeline is so fast! It really does feel like having super powers. I want more super powers. What are we going to do about it? […] I think the optimists have half of the right idea. Reducing review stages, even to an uncomfortable degree.

Imagine an old-school U.S. auto manufacturer buying parts from Japanese suppliers; wow, these parts are so well made! Now I can start removing QA steps elsewhere because I can just assume the parts are going to work, and my job of “assemble a bigger widget from the parts” has a ton of its complexity removed. I like this view. I’ve always liked small beautiful things, that’s my own bias. But, you can assemble big beautiful things from small beautiful things. It’s a lot easier to build those individual beautiful things in small teams that trust each other, that know what quality looks like to them.

This is why good teams right now are a few real seniors augmented with AI. Because a defining trait of seniority is knowing exactly where and when you can cut corners and when you need extra confidence. And if the whole team is senior then you can trust your colleagues with that. Of course this is not very sustainable…

Maybe tooling, including software verification, can help with this too (see next link).

When AI Writes the World’s Software, Who Verifies It? - Leonardo de Moura

The problem is not that everything is broken. It is that AI is changing the scale and speed of software production faster than our ability to verify it. What works at human pace may not survive AI pace.

The productivity gap is widening: teams with the best tools are pulling further ahead while others stagnate. Verification will be a decisive advantage.

Mistral also shipped a coding agent designed for Lean.

My heuristics are wrong. What now? - Marc Brooker

Many of the heuristics that we’ve developed over our careers as software engineers are no longer correct. […] What it means for a system to be maintainable. How much it costs to write code versus integrate libraries versus take service dependencies. What it means for an API to be well designed, or ergonomic, or usable. What it means to understand code. Where service boundaries should be. Where security and data integrity should be enforced. What’s easy. What’s hard.

Your taste, your high standards, your understanding of your business and customers and the deep technical trade-offs in your area are more valuable than ever before. This is like that fantasy that people have of going back to middle school knowing all the things they know now. You’re ahead of the pack in many ways. But you also need to really deeply question the things you know, and the things you assume. Before you share one of your rules of thumb, you need to deeply examine whether it’s still right.

Over the next couple of years, the most valuable people to have on a software team are going to be experienced folks who’re actively working to keep their heuristics fresh. Who can combine curiosity with experience. Among the least valuable people to have on a software team are experienced folks who aren’t willing to change their thinking.

The Structure of Engineering Revolutions - John Allsopp

The resistance to AI-assisted software development among experienced software engineers isn’t random or capricious–it follows the pattern Thomas Kuhn identified in scientific revolutions sixty years ago.

The paradigm is shifting. The evidence is in. The question is not whether the shift will happen, but whether the people who built remarkable things within the old paradigm can find their place in the new one. Kuhn would say probably not. Planck would say wait for the funerals. But the compressed timeline of this particular revolution, the example of the likes of Armin Ronacher, means there’s still a choice available.

Engineering as optimization

Almost 20 years ago, when I was still an engineering student, a retired network engineer from the ITU told me:

An engineer does what a technician can do, but 30% more efficiently and for 50% of the cost.

I always liked that definition of engineering, but only today did I figure out — thanks to Marc Brooker — its likely origin. It is a quote from Arthur Wellington:

It would be well if engineering were less generally thought of, and even defined, as the art of constructing. In a certain important sense it is rather the art of not constructing: or, to define it rudely, but not inaptly, it is the art of doing well with one dollar, which any bungler can do with two after a fashion.

My current agentic coding setup

This changes so fast I think I’ll include an update in most Thoughts posts. The previous update is here.

At work, I use Claude Code exclusively. My editor is now almost read-only, I even do most of my single-line changes through the agent so it has them in its context. I mostly use Opus 4.6 but sometimes reduce effort to improve latency.

My current “free and personal” setup is GLM 5 in pi through OpenCode Zen. It is slower but mostly as good as Claude. I have tried the other two free models available (Kimi K2.5 and MiniMax M2.5), and I much prefer GLM 5. I still use Amp Free sometimes but the credits don’t last long.

EDIT (2026-02-26) — GLM-5 is no longer free on OpenCode Zen (they introduced an OpenCode Go plan for $10 / month). It is still free on Modal until the end of April. To use it with Pi, go here to obtain a token and put this in ~/.pi/agent/models.json:

{
  "providers": {
    "modal": {
      "baseUrl": "https://api.us-west-2.modal.direct/v1",
      "api": "openai-completions",
      "apiKey": "modalresearch_YourToken123",
      "authHeader": true,
      "models": [{ "id": "zai-org/GLM-5-FP8" }]
    }
  }
}

End of Productivity Theater - Murat Demirbas

If AI absorbs all the shallow work, the only things left that genuinely require a human are the core parts that demand genuine creativity, judgment, taste, and the type of thinking that can’t be prompted away. […] That kind of deep creative work is best done away from the glowing rectangle.

I aspire to get bored in the new year - Murat Demirbas

The problem is that I no longer get bored. That is bad. I need to get bored so I can start to imagine, daydream, think, self-reflect, plan, or even get mentally prepared for things (like the Stoics talked about). I badly need that empty space back.

The AI Vampire - Steve Yegge

AI does actually make you 10x more productive, once you learn how. […] But who actually gets to keep that value?

It would seem that we are addicted to a new drug, and we don’t understand all of its effects yet. But one of them is massive fatigue, every day. […] We’re all setting unrealistic standards for everyone else. […] That’s a race that ends, in my opinion, with everyone collapsing in exhaustion without actually winning the race.

Steve’s argument is that if AI makes you 10x more productive and you don’t change your salary or working hours your employer captures 100% of that value, whereas if you work 10x less you capture 100% of the value.

It’s not even remotely sustainable for companies to capture 100% of the value from AI. And when employees capture 100% of the value, it will be temporary at best: that company gets beat by someone who’s got the dial turned higher. […] The right setting is in the middle somewhere. Companies will try to drag it higher. You need to fight to drag it lower. […] The new workday should be three to four hours. For everyone. It may involve 8 hours of hanging out with people. But not doing this crazy vampire thing the whole time. That will kill people.

I am the bottleneck now - Thorsten Ball

This and watching the Claude Code team take feature requests on X and implement them the same day make me consider one way software development team balance should probably change in the short term for user-facing products: support should become much more skilled and important.

Years ago a French startup called Capitaine Train (since acquired by Trainline) was famous for having skilled product people work support and use the feedback to improve the product. I have long admired this and pushed for similar approaches (e.g. having developers participate in support rotation to fill their pipelines).

But now if we can eliminate the developers bottleneck for simple bug fixes, this changes the game. We are close to a point where fixing a bug reported by a customer and confirmed could be a simple button click in Intercom. That would be huge for complex products such as Inch.

Single-customer frontends

In the section above I described a short-term solution to improve products that are complex because of the 80/20 myth. But on a slightly longer timescale I think we can just fix that problem entirely, at least on the frontend. I wrote a LinkedIn comment in French about this.

The complexity comes from customers using a different subset of the product. But if the cost of creating an interface becomes close to zero, customers could just create their own. SaaS products would then just become APIs that customers could query and integrate exactly how they want using agents.

I know it works because I do this for myself. I have a vibecoded personal Web application that is a kind of dashboard where I integrate features from several SaaS as I need them. Note that when I say vibecoded I still read the code to some degree, this is not accessible to non-developers yet. But to a small company with a single developer, I think it is.

John Nunemaker on skills transferring to AI-assisted work

I’m not worried about AI taking over. I’m obsessed with it. […] The fundamentals I learned the hard way? They translate perfectly to telling Claude or ChatGPT exactly what I need. The key is knowing what to ask for. Twenty years of programming principles don’t disappear, they become the foundation for working with AI effectively.

Brex’s AI Hail Mary

Brex realized that speed and execution shouldn’t come at the cost of increasing headcount.

Honestly, that has long been true and I have defended that point of view my whole career. My about page has been saying this for ages:

I believe small teams can achieve a lot. Do not ask me to manage the 10x growth of your engineering team, ask me to find ways to prevent that from happening in the first place.

If AI helps companies realize this more easily, all the better.

Why I Stopped Using nbdev - Hamel Husain

Joy is a valid reason to choose a language, even with less AI support. It’s just not my focus right now. My joy resides in solving problems, and I want tools that maximize my leverage. For that, conventional wins.

Agreed on all counts, except that I think agents are actually reasonably good at using exotic software. For instance, the personal web app I mentioned earlier is written in Teal using mote. In my experience the worst is stacks that are somewhat popular but with unstable APIs (e.g. Zig).

In any case, a lot of people are left wondering if code was just a tool or something more, and about the role of fun (David in French, Ori Bernstein, David Cells…).

Personally, I already touched it here: it is both. I have always needed both a goal and some enjoyment. I don’t think the agentic switch will necessarily change that.

Romain Guy on pushing projects at companies

Many times, you try to make a project take off that by definition nobody asked for. You are convinced that if you do it people will go “Oh great! I can do something awesome with this!”, but nobody will ask for it until it exists. And the problem is at a large company you’ll be asked “can you show me figures that prove it is what we should do?” and you go “Well, no. I have to do it first.” And that’s scary for large organizations because it’s a jump into the unknown, […] even at a small scale.

Typical. When it can be tested in a short timeframe the solution is “ask forgiveness not permission”. As soon as you mention it you get into arguments that will take you more time than actually prototyping.

For things that take more time it is not always possible, and that’s when you need to play politics to get buy-in. But when you can, (vibe)code and ask later. Remember Claude Code was a side project.

The culture war that we won - Daniel Lemire

In some sense, building software that can do financial, political and legal analysis is the latest weapon in the arsenal of the computer people. Many despair about what AI might do to software developers: I recommend looking at it in the context of the hacker culture war.

Returning to Stanford

A beautiful post about the true value of college. Few students appreciate it in the moment. I think I did, but not as much as I do now. The author has founded two startups in the meantime, it clearly gave him the necessary hindsight.

I often wonder how we can realistically emulate parts of this later in life. Recurse Center is a step in that direction.

From craftsman to wage-worker - Karl Dubost

Extract translated from French (original title: De l’artisan au prolétaire).

Note that Karl writes extremely well and I am not fluent enough to translate how it feels. If you can read him in the original French, do it.

Mastering tools is not merely a level of attainment. It is also a trajectory, especially in an accelerated world.

We do not turn into wage-workers when we delegate our expertise to the machine. It happens when we give up (or are forced to) our desire for expertise. When we give up on discovering and mastering new tools while enslaving ourselves to a political program [including capitalism]. I will not name the specific tool, it is a distraction. It prevents us from thinking globally about what is happening under these circumstances.

How AI helps break the cost barrier to COBOL modernization - Anthropic

Yesterday, Anthropic posted this and IBM stock instantly lost 10%.

I have no doubt that agentic programming will be a game-changer to escape legacy systems.

What the Anthropic post describes is the way professionals already deal with legacy systems:

• start by understanding and documenting it;
• define the target architecture and rules;
• write extensive tests to ensure the outputs of the system stay the same;
• replace component after component while ensuring tests keep passing.

This is called the Golden Master pattern.

The challenge, however, will be to give human stakeholders high-level indicators and tools to understand the progress and ensure everything is going well. And then a way to identify and fix the defects that will slip through the net and end up in production.

I have no doubt that AI, used by capable people, can help with all this. And this is not only about COBOL; I have recently met the founder of a (still stealth) startup tackling a similar problem with a different stack. The total market for legacy software migration is unfathomable.

Maybe Anthropic is still undervalued after all!

What Finegrain does

I am going to skip things we have done in the early days that have become irrelevant and focus on what we are doing now.

So first, we make image editing models. We train them, we evaluate them, and we ship them in a format customers can integrate. We have always focused on keeping the models small and efficient, initially so we could have a fast and cheap API, and now so they can run entirely on edge devices such as mobile phones.

We sometimes build end-user tools to demonstrate those models too. Among those you may have seen in our API era, there was a Web-based image editor, ComfyUI nodes, a chatbot for image editing, and an ad generation assistant. All those are gone, but we still have a few public Hugging Face Spaces. Now our flagship product is the mobile SDK, we have a free mobile application to demonstrate it.

Now, let’s get a bit more into what I am working on exactly.

PHX, the server-side inference stack

The first important thing I have built is our server-side inference stack. What does this include?

A model is a bunch of layers, which basically means matrices of numbers corresponding to operations (multiplications, convolutions, etc) organized into a graph with non-linearities sprinkled in-between. The model takes tensors as input and returns tensors as output.

When you want to do something with a model, you need some preprocessing to feed it data and postprocessing to take it out. In the case of text, that often means tokenizing. In the case of latent image models that means encoding and decoding — which by the way also involve models.

You also need a lot of things to optimize the model’s operation. That includes the mode of execution (compilation, quantization, numerical precision, etc). Crucially, in the case of models that you call several times in a loop such as diffusion / flow matching models, this also includes solvers (samplers / schedulers), which can be the most complex and math-heavy part of an inference pipeline.

In our inference stack, which is called PHX (a reference to the Phénix airplane), all of this is packaged in a processor (which some call a pipeline). Processors can be used locally or deployed to Modal — which we use e.g. for model evaluation purposes — but in production they are deployed on GPU servers and exposed to callers using NVIDIA’s inference server Triton.

Disciple, the Web backend

PHX is not exposed to customers or applications directly. We provide a Web API that is easier to call, asynchronous (because although our inference is fast it is still slower than a typical Web request), and deals with authentication and payment.

In addition to this API, we have a Web backend for customers, a bit of admin and monitoring… All of this lives in a modular monolith called Disciple, a reference to a character of the comic strip Léonard whose most famous line is “I serve science, and that is my joy.”

The stack is nothing too fancy; I picked technologies I had previous experience with and knew to be reliable. We use Quart with Blueprints and PostgreSQL. For asynchrony we use Nchan for SSE and Beanstalk for background jobs. For efficient image processing we rely on VIPS.

The iOS SDK

For some time now, the main focus of Finegrain is running all our models on mobile devices directly. This means we had to port our inference code to run on iOS (for now) using Apple’s Neural Engine (ANE).

This involves, among other things, compiling and quantizing the model using Apple’s Core ML Tools. This is not a straightforward step; most models cannot be compiled out of the box and require architectural changes first. As for quantization, obtaining high average quantization ratios while retaining precision is a craft that would deserve its own blog post.

Once you have that model, you need the equivalent of PHX’s processors, but for the mobile device. I use PHX as the reference codebase, but porting to Swift on iOS is not as straightforward as one may think. On mobile anything can quickly become a bottleneck, and if you are not careful you end up spending more time doing pre- and post-processing than running the model. We use Apple’s Accelerate framework extensively to speed up those parts.

Science and hacks

All of this is advanced but straightforward engineering, but if you want to be good in this field you need to go beyond that and implement cutting-edge research or innovate.

A large part of it is in model training and the datasets we use, of course. I am hardly the person who trains the most models at the company — I do a little, mostly focusing on performance-related things. We all read papers in the domain and discuss approaches to try all the time though. Personally, I have been studying the things “around” the main model a lot, including solvers, NFE (Number of Function Evaluations) reduction techniques, and recently auto-encoders.

In addition to that, there are a few ideas I have had and implemented that make a difference. The most important one is the set of techniques we use to edit high-resolution images with a model backbone that works at much lower resolution. We call them “fixes” and let’s just say having a background in signal processing and classical computer vision as well as a good understanding of latent spaces helps a lot there. I have spent — and still spend — a lot of time on this, and it is a crucial part of what we do, so if you were to ask me what I am most proud of so far, that would be it.

Another notable thing I have implemented recently is the ability to swap the capabilities (skills) of the CoreML model. Core ML has Multifunction Models, but in theory they are built by compiling each function and merging them once on a Mac. With the tooling I have designed, we can create new functions for a given backbone in pure PyTorch, and add / remove / swap functions in the model almost instantly on any machine including an iPhone. This makes it possible, for instance, to update a single function in an application without re-downloading the whole model.

As always, I haven’t talked about everything I do. We are a small company and I am still a jack of all trades kind of guy, so I always write tooling and fix bugs here and there. That should give you a decent idea of what my day-to-day is about. As always, if something piqued your interest, feel free to get in touch.

A digression about agents

This year I didn’t write the list above by hand. I gave the URLs to an agent and the format I wanted and told it to make it. It’s a detail, but I do things like this many times a day now.

I also had agents work for me over the FOSDEM weekend. They ported a personal web application I have from a Lua Web server to another Lua Web server, from a CSS framework to another CSS framework, and added end-to-end tests with Playwright. Oh, and the application is in Teal so they wrote the definition files for mote too. All I had to do was describe the task and nudge the agents in the right direction in between talks. If I did not have those agents, I would just never have done all this.

At work, I’m back to training models this week, after a long time working solely on inference. Going from zero to GPUs going brr with nontrivial things like adversarial learning and brand new, (almost) working experiment tracking (because OpenAI is killing the old one) took me roughly a day.

And that was with the old agents. I can’t wait to see how well my new minions do tomorrow.

Examples are the best documentation

This was true before, and this is even more true for the coding agents era.

AI coding agents rely too much on fallbacks

I think agents are improving on that front, and Claude does it less than the others, but I still instruct them to fail early and loud and pay attention to this in reviews.

Start Small, Scale Smart: The Real Value of Incremental Architecture

Some things in this post match my architecture principles.

To scale efficiently, you need cross-functional, independent teams with ownership. This is applied Gall’s Law.

Shorten feedback loops.

Instead of asking: “how long will this take?”, ask: “Can we make this smaller?”

Werner Vogel’s predictions for 2026 and beyond

For adults, AI is a tool. For Generation Alpha, it’s an extension of thinking. They’ve deleted “impossible” from their operating system and replaced it with “not yet.”

Tarek Ziade on Two Years of Building AI in Firefox

Mozilla is still doing some interesting things. I will follow this with attention, especially the WebNN part which is relevant to my work. I only work on native on-device AI for now, but we will probably tackle the browser as well at some point.

Geoffrey Litt’s articles

Found through Pierre’s 2025 recap. I don’t know how I did not follow his feed until now, I follow most current & former Ink & Switch people. Catching up with three articles.

I want to work on core product

This means I want to work directly on the product my employer is selling or something that makes it possible. Support functions matter and I have huge respect for them but they are not for me.

My sweet spot is working on B2B products and interacting directly with customers.

I want to work on AI R&D

Historically, my specialty is Distributed Systems, but I happened to work on AI (for Computer Vision) “before it was cool.” I left the field, re-joined it, and I intend to stay in it for now.

Note that this does not just mean using LLMs through APIs. It means training, operating and/or optimizing models. If reading and implementing research papers is not part of the job description, I am not interested.

I want to work hybrid-remote in Paris

I believe close-knit teams that meet regularly learn to work better together. I know some people believe this can be achieved fully remotely but I do not. However, I also believe working from home part of the time has benefits including cutting down on commute and better focus.

This is not a hard red line, I could go with full remote or full in-office under some conditions. My location is though: my life is in Paris, France and I am not considering relocating anywhere.

Fields I avoid / favor

I do not want to work in finance-related fields, including banking and crypto. I would also like to avoid working for HR / recruiting companies.

Fields that specifically interest me include energy and physical automation / robotics. More generally, industrial fields involving non-standard modalities.

Management

I am highly autonomous and self-directed. All my direct managers so far have been the company founders. I can deal with some politics, but not much.

In most cases I prefer being an IC with a Staff+ role. I am fine with having a few skilled IC reports; I do not manage managers.

Culture

I recognize the necessity of process, but I highly dislike its abuse.

I think small teams with real ownership work best.

I think people who care can — sometimes should — disagree and argue passionately, and still remain friends.

A trade ending or evolving

A post about our job by Éric, from last February (in French). He shares a lot of my thoughts on the topic. Here is a (translated) selection.

I fear a real crisis in our trade in a few years. Some — many — will be left on the sidelines.

If you are very tech-minded, you should do math, data manipulation, statistics and generally AI. It will be harder and require more advanced competencies than being a developer, but there will be jobs. If you want to create, for me the future lies in product-related roles, product management with a technical emphasis and interest. It means caring about the business, market, customers, etc.

For those of you still mostly writing code, I would switch to LLMs ASAP. I know you don’t feel the need, that those tools make mistakes you don’t, that today it slows you down. […] There will be room for older code experts for a long time, in maintenance and large companies that are several generations behind. We still need COBOL developers today. But is it really the position you aspire to?

Breaking rules

A post by Sean Goedecke about how strong engineers break rules and get away with it. (On a side note, I am impressed by how Sean manages to publish interesting content so frequently.)

Like I said on Bluesky it is a very important point to learn mid-career. The caveat is that most people will take it too far at some point. You should do it anyway — you need to learn exactly the right balance and nobody will teach you.

What kind of work

Sean (again!) has written a post about what kind of work he wants. Actually, he did it twice; this is an update from his first post back in 2021. I think this is a very interesting exercise and I should do it sometime. (It is hard for me because I have conflicting interests so I keep oscillating between technically exciting, product-focused roles and ones that are more business- and customer-facing.)

Fallacies of code generation

My favorites:

• Writing code is the hardest part of software development
• A system can be maintained without being understood

Some I am not sure about:

• LLMs will become reliable decision makers
• Users will tolerate bad quality

For the first one, I do not think anyone on Earth is competent to guess what LLMs or whatever will replace them will be able to do or not in a few years. People who say they will never be able to do something typically have either a very shallow understanding or a restrictive (and irrelevant) definition of LLMs.

For the second one, I sadly know from experience that users do tolerate bad quality. Not all users, but often enough for the worse alternative to win.

Happy Ears

A long-form post on X (that should be a blog post…) about something every engineer involved in a sales process — or talking to customers in any capacity — needs to learn.

Interacting with customers as an engineer is very different and sometimes contradicts what an engineer working on a product does. I love doing both — and I think it is important that some people do both — but it is always going to be hard and feel a bit schizophrenic.

Policy of transience

A post by Simon Tatham. His policy is:

Things should either be deliberately permanent in an organized way, or strictly temporary.

I do the same. The first thing I do when I log onto a new machine is disable persistent shell history, I turn my computer off every night and configure my browser not to remember tabs.

I really liked that post because I have been doing this forever — and I know most people do not — but I never really thought about why or formalized it. In my case it was initially because I used a lot of different and often shared machines, but now I just prefer it that way. I guess it is somewhat similar to how some cooks are obsessed with clearing the worktop as soon as they can.

BitTorrent generation

A (sub)post by Karl Dubost about copyright and that message posted by someone else on Bluesky:

20 years ago we were suing teenagers for millions of dollars because they were torrenting a single Metallica album and now billionaires are demanding the free right to every work in history, so they can re-sell it. The law only ever serves capital.

Karl writes (translated from French):

Sam Altman was born on 22 April 1985, so he is 40. BitTorrent came out in 2001, 24 years ago. […] Sam Altman was 16. So he’s only doing now what he was doing when he was a teenager. My answer: teenagers from that era have grown up and some became CEOs. There’s no mystery.

2001 was the year I really went online and started getting involved with the F/OSS movement. I am from the same generation as Altman, the BitTorrent (and Napster) generation, the Aaron Swartz generation, and I will always be against policies that prevent copying and sharing digital content.

On the same topic, Karl also wrote here:

I am a creator. I constantly steal others’ work. It is the nature of creation. We do not live in a vacuum. We imitate life around us. Babies, kids do that. We keep doing that our whole life. We do not ask consent from all those who have created something before us to create ourselves. We train on everything we see. Stealing is not the problem. […] Consent is not the problem. The main problem is ownership and the ability of certain actors to capture that ownership and prevent others from enjoying it.

(On a side note: I do not enjoy translating Karl. I really enjoy his style in French; I always feel like I do not do him justice.)

Go is a good language for agents

Armin Ronacher writes:

Rob Pike famously described Go as suitable for developers who aren’t equipped to handle a complex language. Substitute “developers” with “agents” and it perfectly captures why Go’s simplicity benefits agentic coding.

But I think the most important part is the structural typing:

Interfaces in Go are structural. If a type has the methods an interface expects, then it conforms. This is incredibly easy for LLMs to “understand”. There is very little surprise for the agent.

Other important points are speed (both of execution and compilation) and low ecosystem churn.

Breaking the Spell

A year ago, this text came out in Phrack. It is about how we need hackers more than ever to keep understanding the complex and ever-changing systems AI unleashes on us. (Go read it now and come back!)

A big reason I left the AI field and the end of 2013 is I did not like deep learning. I was a proponent of simple, sharp tools in the Unix tradition. I wanted to understand things entirely despite the complexity, like I did with distributed systems. I said “I am a physicist, not a biologist”.

I came back in 2022 because AI is there to stay. It will run the world, and we will need hackers to poke at both its physics and biology. A lot has changed since 2013; interpretability research is a thing now, and we have had tools to build something we understand from something we do not understand for a long time.

So I am here to stay and understand as much of it all as my feeble human brain lets me.

On a related note, I liked that on Henry Ko’s About page:

I am interested in ideas that share a theme of “doing more with less”.

He is talking about less energy, less data… and I would add less architectural complexity. That’s how we hackers can help.

Code is scaffolding

Every program is just scaffolding for your next understanding. […] The code isn’t just temporary because we’ll rewrite it. It’s temporary because our understanding keeps evolving. […] Your current understanding is probably wrong. Build simple scaffolding that lets you discover why.

One way to understand why AI-assisted coding is so powerful is to see it as an aerial lift. Sometimes you can get to a better understanding in seconds without bothering with all the scaffolding.

Robotics

There has been movement in the field, including in France, from Hugging Face’s LeRobot to the founding of Genesis AI.

That is something I am really excited about. I mean come on, we can live in Star Wars! More seriously, automating physical jobs and tasks is still one of the top things that could change our lives. It will disrupt them too, sure, but I am optimistic about the end result on this one.

Shorten your feedback loop

When asked “What trick of the trade took you too long to learn?”, Hillel Wayne answered:

Shorten your feedback loop. No, shorter than that.

This is such a good answer, and one that I have agreed with forever (I mean, I have been using Lua for 20 years…) that I have added it to my core principles.

Claude Code

The most important change is that I now use Claude Code. I don’t use it all the time, but here are a few ways I do:

• When I implement features in a codebase I know well, I write a detailed prompt describing how I want the feature implemented and give it to Claude.
• When I need to kickstart a project quickly with technology I do not know well, I ask Claude and it typically does a better job than me — and most importantly faster.
• When I get an error message I cannot immediately understand I paste it into Claude Code, and more often than not it finds how to fix it.

Here is one way I do not use Claude: when I work on hard problems where I do not know the solution. I use code as a thinking tool, and using an LLM disturbs that process.

In general, I see Claude as a junior but very fast pair programmer. It won’t help with hard things but it will gain you a lot of time for the simple ones if you guide it well and check its work.

Lovable

Another tool that I used a lot is Lovable, and I have a complicated relationship with it. The way I used it was, someone else vibe-coded an app and I stepped in when things became too complex for Lovable.

At first I was very impressed. In a few prompts anyone can get a usable application prototype, all in the browser. This makes most no-code frontend development tools irrelevant.

The more I used it, though, the more I saw its limits. The recently released agent mode makes it better, but it still has flaws such as writing spaghetti code and not cleaning up dead code. As time progresses, its context becomes polluted by the messy codebase and it gets lost. I think the stack they chose (React and Radix) does not help with that.

The way to fix it is what I did: clone the repository, refactor by hand or with the help of Claude Code, and put things back in a state where the model will start making the right decisions again.

But the main issue with Lovable is not the model, it is everything around it. It uses Supabase as its only backend, which is very limited. Most importantly there is no proper way to have separate environments, not even a development environment separate from production. They need to fix that and understand that if they want to have successful applications in production they must consider the fact that they will be worked on by teams.

Others

I still use GitHub Copilot for autocomplete. It is just better than the default, especially in Sublime Text.

I also use Mistral’s Le Chat for random things, like most people use ChatGPT.

I have not yet tried alternatives to Claude Code except for Aider which I used earlier. I am most interested in Amp for looking well-made, opencode for being Open Source and gemini-cli for being basically free (as in beer).

I think in a few months to years more asynchronous tools such as Devin, OpenAI’s Codex or Google’s Jules may become interesting, but they are not for me just now.

If you are looking for content elsewhere to make up your mind, I think this post by Simon Willison is very relevant, and I like what David and Armin post.

I miss FAKEGRIMLOCK

If you know, you know. If you don’t, picture this: it’s the second tech boom (early 2010s), most people did not experience the dot-com bubble and are figuring this stuff out, and some of the best advice you can find online comes from a robot dinosaur that writes in all caps. You can find some of its content online, for instance on Fred Wilson’s blog.

Paul Graham on Yahoo (2010)

In technology, once you have bad programmers, you’re doomed. I can’t think of an instance where a company has sunk into technical mediocrity and recovered.

In the early days Facebook made a point of hiring programmers even for jobs that would not ordinarily consist of programming, like HR and marketing.

Hacker culture often seems kind of irresponsible. That’s why people proposing to destroy it use phrases like “adult supervision.” […] But there are worse things than seeming irresponsible. Losing, for example.

Quentin Adam on EU tech

This is a tribune by Quentin Adam on AI and industrial policy in the EU. Excerpt translated from French:

We need companies led by technology leaders, trained engineers who deeply understand the challenges of the sector. […] It is time to stop only investing in managers and to fund those who know how to build.

Two bitter lessons from the rebuild of Notre Dame

Philippe Silberzahn writes on how France can do great things, but 1) to preserve the past and not build the future and 2) only by circumventing legislation.

Translated from French (there is a pun in the original I could’t find a good translation for):

I meet a lot of [public and private sector leaders], they all have the same lament: we are suffocating under millimetric regulations. The country belongs to inspectors who spun out of control. Excellence has become impossible.

“More human” management is meaningless

Philippe Silberzahn again, translated from French again.

Management without exigence is inhuman. How many talented engineers […] languish in companies where nothing ever happens except for a continuous flow of bureaucratic events? […] Accepting the mediocrity of an employee is disrespectful. They are invisibilized: “You are doing a bad job but I won’t tell you, and even less demand you improve.” It means you’re not worth it, I won’t help you produce work you can be proud of. […] And collectively, we will make crap, or even better: we will make nothing at all. And we will die slowly.

Upspin is shutting down

Upspin, like Unix and Plan 9, was intended to foster communities of sharing, but has been less successful at that than we hoped. As a consequence, with regret, we have decided to turn down the central infrastructure such as the keyserver over the coming months.

AI hardware

Last December I bookmarked this:

Nvidia’s not the only one that can increase memory capacity of course. ASICs can do this and in fact AMD may be well positioned due to their higher memory capacity versus Nvidia […] Well except Santa Huang has a Red-Nosed Reindeer called NVLink. No other accelerator in the world has all-to-all switched connectivity. No other accelerator in the world can do all reduce through a switch.

Then DeepSeek proved this was not the moat we thought it was. Meanwhile Mistral is doing over 1000 tokens per second on a 123B model by leveraging Cerebras for inference, and Tenstorrent is shipping… fun times!

George Hotz - The Soul

You are a machine learning algorithm. You have some priors in your DNA. You learn on data, RL style because your actions affect the next data you see; the dataset depends on the model. […] How much of [Christianity’s decline] is people starting to figure this out?

Carson Gross - Coding Dirty

The author of htmx on some “dirty” code practices he considers good:

• (Some) big functions are good, actually
• Prefer integration tests to unit tests
• Keep your class/interface/concept count down

Engineers know

Avery Pennarun, founder of Tailscale, on X:

Engineers (who aren’t kept isolated in a back room) are pretty good at knowing what’s important to work on. They just don’t think they are. And they especially don’t think other engineers are.

I answered:

And French engineers are probably even worse… You have no idea how many think you need a business degree to understand a customer, or anything remotely related to strategy.

He answered:

I keep sensing the “people live up (or down) to the expectations you put on them” effect. If you expect people to have bad ideas, they won’t be careful. If you tell people they get to choose what’s important, there’s no safety net, they’ll live up to what’s needed.

Insightful.

Reflections on Palantir

A long article by Nabeel S. Qureshi on his experience at Palantir. The most interesting part for me is how Palantir uses Forward-Deployed Engineers differently from most companies.

[FDEs] gain intricate knowledge of business processes in difficult industries (manufacturing, healthcare, intel, aerospace, etc.) and then use that knowledge to design software that actually solves the problem. [Product Development] engineers then “productize” what the FDEs build, and — more generally — build software that provides leverage for the FDEs to do their work better and faster.

The One Developer Framework

I like how DHH increasingly focuses on single-developer teams. On X:

I view everything I work on through that One Developer Framework lens. Not interested in tooling, approaches, or concepts that demand a team of thinly-sliced specialists to tick. One developer must be able to understand/use it all. From Hello World to IPO! Let’s go, nerds :)

Stop trying to make everyhting too easy

Howard Chu on Mastodon:

A lot of user interactions these days are too user friendly, and when deployed in the real world you discover that you really need a bit of friction, to prevent things from being too easy. Child-proof packaging on medicines is one example where you purposefully raise difficulty. […] Stop trying to make everything “so easy a child can do it.” Some things need to have a higher barrier to entry. Focus on educating people to be smart enough and responsible enough to clear the hurdles.

Scott Chacon - Why GitHub Actually Won

We won because we started at the right time and we had taste. We were there when a new paradigm was being born and we approached the problem of helping people embrace that new paradigm with a developer experience centric approach that nobody else had the capacity for or interest in.

Hugo Lassiège — The role of Staff+ software engineers

On YouTube, translated from French:

As you progress, you must be able to analyze a situation more and more remote from pure technology, and it is interesting to do so because you will also need to understand marketing, sales, finance, and so on. No company that does not have technical people involved with business strategy can have a tech or product approach anyway. All those companies I hear complain like: “We do not innovate enough! We hired the best developers but can’t see a significant impact…” Systematically, we notice no tech person is working on strategy. There is no tech person at product meetings, no tech person at executive committees, etc.

Hugo Lassiège — Security at Startups

On YouTube, translated from French:

At a startup, what I call “human capital” is rare and limited. You cannot hire armies of people, so you need rare birds able to do several things well, or at least people with the potential to become them.

The literal translation of what he says in French is “you need five-legged sheep”, I couldn’t find a good translation in English.

Setup the LSP

Install tree-sitter-cli which is a system dependency, clone the repository somewhere, initialize a local Lua environment and install the dependencies with LuaRocks:

sudo pacman -S tree-sitter-cli
cd /.../git
git clone [email protected]:teal-language/teal-language-server.git
curl https://loadk.com/localua.sh -O
sh localua.sh .lua
luarocks make
luarocks install tlcheck

Run teal-language-server --help to check it works.

Setup Sublime Text

Open the LSP settings (Preferences > Package Settings > LSP > Settings) and configure Teal like this:

{
    "clients": {
        "teal": {
            "enabled": true,
            "command": [
                "/.../git/teal-language-server/.lua/bin/teal-language-server"
            ],
            "selector": "source.teal"
        }
    }
}

Add a tlconfig.lua file to your project

If you do not use this step, the LSP will only lookup dependencies in its own Lua environment. You want it to look in your project instead.

The contents of tlconfig.lua can look like this:

return {
  source_dir = ".",
  include_dir = { ".", ".lua/share/lua/5.4" },
  gen_compat = "off",
  gen_target = "5.3",
}

The important part is .lua/share/lua/5.4 in include_dir. For the rest, settings depend on your project.

Note that even if you use Lua 5.4 gen_target must be 5.3, otherwise the LSP will not work currently even if you specify gen_compat = "off".

Tolerance

In 1959, Bertrand Russell was asked on BBC what would be worth telling later generations.

His answer was divided into two parts, intellectual and moral. The latter, I think, may be of some use today.

Love is wise, hatred is foolish.

In this world, which is getting more and more closely interconnected, we have to learn to tolerate each other. We have to learn to put up with the fact that some people say things that we don’t like.

We can only live together in that way. And if we are to live together and not die together, we must learn a kind of charity and a kind of tolerance which is absolutely vital to the continuation of human life on this planet.

We tend not to do that. We take sides arbitrarily, probably more easily online than offline. We create fractures that make participation in any community more tiring than it should be. Albert Wenger puts it like this:

The world is continuing to descend back into tribalism. […] Words have been rendered devoid of meaning and reduced to pledges of allegiance to a tribe. […]

I don’t belong to any of the tribes nor would I want to. But the effort required to maintain internally consistent and intellectually honest positions in such an environment is daunting. And it often seems futile.

Mental health

I generally avoid that topic online, but I will make an exception.

Brain chemistry is complicated, and mental health is a multi-dimensional spectrum. We vaguely mark regions of that spectrum and give them disorder names; it is a necessarily simplified model. The frontiers of those regions are blurry and differ between cultures and over time. Two decades ago, for instance, behavior considered normal for kids in France would often have resulted in medication for ADHD in the US.

Today the number of diagnoses and self-diagnoses is clearly exploding. Part of the reason is that influencers, sometimes paid by medical companies, have made neurodivergence “trendy”: people feel like having a disorder makes them belong to a community. It is even worse for kids, with parents “diagnosing” them and managing to find medical professionals to confirm those diagnoses.

The problem is that studies have shown that believing that you have a mental disorder when you do not can make you exhibit the same symptoms. However they do not come from the same causes, so medication will not work beyond the placebo effect, but it will mess with your brain chemistry.

A psychiatrist once told me: “If we can avoid treating something like a disease, we should.” I think this is the right approach. We must go back to considering more zones of the spectrum “normal”.

Note that I am not saying you should not go to therapy! If anything learning CBT basics could benefit everyone. But we should normalize therapy instead of labeling everyone with a mental disorder.

Although I have been thinking about this topic for a long time, posting about it here was prompted first by a post by DHH, and more recently by a video in French where I learned about the influence of medical companies like Done and Cerebral. I also have a book about how the American approach to mental health impacts the rest of the world on my reading list.

Have fun

DHH (again) explains very well why I do not use languages like Rust or C++. (*)

There is no universal set of trade-offs that’ll make something objectively “work best”. Half the programming conundrum lies in connecting to an enduring source of motivation. I wouldn’t be a happy camper if I had to spend my days programming Rust (but I LOVE so many of the tools coming out of that community by people who DO enjoy just that).

I work mostly with Python nowadays, it is not my favorite language but I tolerate it. Tools written in Rust are making my life much better, yet I would not pick Rust if I had to write such tools, even if it may be the best tool for that job.

Beginners often ask me what programming language or algorithms they should learn, and I tell them it does not matter at this stage: they should learn what motivates them the most. The older I get, the more I think this remains somewhat true your whole career.

Programmers need slack and fun projects, or their output degrades real fast. Young people with no kids can compensate for boring work with side projects, but as you get older you must make sure you find some fun at work. Jason Cohen talks about procrastinating for success.

Finally, sometimes you’ve got to let yourself do fun stuff at the expense of priorities. Most of what you do in a little company isn’t what you enjoy doing; it’s easy to become frustrated and burned-out. So sometimes you need to allow yourself to recover with plain fun. Whatever the cost of putting off duties, burn-out is even more costly.

I would add a little nuance to this: it is perfectly possible that, at some point in your life, your work will motivate you so much that you won’t want to do anything else. And that is perfectly fine, you just have to recognize when that is no longer the case.

Anyway, happy people become successful, not the other way around. There is a book about this. We are all different, so figure out what you need to be happy now, and go get it.

(*) The reason I do not like C++ or Rust is not because they are efficient, statically-typed programming languages, it is the complexity. I did write a lot of C and I am enthusiastic about Zig, for instance.

Differences between Google and Amazon

It is always interesting to compare those two companies. They differ in their fundamental principles and goals, the way they operate, their technical stacks…

The mother of all comparisons is Steve Yegge’s famous rant. More recently, I have also really enjoyed posts by Carlos Arguelles who went from Amazon to Google and (very recently) back.

In the disaggregated write-ahead log, Shikhar Bhushan highlights an important difference between the two giants’ architectures. At Google the distributed filesystem (Colossus, formerly GFS) is a fundamental building block. It is not global but zonal, and replication is built on top of it. At AWS the globally distributed log — which is not exposed as a public service — is a fundamental primitive, and S3 is built on top of it. Shikhar is now building that service.

Jaana Dogan also recently tweeted about another difference:

At Google, a lot of time is spent on rewriting infra or migrating to new infra. AWS rarely kills its services, hence if there are no new hard problems due to growing scale or something, it’s fairly cheap to keep the lights on.

That would be part of the explanation for why Google often kills its services when Amazon maintains them. Although I think Amazon’s customer obsession is crucial too.

Is radicality needed to solve Big Problems?

I have been re-reading some older articles by Philippe Silberzahn (in French) and this one deserves a mention.

He describes how many people have a view that “Big Problems” such as climate change can only be solved with radical actions. He posits that it stems from at least six mental models:

1. A Big Problem requires a Big Solution
2. There is no doubt about the existence of the Big Problem
3. The end justifies the means
4. There is a simple solution to a complex problem
5. There are no adverse effects
6. The resolution of the problem does not require compromise

His view is that Big Problems often have a large social component, and that the large change required to solve them does not usually come from radicality but from small actions carried out by people with alternative mental models.

Statically typing dynamic languages

Note: I am expanding on something I posted elsewhere here.

I think I finally understood a problem I have with many type systems added on top of dynamic languages. In practice, they work against the I and D of SOLID. You can make Interface Segregation work, but you need granular interfaces and intersection types. As for Dependency Inversion, typing makes boundary code very verbose, and using Dependency Inversion reduces the advantages of using typed libraries anyway, since if you do it well library types should not leak from the boundary into client code.

It is also interesting to note that AI appears to be bad at writing code with inverted control. It shows it “reasons bottom-up”, which makes sense when you know how it works. So the end result of typing + copilot is that I write code that is less flexible and that requires larger diffs to change. It is not necessarily harder to change, because the tooling helps.

I have a feeling it is not always good or bad, it probably depends on the kind of code I write. SOLID is not absolute. But it is annoying me that things that I still view as “mere tools” introduce so much friction to achieve patterns that I consider elegant and useful.

On a related note, Armin Ronacher — who mostly switched to Rust nowadays — has an interesting blog post about his view of typed Python too.

Soft skills posts

Daniel Mangum thinks the most important skill in startup engineering leadership is pacing. I agree wholeheartedly. Pacing in startups is hard to get right: it depends on many forces, it is often a tradeoff, and getting it wrong can be disastrous.

Phil Eaton — who recently changed jobs — wrote a very good post about what to do when you join a new company or team in a senior position. He also mentions internal blogs, which is something I rarely see but always thought was worth doing since I read about it in The Year Without Pants.

In Both pyramids are white, Vicki Boykis talks about a specific kind of groupthink. Unlike the pyramids, it is not black or white, there is a tradeoff between dissenting when necessary and not dissenting when not. We all have a natural tendency for one or the other, that we need to acknowledge and fight to find the right balance.

More technical posts

Dave Paola explains why non-DRY specs are more maintainable. There was no need to convince me on this. He quotes Richard Gabriel:

The primary feature for easy maintenance is locality. Locality is that characteristic of source code that enables a programmer to understand that source by looking at only a small portion of it.

Speaking of Richard Gabriel, I have read a few articles about Rust and Zig recently. The more I do the more I think it may be a Worse is Better situation in modern systems software. It has never been clear if one is actually better. The Better (Rust) side seems to be winning market share for now, but personally I am more drawn to Zig.

Tom Tunguz thinks databases won’t charge for storage in the future. The pendulum keeps swinging between colocating compute with data and separating them. Both have advantages and drawbacks. I have successfully argued for colocation in several cases (“bringing compute to the data”), but it is true that for some categories of systems they tend to be decoupling successfully today. Quickwit is a great example.

Finally, Justin Jaffrey points to cases where compositionality fails. We love clean abstractions that do not leak, but when you get into the nitty-gritty details of some kinds of software, you cannot realistically have them. Justin mentions distributed systems and query planners, and this also makes me think about compilation stacks. You might think there is a nice layered model, but when you look at the details it is often far from being the case.

Automate (But Automate Last)

Matt Rickard talking about the fifth step of Elon Musk’s manufacturing algorithm. Automating too early is one of the mistakes I have seen the most in software companies. Once people see the benefits of automation, they tend to abuse it. Doing that results in poor comprehension of the system and friction when changes are needed.

All of the steps of Musk’s algorithm are relevant to software too, as is the fact that they are in chronological order. I won’t copy/paste them here, go read the end of Matt’s post. As far as I know, its source is Walter Isaacson’s book.

The model is not the product

Matt (again) on how the way we see Generative AI today is only a temporary state. In most cases, raw models will not be exposed to users. We will often want AI use to be more subtle, hidden in the background even. To achieve that we will need a better, more deterministic and flexible interface around the models. This is why I predict a project such as Outlines or LQML is likely to become very important soon.

DHH on zero-interest rates and low productivity

In his Rails World keynote, DHH talked about how the low interest rates we had for over a decade were a likely cause of the success of complex, over-engineered tools and the hyper-specialization of software jobs.

I think this may be true, and if so I hope the pendulum is swinging back.

Toward a shallower future

This is a long and very interesting post. Greatness is often born of terrible situations: misery, war, disease… But then if progress can eliminate those issues, should we do it? Noah Smith argues that yes, we should, hence the subtitle:

Adversity isn’t worth the price of adversity.

You may note that this theme is somehow related to this post. The related meanings of life and progress is a topic I’m thinking about a lot at the moment.

Archive your projects

When you ship something significant you should take some time to either ensure it will be preserved and accessible by you forever, or at least that you can remember it years later, by writing about it and taking screenshots for instance.

Note that I disagree with the “It’s never too late” part. I have not done that for some early work projects in my career, and I regret it now. Do it while you can.

Easy vs Simple

The tension between Simple and Easy has always been an important point for me. Hajime Hoshi gives an interesting definition:

Easy: how quickly you can achieve what you want
Simple: how quickly you can understand the model

Easy is nice to have, Simple is crucial. The effect of “easy” is linear whereas the effect of “simple” is exponential. Never trade the latter for the former.

AI, an energy bomb?

An article in French and paywalled, sorry. But the issue of energy generation and consumption has been on my mind for a long time, and is even more pregnant now. Also a Talos Principle 2 issue!

EU CRA: What does it mean for open source?

Regulation is probably a niche topic for this blog, but it is one I have been following for a long time — ever since I got involved in activism against software patents in the mid-2000s.

This is the best article I have found on the Cyber Resilience Act. Like most EU regulations it is not black and white. If you are interested, go read it and make up your own mind.

The main problem I have with regulation is that the effects are not always easy to predict and not properly re-evaluated. For instance, I believe several laws passed to protect EU citizens and industry against US giants are having the opposite effect. We often forget that added complexity often introduces economies of scale.

There Is No Business Case For Civilisation

OK, there is no question that this one is a Talos topic :)

Not everything in life is a pitch to a VC. Organising human activity purely on the basis of return on investment is a dead end philosophy masquerading as rationalism.

Better just to be brave, hold your head up and say “this is what I believe”, and then apply rationality to how you should shape the universe to realise your values. A great country does great things because of greatness itself.

Don’t write bugs

Some programmers just want to ship fast and don’t care about writing fragile or broken code, they will “fix it later”. It might work for your personal project, but as soon as you work with other people you’re losing everyone else’s time by doing that.

It sounds obvious, but I see people blame reviewers too often. No. The purpose of code review is to discuss design decisions, not find the bugs you wrote because you didn’t pay attention to edge cases or side effects.

Oh, and sure a few unit tests may help too! But they’re not the main point. They’re more a consequence, and a proof that you thought about things, as well as a way to prevent someone else — or future you — from breaking assumptions in the future.

There is no compromise

A good blog post about team culture. In particular, acknowledging that some diversity is good, but that it does not mean there should be no shared culture at all within a team or a company.

The remote vs co-located work example is good. I know great fully-remote companies, and some where everyone works 100% from the office. Many of the people who love the former would hate the latter, and vice-versa. Why would you insist on joining a company that works differently from what you like and change its culture?

Similarly, there are companies where everyone is very close and others where relationships are purely professional. Why would you insist that one or the other is wrong? If you want to join a team, adapt to its culture, or find another one that fits you better.

Culture and values are not something you write on a document and mandate top-down, they’re bottom-up and organic. The best way you can shape culture if you really want to is by picking who you hire carefully. The worst thing you can do is mission someone to change a team or a company’s culture. If you ever are in a context where someone tries to do that and you cannot reason with them, there is only one thing to do: run.

Still using Beanstalk

This is not new, but I am using Beanstalk for Finegrain. I was already using it at Moodstocks over a decade ago. In the meantime, I have used several other queues, but Beanstalk is still my favorite. It works well and it is simple. I love tools like that.

I am using Beanstalk in a Quart application with aiostalk, but I still maintain my Lua client so maybe this will be how I sneak some Lua at yet another company this time… :)

Lightning talks

I started my Saturday with a series of lightning talks.

Writing a consistent-hashing load balancer for the Kong API gateway (ketama principle) (Thijs Schreijer)

Kong is using consistent hashing in the load balancer to ensure a client always gets the same backend, for cache optimization purposes. Thijs explained consistent hashing and the DNS-specific complexities of using it (this talk was in the DNS room). Someone from the audience suggested Rendezvous hashing which I’d heard about but never looked at in-depth, this prompted me to read the Wikipedia article which was interesting.

Magic and Software (Steven Goodwin)

A very fun talk by an actual magician about how they share and protect their tricks. Sadly the amphitheater was far from full, probably because it was the last talk of the day. The video deserves a watch.

You too could have made curl! (Daniel Stenberg)

Daniel’s talks are always great and this one was no exception despite technical issues at the beginning. It was about his experience running the cURL project which is now old enough to drink in Europe. The content of the talk pretty much matches his online book Uncurled which I had read before and recommend.

Introducing ‘Refiners’ – A Micro-Framework for Seamless Integration of Adapters in Neural Networks (Benjamin Trom)

This was a talk by a colleague from Finegrain on our Open Source micro-framework for foundation models adaptation Refiners. Obviously, you should go check it out immediately and star the GitHub repository :)

Five years of Teal: minimalism versus growth in language design (Hisham Muhammad)

Teal, a typed dialect of Lua, is another project I use and occasionally contribute to. This year Hisham talked about upcoming features, and how and why he picked simple but somehow less powerful solutions instead of more complex alternatives.

So you think you know Git (Scott Chacon)

Scott Chacon is an incredible speaker, and the FOSDEM crowd apparently knew it: the amphitheater was full and many people could not get in. The talk was a whirlwind of recent and older lesser-known Git options; watch it and you will certainly learn something.

Toward the end, Scott presented his current project GitButler, a graphical Git client leveraging virtual branches. It was recently open sourced and I tried it earlier this week, because I like the idea which could work very well with my workflow. I found it promising but still too rough around the edges to use it at work for now. I might look at the codebase someday because I am curious about the stack (Tauri and Svelte).

Version control post-Git (Pierre-Étienne Meunier)

A talk about Pijul, a DVCS I know about but never actually used — although I did use its spiritual predecessor Darcs back in the day. It uses a very different approach from Git, representing edits as a graph that does not include the content. Having worked a lot with tree CRDTs (operation-based and state-based) and things like Interval Tree Clocks I can certainly see the advantages but also some of the drawbacks of such an approach. The project also has a fast K-V store in Rust somewhat similar to LMDB and a forge hosting its source code and offering free hosting.

Conclusion

FOSDEM is a bit tiring, but FOSDEM is great. Like I said, the talks sure are nice, but you should also go for the people (and the beer!). This will certainly not be my last FOSDEM, see you next year in Brussels!

One more thing

I wasn’t sure if I wanted to mention this or not, but I ended up editing the post because there is something that bothers me with this function, and it is that business with the oven.

First, pre-heating the oven is self-contained and should probably be a method of the oven itself. But beyond that, this code makes no sense: why would you create a whole new oven to make a pizza? In real life, you get an oven once, and then you bake a whole lot of pizzas with it, without going through the whole heating cycle.

I know this is a synthetic example but this kind of issue actually occurs in real code and sometimes causes performance issues. It is likely that this code should take the oven as a parameter. Providing it is the job of the caller.

(And since you put the pizza in the box, you probably want an interface where you return the box, not the pizza. Oh well.)

Microblogging

First a small announcement: I have joined Bluesky. I like it so far and I think it has a good founding team, so I want to believe in its long-term potential. On the other hand, I will likely stop using Mastodon soon; it is not working for me. I will stay on X for now, but I can see Bluesky replacing it entirely after a while.

The future is Solarpunk

Let’s acknowledge the difficulties of our times but without accepting defeat in the process of trying to solve those issues. […] We are at the point where we can decide if we want to go full Cyperpunk or something more positive. Solarpunk is the movement, the idea behind envisioning and then creating a better future based on humans which live in harmony with nature.

If there is one thing I am certain of regarding climate change, it is that no amount of moderation will be enough to solve it without terrible consequences on a planetary scale. But there most likely won’t be a magical technological breakthrough either. We need to seriously start looking at technologies and ways of life that are both sustainable and acceptable for the whole planet.

On a side note, I also think important technological breakthroughs are not easily predictable in detail. For instance, I think it is highly likely that AI will play a role in fighting climate change, but I currently have no idea how.

Stochastic/Deterministic

Generative AI introduces much more stochasticity into programming. […] Ironically, this makes the deterministic parts that much more important.

One of the first things I did at Finegrain was making sure I knew about all the sources of randomness, and then adding deterministic end-to-end tests.

MiniGPT-4

The authors have taken the visual understanding part of BLIP-2 (a ViT and a Querying Transformer) and aligned it with Vicuna. They achieved that by connecting them through a single 5120x768 linear layer. This illustrates well the unintuitive effectiveness of simple linear projections in high-dimensional spaces.

What took you here won’t take you there

It is hard to grasp for some engineers early in their careers what took them here won’t take them there. Shipping features will probably take you to senior, shipping 10x more features is not how you’ll become a principal. What will get you to principal is removing chaos, unblocking others, fighting for things to be built in the right order, fighting scope creep, having backbone while knowing when to delegate decision making.

– Jaana Dogan (@rakyll) on Twitter^WX

Why it’s necessary to shoot yourself in the foot

To fully understand a “best practice” or why something is necessary, it’s important to experience how things go wrong without it. When teaching programming, we should let people make these mistakes, and [only] then show them the tools to correct them.

The Happiness Advantage

There is a whole book dedicated to the fact that happiness fuels success and (mostly) not the other way around, and how to take advantage of it.

I haven’t read the book (yet) but I have come to that conclusion independently for a while now. I found the book through this post which is one of several about people moving back from management to engineering.

Navigating the unpredictability of everything

Jason Cohen posted this long article on his blog A Smart Bear. It is about how despite - and really because of - us being bad at predicting anything, the best way to move forward is still to have a strategy.

Making a decision and moving forward is often more effective than extended deliberation about the decision. Deliberation assumes we know how to reason about the future, but even experts aren’t good at that. Making decisions and gaining experience is how to find the right answers, if the organization is introspective enough to also face the truth when it turns out the original strategy is incorrect.

Still, some strategies are better than others. One is having several ways to succeed to reduce unpredictability. Perhaps less obviously, doing something seemingly impossible with a very high upside is also a good strategy.

Go read the whole post when you have 30 minutes, it has other motivated advice seen elsewhere such as betting on things that don’t change, looking at what customers do, and keeping things simple. It also features an interesting history of Flickr and Slack.

Growth of AI Through a Cloud Lens

Mitchell Hashimoto compares the growth of AI today to what happened with Cloud Computing and finds many similarities. He also points how that was not the case for Web3 (cryptocurrencies).

Duct Tape Removal Engineer

I love creative job titles and Jaana Dogan found a great one. When you have the experience, you immediately understand what it means.

At Lima my “unofficial” job title was VP of Data Corruption, and Denis was Chief Packet Loss Officer. I never went as far as putting it on my LinkedIn though ;)

The Diamond Age

Scott Chacon tweeted about how he thinks that personalized AI-based tools will change education and his concern that some countries will not adapt fast enough. I share both parts of this opinion.

I have had my own issues with how education works in France and I have been thinking about this for a long time. Neal Stephenson’s book The Diamond Age is a must-read about that topic, and also yet another warning to make sure AI does not end up ossifying the privileges of an elite class.

I want the Primer, but IMO the current breed of foundational models is poorly adapted to the task. We need online training and scalable user personalization. I am hopeful that will get there though, because there are other lucrative use cases with similar requirements.

What is a good engineer?

When I was a student at IMT Atlantique (back when it was still called Telecom Bretagne) I took an optional class about the history of telecommunication systems. It was mostly taught by retired engineers from the field.

It was a fascinating class. A recurring theme was what an engineer’s job was about. One of the teachers, who had been instrumental in the development of electronic switching in Europe, said something among those lines: “a good engineer delivers systems that are scalable, cheaper, and ship faster”.

I have been thinking about that topic lately, and even though I will not deny the “it’s all about trade-offs” line I have been using for years, it is good to remember that those trade-offs should be made only on the optimum curve.

Add to that the idea that engineers should know what systems not to build, and you have my definition of what makes a good engineer.

Using the word We

Karl wrote something interesting about using the word We to talk about a company or a team. He wrote it in French so here is a rough translation:

I often see a company’s employees use the word We (“nous” in French). For a very long time I have been trying to avoid using this turn of phrase in my mail, bugs, etc. For instance, at Mozilla, I used to say “the Webcompat team” or “Mozilla”, or to change the sentence to focus on the product itself.

Instead of “We should add feature H to solve this bug”, I would say “Product A needs feature H to solve this bug”.

When I notice myself unconsciously using “we”, I correct it immediately. But why? To preserve the emotional distance between work and pseudo-belonging to a company.

“We” has a very inclusive meaning of identity for me, and so I avoid it in contractual relationships. The company no longer has a “we” when the time to reduce its workforce for financial gain comes. It is one of my struggle techniques, a small resistance.

I understand what he is saying and it makes a lot of sense, yet I make a different choice. In fact, I cannot fathom working for a company or at least a team where I would not feel comfortable saying “we”.

It’s the same uneasy feeling I have when I see people say that small startups branding themselves as family is a red flag. I understand why they would say that. And sure, we are not family, but when you work that closely with people for years, I think there should be some sense of belonging. Maybe “friends” is a better word.

Also, I am never going to work for a company if I am not aligned with its mission. If I start trying to distance myself from it and notice it, things will have to change one way or another.

Opportunity Cost

This tweet by Jaana Dogan made me remember this blog post by Erik Bernhardsson, which I often find to be true, especially when dealing with startups.

Never attribute to stupidity that which is adequately explained by opportunity cost.

Theory of Mind

A great post by Andrew Bosworth (CTO of Meta) about how some people such as John Carmack can have an influence even after they’re gone.

Chrome, 10 years later

A long post by Evan Martin about his experience working on Chrome from 2007 to 2012. Toward the end, he turns to the state of Chrome and the Web today.

I love this kind of retrospective post, which are full of anecdotes, deeper insight, and food for thought.

By the way, Evan posts interesting things on a variety of topics. I have been subscribing to his feed for a very long time, I think for Arch-related content initially. You may want to look at his other posts as well.

Invest in things that don’t change

Here is DHH saying something I’ve been saying for a very long time as well, but it bears repeating. There is way too much stuff happening to follow it all. If you want to last in this job, especially as a generalist, you must figure out the fundamental things that will matter even more in 10 years than today, and ignore most of the rest unless it’s a deliberate strategy.

CRDTs, Automerge and Braid

Even though it’s not my main focus anymore, I am still following CRDT-based state synchronization systems attentively. Two projects have caught my attention recently. The first one is Braid which has been progressing steadily, and the second is Automerge which has announced a major 2.0 release.

General information on technical ladders

Typical early career levels for software engineers (or developers) are relatively well-known:

• Junior Software Engineer
• (Mid-level) Software Engineer
• Senior Software Engineer

Progression is based on experience and it is expected that a good engineer will reach mid-level after 2 to 5 years of experience and senior between 5 to 10.

Senior developers have two ways to evolve: branch into management or stay on the technical track. This document describes the second option.

Let’s start with a few key things to understand regarding management. A development “team” (project or run) is typically 3 to 10 people. It has a “lead”, which is not a level or a management position. It is a technico-organizational role assumed by a Senior+ developer at a point in time.

The names of levels after Senior differ depending on companies but can be for instance:

• Staff Software Engineer
• Senior Staff Software Engineer
• Principal Software Engineer
• Distinguished Software Engineer
• Fellow

At some large tech companies the “Distinguished” level is equivalent to VP and Fellow to SVP or CXO. At Inch, Staff corresponds to level F. The difference between Staff+ levels is often defined by impact:

• Staff / Senior Staff -> impact at team level
• Principal -> impact at company level (strategy)
• Distinguished -> external impact (aura)

Those levels constitute a pyramid where each stage must be much smaller than the one below (e.g. by a factor of 5 at least) otherwise they stop making sense.

Given the size of Inch, only the Staff level makes sense. A Principal level may not be created until the company is large enough to have several independent technical teams (at least 12 developers, ideally around 30). Consequently, some things relating to strategy and aura are expected from the Staff level at Inch.

Becoming a Staff Engineer

Expectations

References

The SSO Tax

I have had several discussions about how SSO should be priced by SaaS providers. On the one hand, making it prohibitively expensive weakens security for everyone, and there is a website about that. On the other hand, features that distinguish between business categories are often related to security and compliance (audit logs, SLAs, etc).

My personal position - and I do not speak for anyone here including my employer - is the following:

• not having any SSO for free plans is fine;
• any paying customer should be able to use protocols such as OpenID Connect;
• charging extra for SAML support is fair.

Amazon’s Distributed Computing Manifesto

I often say SOA started to become mainstream when Jeff Bezos mandated it at Amazon around 2002, a story famously told by Steve Yegge. But Werner Vogels has shared a document that shows the move began in 1998 and was advocated by engineers.

The Manifesto already emphasized moving data to the process, hiding data models from clients, data consistency issues… A fascinated read, almost 25 years later.

Fix the machine, not the person

This essay by Aaron Swartz is ten years old. A good time to read it again.

An organization is not just a pile of people, it’s also a set of structures. It’s almost like a machine made of men and women. […] True, sometimes you have the wrong gears and need to replace them, but more often you’re just using them in the wrong way. When there’s a problem, you shouldn’t get angry with the gears — you should fix the machine. […] You can’t force other people to change. You can, however, change just about everything else. And usually, that’s enough.

Journal of Comprehensible Explanations

Greg Wilson would like a journal of peer-reviewed summaries of research findings. No new results, just good explanations. Me too!

How I use Dependency Injection

If you have already seen dependency injection used in practice, you may believe it requires the use of a framework such as Dagger. Although I did use such frameworks - especially in Angular where it is native - most of the time I use a simpler form of DI where I just pass dependencies to functions or constructors.

For instance, in Lua object-oriented code, instead of writing this:

local json = require "dkjson"

local obj_mt = {
    __index = {
        as_json = function(self)
            return json.encode(self.data)
        end
    }
}

local function new_obj(data)
    return setmetatable({data = data}, obj_mt)
end

local o = new_obj({foo = 5})
o:as_json()

I could write something like this:

local obj_mt = {
    __index = {
        as_json = function(self)
            return self.json_encoder(self.data)
        end
    }
}

local function new_obj(data, json_encoder)
    return setmetatable(
        {data = data, json_encoder = json_encoder},
        obj_mt
    )
end

local json = require "dkjson"
local o = new_obj({foo = 5}, json.encode)
o:as_json()

Note that this follows the Dependency Inversion principle, which is often confused with dependency injection but not exactly the same thing. It may not be obvious in a dynamic language such as Lua but the implementation of obj implicitly defines the signature of the JSON encoder it expects, and you may substitute the encoder for something else.

Also, on a stylistic note, in a larger program I would probably not pass every injected dependency as a different parameter, instead I would use a single container for this.

In case you’re interested, here is the same example closer to how I’d write it in practice in Teal:

local type JsonAble = {string: any}
local type JsonEncoder = function(JsonAble): string

local record ObjDeps
    json_encoder: JsonEncoder
end

local record Obj
    data: JsonAble
    as_json: function(Obj): string
    _deps: ObjDeps
end

local obj_mt = {
    __index = {
        as_json = function(self: Obj): string
            return self._deps.json_encoder(self.data)
        end
    }
}

local function new_obj(data: JsonAble, deps: ObjDeps): Obj
    return setmetatable({data = data, _deps = deps}, obj_mt)
end

local json = require "dkjson"
local o = new_obj(
    {foo = 5},
    {json_encoder = json.encode as JsonEncoder}
)
o:as_json()

Now let us see a few use cases for this.

Testing I/O

Probably the most well-known use case for DI is simplifying testing of code that does I/O, for instance code that makes network requests or interacts with a database. It makes it easy to replace the problematic part with a mock. You can mock just what you need or the entire dependency, for instance you can replace Redis with fakeredis.

Supporting different implementations of dependencies

This is a less well-known but very good use-case, especially in Lua. Let us consider my JSON example above. Users of this code may want to use dkjson, but maybe they want to use a faster, pure C module, or maybe they are running in OpenResty and have cjson available. With dependency injection, it’s easy to just use it.

Eliminating randomness

This is in fact the use case that prompted me to write this blog post. I was writing some algorithmic code in Teal for which I had a reference in Python, and I wanted to apply the dual-implementation comparison technique I like. However, this was AI code that had random parts.

What I did was that I refactored the code in both languages so that all sources of randomness were injected, and then I injected deterministic pseudo-random values instead, which meant I could now compare the state of both implementations at all times and easily debug things such as numerical problems.

When not to use DI

DI is a great tool, but it tends to make code a bit more complicated, so for instance I do not use it to inject pure functions or libraries from the same codebase.

If you have used Angular with Storybook, you may know what abuse of DI looks like. It can be super verbose because sometimes you need to inject a lot of things you do not really care about for it to work.

A way to work around that if you still want to use DI may be to have sensible defaults for some injected objects. That way you can still override them if needed, but you can still have simple sample code.

Polling

Suppose you are responsible for a “server” system. That system generates events (messages), and authenticated “clients” need to receive those messages.

A simple solution is to provide some kind of feed: the client authenticates and provides a small piece of state (a timestamp or a cursor) which allows the server to return the messages they haven’t processed yet. Ideally individual messages also have properties which allow the client to process them in an idempotent way, e.g. skip messages it has already processed.

EDIT (2022-04-04): I was made aware through HN of a spec called HTTP Feeds which looks like a very good format for those feeds. Something like Atom can still work too.

The client can then just process the feed in a loop, in pseudocode:

cursor = INCEPTION

loop do
    messages, new_cursor = fetch_messages_since(cursor)
    for message in messages do
        if not processed(message)
            process(message)
        end
    end
    cursor = new_cursor
    sleep(DELAY)
end

This solution is called polling, but the main problem with it is the DELAY variable. If it is set too low, both the server and the client will keep doing work and exchanging data for nothing, but if it is too high messages won’t be received by the client in a timely fashion.

Server Push

If we want message reception to be realtime, we may use a push-based design instead. In that model, the client starts by registering with the server. The server will then send the messages to the client as they arrive.

In practice you might not really want to send each message on its own so you will send message batches instead and client code looks like that:

def messages_handler(messages)
    for message in messages do
        process(message)
    end
end

register_with_server(messages_handler)

This design, however, has a few issues as well.

First, there is the bootstrap problem. The client might need messages that arrived before they registered. The server could theoretically just send all the previous messages to every client on connection, but in practice this is almost never what you want because of volume, performance, etc.

Even if you don’t need to bootstrap, what should the server do when it tries to send a message to the client and it doesn’t work? It could be that the client has gone down, or it could just be a network issue. In some systems the server keeps the failed messages for a while and retries, but that complexifies the design because it means the server needs to keep per-client state.

Finally, in practice, authentication is often more complicated that way. The server needs to authenticate the client, of course, but the client also needs a way to make sure the data it receives comes from the server. Here we might need to take a few real-life examples: the client could be a single-page app, in which case the push channel would be something like SSE or websockets; it could also be two HTTP servers communicating in which case we are talking webhooks and might need some ad hoc signature scheme. And by the way, the GitHub scheme does not have any form of protection against replay attacks, which might or might not be a problem for your use case…

Push-to-poll

The push-to-poll pattern combines the best of both worlds. The idea is simple: start with the same feed as polling, but instead of using a delayed loop to decide when to fetch it let the server tell you like with the push pattern.

It looks like this:

cursor = INCEPTION

def poll_now()
    messages, new_cursor = fetch_messages_since(cursor)
    for message in messages do
        if not processed(message)
            process(message)
        end
    end
    cursor = new_cursor
end

poll_now()
register_with_server(poll_now)

It solves the problems from the poll design, because there is no more arbitrary delay. The server tells clients when to poll, which might be every time a new message comes, or with a slight delay to optimize the number of requests and ensure messages are processed in batch. Clients might decide not to poll immediately when they receive a server push as well.

There is no more bootstrap problem or reliability problem since it is built on the poll design: every time you poll, you catch everything up. If you want to be extra safe and messages are very rare, you can decide to poll even if you didn’t receive any message after a reasonably long delay.

The authentication story is better than in the push case, because pushes contain no data. All data access uses regular, client-calls-server authentication. It is still a good thing to authenticate server pushes (for various reasons I won’t go into) but mistakes there have less impact, especially if you have client-side rate-limiting.

There are many small tweaks that can be added on top of that design, for instance you can include a little bit of metadata in the push messages to let clients decide whether they should actually poll or not.

Finally, note that you get polling for free! Implementers who do not need realtime can just use the feed endpoint with the polling pattern.

The only drawback is that it takes an extra request for every exchange, but in practice I have seen very few cases where the server keeps message history where it would not have been an improvement over server push. So I hope you think about it next time you consider serializing data into websockets or webhooks.

My own posts

Let’s get started by mentioning posts from this blog which transcribe the opinions of two influential people, which I share (on those topics).

• Werner Vogels on Skills, where the CTO of Amazon.com describes the qualities he is looking for in leaders at Amazon, in particular Ownership and the willingness to get their hands dirty doing operations.

• Hacker Founders, where Paul Graham, founder of Y Combinator, explains how programmers have an edge as company founders, including as CEOs.

Titles are Toxic

Michael Lopp (Rands) on issues with giving people titles. Personally I have never liked titles, which can be a source of conflicts and bad decisions. I have also often held titles that didn’t really correspond to the job I did - which was almost always multi-faceted. Yet, titles are a proxy used by people who don’t know you to understand what you do at work.

Like Rands I don’t have a good solution to that. I write some blog posts explaining what I do in more details that a title ever could, for people who care.

In any case I always suggest avoiding giving some titles to people too early, including to founders: having a “technical co-founder” instead of a CTO will make things easier if you grow fast and need to hire someone later on.

Creating a Software Company? 9 Decisions You Have to Make

A list of questions to ask yourself when founding a software company, by Michael Marsiglia, CEO of Atomic Object. It can also be a good idea to ask them before joining one.

The 40% rule

A simple but powerful rule of thumb for SaaS companies by Brad Feld, relayed by Fred Wilson:

annual revenue growth rate + operating margin should equal 40%

Why it’s nice to compete against a large, profitable company

Jason Cohen explains how a startup can disrupt a large competitor by making a part of their business unprofitable and forcing them to abandon it. It’s not as simple as this summary makes it sound, and it’s not the Innovator’s Dilemma. Read the article :)

Product design at GitHub

Kyle Neath on how, in the early days of GitHub, everyone was a “product designer”. Many successful SaaS products started with the same concept:

We only hire smart people we trust to make our product better. We don’t have managers dictating what to work on. We don’t require executive signoff to ship features. Executives, system administrators, developers, and designers concieve, ship, and remove features alike.

Actually, I think I have never seen a company ship good product fast without doing some version of this.

Joel Spolsky’s Strategy Letters

If you don’t know that series of blog posts, you are missing something. They are super insightful essays by a multi-successful founder about building companies.

• Ben and Jerry’s vs. Amazon - How to decide between slow profitable growth or fast VC-funded growth.

• Chicken and Egg Problems - Understanding the Chicken and Egg problem and some ways to solve it.

• Let Me Go Back! - On barriers to entry, and how letting customers understand they can easily move out of your product (maybe back to what they used before) is important.

• Bloatware and the 80/20 Myth - 80% of software users use 20% of features, so you can win the market with just 20% of the features, right? No, because not everyone uses the same 20%! I can’t state how often I meet founders who don’t understand this.

• Strategy Letter V - How companies have an advantage in commoditizing their complements, and how this can explain some investments in Open Source software.

(There is at least a sixth strategy letter, but it is less interesting IMO.)

Ev William’s 10 rules for Web Startups

This post used to be on Ev William’s blog which went offline some time ago, thankfully I mirror articles that interest me that much.

Your company’s most valuable resource

Just a tweet, but an important one, by William Pietri:

Your company’s most valuable resource is people giving a shit. Ask yourself: does your system encourage or discourage that?

Wants, needs, and chasm-crossing

A recent post by Avery Pennarun, Tailscale CEO. It shows how the best strategy to acquire customers is often to solve a single Want - something their current solution doesn’t provide - and then to resolve all their Needs (blockers) so they can actually flock to your product instead of attempting to implement any more Wants.

Financial modeling for startups

A slide deck by Rodrigo Sepulveda with a simple and effective approach to business models and PnLs for startups.

Startup Therapy: Ten questions to ask yourself every month

Jason Cohen (again) shares his alternative to business plans: a list of 10 questions to help you figure out where you are and where to go next.

The Small Teams Manifesto

Jason Gorman says small teams win because:

• the probability that a project fails increases rapidly with size;
• small teams interact more effectively with the customer and/or end users;
• small teams are easier to build (hiring good people is hard).

Innovating with purposeful anger

One of the very interesting tools using anger as a driver to innovate is the “You Got Fired!” game.

Get a bunch of managers in a business unit and role-play that their whole team is fired. Indicate also that as a cockup from legal their non-compete clause isn’t applicable.

Then, give them an hour to build a competitor startup that will take revenge on their company on a “That’ll teach them!” mode.

How To Decide What To Build

Daniel Gross (a founder I have been following since his first YC startup, Greplin) on ways to decide what company to build on a personal level. I wish he had published that before my last attempt.

Qu’est-ce qu’un projet entrepreneurial viable?

A blog post in French by Philippe Silberzahn. A viable project stems from an individual acting on an idea and convincing people.

Dealing with integrations

That is my main role, and the reason I was hired in the first place. See, our software is used by our customers to manage the relationship with their own customers and suppliers, but they invariably already have pre-existing sotware with overlapping data: their accounting software or ERPs. Some have built them in-house, but most picked from a very fragmented market of industry-specific software which we have to integrate with. So I maintain somewhere between 20 and 30 different integrations with those pieces of software you have probably never heard about.

You may not understand the complexity until you realize that none of those integrations looks the same, because the software itself doesn’t. Some is cloud-based, some is on-prem. In some cases they provide APIs or flat-file data export mechanisms, but sometimes we have to go get the data directly in their database. Of course, we almost never have technical documentation for their data model, and the databases themselves are all different. I have code for over 10 different major DBMS, not all of them using SQL, and not all of them running on typical OSs (hello AS400, hello SCO…).

Add to that integration with a few APIs we use, the maintenance of our own APIs so others can integrate with us and a few custom features for large customers, and you will have an idea of the scope. It is a role that requires domain knowledge, technical insight about many systems and the ability to understand new ones quickly. And maybe most importantly, I have to model our own software and systems so that they can support all of this. If you think about it it’s all about synchronizing data between systems, so I am not all that far from my specialty after all. :)

This role means that when we talk to large customers or prospects I am often the person representing the technical side of the company (hence the CTO thing).

Operations and security

You know the pattern now: I arrive at a startup with a small team of developers where nobody has significant operations experience and I end up becoming responsible for them. It never bothered me, quite the opposite actually: I have been doing systems administration for 20 years and I have always liked it.

In this case, there had been some people who knew what they were doing on the team before, so it was not that bad. For instance, they had a reasonable backup strategy already (yay!).

One of the first initiatives I took was look at spending, which was too high, rationalize a few things and get the hosting bills down. Also, Inch had four main hosting providers for historical reasons: AWS, GCP, Digital Ocean and OVH. The plan was to move everything to GCP, for simplification and compliance reasons. I’m still not done but I got out of Digital Ocean and reduced our usage of AWS by a lot.

We still had three servers left at OVH when their DCs burned last March. We lost two which served as integration gateways, and I took the opportunity to move them to GCP. Losing those servers has made me very busy in the last two months because their IP adresses were all over our customers’ and partners’ config, so I had to get in touch with them all to fix it, since there was sadly no way we could get back the IPs from OVH in time. That won’t happen again now, I used reallocatable IPs this time. The third server will remain at OVH for the time being because it is a very special machine with very specific hardware hosting needs.

At small companies like this, operations also come with a lot of security-related responsibilities. Security is pretty important for Inch, which deals with a lot of personal data. So far, I have worked on improving our operational practices, added a team password manager, rationalized internal auth{n,z} around the GCP tools, and written some documentation. I have also upgraded several important software components and reviewed / fixed all the usual configuration (systems, TLS, CORS, CSP…). I have also started working on the Web application security itself, I have less experience in that space - especially regarding the frontend - than in systems security but I am learning and we have a few experienced Web developers on the team to help.

As usual, nothing is ever done in that space and I still have a lot of work coming up, but at least I believe we’ve significantly improved since I joined.

Fullstack Web development

Finally, the third leg of my role is the same as every developer at Inch: working on the product.

I know the “fullstack” wording makes some people cringe. Of course we all have our favorite part of the stack but I think its is extremely important, especially on small teams, that everyone be able to write or at least understand a feature in its entirety.

With the other two roles taking much of my time, I don’t do as much feature work as the rest of the team but I still try to implement features myself regularly. I had never done frontend stuff seriously before Chilli and it was a different stack so I learn a lot about React and its ecosystem. I do my share of performance and bug fixes too. On the backend especially, I also do quite a lot of architecture and code reviews.

Other things

Besides those three main parts of my role, there are of course other things I take part in. Inch encourages its employees to participate in many cross-cutting activities.

An important one is that we all do support in rotation, which means the whole team ends up knowing the customers and identifying the parts of the product that need improvement. This also means everyone can, and does, participate in product decisions.

But that doesn’t stop at product. Inch founders are very transparent about the business, our finances and future plans. Everyone is also encouraged to chime in on strategic decisions for the company.

This article is already long enough so I’ll stop here. I hope that gives you an idea of what I’ve been doing at work since November 2019. If any of this interests you in any way, don’t be shy, get in touch. :)

Shutting down Chilli

I haven’t posted about it here yet, but my co-founder Julien and me decided to shut down Chilli at the end of last year, in agreement with eFounders. Basically, some of our initial hypotheses about the state of the market we were in (french SMBs) were wrong and the business couldn’t be profitable enough.

In retrospect, it was an interesting experience. I really appreciated being part of eFounders, I do recommend it for people who want to start a B2B SaaS company.

However, I wanted to get out of my comfort bubble too much with this one, by tackling a problem in a market I didn’t know. Because of that, I had to rely on others regarding core business choices, and while that was fine with me initially it ended up being frustrating in the end when things didn’t work well and I couldn’t help much. So if I start another company someday, it will be one where I know the problem domain better.

Joining Inch

After we shut down Chilli, I decided to join a small startup editing a SaaS CRM and ticketing solution for property managers called Inch, only available in the French market for now.

I didn’t pick Inch randomly, I have known one of the founders for years since he was a Moodstocks user! Fun fact: at the time, I had met him and he was looking for a technical co-founder for his project. I told him he should learn to code instead and gave him Ruby sample code… and this is why I am back to Rails today. Karma? :)

Anyway, I had been following Inch with interest since its creation, because it is the company I like: solving a real need in a market where tools were either terrible or non-existent. Now they have a unique place in the market and some interesting technical challenges to solve, so I decided to join and help.

Starting a family

And here comes the last piece of news: despite all the madness that happened this year, the biggest change for me is that I am now a father! My son was born last week, and I took a holiday until the end of the year to spend time with him and his mother. I don’t intend to post too much personal news here, but this one deserved it. =)

Complements

It was a post on Lobsters that prompted me to post this. Someone from the comments and a former colleague on Twitter told me about a pass extension I didn’t know about which does almost the same thing.

Also, some people think that putting 2FA codes in a password manager defeats the purpose. But in practice TOTP 2FA does not really add much more to the security of my accounts than the strong random passwords I generate with pass. The “second factor” part isn’t really the true benefit.

One actual advantage is that nobody on the network can sniff all of my credentials (like digest-based password verification methods). Another, and I think this is the main one, is that the owner of the website has chosen part of the credentials and hence ensured some degree of strength. What I do preserves both of those properties, so I’m fine with it. By the way, note that password managers like 1Password and Passbolt do the same thing.

One thing you can do to improve the security of the whole thing is use 2FA to access pass by storing the GPG key in a Yubikey.

Moodstocks (2010 - 2013)

• Fastmail - E-mail
• GitHub - Code hosting & issues
• Basecamp - PM
• Campfire - Chat & Customer Support
• Salesforce - CRM
• Desk.com - Customer Support
• Gandi - DNS
• Uptime Robot - Alerting
• AWS - Hosting
• Mandrill - Transactional e-mails
• Kong - API Gateway

Lima (2014 - 2019)

• GSuite - E-mail & Calendar
• GitHub - Code hosting, issues & PM
• Flowdock then Slack - Chat
• Lucca then Payfit
• Segment - Customer workflows
• customer.io - Customer base, emailing
• Zendesk - Customer support
• Invision - Design work
• OneSky - App translation
• Talent.io - Hiring
• Gandi - DNS
• Datadog - Monitoring
• Uptime Robot - Alerting
• Pushover - Alerting
• Sentry - Error monitoring
• AppVeyor - Windows build & CI
• Travis - Build & CI
• Firebase (Fabric…)
• AWS (+ many others including OVH and Scaleway) - Hosting
• Sendgrid - Transactional e-mails
• Cloudflare - CDN

Chilli (eFounders, 2019)

• GSuite - E-mail, Calendar, Drive
• GitHub - Code hosting
• Slack - Chat
• Slite - EDM
• Airtable - Process modeling
• Pipedrive - CRM
• Trello - PM
• Payfit - Payroll, Leaves & absences
• Spendesk - Expenses
• Zoom - Video calls
• Intercom - Lead acquisition
• Aircall - Centralized phone calls
• Moqups - Design work
• Invision - Design work
• Google Analytics - Analytics
• Squarespace - Blog
• 1Password - Team password management
• Codility - Hiring (screening)
• Hiresweet - Hiring (sourcing)
• Workable - Hiring
• Gandi - DNS
• Uptime Robot - Alerting
• AWS - Hosting
• Mailjet - Transactional e-mails

Inch (2019 - 2023)

• Google Workspace - E-mail, Calendar, Drive
• GitHub - Code hosting & issues
• Slack - Chat
• Notion - EDM
• Hubspot - CRM
• Payfit (previously Lucca) - Payroll, Leaves & absences
• Mixpanel - Analytics
• Intercom - Customer support
• Aircall - Centralized phone calls
• Uptime Robot - Alerting
• Rollbar - Error monitoring (backend)
• Sentry - Error monitoring (frontend)
• Scout - APM
• GCP (+ OVH) - Hosting
• Mandrill - Transactional e-mails
• Xero - Accounting
• Postman - API documentation
• Pipefy - Customer lifecyce
• Vonage and smsmode - Transactional SMS

Finegrain (2023)

• Google Workspace - E-mail, Calendar, Drive
• GitHub - Code & documentation hosting
• Linear - Project management and issues
• Slack - Chat
• Notion - EDM
• Uptime Robot - Alerting
• Sentry - Error monitoring
• Vercel - Frontend platform

Personal

• Fastmail - Email
• GitHub - Code hosting
• Pinboard - Bookmarks
• Gandi - DNS
• Uptime Robot - Alerting
• Vultr - Hosting

Forming opinions

People who have spent significant time around me know all too well how I behave when I get interested in a topic. I can spend weeks (and sometimes even much longer) reading all I can find about it until I know its ins and outs. This results in me being able to discuss and have opinions about things which are apparently completely random, but which I ended up picking up along the way for some reason.

For the longest time, I have believed in strong opinions (somewhat) weakly held, and for the most part I still do. In particular, I agree with the “somewhat” part, which says that the strength with which you should hold an opinion depends on how you formed it.

What has changed in the last 5-7 years is my view on how explicit those opinions should be, and how much I should fight for them.

Expressing opinions

I was strongly influenced by the Open Source community of the early 2000s where, if you wanted something to happen, you had to state your opinion and be ready to defend it - sometimes ferociously - with arguments. If you didn’t, you would be ignored.

What outsiders often overlook about this method is that it worked, and it was pretty efficient. Most people got used to it, and in general it did not turn into endless debates, because in the end we had benevolent dictators to settle the matter.

But a problem I could not see for a long time is that this excludes a lot of people from the discussion. People who, because of their education or the position society put them in, won’t express their own opinion if it contradicts the current consensus or that of someone they either respect or fear; or people who won’t engage in anything remotely resembling conflict because that’s not in their nature.

It is especially easy to ignore the existence of these people when the only way you are communicating is through asynchronous text over the Internet. However they often have very interesting things to say, with different points of view that can make headway or prevent big mistakes.

There are lots of ways to work around that issue, including timeboxing contributions on a topic while keeping them secret, or simply having people in positions of power and people more comfortable with their opinion express it last.

Defending opinions

Another matter is how much you should fight for your opinion. I am not talking about the famous XKCD comic, I know I can be this guy sometimes, but that is something else.

Some time ago I was struggling with how to handle disagreement at work, and I became an adept of the way Amazon does things, its Leadership Principles, and in particular Disagree and Commit:

Leaders are obligated to respectfully challenge decisions when they disagree, even when doing so is uncomfortable or exhausting. Leaders have conviction and are tenacious. They do not compromise for the sake of social cohesion. Once a decision is determined, they commit wholly.

When organizations follow this principle, it helps a lot with the issue, because it makes it clear that you can - and should - express dissenting opinion, and how you can align on things you do not agree with without making it look like you changed your mind when you did not.

(On a side note, I even think in some cases making dissent mandatory by instituting a Tenth Man / Devil’s Advocate rule can be beneficial.)

However, it turns out this principle is more complicated than it sounds, and there are two important points I did not immediately understand, which are much more explicit in Bezos’ 2016 letter to shareholders.

The first is that the person who “disagrees but commits” is not necessarily the subordinate in a power relationship. Bezos says:

This isn’t one way. If you’re the boss, you should do this too. I disagree and commit all the time. We recently greenlit a particular Amazon Studios original. I told the team my view: debatable whether it would be interesting enough, complicated to produce, the business terms aren’t that good, and we have lots of other opportunities. They had a completely different opinion and wanted to go ahead. I wrote back right away with “I disagree and commit and hope it becomes the most watched thing we’ve ever made.” Consider how much slower this decision cycle would have been if the team had actually had to convince me rather than simply get my commitment.

The second point I missed is the meaning of the sentence “they do not compromise for the sake of social cohesion.” The “compromise” part is clear enough; here compromising would have meant giving the green light but with, say, fewer resources. From experience compromises like this often end up poorly. But the hard part is “for the sake of social cohesion”. Here is what Bezos has to say:

Note what this example is not: it’s not me thinking to myself “well, these guys are wrong and missing the point, but this isn’t worth me chasing.” It’s a genuine disagreement of opinion, a candid expression of my view, a chance for the team to weigh my view, and a quick, sincere commitment to go their way.

Unlike Bezos my current (weakly held) opinion is that sometimes keeping social peace is worth not engaging in some minor issues. Some people will always appreciate being told when you think they’re mistaken, but others won’t even if they end up admitting they were wrong in the end, so it only makes sense to contradict them if it is worth risking being resented for it.

Tolerating opinions

The first step for all this to work is probably to admit that opinions can exist without being “right” or “wrong”.

That may be obvious to you but for a long time it was not for me. When I was a kid I looked at things in binary: true or false, right or wrong, better or worse. This made me enjoy CS and math, but paradoxically learning more advanced math (partial orders, Simpson’s paradox, Gödel’s incompleteness theorems…) showed me that even in the hardest of sciences things were more nuanced.

Anyway, there are reasons why people come to hold a set of beliefs which makes sense at least locally. Learning about this background is important, whether it helps you convince them or changes your own mind.

Don’t worry though, I still hold a few strong opinions, both in my field (some quite strongly - you won’t easily make me change those for instance) and outside of it!

April 2020 update

This article is out of date. Arch Linux stopped shipping OpenSSH with socket activation due to the risk of DoS attack. Now you can just set Port in sshd_config as usual.

Original article

I often change the default SSH port from 22 to something else on servers I run. It kind of is a dangerous operation, especially when the only way you have to connect to that server is SSH.

The historical way to do this is editing sshd_config and setting the Port variable, but with recent versions of Arch Linux and the default configuration, this will not work.

The reason is that SSH is configured with systemd socket activation. So what you need to do is run sudo systemctl edit sshd.socket and set the contents of the file to:

[Socket]
ListenStream=MY_PORT
Accept=yes

where MY_PORT is the port number you want.

I hope this short post will avoid trouble for other people, at least it will be a reminder for me the next time I have to setup an Arch server…

What happened to the company exactly?

Lima as a company no longer exists. It ran out of money. Its employees (including me) have all been fired, and its assets will be sold to pay its debts.

Regarding why the company died, it is a long story and it is not my place to tell it all. What I can say is that it ran into unexpected funding problems in early 2017, shortly after we started shipping the Lima Ultra. During most of 2017, there was strong demand for the product but we could not fulfill it because we did not have enough cash to pay for production and shipping (Remember the never-ending waiting list?) At the end of the year, we had to fire a large part of the team and we switched our business model to sell our software to other companies. We made a deal where we worked for another startup. The deal was good enough to keep the company afloat and the product alive for a year, but it forced us to stop selling Lima devices. What happened recently is that this deal eventually fell through, leaving us with no viable options.

This past year was not the best time of my life, or for any of the other employees who stayed. Many of us could have left for much better jobs at any time, some did and I cannot blame them. All those who stayed on board all this time did so hoping for a better end for the company and its customers.

What will happen to the devices?

Once Lima’s servers shut down, Lima will keep working on your local LAN with the devices you have already paired with it. However, a lot of things will stop working.

First, it won’t be possible to add new devices to the system. That’s because, when you log a new device into Lima, you do so with an email and password. To find out which Lima those credentials belong to, the system asks a server, and that server won’t answer anymore.

Second, it won’t be possible to reset your password, because email confirmation will be broken. If you have forgotten your password, change it now while the servers are still up.

Third, the sharing feature will be broken, because it relies on sending HTTP requests to relay servers which will go down as well.

Finally, it won’t be possible to access Lima from outside your home. This is a little harder to explain than the rest. Basically all communications between anything related to Lima (Lima devices, your devices, servers…) happen in a peer-to-peer VPN. To “locate” devices within the VPN (basically figure out how to talk to something), devices rely on a node which is called the ZVPN master. The IP address and public key of that node are hardcoded into every Lima client, and that node will go down as well. The use of that node is not needed on local networks because Lima devices and applications have a protocol to pair with other devices associated to the same account on a LAN without talking to any server.

Is there a risk for my personal data?

At that moment, not that I know of. Your data was never stored on Lima’s servers, and all data traffic going through relay servers is end-to-end encrypted, which means that even if an attacker took control of one they couldn’t decipher your data.

However in the long run there are two issues.

First, we won’t be able to publish updates for the Lima firmware and applications anymore. If a security issue is found in one of the components they use, they may become vulnerable with no way to fix them.

Second, if someone was to acquire all the assets or Lima, including the domain and code signing certificate, they could theoretically do everything Lima was able to do, including publishing updates. That means they could publish malicious updates of the applications and firmware.

That second issue sounds scary but I do not think there is any chance it will happen. Potential acquirers will probably be more interested in Lima’s technological IP, there are very few chances that an acquirer will get all the assets necessary for such an attack, and even if they do they probably won’t have an interest in performing it. Even if it did happen, it would be easy to notice. Still, I have to mention it for transparency.

What I will personally do now, and what I advise users to do as well, is export all my data out of Lima, unplug the device and uninstall all the applications.

Note: If you have problems when trying to recover your data (due to e.g. a hardware issue with the USB drive), do not uninstall the applications. The data on your desktop might sometimes help recovering some of the files.

If you have an issue with the Decrypt Tool, check here for potential answers.

EDIT (2022-02-06): What I feared when I wrote this post did happen a while ago. Someone did get hold of the meetlima.com domain and uses it to host what is clearly a scam. They want you to think they are us, but they are not. I do not think they have performed any attack on the devices themselves, but it is definitely not safe to keep using the original software anymore.

What can users replace Lima with?

It depends on the users. I don’t know anything that is exactly like Lima. There was Helixee, which I have never tried out, but I just found out they are shutting down as well. I also learned that a project I had never heard about before called Amber had a special offer for Lima customers.

For technical people, you can probably do most of what you were doing with Lima with a Synology NAS, or a setup based on some small computer and Open Source software such as Nextcloud or Cozy Cloud.

However, Lima was never designed for technical customers. It was built for, marketed to and mostly bought by non-technical people. For them, I don’t have a good answer. I heard that WD My Cloud Home had become a lot better than it once was, but I have not tried it personally.

Can you open-source the code?

To the best of my knowledge, there is no way that can happen. This makes me extremely sad, especially since I know there are parts of the code I would love to reuse myself, and that could be useful to other projects.

The reason why we cannot open-source is that the code does not belong to us, the employees, or the CEO. Intellectual property is considered an asset of a bankrupt company, and as such will be sold to the highest bidder to pay the company’s debts.

That being said, Lima has contributed some source code to a few Open Source projects already. Most importantly we fixed the issues in OSXFUSE that prevented it from being used for something like Lima, and those fixes are now in the main branch.

Completely independently from the company, the former CTO of Lima has also released a project which looks a lot like a second, fully decentralized iteration of the Lima network layer ZVPN (using a DHT instead of a master node, and WireGuard instead of TLS). Let me be clear: this project contains no code or IP from Lima, it is a clean room implementation.

Can you give us root access to the device?

For Lima Original, no, I think that would be impossible (or rather, I can’t see a solution that doesn’t involve soldering…). The device is not worth much today anyway, its specs are so low I don’t think you could run any other private cloud software on it.

For Lima Ultra, a few of us ex-Lima employees (and the CEO) are trying to figure out a way to let users get root access. We can’t promise anything, but we will keep you informed if we do.

EDIT (2019-02-18): We did it, check this out!

Why does it say something different in the Kickstarter FAQ?

Some people have mentioned that what was happening was not in line with what had been said in the Kickstarter FAQ.

This FAQ has been written in 2013, before I or any other Lima developer joined the company. At the time Lima was a very small project with two founders trying to raise $70,000 to make their dream happen. Instead they raised $1,229,074, hired 12 people (including me), and the rest is history.

I do not think we have not communicated like that ever since, especially regarding decentralization. As far as what I know we have been transparent that our servers were needed for some major features of the product, as it was obvious the few times they went down. You may ask why we didn’t amend this page then, and the answer is (I think) that it is technically impossible to edit it after the campaign is over.

Regarding Open Source, I sincerely believe the CEO of Lima would have done it if it was possible, but with the success of the Kickstarter the company had to take VC funding very early on (see below), and from that moment on I do not think it was in his hands.

Where did all that Kickstarter money go?

OK, let’s address this last. What Kickstarter money?

Yeah, the founders raised over a million dollar. But do you remember how much the backers paid for those devices? From $59 to $79 each. Well, as bad as the hardware was, it was planned for about 1000 devices, not over 10,000. And it was pretty expensive.

I don’t know the exact figures, but basically Lima did not make money on those devices, or no significant amount of money at least. Which is why it raised extra cash from VCs just afterwards, to pay the team that worked on the project, the production of more devices to sell, etc…

If you still think something shady went on with that money, rest assured: when a company like Lima goes bankrupt, its books are closely investigated by the state, which is one of its main creditors. So if you are right, the people responsible will end up in jail. (Spoiler: I really don’t think it will happen.)

What are you going to do next?

Yes, I have plans.

No, they are not in any way related to Lima.

I will tell you more next month, probably.

Distributed filesystem synchronization

For the public, Lima makes a device which turns a USB hard drive into personal “cloud” data storage. We actually shipped two devices, Lima Original in July 2015 and Lima Ultra in Decemner 2016.

Technically, Lima is really a personal distributed filesystem. Every single machine running a Lima application logged into to the same user account is a node in the system. All nodes keep a copy of all the filesystem metadata, which means the user can always browse the whole file tree, create or remove files, move them around, etc. The actual data, however, may or may not be on the device, which makes it possible to access a multi-terabyte filesystem on a mobile device. In theory, the Lima device is just another node in the system, except it always stores all the data.

To make this all work, we need a mechanism to keep the filesystem metadata synchronized on all nodes. I have designed and implemented the algorithm and protocol behind this. It is tricky, because we always want users to be able to write to their filesystem, even on an offline device, so when devices come back online there may be conflicts to resolve. It is sometimes tempting to do something simple and correct, but which would not result in what the end user would expect. Yes, distributed systems are a UX problem…

I can’t get into too much detail here, this being proprietary technology and all, but I have drawn a lot of inspiration from CRDTs as well as older systems like Bayou and WinFS.

Minicore: fitting into a tiny box

Like I said, in theory, the Lima device just runs a slightly modified Lima application. And in theory, theory and practice are the same thing. But in practice…

The Lima Original hardware was very limited, with a slow exotic CPU, 32 MB of RAM and 8 MB of Flash. At some point we realized that we would never succeed in running our software on it with reasonably good performance, so shortly before shipping we decided we had to do something radical: write entirely different software for it. We called that software “minicore”, and I wrote its first version in a 6-week rush.

Unlike the Lima filesystem, minicore is designed to be very conservative about its use of memory and I/O resources. It uses a different data model and synchronization protocol, which relies on client machines running Lima to do the heavy lifting. It prioritizes what Lima devices have to do best, which is downloading or serving file data.

The trade-off here was a huge increase in the complexity of the system. Minicore itself is not that complex, but we had to introduce a separate protocol between client nodes and Lima devices, and a third protocol to deal with replication between Lima devices (if you add several Lima devices to the same account they mirror one another).

Other things

I won’t get into too much detail about all the other things I have written or worked on over the last four years, but important ones include:

• our build system, which builds our software and its dependencies on five different OSs and three major architectures ;
• an integration test suite designed among other things to replicate timing-related edge cases for the synchronization protocol ;
• our “identity server” which lets users log into their account with just an email and a password ;
• our Lua scripting layer, which I didn’t implement initially but maintain now, and on which we rely more and more.

What’s next

Lately, my role has changed as I have become responsible for software architecture for the whole company. I now work more closely with the developers outside of my team (Core Engineering), who implement the user-facing parts of our stack. I also do some hiring - and by the way, if you are interested, get in touch. :)

All in all, it was four interesting years. Things have not always been easy to say the least, but a relaxing job is not what I expect from a startup anyway. We have some really great people on the team, challenging plans for the future, and we learn a lot, so here’s to the next years of Lima!

On Call

Alice Goldfuss started a debate about having developers on call. I am firmly in the camp of those who think it is a good thing, probably because I was a sysadmin before being a software engineer. But this means those developers must have the power to refuse to ship broken code that may make the pager go off as well.

In fact, it is all about fostering the sense of ownership, which is the N.1 trait I look for in developers. And this won’t be the first time I have quoted great people saying similar things!

The value of metrics

Here is something people who haven’t worked in B2B software companies may not understand: sometimes knowing your product’s metrics, how its users behave and the market it is operating in can be more valuable than the product itself. Small B2B2C software vendors often end up knowing things about their (huge) customers’ clients that the customers themselves don’t.

What that means is: if you provide third-party software to your users, ask the third-party vendor what data points they collect, what they can do with them, and if they can feed them back to you. And if you are the third-party vendor, consider the value of that data, and maybe find clever ways to turn it into a competitive advantage.

Why did the robot do that?

Explaining why AI makes decisions is going to be very important in the years to come. Trusting algorithms is hard for humans. This slows down adoption and progress the most in areas that really matter. So we need a way to let concerned people “feel” why machines do things the way they do it. Because writing papers is nice, but we all know simply telling people to trust experts is never going to work. This means those topics are now also UX problems.

Can’t you just…?

Someone made a nice web page to answer a question we hear pretty often. :)

EDIT: That was about cantyoujust.no which does not exist anymore.

Applications and Data

Here is something I have been believing in for a long time: there is no good technical reason why most Web-based application vendors force you to store your data on their servers. It should be possible to use, say, a presentation application from Google to edit slides stored on Microsoft’s servers.

Sadly, for reasons that probably have a lot to do with business models, things like remoteStorage did not really take off. But I hope someone will revisit that idea someday, and find a way to make it take off.

Thoughts

I have not found the motivation to write here for a while, so I will be trying something new. This is the first of a series of posts which will be composed of a few smaller unrelated snippets, mostly things I would usually post in a smaller form on Twitter. They will relate to anything I will have found interesting recently. Some will be practical and some will just be reflections about a given topic.

MSYS2

If, like me, you are mostly a Linux user but sometimes have to work on a Windows machine, have a look at MSYS2. It allows you to quickly get a decent environment set up, with Bash and a package manager, (pacman), which you can use to install things like Git.

Sailfish

I have had my Jolla C for two months now. I have not spent as much time as I would like playing with it, but so far I am pretty happy with Sailfish. Like WebOS, it relies heavily on gestures, which makes most of the features of the phone usable with the thumb of a single hand. There are not many applications in the store so far but the default applications are good in general and most Android applications work well, I could run Lima on it without any problem for instance.

From a programmer’s point of view, it is all I hoped for. I went from zero to Hello World on the device in less than an hour. Coding happens in QT Creator and the UI of the application is described with QML. The SDK ships with an emulator that runs in VirtualBox, but also with a second virtual machine which is used to compile code, which means you don’t have to install a cross-compiler on your local machine. Applications, which are technically RPM packages, can be deployed to hardware devices via USB or SSH over WiFi.

It also has to be noted that the native terminal application is very good, much better than anything I saw on Android. A SSH client is included so it can occasionally be used to log into a server.

The main reason I have not made that device my main phone yet is that its battery life is much worse than that of my Android phone, but hopefully that is something that can be solved.

The fractal dimension of software

I stumbled upon an insightful tweet about software development time estimation. The larger the tasks we try to estimate, the more wrong we will be. Maybe an estimate as a single number representing time is not enough; maybe we should also try to figure out the fractal dimension of the job to be done.

Distributed systems are a UX problem

Everyday I grow more convinced that this is true. Not only of failure conditions, but even more so of things like reconciliation strategies in case of causally concurrent events, for instance how to represent data with CRDTs.

It is a problem, because people doing distributed systems often either do not want to do UX, do not have the competence to do it, or just do not have the permission. We tend to be at the bottom of the stack, and some people think it would be a disaster to let us work on anything user-facing. On the other hand, most people actively doing UX do not understand or care for distributed systems issues.

The result of this is that, if we design without taking technical constraints into account, we end up with systems that cannot be implemented without relying on central authority. Add to that the fact that it is almost always much easier to implement something in a centralized way, and you have one reason why the Cloud is winning and distributed alternatives to popular services never seem to take off.

Palm

I stumbled upon two fantastic documents related to Palm recently. The first is a series of blog posts called A History of Palm. Very interesting, even though I disagree with the conclusion of the fifth post. What killed the Pre line at HP was not that WebOS did not take off; they never let it a chance to. I have been using a Pre 3 for a long time (I had pre-ordered it so I was one of the lucky few to have it in France) and it was a fantastic device, extremely influential on future evolutions of both iOS and Android (after all, the interface was the brainchild of Mathias Duarte). What ended Palm’s legacy was the eviction of Mark Hurd and his replacement with Léo Apotheker, who hates hardware. Probably one of the worst mistakes the HP board ever made.

The second document is the Zen of Palm, a design guide for Palm OS from 2003 which is still pretty relevant today in some ways.

Is preemption the root of all evil?

Two decades years ago cooperative scheduling was more frequent than preemptive scheduling. Windows 3.1 was completely cooperatively scheduled, which means that all applications had to yield to the system scheduler, or they would block the whole system. In Mac OS 9, the threading API was cooperative as well within a process, even though different applications would be scheduled preemptively by the kernel.

Modern OSs do not make much distinction between a thread and a process anymore, threads are basically processes which share memory, and they are scheduled preemptively. I have come to think that this design results in code of lower quality, and which is harder to reason about. Of course, it is still possible to use event loops or programming languages which do their own preemptive scheduling (like Lua)…

Portability

The first thing to know is that the data.mdb format is platform-specific, which means that you cannot necessarily open a database created on a machine on another one. In practice, on ARM and x86, the only thing you have to care about is whether the machine is 32 or 64 bits. On a 64 bit machine, you can read the data from a 32 bit database by exporting it to a text format using the mdb_dump utility compiled in 32 bits and reimporting it with a 64 bits mdb_load. With lua-mdb, you can read a 32 bits database directly by passing the bits option to the constructor.

Pages

A data.mdb file is made to be mapped into memory, and as such is organized into a set pages of 4 kB. All pages start with a header which contains the page number and some flags which determine the type of the page. For all page types except overflow pages, the header also includes the bounds of the free space in the page.

Meta pages

The first two pages (page numbers 0 and 1) are called meta pages. Meta pages start with a magic number (0xBEEFC0DE), a version (currently 1), and the address and size of the memory mapping. The address is only valid if the mapping is fixed, i.e. if LMDB is used with the option MDB_FIXEDMAP.

After that, we find two structures describing what LMDB calls databases. The first one is the “free” database, which is used to track free pages, and the second one is the “main” database.

Finally come the number of the last page used in the file and the ID of the last transaction that wrote to the page successfully. Indeed, those meta pages is the mechanism LMDB uses to implement MVCC. There can be a single writer at a time, and it will always write to page ID N % 2, where N is the current write transaction ID. Meanwhile, readers will read page N % 2 + 1, providing readers - writer isolation. This means that, when opening a .mdb file, the latest data can be found in the databases pointed to by the meta page with the highest transaction ID.

Databases

As we saw, a meta page contains two structures describing databases. Databases in LMDB are technically B+ trees. The structures contain some environment flags used at runtime, information about the tree including its depth, the number of pages of each type and the total number of entries, and finally the page number of the root of the tree.

Pages of the tree itself can be one of three types: branch pages, leaf pages and overflow pages. There are two other types (BRANCH2 and SUBP) which are only used with specific DB options (MDB_DUPFIXED and MDB_DUPSORT) and that we will ignore.

Branch pages

Branch pages represent internal nodes of the tree. After the header, they contain an array of “pointers” (technically, 16 bit indices within the page) to “branch nodes” within that page. The number of branch nodes, which is the number of keys in the page, can be deduced from the boundaries of the free space in the header.

Branch nodes are allocated from the bottom of the page up. They start with a page number, then a key prefixed by its size. Branch node pointers are sorted according to the key and the current key comparison function, and the page number indicates the page which should be visited to look up keys between the key of the node (included) and the key of the next node. That page can be a branch or a leaf.

Leaf pages

Like branch pages, leaf pages start with a number of pointers to leaf nodes. Leaves store the actual keys and values contained into the DB. Small values are stored in the leaf node itself and large values are stored in overflow pages.

Leaf nodes are slightly more complex than branch nodes. They contain flags, which determine if the value is contained in the node or in an overflow page, and the key which corresponds to the node. For small values, they contain the value itself, and for larger values they contain the page number of the corresponding overflow page.

Overflow pages

Large values are stored in so-called overflow pages. Those pages are pretty simple: after the header, they contain the raw value (not prefixed by its size). This value can be larger than the size of a page, which explains the name: they overflow on later pages. Note that this means those later pages do not start by the usual header.

Learning more

I have explained the basics of the data.mdb file format. Should you have to debug broken databases, it will help you get started. If you need more, read the lua-mdb source code or the LMDB source code (it is only about 10000 lines, although sometimes not easy to understand given that its author adopts a style that favors performance to readability).

The Year Without Pants - Scott Berkun

Scott Berkun was a management writer and consultant when he was hired by WordPress. Before that, he had been a UX designer and team lead at Microsoft, on the Internet Explorer project. When he joined WordPress, the company had over 50 employees, it was completely flat and had a culture of remote work. The founder wanted to experiment with a team-based structure, and Scott was to be the first team lead.

The book is not a story about how WordPress was a doomed company, and how Scott came in to save the day. On the contrary, WordPress was already working much better than most companies that size, and Scott was part of a series of changes they implemented to scale to the next level without losing too much of their culture.

That culture, rooted in Open Source and documented by Berkun, an outsider, is what makes the book interesting. I would say it is a must read for anybody considering starting or leading a team at a small software company.

The Open Organization - Jim Whitehurst

This book is somehow similar to the one above, but on a larger scale. Jim Whitehurst is the current CEO of Red Hat, a company with over $1.5B in yearly revenue, over 8000 employees, and a culture firmly rooted in Open Source (I started using Linux with their distribution, Red Hat 6.2, in 2000).

Before joining Red Hat, Whitehurst was COO of Delta Airlines. When he arrived at Red Hat, he quickly realized it was a company that worked in a very different way, following the core values of Open Source, transparency and meritocracy (Whitehurst uses the word with the meaning it had in the Free Software community in the 90s and 00s, without the background that makes it controversial nowadays).

The book is both analytical and prescriptive, and it shows how the way Open Source projects are organized can be successfully adopted by a large company.

The Systems Bible (Systemantics) - John Gall

A 1975 classic. Gall is a (retired) pediatrician, but he is well-known in software engineering circles for this book which contains, among other things, what is now known as Gall’s Law:

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work.

The book is a curious mix of humor and philosophy, written in faux-scientific style and full of things which people working in Distributed Systems will be painfully aware of:

In complex systems, malfunction and even total non-function may not be detectable for long periods, if ever.

Any large system is going to be operating most of the time in failure mode.

Intermittent failure is the hardest case.

One does not know all the expected effects of known bugs.

In setting up a new system, tread softly. You may be disturbing another system that is actually working.

Bad design can rarely be overcome by more design, whether good or bad.

A system that ignores feedback has already begun the process of terminal instability.

In dealing with large systems, the striving for perfection is a serious imperfection.

Remember: this book is originally not about computer systems. Not at all. But read it anyway!

The Four Steps to the Epiphany - Steve Blank

I have read several other books about Lean Startup, but this is the one I would recommend now, especially if you already have some experience with startups.

It exposes the Customer Development model, composed of four steps: Customer Discovery, Customer Validation, Customer Creation and Company Building. The most interesting ones, in my opinion, are the first two, the ones that occur before Product / Market Fit.

Read this book if you want to understand what this Lean Startup thing is about, and offer it to your friends who want to start their own business.

Flash Boys: Not So Fast - Peter Kovac

This book is an answer to Michael Lewis’ Flash Boys and a defence of High-Frequency Trading.

I have long thought that the financial world used HFT as a scapegoat (especially since I read Chris Stucchio’s blog). Flash Boys: Not So Fast follows the structure of Lewis’ book; it provides a nice overview of what HFT is and why it may not be as evil as you think it is.

There is No Now

Most distributed systems are asynchronous, which means that the usual notion of time does not make sense within them. This article by Justin Sheehy, former Basho CTO (now at VMWare), published in ACM Queue in March 2015, is about that fact and its consequences. It inspired me to make this list, because as soon as I read it my first thought was that I should share it with all of my colleagues and every single person I will ever work with. It introduces two important impossibility theorems (FLP and CAP) and adversarial models, discusses the existence of “reliable” clocks and networks, and touches on some solutions such as consensus protocols and CRDT.

Eventually Consistent - Revisited

This 2008 blog post by Werner Vogel (Amazon CTO) explains the trade-off between availability and consistency, introduces consistency models, and hints at the relationship between the replication factor of data and the level of consistency that can be achieved. If you want to go further after that, you can read the famous Dynamo paper (which is not too complicated), preferably in its version annotated by Basho.

Distributed systems for fun and profit

This document, by Mikito Takada (Trifacta), is longer than the others. It is the best short (meaning: not a whole course or book) introduction to distributed systems I know. In a nutshell, it expands on the ideas presented in the first two documents and gives you most of the necessary concepts and vocabulary to understand actual distributed systems papers, should you want to read them.

Building on quicksand

I could hardly leave that 2009 paper by Pat Helland and Dave Campbell out of this list. It starts with the idea that, as the distribution of systems increases, latency makes synchronicity intractable and pushes us towards asynchronous designs. The rest of the paper is a discussion of the trade-offs involved. The very interesting idea in that paper (also found in other papers by Pat Helland) is that it all comes down to a problem of risk management and reconciliation. If asynchronous software can not ensure something will happen, it may make a guess and fix the result later if it was wrong. That may include having to make excuses to an actual human being. An important corollary is that the distributed nature of software permeates all the way to the user interface (which is why you may have seen me ranting about how we need to start forming cross-functional teams, with developers who understand UX and designers who get distributed systems).

Bonus: Why Logical Clocks are Easy

I said on Hacker News that, if I wrote that article today (in May 2016), I would add this article to the list. I was advised to edit the original blog post, so I did.

Published in April 2016 in ACM Queue, this article by Carlos Baquero and Nuno Preguiça may be the best introduction to causality I know. Causality is a very important concept in distributed systems. In a nutshell, the idea is to answer the question: given two events, could one have influenced the other? In other words: when the entity performing the second event did, dit it have any knowledge that the first happened? This article introduces causality and gives usual ways to represent it in theory, with causal histories, and in practice, with vector clocks and version vectors as well as their dotted variants.

Conclusion

When I started programming as a child, I was hooked by the idea that the computer was a perfect machine that, unlike a teacher, would never be unjust: it would work if I got the code right, and if I did not I just had to fix it and try again. Once it worked, it would always work! Everything was reproducible, and all issues could be diagnosed and understood.

Of course, I grew up and the messy reality caught up with me. I went into distributed systems, which are a lot more like physics than maths. There are laws that govern the world out there, and they are always coming in your way. Everything works as expected, until something unexpected happens. And sometimes, you will not even be able to explain what it was after the fact.

Distributed systems work is about that adversarial world out there, and how we write programs to cope with it. It is about dealing with the unreliability of clocks, communications, hardware and, sometimes, people. It is about reading papers, drawing diagrams, writing proofs sometimes, and finding solutions to obtain the guarantees we want. More than anything else, distributed systems is the science (or art? or game?) of trade-offs. It is hard, error-prone and usually broken, terribly complicated sometimes, but it is my field, and so far I like it.

Request/Response

This is the pattern everybody knows about from HTTP, and probably the workhorse of most SOAs. A client sends a request to a cluster, and a single node of the cluster answers it. Iris just does the messaging here: the request and the response are binary blobs, there is no notion of headers.

Here is how it looks in Lua on the client side:

local iris = require "iris"

local c = iris.new()
assert(c:handshake(""))

local req = c:request("echo", "hello", 1000)
c:process_one()
local r = assert(req:response())

c:teardown()

print("reply arrived: " .. r)

and on the service side:

local iris = require "iris"

local c = iris.new()
assert(c:handshake("echo"))

c.handlers.request = function(req)
    print("request arrived: " .. req)
    return req
end

for i=1,5 do c:process_one() end

c:teardown()

Publish/Subscribe

Another well-known pattern. Consumers subscribe to a channel identified by a name, producers send messages into the channel and all subscribers receive them. There is no response from the subscriber. Channels are completely independent from clusters, i.e. channel “X” and cluster “X” can coexist and have nothing in common, and there is no restriction that a channel only works within a given cluster (any entity on the same Iris network can access it).

Here is a publisher to channel “somechan” in Lua:

local iris = require "iris"
local socket = require "socket"

local c = iris.new()
assert(c:handshake(""))

for i=1,5 do
    c:publish("somechan", "message " .. i)
    socket.sleep(1)
end

c:teardown()

and a consumer:

local iris = require "iris"
local socket = require "socket"

local c = iris.new()
assert(c:handshake(""))

c.handlers.pubsub.somechan = function(msg)
    print("message arrived: " .. msg)
end

c:subscribe("somechan")

for i=1,5 do c:process_one() end

c:teardown()

Broadcast

We have seen that requests are sent to a single instance of a cluster. What if you want to notify all members of a cluster of some event? This is what Broadcast is for. It is kind of redundant with Publish/Subscribe (you could have all members of each cluster subscribe to a channel) but it is somehow cleaner to have a separate message type for this. You could use it to build more complicated things on top, for instance a full-fledged service bus.

Here is how to send a broadcast message in Lua:

local iris = require "iris"

local c = iris.new()
assert(c:handshake(""))

c:broadcast("bcst", "hello")
c:teardown()

and how to handle them:

local iris = require "iris"

local c = iris.new()
assert(c:handshake("bcst"))

c.handlers.broadcast = function(msg)
    print("message arrived: " .. msg)
end

for i=1,5 do c:process_one() end

c:teardown()

Tunnel

Tunnels are the last and most complicated way to communicate provided by Iris. To understand why we need them, remember how requests work.

The fact that any member of a cluster can answer a request prevents you from doing anything stateful, because you cannot know if you are talking to the same instance or not in two separate requests. Moreover, the nature of the request and response does not allow any of them to be composed of multiple parts sent in separate messages. A request cannot trigger multiple responses.

Tunnels are a solution to all that. When you open a tunnel, you address a cluster, but what you obtain is a persistent, bidirectional, ordered pipe to a specific instance of that cluster. You can think of it as a TCP socket or a circuit if you come from a telecom background (although there are no latency guarantees). Tunnels implement a throttling algorithm so that both ends of the tunnel can specify a maximum size for their input buffer.

For now, here is what I think of tunnels in the context of a SOA: they are very powerful but tricky to use. If you can avoid them, you should. If not, you should not use raw tunnels anyway: you should define your protocol, write an abstraction for it and never expose the tunnel itself to the application.

One valid use case I can see for tunnels, and which is used in Iris examples, is streaming. But this can go much further. Like the name indicates, you could tunnel most protocols into them, so they could serve as the foundation for some kind of VPN.

Anyway, here is an example client with multiple responses and state:

local iris = require "iris"

local c = iris.new()
assert(c:handshake(""))

local tun = c:tunnel("tunnel", 1000)
tun:confirm()
tun:allow(1024)

tun.handlers.message = function(msg)
    if msg == "data" then
        print("got data")
    elseif msg == "continue" then
        print("got continue, sending request")
        tun:cosend("request")
    elseif msg == "bye" then
        print("got bye, quitting")
        tun:close()
    else
        print("got invalid message: " .. msg)
        tun:close()
    end
end

local xfer = tun:transfer("request")
while not xfer:run() do c:process_one() end

local op
while op ~= iris.OP.TUN_CLOSE do op = c:process_one() end

c:teardown()

and the server on the other end:

local iris = require "iris"

local c = iris.new()
assert(c:handshake("tunnel"))

local MAX_COUNT = 3
local count = 0

c.handlers.tunnel = function(tun)
    tun:allow(1024)
    tun.handlers.message = function(msg)
        if msg == "request" then
            if count < MAX_COUNT then
                count = count + 1
                print(string.format(
                    "got request, sending %dx data + continue",
                    count
                ))
                for i=1,count do tun:cosend("data") end
                tun:cosend("continue")
            else
                print("got request, sending bye")
                tun:cosend("bye")
            end
        else
            print("got invalid message: " .. msg)
            tun:close()
        end
    end
end

local op
while op ~= iris.OP.TUN_CLOSE do op = c:process_one() end

c:teardown()

Conclusion

Iris is a very interesting piece of software which implements a vision of systems that I like and cannot find in any other product out of the box. It is still very young and probably not ready for serious production yet, although I have found it stable during the (limited) testing I have done. Even though my current job does not involve SOA I will probably come back to it someday, so Iris will join the list of tools whose progress I follow attentively.

On a side note, Iris was built for its authors’ thesis, and he is now looking for a sponsor to allow him to continue working on it. If you have the power to make that happen, give it a look. Think how VMWare / Pivotal must be happy of the deal they did with Antirez :)

Finally, if you try my Iris client and find bugs, do not hesitate to report them on Github! I have no real world Iris system to try it on so there are certainly plenty.

Writing a Lisp

I like to learn new tools and concepts by experimenting with small projects whose sole purpose is to help me grasp something better. Working on a larger project in a team with other people is invaluable, but doing things on a smaller scale on my own offers a different perspective.

Having recently started to write a lot more C at work than I used to, I felt that it was a good idea to do a smaller project in the language. It turned out Build Your Own Lisp came out just when I needed it, so I had a go at it.

I cannot recommend this book enough if you feel like you would enjoy building a dynamic language in C from scratch (using just a parser library). It is one of the best tutorial-style books I have ever read. It achieves a perfect balance between being didactic and leaving freedom to the user.

My Lisp ended up being different from the one described in the book in several minor ways. For instance: it has more types, builtins take expressions instead of values, values are based on a union (which saves memory), memory management works differently, the source more organized… In that respect the book delivers exactly what it promised: the reader is encouraged to build “his own” Lisp.

In the end I got just what I wanted from this experience: I clearly improved my C and had the satisfaction to write a working programming language in a few hours scattered over three weekends.

Oh, just in case you were wondering, nobody paid me to write that blog post!

Moving on to Go

Next, I will probably be learning Go. Go is a weird language in that about half of the technical people I follow and look up to like it a lot, and the other half hates it. The latter tend to be users of languages with stricter type systems…

I have to admit that, if I had looked solely at the technical merits of both languages, I would probably have learned more Rust instead. I already know some Rust, but not enough to use it productively. Its main advantages are its (arguably) better type system and the fact that it can run without a GC or a frontend, making it suitable to write dynamic libraries.

However, Rust doesn’t look completely stable yet, whereas Go is already used in production by several serious companies. Also, I am a distributed systems programmer, and most of the interesting codebases I see popping up around in that field are written in Go.

Moreover, Go appears to be a simpler language than Rust. I suspect I can learn enough of it to read code “fluently” and write some much faster. So Go it is, and Rust will probably be next.

Reproducible Builds for Debian (Jérémy Bobbio)

Debian developers want to provide their users a way to verify that the binary packages they distribute correspond to the source. To achieve reproducible builds they have to patch code that depends on things such as timestamps at build time. To make things worse, using a standard VM for builds would help but they refuse to do it “because they’re Debian.”

Is distribution-level package management obsolete? (Donnie Berkholz)

The Gentoo leader thinks distribution are doing a poor job of meeting the needs of users - especially developers and system administrators - in terms of package management. They are increasingly relying on configuration management tools (CFEngine, Puppet, Chef, Ansible…), language-specific package managers (RubyGems, NPM, LuaRocks…) and tools like Docker; however, distribution package managers do not integrate well with those.

This is a topic that interests me, following all the discussion that occurred at the last Lua Workshop. After the talk I asked Donnie whether there was a discussion list somewhere dedicated to those issues. He told me that, to his knowledge, there was not. It may be a good idea to start one.

Minix 3 on ARM (Kees Jongenburger)

Kees talked about the Minix 3 architecture, how it was ported to ARM (with the BeagleBoard and BeagleBone devices as first targets) and plans for the future.

The main priority of the project is porting the NetBSD userland. As a former user I think this is an excellent idea. I may try the OS again someday if it becomes more “usable”, I really like some of the ideas behind it.

Technical introduction to the deeper parts of Sailfish OS (Carsten Munk)

Sailfish is probably the most interesting mobile OS project today, go check it out if you don’t know it yet. Carsten gave us an overview of the main parts of the OS and explained some funny things they did, such as making glibc and the Bionic libc co-exist within the same process. He also provided the audience an SSH access to an actual terminal. I found liblua 5.1 on it :)

Postfix: lessons learned and recent developments (Wietse Venema)

This talk introduced the Postfix least-privilege architecture, and then went on to explain recent improvements. Most of them revolve around fighting spam more efficiently. Postfix also migrated from Berkley DB to LMDB (see later), mostly for licensing reasons.

NixOS: declarative configuration Linux distribution (Domen Kožar)

Nix OS is a Linux distribution built around the Nix package manager. It is one of those few distributions that completely disrupt the FHS, another one being GoboLinux.

The idea of Nix is that a package is the output of a function provided with some arguments and without side effects. There are a lot of interesting ideas there, such as atomic updates based on symlinks. However, doing this involves some heavy patching and that makes me uncomfortable.

Iris Decentralized Messaging (Péter Szilágyi)

Iris is a messaging backend written in Go but with a language-agnostic interface. It is somehow similar 0MQ but goes further by removing the notion of a specific instance or process in favor of sets of those. Each machine runs a single daemon that does all the heavy lifting, and each client connects to it locally to interact with the system.

This is a very interesting project and I was a bit surprised I had never heard about it until now. The slides and live demos were also really good, with multiple Go code snippets running concurrently in a browser. Oh, and Gopher drawings!

Camlistore (Brad Fitzpatrick and Mathieu Lonjaret)

The speakers showed off what the latest release of Camlistore, a personal file storage system, can do. I already knew and liked the technical design of the project, but I was impressed by the progress made on usability since Brad’s talk at dotScale 2013. They have a new Web UI which is not complete yet but looks very promising. I really have to install my own node someday.

Statically compiling Ruby with LLVM (Laurent Sansonetti)

A very interesting talk on LLVM and how it helps when statically compiling a (very) dynamic language such as Ruby. Among other things, the optimization passes do an incredible job. However, it looks like JIT compilation is not as good as static compilation yet.

What’s new in OpenLDAP (Howard Chu)

This was, as I had hoped, mostly a talk about LMDB. It confirmed what I already suspected: at least according to its author’s benchmarks, it outperforms any kind of competition in this space (embedded key-value stores). It is also one of the rare NoSQL DBs to implement MVCC transactions, in less than 10000 lines of C code and 32 kB of object code. If there is a piece of Open Source software I think you should not have missed in 2013, it is this one.

Persistent Memory: Changing the way we store data (Ric Wheeler)

An interesting talk about how the shift away from rotating disk towards persistent memory storage affects the way we develop filesystems, kernels and any code that does I/O. This is not only about SSDs, but also about new kinds of devices that will come out soon and be an order of magnitude faster according to Ric.

Concurrent programming with Python and my little experiment (Benoit Chesneau)

Benoit explained how he ported the Go concurrency model (Goroutines and Channels) to Python. I know Benoit and his offset project so this was not foreign to me, but there are two pieces of interesting news: first, the next version will support multiple processes, freeing users from the GIL (yay!), and second he is thinking about changing the API to make it more like Julia and less like Go, because Julia is more similar to Python.

NSA operation ORCHESTRA: Annual Status Report (Poul-Henning Kamp)

This was a fun talk where phk endorsed the role of a NSA agent who mistakes the FOSDEM amphitheater for the European Commission and explains how they sabotage attempts to give the general public more privacy. Lots of conspiracy theory in there obviously, but given the recent events, is this really so absurd? phk argued at the end of the talk that the solution should be political and not technical.

The format of the talk was a very good idea that worked really well on my brain tired by two days of conference, but also by two consecutive nights out filled with Belgian beer and other strong drinks. :)

Conclusion

FOSDEM is still a very good event which you should consider attending next year. Besides the talks, you will enjoy the parties and meeting people you only knew online. I hopefully will see you there.

Goodbye Moodstocks

Friday 20th will be my last day at Moodstocks. I am leaving a company where, after three years and a half, I was the most senior employee. As you may imagine, it was not an easy decision.

Moodstocks has grown up since I joined it. After several pivots, the founders have assembled a great team I will miss, and together I dare say we have advanced the state of mobile image recognition. Recently, we have also been looking at other technologies related to mobile.

We have released two new products. The first one is what was known as Physalis and took the commercial name Winch. This is what I have been mostly involved with for the last few months. I believe it has the potential to become the reference solution to write native mobile applications that react quickly and work offline.

The second product is Overlay, a mobile application that lets you buy products from paper catalogs online. Years after Pikadeo and Notes, it is finally time for a real application available in the stores, leveraging Moodstocks’ image recognition SDK and Winch, showcasing the best of both technologies. But beyond pure tech, it is also a demonstration of Augmented Reality as we like it: fast, predictable, purposeful. No need for 3D models.

With those two products in the pipes, choosing to leave was incredibly difficult. But I did nevertheless, because of an opportunity I could not turn down.

Hello Lima

So, what next for me? After two weeks to see my family, eat too much and have some welcome rest, I will be joining the team developing the Lima as a core developer.

The Lima, which promises no less than to solve the personal data storage problem, is the first hardware product I backed on Kickstarter. I was thinking: “this won’t be easy, but if those guys can do it, I want that device.”

The Kickstarter went (very) well, so they started looking for help and got in touch. I was curious and decided to meet the founders. After a Saturday morning spent discussing the project at Starbucks, I was convinced: they are the right people to do this.

For years I have been convinced that ubiquitous computing is one of the next big things in technology and wanted to be part of it. As I see it, Lima gives me this opportunity today.

So it looks like my short-term future holds some Unix development and a good dusting of my rusty C. I may try to sneak a Lua interpreter in that box at some point, but please don’t tell the others! ;) Severin and Gawen have high expectations and I do not expect much relaxing but hey, it should be fun to live the adventure of Early Stage again.

Harvest, the Pikadeo crawler

When I joined Moodstocks in April 2010 the team was working on a mobile price comparison application called Pikadeo. The pitch was that you could take a picture of any cultural product (CD, DVD, book…) and it would give you a list of places where you could buy it, sorted by price.

The iOS application itself, largely designed by Louis Romero who had interned at Moodstocks and left just as I arrived, was working. The image recognition technology was working too, although it was purely server-side. The team was already researching how to leverage client-side processing but it was really just a crazy idea at that point, so Pikadeo was doing what most “mobile” image recognition software still does: send JPEG frames to the server.

What was missing was the data. We needed to crawl several large e-commerce websites, extract product images and metadata, send the former to the image recognition engine and the store the latter in a database. So I set out to write a crawler in Ruby, which was the dynamic language of choice of the team at the time. Moodstocks was a Ruby / C++ shop due to the background of the founders. Obviously things have changed a lot since then.

I tried to use Hadoop for the job, mostly because it was trending at that time and I had access to Amazon’s Elastic MapReduce. I soon understood that 1) the Hadoop Streaming interface was not quite there yet so I would have to switch to Java and 2) the Map/Reduce paradigm was not the best for the job anyway.

After reading a few papers on crawling (I had to anyway, since that project would be the basis of my MSc thesis, but it actually helped a lot) I ended up writing a kind of Master/Worker system, with work queues in Beanstalkd and metadata storage in Amazon’s SimpleDB, which did the job. It did the job a little too well, actually, since it ended up DDoSing an e-commerce website for a few seconds during a performance test for my thesis. Fortunately I was monitoring it and hit the stop button…

After setting reasonable speed limits and balancing the requests between various websites, Harvest was fast enough for our needs. The bottleneck became the image search engine itself, I will expand on that later.

Due to the deprecation of the Pikadeo product, Harvest is no longer used today. It is probably a good thing: it was my first Ruby program so the code was awful, it was way too complex and too tied to AWS (the master instance would run and kill worker instances, it relied a lot on SimpleDB…). That being said, the crawling model was sound.

Acorn

Once we got crawling sorted out, the image recognition engine itself became the problem. Oak (it has become a tradition to use plant-related names for our projects internally) had been almost entirely written by Cédric, Moodstocks’ CTO. It was a piece of C++ software, with the image recognition parts isolated in dynamic libraries and a Thrift layer to interface with the core Ruby on Rails Web application. It was multithreaded, designed to run on a single multicore EC2 instance.

The scalability pain point, it turned out, was not CPU load. The index was stored in a B+ Tree in Tokyo Cabinet, a tool we like a lot and still use today in other parts of our system. The problem was that when we indexed millions of images the dataset would inevitably become very large, larger than the available memory. The system would still be very responsive on most reads, but writes would invalidate large chunks of the in-memory cache and result in long pauses.

Latency is the enemy when you write image recognition software, so we decided not to sacrifice it: all the index had to fit in RAM. We decided to consider RAM as our primary datastore. That decision would bring about our later choices.

Since rebuilding an index can be very long we wanted something that could persist even if the engine crashed or had to be restarted for an update. Soon it became obvious that Redis could be the answer. However it was missing some commands that we needed, especially one that would insert the same key with different values in different maps (if you have already written inverted indices you may understand why, otherwise have a look at that presentation). Lua scripting was what we needed, but it wasn’t there yet so I ended up forking Redis to develop it in C while lobbying for scripting support. Acorn was, to my knowledge, the first application to run Redis 2.5 in production, and the first one to use Redis scripting too. We never encountered any Redis-related crash.

Now, to have the index fit entirely in RAM, we would have to distribute it across different machines, so Acorn would have to be a distributed system. Knowing that, I chose to make Acorn nodes single-threaded: they would communicate by message passing and we would have several of them per instance.

We chose MessagePack for serialization of Redis values, and I started looking at MessagePack-RPC. It had a lot of the pieces that I wanted for the distribution part, and one major problem: it was only usable in Ruby. But we were not CPU-bound… Would it be sensible to write the engine in a dynamic language? I started investigating that possibility. Our C++ libraries already had Ruby bindings that we used for vision R&D, and the little number crunching Oak Core did (mostly different scoring algorithms) turned out to be fast enough in Ruby.

So Acorn ended up as a distributed system written in Ruby, with MessagePack-RPC for communication and a fork of the development branch of Redis at its heart. It used MessagePack-RPC for communication with our Rails stack too.

In retrospect, relying on two unstable pieces of software was risky. It turned out well, and Redis was definitely the right choice, especially since Lua Scripting now allows us to use a regular, stable 2.6 version. 0MQ would probably have been a better technical choice than MessagePack-RPC, and plain C better than Ruby, but I believe those choices saved us development time, and time to market was important. Acorn is still in production today, doing its job for legacy clients who use online recognition.

Moodstocks Notes

You now know that when I joined Moodstocks it was trying to be a B2C company. However we were seeing interest in licensing our technology, and began to envision a B2B product: Moodstocks API.

When you write an API, especially as a product, you must write applications for it simultaneously. They serve two purposes: demonstrate what your API can do, and help you figure out how it should be improved. We set out to do that with two mobile applications, one of which was Notes.

The original idea I proposed was, I think, simple: Google SideWiki (RIP) for the real world. That is: you walk in the street, you see something interesting, you take a picture of it, you get a comments thread. If you are the first to do so, you get to leave the first comment (yay, first!!1).

As we were looking to add virality to it, that idea developed into a kind of mostly mobile social network where both people and objects could be followed. Objects actually had their own timeline with an associated Atom feed, which you could reach by browsing or, of course, as the result of an image search.

Technically the server-side part of Notes was a rather classic Sinatra application. The most interesting part of it was that it used some kind of CQRS architecture with all reads coming from Redis and all writes going to log-structured storage. The very nice thing about it was that any part of it could be replayed so it was almost trivial to reproduce bugs or replicate production incrementally to a development setup.

The iPhone application, on the other hand, was one of the nicest and most complex ones we have ever written. I wasn’t responsible for it so I won’t get into details here but the latest internal version we never actually released was IMO a thing of beauty.

As it turned out, Notes got a reasonable amount of online press after our CEO showed it to Michael Arrington at the Le Web conference. This got us a few users and we briefly thought about making it a product in its own right. I wrote a wxPython GUI to analyze logs, trying my hand for the first time at techniques like cohort analysis.

Eventually we took the decision not to invest more time in the idea: we were a small team and our now core B2B business needed our attention. Notes’ success was a long shot and would have required significant time and money investment so I guess it was the right decision, although I would love to see someone revisit the idea.

Moodstocks API v2

As I said, Notes was written to help us design our API. Using its feedback and that from the few users of our v1 API, which was more some kind of beta, I set out to write version two.

I will not expand too much on all its aspects here, REST-ish API design being well covered in the literature and online (start here).

The main differences with API v1 were the use of JSON instead of XML, and the ability to index a single image by uploading it to the API using multipart post. Previously, users would upload a XML list of image URLs and associated IDs; we would download them and tell you when indexing was over. Now users index single images and changes are taken into account instantly. The necessity for that was a lesson from Notes and user feedback, and it was made possible by Acorn.

Another interesting choice was the authentication method, HTTP Digest, which we kept from version one. Theoretically, it had all the right properties and was a standard, so it was the best choice. What we had not realized is how many implementations were broken or incomplete (i.e. not supporting nonce reuse, which is a necessity on mobile to reduce the number of HTTP requests). I ended up having to submit patches to a lot of them, and I am not even mentioning .NET land… If I had to do it again today I would probably go with Basic Auth and SSL.

Acorn Quantizer

Earlier, I wrote about how I had made Acorn processes single-threaded. This had some advantages, but also a big inconvenient.

Part of the image search process involves quantizing features, which means associating vectors in a many-dimensional space to integers. To do this the curse of dimensionality forces you to use an approximate nearest-neighbor search algorithm.

The way it works is: take a large number of features from a representative dataset and use some kind of clustering algorithm (e.g. k-means) on them to obtain a bunch of centroids (a “vocabulary”), then process these centroids to obtain a datastructure called kd-forest which will be used to perform nearest-neighbor search (a “dictionary”).

Vocabulary generation is clearly an offline task that requires a lot of number crunching and is done as little as possible. Generating the kd-forest, on the other hand, takes from a handful of seconds to a few minutes depending on the size of the vocabulary, so it is frequently done on engine startup. The kd-tree itself only exists in RAM.

The problem with that was that a kd-forest is a rather large datastructure. In our case it occupied hundreds of MB of RAM and took about one minute to generate. That was OK with Oak, where it was shared between threads, but with Acorn that overhead had to be paid for every process, both in space and time. We had to find a way to share the kd-forest across Acorn nodes on the same machine, and if possible to make startup faster.

The solution I opted for was to rewrite the whole quantizer. Previously we had been using popular Open Source libraries for this, but they didn’t do what I wanted.

I wrote the kd-forest generation algorithm as a LuaJIT program. It was the first Lua program officially used in production at Moodstocks, although as you will see it was only run offline. What it does is take centroids as input, generate a kd-forest and serialize it in a way easily readable in C thanks to the FFI. It can also actually perform nearest-neighbor searches but this is only used for test purpose.

Once the kd-forest is serialized, it can be loaded into system shared memory quite fast. A C library can then be used in every Acorn process to access this shared memory read-only and perform nearest-neighbor searches.

The idea is simple once you stop under-estimating the capabilities of SHM on Linux. By default it usually limited to a few MB so you have to increase it a lot for this to work (it can be done with sysctl). The implementation, on the other hand, is far from trivial. My code uses a lot of pointer arithmetics, I should probably clean it up someday, but in the meantime it does its job perfectly.

Seed

The Acorn Quantizer was the last major improvement to our online search stack. Around that time, we resolved on a major technological shift: we would perform image recognition on mobile devices directly instead of doing it on the server. Of course, initially, we would have an hybrid approach where on-device recognition would work as a kind of cache, but the mobile was where we would focus our efforts.

Doing on-device image recognition, though, almost meant starting from scratch: we had to make different technological trade-offs, use very different algorithms, and that meant writing an almost entirely new image recognition stack. We named that project Seed.

Seed encompasses a lot of things now, but at its core are proprietary Computer Vision algorithms that we set out to develop with Cédric and Maxime, who had joined us by then. We would discuss them as a team, then Maxime and Cédric would implement them in C while I would work on a Lua version.

The big picture is that some processing is done on the server at indexing time to generate signatures which are then sent to the client. Server-side software used to be entirely Lua, client-side software entirely C, but we decided to implement the whole stack in both languages. I think that was one of the best ideas we ever had. Being able to compare results avoided errors on both sides (tricky things like off-by-ones were always noticed thanks to the fact that Lua is 1-based, floating-point math issues were found…). Lua allowed faster prototyping on some parts and it was interesting to compare the different architectural choices we were making.

With the current (second) generation of the Seed algorithms, we are actually mostly using the C implementation through the LuaJIT FFI on the server side now. That is because I have been working on other projects while the rest of the team (which is not as comfortable with Lua) was developing them, so I would have been a bottleneck if we had kept the dual stack approach. I may well bring the Lua branch up to date someday though, who knows?

chksrv and chkcoherence

Moodstocks’ server-side architecture is some kind of SOA. That means we have a lot of different services that run as daemons and need to stay up. chksrv is a medium-sized program in Bash that takes care of this. It is deployed on every instance with a configuration file that indicates which services should be running on that instance, and it makes sure that they are (correctly). It also checks if other instances are up. If something goes wrongs, it warns the “ops team”, who is basically me and Cédric as a backup in case I am not available.

chksrv is a very useful piece of software but I was a bit worried by its growth as we added services. Standardizing the way we deamonize processes helped a lot with that by increasing code reuse (thank you libslack).

chkcoherence is the ideal complement to chksrv: where the latter checks if services are running, the former verifies that they are doing things right. It is also written in Bash at the top level. I have already written about its concept here.

Anemone

Anemone is the project that deals with everything related to metrics and measurements at Moodstocks. It is written in Lua and has quite a few different roles:

• collect logs and data from production instances;
• generate internal daily and weekly reports with business metrics for the platforms and email them to the team;
• generate technical reports for operations (e.g. growth of different datasets);
• generate weekly reports per application and email them to our platform customers;
• generate custom reports tailored to the needs of our enterprise customers.

It also has a web-based dashboard for the team with high-level KPI, written in JavaScript and flot. Someday I might integrate Brett Slatkin’s Cohort Visualizer into it.

Dandelion

I said earlier that with Seed we generate image signatures on the server and send them to the mobile clients where they are used for recognition. Dandelion is the code name of the service responsible for that.

It turns out efficiently sending millions of image signatures per day, over slow and unreliable networks, to devices everywhere in the world, is not trivial. So Dandelion, more than software, is a synchronization protocol and its implementation; a range of tricks to make the best of mobile networks packaged as software. It is one of the reasons (along with all the innovation on CV algorithms and their optimized client-side implementation) why we can propose client-side recognition with databases of thousands of images or even videos, an order of magnitude more than our competitors.

The server part of Dandelion is written in Lua and depends on pieces like Redis and Beanstalkd, which is why I wrote haricot.

Physalis

Finally, Physalis is the project I am currently working on. It has not been released yet so I won’t get into the details, but I can explain the reasoning behind it.

While we were building Dandelion and through our experience with our clients, we learned the following things:

• writing mobile applications that work offline but keep their data up to date is tricky;
• most developers who attempt to implement it from scratch end up with a broken solution or just give up;
• when you succeed it makes your applications a lot better.

So we thought: we have done it, why not make it accessible to everybody? This is what Physalis is: Moodstocks’ image signature distribution system generalized so that you can leverage it for your own mobile application.

Physalis will be available in private alpha for selected users soon, under its real brand name (Physalis is only its “internal plant-themed name”). If you are interested in trying it out, get in touch. The requirements are that you should be making a mobile application and ready to communicate on a regular basis with us: we are doing this alpha to collect useful feedback.

EDIT: Physalis was eventually released in August 2013 as Winch.

Story of a bug

Imagine you have a database where you can index some documents in a full text search engine. Indexing can be turned on and off on a per document basis.

As the size of your dataset increases, you outgrow it and decide to use a separate service for search. So you change your code such that documents which must be indexed are sent to that service, and you add a boolean field in your main datastore to indicate whether the document is indexed or not.

All is fine until one day you refactor this code a bit too fast and part of the logic for document updates becomes something like this:

if document.indexed then
  search_engine:index(document)
  document.indexed = false -- <-- WAT?
end

Of course, in practice, it is less trivial and only happens in rare corner cases, so your tests don’t catch it…

This ain’t normal(ized)

This kind of bug in a system can be very nasty. What happens when it is triggered is that the worldviews of two subsystems (here the datastore and the index) are not coherent anymore.

If you are a programmer not used to distributed systems, you may think that the problem is that the information “the document is indexed” is duplicated. State is bad, but duplicate state is plain wrong, just always ask the index for that information and drop that “indexed” field!

In a normal application setting you would be right, but this is one of the main differences between SOA and OOP. There are two reasons why you do not want to do that. The first one is performance: this may generate more internal network requests. Its importance could be discussed at length (“Is it some form of premature optimization?”).

The second reason is much more important though: if you do that, your index becomes a data-critical service. That means you cannot lose its state without losing information, so you have to back it up seriously. This simple boolean field in the datastore is enough to rebuild the whole index, making it non-critical.

So that leaves us with denormalized data and its own problems. How do we mitigate them?

Invariants are sexy

Once you have denormalized data, your problem is to keep it coherent. That means that there are invariants that must be verified at all times by the various states of the subsystems. Or rather, because of asynchronous jobs, invariants that shouldn’t remain unverified for too long.

Those invariants are almost always properties on sets. For instance, if you have a forum where only registered users can comment, users who have commented must be a subset of users who have confirmed their email. In my case, the set of documents indexed in the search engine must be equal to the set of documents flagged as indexed in the main datastore.

The big idea is that every time you denormalize data you should write invariants that ensure coherence. These invariants are checked by scripts that can be run at every transaction in some cases, but are more usually cronned. You should also have procedures to reconcile (repair) the data in case of incoherence. I am not a huge fan of having them run automatically: incoherence often reveals bugs, so humans should check where it comes from and fix it.

In our case, the script responsible for coherence checks warned me that a few documents belonging to the same user were present in the search engine but not flagged as indexed. I asked the application logs what had happened to these documents around the time when the incoherence occurred, and saw they had all been updated. I looked up the relevant code path in application code… and facepalm-ed. I had pushed that code to production half a year ago, and it was obviously wrong.

Conclusion

The moral of it all is not, as I already discussed, that you should not denormalize data. It is not that we should write more tests, either. At least it is not what I want the takeaway to be (we do test these things more carefully now, but edge cases can always slip through).

What I think this story shows is that, if you write distributed systems that handle denormalized data, you should have:

• something to check invariants to detect issues;
• exhaustive logs and tools to diagnose them;
• reconciliation scripts to fix them.

I cannot imagine releasing a distributed system that does not have those things. They are even more important than a comprehensive test suite to me. Moreover, the coherence checks can also be run in the test suite itself (on mocks for instance) so writing them is always a win-win.

The State Machine

Let us consider a system that can take a finite number of states S[1] to S[n] over time.

We can represent the system as a N-node directed graph, where nodes are the states. An arc exists between node S[i] and S[j] if the system can transition from S[i] to S[j]. You probably already know that.

In Chains

We will now observe the evolution of the system over time. We will model time discretely. That does not mean that time itself is discrete, but we will observe the system at fixed-interval “ticks”. We call S(t) the state of the system at tick t.

If the system is known to be in S[i] at tick t (i.e. S(t) = S[i]) we call P(i->j) the probability that it will be in S[j] at tick t+1. Here is a definition for the mathematically inclined: P(i->j) = P(S(t+1) = S[j] | S(t) = S[i]).

A few remarks:

• P(i->j) is considered independent of t;
• P(i->i) exists;
• the sum of outbound probabilities is 1: ∀i, ∑[j=1,N]{P(i->j)} = 1.

By the way, this is called a Markov Chain.

Blurry Sights

Now let’s complicate things a little by introducing a separate system that can take states V[1] to V[M]. This second system V is tied to S: at every tick t, the probability distribution of V(t) only depends on S(t). We will note: O(i,j) = P(V[j] | S[i]).

You can think of V as something that observes S and reports on its state in an incomplete and probabilistic manner.

If you have understood everything up to now, you should easily get what this means: ∀i, ∑[j=1,M]{O(i,j)} = 1.

If you have not, let’s take a really simple example with N = M = 2. You can imagine that both S and V are lights that can be either red or blue. The relationship between S and V can be expressed by something like this: “If light S is red, then there is 90% chance that V is red too. If light S is blue, then there is only 40% chance that V is red.”

Now, add more possible colors to S and V, not necessarily the same number for both, to generalize the system.

The Black Box

Now, here is our problem: imagine that you know how system S works (the probabilities of transition between the states) but that it is hidden in a black box such that you cannot observe it. What you can observe is system V. Is there any way to guess what system S is doing at any time in this setting? If yes, how?

The answer to the first question is: “sometimes, yes”. “Sometimes” means “if the probability distributions help us”.

The second question is the one my favorite algorithm, the Viterbi algorithm, answers. Let’s see how it works.

Make it Code

First we need to represent the problem in a way that can be understood by a computer. I will use the Lua programming language for that.

We start by defining a few parameters. We will choose N = 3 and M = 2 for simplicity. We will observe the system over 1000 ticks.

local N = 3
local M = 2
local L = 1000

The probabilities of transitions in S will be represented by a NxN matrix, with P(i->j) as T[i][j]. We use percentages because they are easier to read and avoid some floating point calculation issues.

local T = {
  {  60, 20, 20 },
  {   0, 70, 30 },
  {  70, 10, 20 },
}

For instance, there is a 70% chance to transition from S[3] to S[1].

The probablitities of observations will be represented by a NxM matrix, with O(i,j) as O[i][j].

local O = {
  { 5, 95 },
  { 55, 45 },
  { 90, 10 },
}

Let us make sure we didn’t make too many mistakes thanks to our two probability equations from earlier:

local sum = function(t)
  local s = 0
  for i=1,#t do s = s + t[i] end
  return s
end

for i=1,N do
  assert(sum(T[i]) == 100)
  assert(sum(O[i]) == 100)
end

This is what our example system looks like in graph form:

The green part is the Markov chain itself, the blue part corresponds to the observation device. Arcs are labeled with probabilities of transition or observation.

Stepping Forward

Now we can simulate the behavior of the system.

local proba_pick = function(v)
  local p,s = math.random(1,100),0
  for i=1,#v do
    s = s + v[i]
    if p <= s then
      return i
    end
  end
  assert(false)
end

local S,V = {},{}

local transition = function(t)
  assert(S[t-1] and not S[t])
  S[t] = proba_pick(P[S[t-1]])
end

local observe = function(t)
  assert(S[t] and not V[t])
  V[t] = proba_pick(O[S[t]])
end

S[1] = math.random(1,N)
observe(1)

for t=2,L do
  transition(t)
  observe(t)
end

This code is slightly more complicated, but what you should understand is that, at the end of it, S is a vector of L states of the system and V is the corresponding vector of observations. The initial state S[1] is chosen randomly with uniform probabilities.

Seeing Through the Box

Now, let us proceed to guess what happens inside the black box. We do so iteratively: at each tick t, we calculate the most plausible sequence of events that could have led to each possible state of the system, given the observation at t and the results for t-1.

local trellis = {{}}

local prb = function(x)
  return math.log(x/100)
end

local viterbi_node = function(t,i)
  local prev = assert(trellis[t-1])
  local idx,proba = 0,-math.huge
  local p
  for j=1,N do
    p = prev[j].proba + prb(T[j][i]) + prb(1/N*O[i][V[t]])
    if p > proba then proba,idx = p,j end
  end
  return {
    state = i,
    proba = proba,
    prev = prev[idx],
  }
end

local viterbi_step = function(t)
  assert(not trellis[t])
  trellis[t] = {}
  for i=1,N do
    trellis[t][i] = viterbi_node(t,i)
  end
end

-- initialize the trellis
for i=1,N do
  trellis[1][i] = {
    state = i,
    proba = prb(1/N*O[i][V[1]])
  }
end

-- run the algorithm
for t=2,L do
  viterbi_step(t)
end

Again, not trivial code, but this is the Viterbi algorithm itself. Moreover we use logarithmic sums for probabilities to avoid numerical problems.

What the algorithm does is build a trellis, a special kind of graph. For every step t we calculate the most plausible sequence of events (a path through the trellis) which ends up with the system in each of the possible states. When we reach the end of the simulation, we take the most plausible state of the system and take the path that led to it as our estimate of what happened.

To avoid memory usage explosion, we do not actually store the paths themselves in the nodes of the trellis. Instead, in a node at tick t, we store a pointer to the previous node (at t-1) that led there. This is why we have to backtrack through the trellis to find the actual path:

local getpath = function(node)
  local r = {}
  repeat
    table.insert(r,1,node.state)
    node = node.prev
  until (not node)
  return r
end

local best_node = function(nodes)
  local r = {proba = -math.huge}
  for i=1,#nodes do
    if nodes[i].proba > r.proba then
      r = nodes[i]
    end
  end
  return r
end

local bestpath = getpath(best_node(trellis[L]))

Does this thing really work?

To check if this work, let’s take the actual system out of the box and calculate how much it looks like our estimate:

local ok = 0
for i=1,L do
  if S[i] == bestpath[i] then
    ok = ok + 1
  end
end

print(100*ok/L)

The result I got with those parameters is 72% accuracy, to compare to what a naive random process would get (33%).

Why I love it

Let’s go back to the title of this post: now that you know what it does, why is Viterbi’s algorithm my favorite?

I admit that the elegance of the representation of the problem as a trellis is part of the reason, but mainly it’s because of its implications. Think about it: fundamentally it is an algorithm that allows a machine to explain what it observes. Isn’t that crazy?

Yes, it is. And because of that it has a whole range of applications, from advanced orthographic correction and speech-to-text to the decoding of convolutional codes used in voice codecs. And I suspect its potential has not been fully exploited yet.

Algorithms & Data Structures

For a long time I have thought that there were only two fundamental parts to Computer Science: Algorithms and Data Structures. This is how it used to be taught, and how I think it should still be taught to beginners.

There are two main categories of parts in a computer: processing units (CPUs, GPUs…) and data storage units (caches, RAM, disk…). Your typical information processing program takes data from a durable storage unit, moves it to a temporary storage unit, modifies it thanks to a processing unit and stores the result in a permanent storage unit. Of course there are variations around this theme, and some programs do a bit of extra work such as handling user input, but that is the idea.

We call “algorithms” the different ways we instruct the processing units to act on the data and “data structures” the different layouts we use to store data in the storage units.

A very simple view of fundamental CS could be summed up to these two disciplines. You may argue that other things are indispensable, for instance theoretical aspects like lambda calculus or Turing machines, or maybe paradigms. You would be right, these things are important, but if you really want the baseline, it all comes down to algorithms and data structures.

Or so I thought.

Protocols

It turns out looking at the aforementioned parts of the computer is not enough. We need to look at the negative space too: how does the data move around? Inside the computer, data mostly circulates in buses. Between computers, data may circulate in networks, on detachable storage units…

With computers becoming increasingly parallel, the Internet reaching ubiquity and programs turning into complex distributed systems, understanding the interactions between computer parts, software parts or entirely different systems has become a necessity. This reveals a third fundamental side of CS, on equal footing with algorithms and data structures: protocols.

Protocols are more present in CS than you may think. Even the original vision of OOP by Alan Kay was mostly about protocols. Joe Armstrong, a father of Erlang, recently insisted on the necessity to teach protocols (and algorithms - if he had added data structures this post would look like plagiarism ;p).

CS classes at engineering school did not teach me much or influence the way I write programs, but majoring in network engineering did. It made me start to think in terms of independent blocks and the protocols they use to communicate. I cultivated that vision during my MSc in distributed systems (it used to be called “grid computing”) and I still have it today.

So when I see people arguing endlessly about static versus dynamic typing or such matters, I think: OK, that may be important, but do not forget CS is, at heart, about algorithms, data structures and protocols.

What is SOA?

When people say SOA (Service Oriented Architecture) they often mean different things. My personal definition of SOA is rather loose and, like all the quotes in this article, comes from Werner Vogels, the CTO of Amazon:

For us service orientation means encapsulating data with the business logic that operates on the data, with the only access through a published service interface.

If you are an Object Oriented Programming person this should ring a bell. Indeed, SOA is essentially encapsulation at the system level. Actually, I strongly believe OOP is a good idea (encapsulation) applied at the wrong layer.

The opposite of SOA in the context of the Web is monolithic applications. Think textbook Ruby on Rails or Django, for instance. Monolithic applications are fine as long as they are small, but I think SOA is a better fit for larger projects. If you are a technological startup, that means you should at least start thinking about what SOA could bring you if you are starting to have a clear idea of what your product is. Monolithic applications are great for prototypes and MVPs.

If you are wondering if what you are currently doing is closer to SOA than to monolithic applications, ask yourself these questions:

• How many different repositories does our code live in? (If you do not use source control you have more urgent problems to think about than SOA.)

• How many different databases do we have? How many database instances do we have? How many connections? How many schemas? (SOA and NoSQL tend to mix well.)

• Do we have a Web application that does significantly more than rendering templates? Is it stateful? What parts of that state could we also use in a mobile application? Does the same application render JSON and HTML?

Advantages of SOA

One of the most obvious benefits of SOA is that services are easier to scale than monolithic applications. Maybe you want to migrate user authentication data to Redis and keep other data in whatever DB you already have? SOA makes it trivial. Maybe some part of your product would benefit from running on an AWS instance that has lots of RAM while another one is CPU-intensive? Just deploy the corresponding services where they run best.

SOA will also help you achieve resilience. You can program user-facing or aggregate services defensively so that they detect the unavailability of backend services and degrade gracefully. You can even use a chaos monkey to ensure this.

However in my opinion the main advantages of SOA are not technical, they are organizational.

As the number of your employees increases you will have to separate them into teams. The classical way to do this is by specialty: first developers and operations, then spin off DBAs… The main problem with that organization is that it hinders innovation. To create a new feature you need to coordinate all these people, and it results in endless meetings and sterile debates (not to mention clan wars).

This is not a fatality. You have probably heard of DevOps, and maybe dismissed it as Yet Another Buzzword. Well SOA is the easiest way to implement True DevOps (TM?), meaning that developers and operations really work together all the time. The trick is that there are no “devs” and “ops”, because you create teams along another dimension: services. In Werner Vogel’s words:

The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software.

These teams are like smaller startups within your growing startup: they have all they need to build new features or even whole products by themselves. And they will keep you nimble if you give them enough leeway:

We allow teams to function as independently as possible. Developers are like artists; they produce their best work if they have the freedom to do so, but they need good tools.

SOA also gives you language agnosticism almost for free. Not much prevents different services from being written in different programming languages. Why would you want to do that? Well, beyond the fact that different problems are better solved in Python, Haskell or C, there is lot of talent out there not using Ruby, Python, PHP, Scala, JavaScript or whatever language you have chosen for your monolithic application. Some of those Scheme, Io, Factor, Lua or F# hackers can be real assets for those clever enough to give them a job.

Finally, I can see a (perhaps more abstract) last advantage to SOA which is related to the rise of the Programmable Web. After all, there is little difference between a Web API and a service, except that you wrote the latter and not the former. Embracing SOA will prepare you to work with external APIs, and more importantly to publish your own. For instance, lots of AWS services have started their life as internal services at Amazon.

Conclusion

I am not saying SOA only has advantages. Such an architecture can be fragile if you do not code with service unavailability in mind. Testing complex operations involving lots of services can be hard, and debugging them even harder. Correctly defining the boundaries and interfaces of services requires a significant amount of reflection and may not be a good idea when you do not know what exactly you are building.

That being said I still think more technological and/or Web startups should consider SOA instead of the “everything within the Framework” approach, not really for future scalability reasons, but because they risk limiting their flexibility, speed of iteration and capacity to hire short term if it is not already the case.

Paye ta TV

On la critique en France mais la redevance TV est bien pire en Angleterre. Si dans l’hexagone elle est perçue par l’État, ici c’est une société privée qui s’en charge. Et contrairement à chez nous, ni les gens qui regardent la télé sur Internet ou sur un téléphone mobile ni les étudiants en résidence ne sont exemptés.

Cette société, TV Licensing, envoie régulièrement des lettres de menace aux délinquants qui osent capter la bonne parole diffusée sur les ondes sans s’acquitter de la dîme. Hmm, un petit arrière-goût d’HADOPI ? J’ai reçu cinq lettres, et je suis d’après leurs dires “scheduled for visit”. Ils peuvent venir, j’ai l’esprit tranquille : je ne regarde pas la TV, la loi anglaise ne leur permet pas d’entrer chez moi et de toute façon c’est tellement difficile techniquement d’atteindre ma porte que ça m’étonnerait qu’ils y arrivent un jour.

Le garçon qui criait au loup

Depuis que j’habite ici, j’ai bien dû être réveillé une vingtaine de fois par des alarmes incendies en pleine nuit. La première fois, enfiler un manteau, des chaussures et descendre devant l’entrée se geler pendant une demi-heure en attendant l’arrivée de la sécurité peut être drôle, mais à la longue ça lasse. Qui plus est, le dispositif d’alarme permet de savoir aisément quel détecteur a déclenché l’alerte, et donc de confirmer rapidement que c’est une erreur, mais la loi anglaise stipule qu’ignorer l’alarme et retourner au chaud dans son lit est un crime !

Si jamais un feu se déclenche vraiment, j’espère que les familles des victimes penseront à faire un procès à tous les responsables du dispositif d’alarme. Un système d’alerte qui génère autant de faux-positifs est totalement inutile, simplement parce que personne n’y prête plus attention. Tous les administrateurs système qui ont déjà essayé de lire des logs doivent bien le savoir…

La vie privée, c’est dépassé !

C’est en tout cas ce que pense Mark Zuckerberg, le patron de Facebook. Il a le droit, et j’ai le droit de ne pas être d’accord. Comme mon ami Dédé, je vais donc logiquement quitter le célèbre réseau social.

De toute façon, les soi-disant réseaux sociaux sont des aberrations, des substituts au Web pour ceux qui ne l’ont pas compris. Vous voulez donner des nouvelles, ou même poster des liens débiles ? Ouvrez un blog (on appelait ça “une page perso” à l’époque révolue du Web 1.0), ou twittez si ça vous amuse ! Il semblerait que la raison qui pousse les gens à rejoindre Facebook soit la simplicité d’ouverture d’un compte. Ouvrir son blog aussi est très simple.

Pour ma part, je ne consulterai pas Facebook en février. Je ne clôturerai mon compte que début mars de sorte que les gens qui n’ont que ce moyen pour me joindre puissent m’en demander un autre par message privé (je les reçois dans ma boite mail). Si vous trouvez que je suis paranoïaque, sachez simplement que l’un des sujets proposés en MSc Thesis à Cranfield consiste à faire du data-mining (collecter plein d’informations sur vous) sur Facebook en utilisant le Grid Computing (un paquet d’ordinateurs, dont des super-calculateurs, en réseau). Si de tels moyens, habituellement réservés à de très grosses industries comme le pétrole, l’automobile ou la mode, sont déployés dans ce but, c’est simplement parce que vos données personnelles valent très cher.

Java m’pendre (ou pas)

Java est un langage tout pourri, si je commençais à énumérer toutes les raisons pour ça il me faudrait autant de lignes que pour coder une application de taille moyenne dans le dit langage, et oui ça fait beaucoup.

En tout cas, étant contraint et forcé d’en faire pas mal en ce moment, j’ai pu bien m’arracher les cheveux sur quelques points :

• string.length(), mais table.length. La cohérence, vous connaissez ?

• Avec Java, Axis et un programme un peu complexe, il suffit qu’une exception se déclenche pour afficher une trace qui fait cinq fois la hauteur de votre console. Bien entendu, le message d’erreur intéressant est en haut.

• Enfin et surtout, c’est lent, à la compilation comme à l’exécution.

Voilà, tout ceux qui l’ont utilisé savent que Java c’est mal. Après, il y en a qui aiment, et je n’ai rien contre les masochistes, mais je supporte déjà moins les sadiques qui forcent des gens sains d’esprit à s’en servir.

YouTube (4e) et Blogger (7e)

Deux services de Google, bizarrement. Tous deux vous permettent d’héberger du contenu, vidéo pour YouTube et principalement textuel pour Blogger. Ce contenu pourra ensuite être commenté par d’autres utilisateurs, c’est ce qu’on appelle Web 2.0. Qu’y a-t-il de mal à cela ? Pas grand-chose, en fait, si ce n’est la situation de quasi-monopole de ces plates-formes.

En offrant un service simple et gratuit, Google [1] s’est imposé sur ces secteurs. Or, même si Google n’est pas maléfique, il est bon de se rappeler que c’est une entreprise dont le business model repose sur deux choses : vous vendre de la publicité et apprendre des choses sur vous. Vous lui donnez une occasion en or de faire la deuxième en vous créant un compte personnel chez eux et en y hébergeant vos données, quant à la première il n’y a qu’à voir le principal changement sur YouTube pour l’utilisateur depuis le rachat [2].

Autre problème commun à tous les services gratuits gérés par de grosses boites : vous ne valez pas grand-chose pour eux, et ils ne feront rien pour vous aider, par exemple, à éviter la censure. Il suffit de voir le nombre de vidéos supprimées de YouTube sans raison valable, sur simple demande de gens qui ne sont souvent pas les ayant-droits. Certains se souviennent peut-être de la vidéo où Mme Albanel prouvait son incompétence en annonçant qu’OpenOffice incluait un pare-feu. Cette vidéo a été censurée à de multiples reprises bien qu’elle ait été filmée de manière officielle à l’Assemblée Nationale.

Ceci dit, ces problèmes sont mineurs, il suffit d’en être informé, et ces services ont une raison d’être sur Internet. Pour les deux qui suivent, c’est une autre histoire.

[1] OK, pour Youtube c’était déjà le cas avant que Google achète…

[2] Pour ceux qui ne l’utilisent pas, je parle de publicités sous forme de bandeaux qui s’affichent en superposition sur la vidéo.

Wikipedia (6e)

Wikipedia, l’un des plus gros succès du libre. Une source d’information à laquelle chacun peut contribuer s’il a les compétences nécessaires, accessible instantanément de tout point du monde. Une merveille !

Attendez, ça ne vous fait pas penser à quelque chose ça ? Ah si, le Web… Sauf que Wikipedia a un modèle centralisé alors que le Web est distribué. Sauf que sur Wikipedia ce sont les modérateurs qui détiennent le pouvoir de décider si une information est valable ou non, alors que sur le Web c’est à chacun de se faire son idée, même si l’opinion dominante se trouvera en premier dans les moteurs de recherche. Sauf que sur Wikipedia tout devra s’appuyer sur des sources “fiables”, comme par exemple des publications scientifiques (hum). Bref, Wikipedia, une formidable machine à imposer une pensée unique à des utilisateurs qui ne demandent pas mieux.

EDIT - Pour ceux qui ne s’en seraient pas rendu compte, j’exagère volontairement. Ne prenez pas tout pour argent comptant non plus, la diversité d’opinion existe bien sur Wikipedia dans une certaine mesure.

Facebook (2e)

J’ai gardé le meilleur pour la fin. Juste derrière le roi des moteurs de recherche, le sultan des réseaux sociaux. Facebook, la toile dans la toile, mais une toile encore une fois centralisée et où on ne communique qu’avec ses “amis”. Pratique, je peux dire des bêtises comme ça, personne ne le verra. Et si jamais un jour je ne les assume plus, j’efface.

Mouais. Vous faites vraiment une confiance absolue à tous vos contacts, vous ? Si c’est le cas, bravo. Sinon, une fois que vous aurez réalisé que tout ce que vous mettez sur Facebook est du numérique, donc de l’information copiable, vous vous rendrez compte que, quels que soient les réglages de votre compte, il vaudrait mieux considérer que tout ce que vous y postez est public.

Et puis même au-delà de ça, comment comprendre qu’un tel principe ait du succès ? Parce qu’après tout, soit vous n’ajoutez pas de contenu et vous ne faites que poster des liens vers des choses que vous avez trouvées en surfant sur le Web ou partager ceux des autres, et ça ressemble fortement aux chaînes d’emails que personne ou presque n’aime, soit vous mettez des choses nouvelles en ligne et c’est souvent égoïste parce que vous réservez (théoriquement) à vos pseudo-amis l’accès à de l’information qui n’est en général pas si privée que ça et qui aurait peut-être pu servir ou faire plaisir à d’autres. Regardez un peu en arrière, même les kikoolols de la génération Caramail préféraient en général les forums publics aux forums privés.

Le cœur du problème

Oui, parce qu’il faut bien le dire, il y a un problème. Et ce n’est pas que ces trucs-là existent, ils en ont le droit, c’est même un des fondements d’Internet : n’importe quel service a sa chance.

Non, le problème, c’est que bien que je voie tous leurs défauts gros comme des maisons, je les utilise ! J’ai un compte YouTube, un compte Facebook et ça m’arrive régulièrement de consulter des pages Wikipedia. Seule exception : ces quelques pages ne sont pas chez Google mais bien sur mon VPS, sous mon nom de domaine. Ouf.

La raison pour laquelle je les utilise, c’est que bien que conceptuellement et idéologiquement ils soient percés de partout, techniquement, ils vont de pas trop mauvais (Facebook, Wikipedia) à très bons (tout ce que fait Google). C’est un peu la différence avec Windows ou Internet Explorer, vous voyez : le fait qu’ils sont techniquement mauvais fait que leurs alternatives sont crédibles, pas vraiment la question éthique qui est derrière.

Conclusion

Conclusion ? Rien. Que voulez-vous que je dise de plus ? Voir que ces sites-là sont parmi les plus visités du monde me rend un peu triste mais ce n’est pas pour autant que je vais arrêter de les utiliser. Pas tout de suite en tout cas.

Peut-être qu’un jour j’aurai le temps de mettre en place un système simple pour héberger mes vidéos sur mon serveur. Peut-être que je me lasserai de Facebook (les gens que dont je voudrais avoir des nouvelles n’y postent plus trop de toute façon) et que je fermerai mon compte. Peut-être qu’on reverra des annuaires collaboratifs de liens vers des pages qui donnent des avis variés sur divers sujets, avec une structure d’arbre ou de graphe par thématique et un hébergement décentralisé, de la réplication spontanée, du Web quoi… Houlà, je m’envole ;)

En attendant le miracle, je profite de la qualité de ces services, comme tout le monde. Je ne suis pas assez fanatique pour les boycotter. J’essaie juste de me rappeler de temps en temps qu’on est bien loin de l’utopie du Web de 90.

La prochaine fois, je parlerai peut-être de Chrome OS, un sujet pas si éloigné finalement.