Observability

2024-09-19T00:00:00+00:00

Imagine a collection of values T</code>. This collection can be updated by inserting new values, removing existing ones, or the collection can truncated, cleared… This collection acts asthe standard Vec</code></a>. However, there is a subtlety: This collection is observable. It is possible for someone to subscribe to this collection and to receive its updates.

This observability pattern is the basis of reactive programming. It applies to any kind of type. Actually, it can be generalised as a single Observable<T></code> type. For collections though, we will see that an ObservableVector<T></code> type is more efficient.

I’ve recently played a lot with this pattern as part of my work inside theMatrix Rust SDK</a>, a set of Rust libraries that aim at developing robust Matrix</a> clients or bridges. It is notoriously used by the next generation Matrix client developed by Element</a>, namely Element X</a>. The Matrix Rust SDK is cross-platform. Element X has two implementations: on iOS, iPadOS and macOS with Swift, and on Android with Kotlin. Both languages are using our Rust bindings to Swift</a> and Kotlin</a>. This is the story for another series (how we have automated this, how we support asynchronous flows from Rust to foreign languages etc.), but for the moment, let’s keep focus on reactive programming.

Taking the Element X use case, the room list –which is the central piece of the app– is fully dynamic:

Rooms are sorted by recency, so rooms move to the top when a new interesting message is received,</li> The list can be filtered by room properties (one can filter by group or people, favourites, unreads, invites…),</li> The list is also searchable by room names.</li> </ul> The rooms exposed by the room list are stored in a unique observable type. Why is it dynamic? Because the app continuously sync new data that update the internal state: when a room gets an update from the network, the room list is automatically updated. The beauty of it: we have nothing to do. Sorters and filters are run automatically. Why? Spoiler: because everything is a Stream</code>. Thanks to the Rust async model, every part is lazy. The app never needs to ask for Rust if a new update is present. It literally just waits for them. I believe this reactive programming approach is pretty interesting to explore. And this is precisely the goal of this series. We are going to play with Stream</code> a lot, with higher-order Stream</code> a lot more, and w… Le Comte </picture> </div> Hold on a second! I believe this first step is a bit steep for someone who's not familiar with asynchronous code in Rust, don't you think? Before digging in the implementation details you are obviously eager to share, maybe we can start with examples. </div> </div> Alrighty. Fair. Before digging into the really fun bits, we need some basis. Baby steps with reactive programming#</a> </h2> Everything we are going to share with you has been implemented in a library called eyeball</code></a>. To give you a good idea of what reactive programming in Rust can look like, let's create a Rust program: $ cargo new --bin playground Creating binary (application) `playground` package $ cd playground $ cargo add eyeball Updating crates.io index Adding eyeball v0.8.8 to dependencies Features: - __bench - async-lock - tracing Updating crates.io index Locking 3 packages to latest compatible versions</code></pre>// in `src/main.rs` use eyeball::Observable; fn main() { let mut observable = Observable::new(7); let subscriber = Observable::subscribe(&observable); dbg!(Observable::get(&observable)); dbg!(subscriber.get()); Observable::set(&mut observable, 13); dbg!(Observable::get(&observable)); dbg!(subscriber.get()); }</code></pre> What do we see here? First off, observable</code> is an observable value. Proof is: It is possible to subscribe to it, see subscriber</code>. Both observable</code> and subscriber</code> are seeing the same initial value: 7. When observable</code> receives a new value, 13, both observable</code> and subscriber</code> are seeing the updated value. Let's take it for a spin: $ cargo run --quiet [src/main.rs:8:5] Observable::get(&observable) = 7 [src/main.rs:9:5] subscriber.get() = 7 [src/main.rs:13:5] Observable::get(&observable) = 13 [src/main.rs:14:5] subscriber.get() = 13</code></pre> Tadaa. Fantastic, isn't it? Le Comte </picture> </div> I… I am… speechless? Is it really reactive programming? Where is the reactivity here? It seems like you've only shared a value between an owner and a watcher. You're calling them observable and subscriber, alright, but how is this thing reactive? I only see synchronous code for the moment. </div> </div> Hold on. You told me to start slow. You're right though: the Observable</code> owns the value. The Subscriber</code> is able to read the value from the Observable</code>. However, Subscriber::next</code> returns a Future</code></a>! Let's add this: // in `src/main.rs` // … fn main() { // … dbg!(subscriber.next().await); }</code></pre>$ cargo run --quiet error[E0728]: `await` is only allowed inside `async` functions and blocks --> src/main.rs:16:28 | 3 | fn main() { | --------- this is not `async` ... 16 | dbg!(subscriber.next().await); | ^^^^^ only allowed inside `async` functions and blocks</code></pre> Indeed. Almighty rustc</code> is correct. The main</code> function is not async</code>. We need an asynchronous runtime. Let's use the smol</code> project</a>, I enjoy it a lot: it's a small, fast and well-written async runtime: $ cargo add smol Updating crates.io index Adding smol v2.0.2 to dependencies [ … snip … ]</code></pre> Now let's modify our main</code> function a little bit: // in `src/main.rs` use eyeball::Observable; fn main() { smol::block_on(async { let mut observable = Observable::new(7); let mut subscriber = Observable::subscribe(&observable); dbg!(Observable::get(&observable)); dbg!(subscriber.get()); Observable::set(&mut observable, 13); dbg!(Observable::get(&observable)); dbg!(subscriber.get()); dbg!(subscriber.next().await); }) }</code></pre> Please rustc</code>, be nice… [src/main.rs:9:9] Observable::get(&observable) = 7 [src/main.rs:10:9] subscriber.get() = 7 [src/main.rs:14:9] Observable::get(&observable) = 13 [src/main.rs:15:9] subscriber.get() = 13 [src/main.rs:17:9] subscriber.next().await = Some( 13, )</code></pre> Hurray! We can even have a bit more ergonomics by using the smol-macros</code> crate</a> which sets up a default async runtime Executor</code></a> for us. It's useful in our case as we want to play with something else (reactive programming), and don't want to focus on the async runtime itself: $ cargo add smol-macros macro_rules_attribute Updating crates.io index Adding smol-macros v0.1.1 to dependencies Adding macro_rules_attribute v0.2.0 to dependencies Features: - better-docs - verbose-expansions Updating crates.io index Locking 4 packages to latest compatible versions Adding macro_rules_attribute v0.2.0 Adding macro_rules_attribute-proc_macro v0.2.0 Adding paste v1.0.15 Adding smol-macros v0.1.1</code></pre> We will take the opportunity to improve our program a little bit. Let's spawn a Future</code> that will continuously read new updates from the subscriber</code>. use std::time::Duration; use eyeball::Observable; use macro_rules_attribute::apply; use smol::{Executor, Timer}; use smol_macros::main; #[apply(main!)] async fn main(executor: &Executor) { let mut observable = Observable::new(7); let mut subscriber = Observable::subscribe(&observable); // Task that reads new updates from `observable`. let task = executor.spawn(async move { while let Some(new_value) = subscriber.next().await { dbg!(new_value); } }); // Now, let's update `observable`. Observable::set(&mut observable, 13); Timer::after(Duration::from_secs(1)).await; Observable::set(&mut observable, 17); Timer::after(Duration::from_secs(1)).await; Observable::set(&mut observable, 23); // Wait on the task. task.await; }</code></pre> The little Timer::after</code> calls are here to pretend the values are coming from random events, for the moment. Let's run it again to see if we get the same result: $ cargo run --quiet [src/main.rs:16:13] new_value = 13 [src/main.rs:16:13] new_value = 17 [src/main.rs:16:13] new_value = 23 ^C</code></pre> Here we go, perfect! See, ah ha! It's async and nice now. Le Comte </picture> </div> I believe I start to appreciate it. However, I foresee you might hide something behind these Time::after</code>. Am I right? And this task.await</code> at the end makes the program to never finish. It explains the need to send a SIGINT</code> signal</a> to the program to interrupt it, right? </div> </div> You're slick. Indeed, I wanted to focus on the observable</code> and the subscriber</code>. Because there is a subtlety here. If the Timer::after</code> are removed, only the last update will be displayed on the output by dbg!</code>. And that's perfectly normal. The async runtime will execute all the Observable::set(&mut observable, new_value)</code> in a row, and then, once there is an await point, another task will have room to run. In this case, that's subscriber.next().await</code>. The subscriber only receives the last update, and that's pretty important to understand. There is no buffer of all the previous updates here, no memory, no trace, subscriber</code> returns the last value when it is called. Note that this is not always the case as we will see with ObservableVector</code> later, but for the moment, that's the case. And yes, if we want the task</code> to get a chance to consume more updates, we need to tell the executor we will wait while the current other tasks are waken up. To do that, we can use the smol::yield_now</code> function</a>: // Now, let's update `observable`. Observable::set(&mut observable, 13); // Eh `executor`: `task` can run now, we will wait! yield_now().await; // More updates. Observable::set(&mut observable, 17); Observable::set(&mut observable, 23); // Eh `executor`: _bis repetita placent_! yield_now().await; drop(observable) task.await; }</code></pre> Let's see what happens: $ cargo run --quiet [src/main.rs:14:13] new_value = 13 [src/main.rs:14:13] new_value = 23</code></pre> Eh, see, new_value = 17</code> is not displayed, because the observable</code> is updated but the subscriber</code> is suspended by the executor. But the others are read, good good. Note that we are dropping the observable</code>. Once it's dropped, the subscriber</code> won't be able to read any value from it, so it's going to close itself, and the task</code> will end. That's why waiting on the task with task.await</code> will terminate this time. And thus, the program will finish gracefully. And that's it. That's the basis of reactive programming. Also note that Subscriber<T></code> implements Send</code></a> and Sync</code></a> if T</code> implements Send</code> and Sync</code>, i.e. if the observed type implements these traits. That's pretty useful actually: it is possible to send the subscriber in a different thread, and keep waiting for new updates. Attack of the Clones#</a> </h2> However, at the beginning of this episode, we were talking about a collection. Let's focus on Vec</code></a>. Le Comte </picture> </div> Why do we focus on Vec</code> only? Why not HashMap</code>, HashSet</code>, BTreeSet</code>, BTreeMap</code>, BinaryHeap</code>, LinkedList</code> or even VecDeque</code>? It seems a bit non-inclusive if you ask me. Are you aware there isn't only Vec</code> in life? </div> </div> Well, the reason is simple: Vec</code> is supported by eyeball</code>. It's a matter of time and work to support other collections, it's definitely not impossible but you will see that it's not trivial neither to support all these collections for a simple reason: Did you notice that Subscriber</code> produces an owned T</code>? Not a &T</code>, but a T</code>. That's because Subscriber::next</code></a> requires T: Clone</code>. It means that the observed value will be cloned every time it is broadcasted to a subscriber. Cloning a value</a> may be expensive. Here we are manipulating usize</code>, which is a primitive type, so it's all fine (it boils down to a memcpy</code></a>). But imagine an Observable<Vec<BigType>></code> where BigType</code> is 512 bytes: the memory impact is going to be quickly noticeable. So th… Le Comte </picture> </div> … Excuse my interruption! You know how I love reading books. I like defining myself as a bibliophile. Anyway. During my perusal of the eyeball</code> documentation, I have found Subscriber::next_ref</code></a>. The documentation says: Wait for an update and get a read lock for the updated value. </blockquote> and later: You can use this method to get updates of an Observable</code> where the inner type does not implement Clone</code>. </blockquote> </div> </div> Can you stop cutting me off please? It's really unpleasant. And do not forget we are not alone… doing sideways head movement You're right though. There is Subscriber::next_ref</code>. However, if you are such an assiduous reader, you may have read the end of the documentation, aren't you? However, the Observable</code> will be locked (not updateable) while any read guards are alive. </blockquote> Blocking the Observable</code> might be tolerable in some cases, but it cannot be generalised to all use cases. A user is more likely to prefer next</code> instead of next_ref</code> by default. Back to our Observable<Vec<BigType>></code> then. Imagine the collection contains a lot of items: cloning the entire Vec<_></code> for every update to every subscriber is a pretty inefficient way of programming. Remember that, as a programmer, we have the responsibility to make our programs use as few resources as possible, so that hardwares can be used longer. The hardware is the most polluting segment of our digital world. So. How a data structure like Vec</code> can be cloned cheaply? We could put it inside an Arc</code></a> right? Cloning an Atomically Reference Counted value is really cheap: it increases the counter by 1 atomically</a>, the inner value is untouched. Nonetheless, we have a mutation problem now. If we have Observable<Arc<Vec<_>>></code>, it means that the subscribers will be Subscriber<Arc<Vec<_>>></code>. In this case, every time the observable wants to mutate the data, it is going to… be… impossible because an Arc</code> is nothing less than a shared reference, and shared references in Rust disallow mutation by default. Using Observable::set</code> will create a new Arc</code>, but we cannot update the value inside the Arc</code>, except if we use a lock… Well, we are adding more and more complexity. Spes salutis</q>1</a>! Fortunately for us, immutable data structures exist in Rust. An immutable data structure is a data structure which can be copied and modified efficiently without altering the original. </blockquote> It can be modified. However, as soon as it is copied (or cloned), it is still possible to modify the copy but the original data is not modified. That's extremely powerful. Such structures bring many advantages, but one of them is structural sharing: If two data structures are mostly copies of each other, most of the memory they take up will be shared between them. This implies that making copies of an immutable data structure is cheap: it's really only a matter of copying a pointer and increasing a reference counter, where in the case of Vec</code></a> you have to allocate the same amount of memory all over again and make a copy of every element it contains. For immutable data structures, extra memory isn't allocated until you modify either the copy or the original, and then only the memory needed to record the difference. </blockquote> Well, taking a deep breath, it sounds exactly like what we need to solve our issue, isn't it? The Observable<Immutable<_>></code> and the Subscriber<Immutable<_>></code>s will share the same value, with the observable being able to mutate its inner value. The subscribers can modify the received value too, in an efficient way, without conflicting with the value from the observable. Both values will continue to live on their side, but cloning the value is cheap. Le Comte </picture> </div> Dare I ask how immutable data structures are implemented? It sounds like complex beasts. I mean… a naive implementation sounds relatively doable but I am guessing there is a lot of subtleties, possible conflicts, and many memory guarantees that I am not anticipating yet, right? </div> </div> Oh… beati pauperes in spiritu</q>2</a>… it is actually really complex. It may be a topic for another series or articles. For the moment, if you interested, let me redirect you to one research paper that proposes an immutable Vec</code>: RRR Vector: A Practical General Immutable Sequence</cite>3</a>. Be cool though, understanding this part is not necessary at all for what we are talking now. It's a great tool we are going to use, no matter how it works internally. Do you know the other good news? We don't have to implement it by ourselves, because some nice people already did it! Enter the imbl</code> crate</a>. This crate provides a Vector</code> type</a>. It can be used like a regular Vec</code>. (Side note: it's even smarter than a Vec</code> because it implements smart head and tail chunking4</a>, and allocates in the stack or on the heap depending on the size of the collection, similarly to the smallvec</code> crate</a>. End of digression) Observable (immutable) collection#</a> </h2> The imbl</code> crate then. It provides a Vector</code> type</a>. eyeball</code> provides a crate for working with immutable data structures (how surprising huh?): this crate is eyeball-im</code></a>. Instead of providing an Observable<T></code> type, it provides an ObservableVector<T></code> type</a> which is a Vector</code>, but an observable one! Let's see… what do we have… scroll the documentation, hmm, interesting, scroll more…, okay, that's interesting: First off, there is methods like append</code>, pop_back</code>, pop_front</code>, push_back</code>, push_front</code>, remove</code>, insert</code>, set</code>, truncate</code> and clear</code>. It seems this collection is pretty flexible. The vocabulary is clear. They all take a &mut self</code>, cool.</li> Then, there is a with_capacity</code> method, this is intriguing, add to notes,</li> Finally, we find our not-so-ol' friend subscribe</code>, but this time it returns a VectorSubscriber<T></code></a>.</li> </ul> Let's explore VectorSubscriber</code> a bit more, would you? Scroll the document, contrary to Subscriber::next</code></a>, there is no next</code> method. How are we supposed to wait on an update? Le Comte </picture> </div> Confer to the assiduous reader! If you read carefully the documentation of the Subscriber::next</code> method, you will see: This method is a convenience so you don't have to import a Stream</code> extension trait such as futures::StreamExt</code> or tokio_stream::StreamExt</code>. </blockquote> </div> </div> … fair enough. So Subscriber::next</code> mimics StreamExt::next</code>. Okay. Let's look at Stream</code></a> first, it's from the futures</code> crate</a>. Stream</code> defines itself as: A stream of values produced asynchronously. If Future<Output = T></code> is an asynchronous version of T</code>, then Stream<Item = T></code> is an asynchronous version of Iterator<Item = T></code>. A stream represents a sequence of value-producing events that occur asynchronously to the caller. The trait is modeled after Future</code>, but allows poll_next</code> to be called even after a value has been produced, yielding None once the stream has been fully exhausted. </blockquote> We aren't going to teach everything about Stream</code>: why this design, its pros and cons… However, wave its hand to ask you to come closer, did you notice how Future::poll</code></a> returns Poll<Self::Output></code>, whilst Stream::poll_next</code></a> returns Poll<Option<Self::Item>></code>? It's really similar to Iterator::next</code></a> which returns Option<Self::Item></code>. Let's take a look at Poll<T></code></a> don't you mind? It's an enum with 2 variants: Ready(value)</code> means a value</code> is immediately ready,</li> Pending</code> means no value is ready yet.</li> </ul> Then, what Poll<Option<T>></code> represents for a Stream</code>? Poll::Ready(Some(value))</code> means this stream has successfully produced a value</code>, and may produce more values on subsequent poll_next</code> calls,</li> Poll::Ready(None)</code> means the stream has terminated (and poll_next</code> should not be called anymore),</li> Poll::Pending</code> means no value is ready yet.</li> </ul> It makes perfect sense. A Future</code> produces a single value, whilst a Stream</code> produces multiple values, and Poll::Ready(None)</code> represents the termination of the stream, similarly to None</code> to represent the termination of an Iterator</code>. Ahh, I love consistency. We have the basis. Now let's see StreamExt</code></a>. It's a trait extending Stream</code> to add convenient combinator methods. Amongst other things, we find StreamExt::next</code></a>! Ah ha! It returns a Next</code> type which implements a Future</code>, exactly what eyeball</code> does actually. Remember our: // from `main.rs` while let Some(new_value) = subscriber.next().await { dbg!(new_value); }</code></pre> It is exactly the same pattern with StreamExt::next</code>: // from the documentation of `StreamExt::Next` use futures::stream::{self, StreamExt}; let mut stream = stream::iter(1..=3); assert_eq!(stream.next().await, Some(1)); assert_eq!(stream.next().await, Some(2)); assert_eq!(stream.next().await, Some(3)); assert_eq!(stream.next().await, None);</code></pre> Pieces start to come together, don't they? End of the detour. Back to eyeball_im::VectorSubscriber<T></code> . It is possible to transform this type into a Stream</code> with its into_stream</code></a> method. It returns a VectorSubscriberStream</code></a>. Naming is hard, but if I would have to guess, I would say it implements… a… Stream</code>? // from `eyeball-im` impl<T: Clone + Send + Sync + 'static> Stream for VectorSubscriberStream<T> { type Item = VectorDiff<T>;</code></pre> Yes, it does</a>! Dust blown away, the puzzle starts to appear clearly. Let's back on coding! $ cargo add eyeball-im futures Updating crates.io index Adding eyeball-im v0.5.0 to dependencies Features: - serde - tracing Adding futures v0.3.30 to dependencies Features: + alloc + async-await + executor + std - bilock - cfg-target-has-atomic - compat - futures-executor - io-compat - thread-pool - unstable - write-all-vectored [ … snip … ]</code></pre>// in `src/main.rs` use eyeball_im::ObservableVector; use futures::stream::StreamExt; use macro_rules_attribute::apply; use smol::{future::yield_now, Executor}; use smol_macros::main; #[apply(main!)] async fn main(executor: &Executor) { let mut observable = ObservableVector::new(); // Subscribe to `observable` and get a `Stream`. let mut subscriber = observable.subscribe().into_stream(); // Push one value. observable.push_back('a'); // Task that reads new updates from `observable`. let task = executor.spawn(async move { while let Some(new_value) = subscriber.next().await { dbg!(new_value); } }); // Now, let's update `observable`. observable.push_back('b'); // Eh `executor`: `task` can run now! yield_now().await; // More updates. observable.push_back('c'); observable.push_back('d'); // Eh `executor`, same. yield_now().await; drop(observable); task.await; }</code></pre> Time to show off: $ cargo run --quiet [src/main.rs:18:13] new_value = PushBack { value: 'a', } [src/main.rs:18:13] new_value = PushBack { value: 'b', } [src/main.rs:18:13] new_value = PushBack { value: 'c', } [src/main.rs:18:13] new_value = PushBack { value: 'd', }</code></pre> Do you see something new? Le Comte </picture> </div> Hmm, indeed. With Observable</code>, some values may “miss” because Observable</code> and Subscriber</code> have no buffer. The subscribers only return the current value when asked for. However, with ObservableVector</code>, things are different: no missing values. There are all here. As if there… was a buffer! And the values returned by the subscriber are not the raw T</code>: we see PushBack</code>. It comes from, check the documentation, VectorDiff::PushBack</code></a>! </div> </div> Good eyes, well done. First off, that's correct that PushBack</code> comes from VectorDiff</code></a>. Let's come back to this piece in a second: it is the cornerstone of the entire series, it deserves a bit of explanations. Second, yes, VectorSubscriber</code> returns all values! There is actually a buffer. It's a bit annoying to continue with a task</code> as we did so far, let's use assert_eq!</code></a> instead. // in `src/main.rs` use eyeball_im::{ObservableVector, VectorDiff}; // ^^^^^^^^^^ new! // … #[apply(main!)] async fn main(_executor: &Executor) { let mut observable = ObservableVector::new(); let mut subscriber = observable.subscribe().into_stream(); // Push one value. observable.push_back('a'); assert_eq!( dbg!(subscriber.next().await), Some(VectorDiff::PushBack { value: 'a' }), ); // Push another value. observable.push_back('b'); observable.push_back('c'); observable.push_back('d'); assert_eq!( dbg!(subscriber.next().await), Some(VectorDiff::PushBack { value: 'b' }), ); assert_eq!( dbg!(subscriber.next().await), Some(VectorDiff::PushBack { value: 'c' }), ); assert_eq!( dbg!(subscriber.next().await), Some(VectorDiff::PushBack { value: 'd' }), ); }</code></pre>$ cargo run --quiet [src/main.rs:16:9] subscriber.next().await = Some( PushBack { value: 'a', }, ) [src/main.rs:26:9] subscriber.next().await = Some( PushBack { value: 'b', }, ) [src/main.rs:30:9] subscriber.next().await = Some( PushBack { value: 'c', }, ) [src/main.rs:34:9] subscriber.next().await = Some( PushBack { value: 'd', }, )</code></pre> Beautiful! However… the code is a bit verbose, isn't it? Desperately waiting for an affirmative answer, okay, okay, something you may not know about me: I love macros. There. I said it. Let's quickly craft one: // in `src/main.rs` // before the `main` function macro_rules! assert_next_eq { ( $stream:ident, $expr:expr $(,)? ) => { assert_eq!(dbg!( $stream .next().await), Some( $expr )); }; }</code></pre> This macro does exactly what our assert_eq!</code> was doing, except now it's shorter to use, and thus more pleasant. Don't believe me? See by yourself: // in `src/main.rs` // at the end of the `main` function // Push one value. observable.push_back('a'); assert_next_eq!(subscriber, VectorDiff::PushBack { value: 'a' }); // Push another value. observable.push_back('b'); observable.push_back('c'); observable.push_back('d'); assert_next_eq!(subscriber, VectorDiff::PushBack { value: 'b' }); assert_next_eq!(subscriber, VectorDiff::PushBack { value: 'c' }); assert_next_eq!(subscriber, VectorDiff::PushBack { value: 'd' });</code></pre> There we go. Having a scientific and rigorous approach is important in our domain. We said ObservableVector</code> seems to contain a buffer, and VectorSubscriber</code> seems to pop values from this buffer. Let's play with that. I see two things to test: Modify the ObservableVector</code>, and subscribe to it after: Does the subscriber receive the update before it was created?</li> How many values the buffer can hold?</li> </ol> let mut observable = ObservableVector::new(); // Push a value before the subscriber exists. observable.push_back('a'); let mut subscriber = observable.subscribe().into_stream(); // Push another value. observable.push_back('b'); assert_next_eq!(subscriber, VectorDiff::PushBack { value: 'b' });</code></pre> If the subscriber</code> receives a</code>, it must fail, otherwise no error: $ cargo run --quiet [src/main.rs:25:5] subscriber.next().await = Some( PushBack { value: 'b', }, )</code></pre> Look Ma', no error! Le Comte </picture> </div> We have learned that a VectorSubscriber</code> is aware of the new updates that are made once it exists. A VectorSubscriber</code> is not aware of updates that happened before its creation. In the example, VectorDiff::PushBack { value: 'a' }</code> is not received before subscriber</code> was created. However, VectorDiff::PushBack { value: 'b' }</code> is received because it happened after subscriber</code> was created. It makes perfect sense. It suggests that the buffer lives inside VectorSubscriber</code>, and not inside ObservableVector</code>. Or maybe the buffer is shared between the observable and the subscribers, with the buffer having some specific semantics, like a channel. We would need to look at the implementation to be sure. </div> </div> Agree. This is left as an exercise for the reader, wink to you. We have an answer to question 1. What about question 2? The size of the buffer. // in `src/main.rs` let mut observable = ObservableVector::new(); let mut subscriber = observable.subscribe().into_stream(); // Push ALL THE VALUES! observable.push_back('a'); observable.push_back('b'); observable.push_back('c'); observable.push_back('d'); observable.push_back('e'); observable.push_back('f'); observable.push_back('g'); observable.push_back('h'); observable.push_back('i'); observable.push_back('j'); observable.push_back('k'); observable.push_back('l'); observable.push_back('m'); observable.push_back('n'); observable.push_back('o'); observable.push_back('p'); assert_next_eq!(subscriber, VectorDiff::PushBack { value: 'a' }); // no need to assert the others</code></pre>$ cargo run --quiet [src/main.rs:36:5] subscriber.next().await = Some( PushBack { value: 'a', }, )</code></pre> Hmm, the buffer doesn't seem to be full with 16 values. Let's add a couple more: // in `src/main.rs` // [ … snip … ] observable.push_back('n'); observable.push_back('o'); observable.push_back('p'); observable.push_back('q'); // ^ new! observable.push_back('r'); // ^ new! assert_next_eq!(subscriber, VectorDiff::PushBack { value: 'a' });</code></pre>$ cargo run --quiet [src/main.rs:38:5] subscriber.next().await = Some( Reset { values: [ 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', ], }, ) thread 'main' panicked at src/main.rs:38:5: assertion `left == right` failed left: Some(Reset { values: ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r'] }) right: Some(PushBack { value: 'a' }) note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace</code></pre> Oh! An error, great! Our assert_next_eq!</code> has failed. subscriber</code> does not receive a VectorDiff::PopBack</code> but a VectorDiff::Reset</code>. Let's play with ObservableVector::with_capacity</code></a> a moment, maybe it's related to the buffer capacity? Let's change a single line: let mut observable = ObservableVector::with_capacity(32); // ^^^^^^^^^^^^^^^^^ new!</code></pre>$ cargo run --quiet [src/main.rs:38:5] subscriber.next().await = Some( PushBack { value: 'a', }, )</code></pre> Le Comte </picture> </div> We have learned that ObservableVector::with_capacity</code> controls the size of the buffer. The name could suggest that it controls the capacity of the observed Vector</code>, à la Vec::with_capacity</code></a>, but it must not be confused. For a reason we ignore so far, when the buffer is full, we receive a VectorDiff::Reset</code>. We need to learn more about this type. </div> </div> Observable differences#</a> </h2> The previous section was explaining how immutable data structures could save us by cheaply and efficiently cloning the data between the observable and its subscribers. However, we see that eyeball-im</code></a>, despite using imbl</code></a>, does not share an imbl::Vector</code></a> but an eyeball_im::VectorDiff</code></a>. Why such design? It looks like a drama. A betrayal. An act of treachery! Well. Firstly, eyeball-im</code> is relying on some immutable properties of Vector</code>. And secondly, the reason for which VectorDiff</code> exists is simple. If a subscriber receives Vector</code>s, how is the user able to see what has changed? The user (!) would be responsible to calculate the differences between 2 Vector</code>s every time! Not only this is costly, but it is utterly error-prone. Le Comte </picture> </div> Are you suggesting that VectorSubscriber</code> (or VectorSubscriberStream</code>) calculates the differences between the Vector</code>s itself so that the user doesn't have to? I still see many problems though. I believe the order of the VectorDiff</code>s matters a lot for some use cases. For example, let's consider two consecutive Vector</code>s: ['a', 'b', 'c']</code> and</li> ['a', 'c', 'b']</code>.</li> </ol> Has 'b'</code> been removed and pushed back, or 'c'</code> been popped back and inserted? How can you decide between the twos? </div> </div> We can't —it would be implementation specifics anyway— and we don't want to. The user is manipulating the ObservableVector</code> in a special way, and we should ideally not change that. These VectorDiff</code> actually comes from ObservableVector</code> itself! Let's look at the implementation of ObservableVector::push_back</code></a>: pub fn push_back(&mut self, value: T) { // [ … snip … ] self.values.push_back(value.clone()); // ^^^^^^ this is a `Vector`! self.broadcast_diff(VectorDiff::PushBack { value }); // ^^^^^^^^^^ here you are… }</code></pre> Each method adding or removing values on the ObservableVector</code> emits its own VectorDiff</code> variant. No calculation, it's purely a mapping: ObservableVector::…</code></th> VectorDiff::…</code></th> Meaning</th></tr></thead> append(values)</code></td> Append { values }</code></td> Append many values</code></td></tr> clear()</code></td> Clear</code></td> Clear out all the values</td></tr> insert(index, value)</code></td> Insert { index, value }</code></td> Insert a value</code> at index</code> </td></tr> pop_back()</code></td> PopBack</code></td> Remove the value at the back</td></tr> pop_front()</code></td> PopFront</code></td> Remove the value at the front</td></tr> push_back(value)</code></td> PushBack { value }</code></td> Add value</code> at the back</td></tr> push_front(value)</code></td> PushFront { value }</code></td> Add value</code> at the front</td></tr> remove(index)</code></td> Remove { index }</code></td> Remove value at index</code></td></tr> set(index, value)</code></td> Set { index, value }</code></td> Replace value at index</code> by value</code></td></tr> truncate(length)</code></td> Truncate { length }</code></td> Truncate to length</code> values</td></tr> </tbody></table> Mappings of ObservableVector</code> methods to VectorDiff</code> variants. </figcaption> </figure> See, for each VectorDiff</code> variant, there is an ObservableVector</code> method triggering it. Le Comte </picture> </div> And what about VectorDiff::Reset</code>? We were receiving it when the buffer was full apparently. You are not mentioning it, and if I take a close look at ObservableVector</code>'s documentation, I don't see any reset</code> method. Is it only an internal thing? </div> </div> You are correct. When the buffer is full, the subscriber will provide a VectorDiff::Reset { values }</code> where values</code> is the full list of values. The documentation says: The subscriber lagged too far behind, and the next update that should have been received has already been discarded from the internal buffer. </blockquote> If the subscriber didn't catch all the updates, the best thing it can do is to say: Okay, I am late at the party, I've missed several things, so here is the current state!</q>. This is not ideal, but the subscriber is responsible to not lag, and this design avoids having missing values. If a subscriber receives too much VectorDiff::Reset</code>s, the user may consider increasing the capacity of the ObservableVector</code>. Filtering and sorting with higher-order Stream</code>s#</a> </h2> We are reaching the end of this episode. And you know what? We have set all the parts to talk about higher-order Stream</code>, chante victory and dance at the same time! At the beginning of this episode, we were saying that the Matrix Rust SDK is able to filter and to sort an ObservableVector</code> representing all the rooms. How? VectorSubscriberStream</code> is a Stream</code>. More specifically, it is a Stream<Item = VectorDiff<T>></code>. Now questions: What's the difference between an unfiltered Vector</code> and a filtered Vector</code>?</li> What's the difference between an unsorted Vector</code> and a sorted Vector</code>?</li> What's the difference between a filtered Vector</code> and a sorted Vector</code>?</li> and so on.</li> </ul> All of them are strictly Stream<Item = VectorDiff<T>></code>! However, the VectorDiff</code>s aren't the same. A simple example. Let's say we build a vector by inserting 1</code>, 2</code>, 3</code> and 4</code>. We subscribe to it, and we want to filter out all the even numbers. Instead of receiving: VectorDiff::Insert { index: 0, value: 1 }</code>,</li> VectorDiff::Insert { index: 1, value: 2 }</code>,</li> VectorDiff::Insert { index: 2, value: 3 }</code>,</li> VectorDiff::Insert { index: 3, value: 4 }</code>.</li> </ul> … we want to receive: VectorDiff::Insert { index: 0, value: 1 }</code>,</li> VectorDiff::Insert { index: 1, value: 3 }</code>: note the index</code>, it is not 2 but 1!</li> </ul> We will see how all that works in the next episodes and how powerful this design is, especially when it comes to cross-platform UI (user interface). We are going to learn so much about Stream</code> and Future</code>, it's going to be fun! Latine expression meaning salvation hope. ↩</a> </li> Latine expression meaning bless are the poor in spirit. ↩</a> </li> Relaxed-Radix-Balanced (RRR) Vector: A Practical General Purpose Immutable Sequence</a></cite> by Sticki N., Rompf T., Ureche V. and Bagwell P. (2015, August), in Proceedings of the 20th ACM SIGPLAN International Conference on Functional Programming (pp. 342-354). ↩</a> </li> Theory and Practise of Chunked Sequences</a></cite> by Acar U. A., Charguéraud A., and Rainey M. (2014), in Algorithms-ESA 2014: 22th Annual European Symposium, Wroclaw, Poland, September 8-10, 2014. Proceedings 21 (pp. 25-36)., Springer Berlin Heidelberg. ↩</a> </li> </ol> </section> I've loved Wasmer, I still love Wasmer 2021-10-04T00:00:00+00:00 This article could also have been titled How I failed to change Wasmer. Today is my last day at Wasmer</a>. For those who don't know this name, it has a twofold meaning: it's a very popular WebAssembly runtime</a>, as well as a startup. I want to write about what I've been able to accomplish during my time at Wasmer (a high overview, not a technical view), and what forces me to leave the company despite being one of its co-founder. I reckon my testimony can help other people to avoid digging into the hell I (and my colleagues) had to endure. I'm available for work, you can contact me at [email protected]">[email protected]</a>, @mnt_io</a>, ivan-enderlin</a> (LinkedIn). From nothing to pure awesomeness#</a> </h2> I've joined the Wasmer company at its early beginning, in March 2019. The company was 3 months old. My initial role was to write and to improve the runtime itself, and to create many embeddings, i.e. ways to integrate the Wasmer runtime inside various technologies, so that WebAssembly can run everywhere. I can say with confidence that my work is a success. I've learned a lot, and I've worked on so many different projects, technologies, hacked so many things, collaborated with so many people, every action was led by the passion. At the time of writing, Wasmer has an incredible growth. In 2.5 years only, the runtime has more than 10'500 stars on Github, and is one of the most popular WebAssembly runtime in the world! It's used by many various companies, such as Confio</a>, Fluence Labs</a>, HOT-G</a>, Brave</a>, Google</a>, Apple</a>, SpaceMesh</a>, Linkerd</a>, SingleStore</a>, CleverCloud</a> or Kong</a> to name a few (for the ones I can name though, however other companies are also using Wasmer in very critical environments). Most of my engineering job happened on the Wasmer runtime itself. At the time of writing, I'm the #2 contributor on the project. I was working on every parts of the runtime</a>: the API, the C API, the compilers, the ABI (mostly WASI), the engines, the middlewares, and the VM itself which is the most low-level foundamental layer of the runtime. The runtime provides so many features. It is an impressively powerful runtime for WebAssembly, and I'm saying that with a neutral and respectful mindset. Not everything is perfect obviously but I did my best to set up a truly user-friendly learning environment, with an important documentation and a collection of examples</a> that illustrate many features. I strongly believe it contributed to Wasmer's popularity to great extent. I would like to highlight the most notable embedding projects I've created: wasmer-c-api</code></a> is the C embedding for Wasmer. It's part of the Wasmer runtime itself, and is fully written in Rust. The documentation, the C examples</a>, everything is super polished to offer the best experience possible. Mark McCaskey</a> and I are the authors of this project.</li> wasmer-python</code></a> is the Python embedding for Wasmer. At the time of writing, it's been installed more than 5 millions times (I'm counting the compiler packages too, like wasmer-compiler-cranelift</code> and so on), and 1300 stars on Github. There is about 300'000 downloads per months, and it continues to grow! The code is written in Rust, and it relies on the awesome pyo3</code> project</a> .</li> wasmer-go</code></a> is the Go embedding for Wasmer. It's hard to know how much total downloads we have because of how the Go ecosystem is designed, but we have about 60'000 downloads per months from Github (I'm excluding the forks of the project), and 1600 stars on Github. The code is written in Go and uses cgo</code></a> to bind against the C API. Almost all blockchain projects that use WebAssembly are using wasmer-go</code>, which is a popularity I wasn't expecting.</li> wasmer-ruby</code></a> is the Ruby embedding for Wasmer. It's not as popular as the others, but it's also very polished and it's finding its place in the Ruby ecosystem. The code is written in Rust, and it relies on the awesome rutie</code> project</a> .</li> I won't detail all the projects, but there is also wasmer-php</code></a>, wasmer-java</code></a>, wasmer-postgres</code></a>… Because of the Wasmer runtime API and C API we have designed, many developers around the globe have been able to create a lot more embeddings, such as in C#</a>, D</a>, Elixir</a>, R</a>, Swift</a>, Zig</a>, Dart</a>, Lisp</a> and so on.</li> </ul> Other fun notable projects are: sonde-rs</code></a>, a library to compile USDT probes into a Rust library,</li> llvm-custom-builds</code></a>, a sandbox to produce custom LLVM builds for various platforms,</li> loupe</code></a>, a set of tools to analyse and to profile Rust code,</li> wasmer-interface-types</code></a>, a “living” (understand an unstable playground) library that implements the WebAssembly Interface Types proposal</a>,</li> inline-c-rs</code></a>, to write and to execute C code inside Rust,</li> in-memory filesystem, that acts exactly like std::fs</code>.</li> </ul> As you might think, I've learned so much. The impostor syndrom was very present because I was constantly trying to do something I didn't know. It's part of the routine at Wasmer: Trying something for the first time. But it's also what kept me motivated, and it was the energy for my passion. This list above shows released projects, but I've also experimented (and sometimes at two hairs of a release) with: Unikernels</a>; this one was really fun, given a WebAssembly module and a filesystem, we were able to generate a unikernel that was executing the given program,</li> Parser; to write the fastest WebAssembly parser possible, it was working, but never released,</li> HAL (Hardware Abstraction Layer) ABI for WebAssembly, so that we can run WebAssembly on small chips super easily (think of IoT),</li> Networking; an extension to WASI to support networking (TCP and UDP sockets), with an implementation in Rust</a>, libc</a>, and even Zig</a>! We were able to compile C programs to WebAssembly like cURL, or TCP servers written with kqueue or epoll etc, and to execute them on any platforms.</li> </ul> All those things were working. It's absolutely crazy what WebAssembly can do today, and I still truly and deeply believe in this technology. I'm not the only one: YCombinator</a> and SpeedInvest</a> are also founders that believe in Wasmer. So. What a dream, huh? The toxic working environment#</a> </h2> WebAssembly is nothing without its community. I won't name people to avoid missing important persons, but all the contributors are doing amazing work, to create something new, something special, something right. Wasmer is a success. The Wasmer runtime is nothing without the incredible, marvelous, exceptional team of engineers behind it. In no particular order: Lachlan Sneff</a>, Mark McCaskey</a>, Julien Bianchi</a>, Nick Lewycky</a>, Heyang Zhou</a>, Mackenzie Clark</a>, Brandon Fish</a>. All of them, with no exception, have put a lot of passion in this project. It is what it is today because of them and also because of the contributors we have been honored to welcome. The open source side of Wasmer was intense but also an important source of joy. It is a respectful place to work. However, the inside reality was very different. All the employees hereinabove have left the company. Almost all of them due to a burn-out or conflicts or strong disagreements with the company leadership. I am leaving due to a severe burn-out. I would like to briefly share my journey to my first burn-out in few points: I've started as an engineer. I love coding. I love hacking. In Wasmer, I've found a place to learn a lot and to express my passion. We had a lot of pressure mostly because our friendly “competitors” had more people dedicated to work on their runtimes, more money, more power, better marketing and so on. That's the inevitable burden of any startup. When you're competing against giants, that's what happens. And that's OK. It's part of the game.</li> During that time, we were delivering more and more projects, more and more features, at an incredible pace. New hats: Release manager, project manager, more product ownership, more customers to be connected with, more contributors to help, more issues to triage, blog writer etc. The pace was accelerating too fast, something we did notice on multiple occasions.</li> The CEO, Syrus Akbary</a>, had evidently a lot of pressure on its shoulders. It sadly resulted in the worst possible way: micro-management, stress, pressure, bad leadership, lack of vision, lack of trust in the employees, changing the roadmap constantly, lies, secrets etc.</li> As one of the older in the company, with a family of two kids, I probably got more “wisdom”. I've decided to create a safe place for employees to express their frustrations, their needs, to find solutions together. De facto, I became the “person of trust” in the company. I got new hats, new pressures.</li> SARS-CoV-2 hit. School at home. Lock-down. More micro-management, more stress. Wasmer was running out of money. I brought a new investor that saved the company. New hat.</li> After too many departures (85% of the engineering team!), I tried to take more space and to take more responsabilities in the company. That was at the beginning of 2021. It was my last attempt to save the company from a disaster before leaving. I couldn't imagine leaving such brilliant and successful projects without having tried everything I could.</li> Then I became a late co-founder of Wasmer. Too many new hats: Doing hiring interviews, accountabilities, helping to define the roadmap (with another awesome person, friend, and employee), handling legal aspects to hire people in multiple countries with non-precarious contracts etc.</li> Obviously, I was also doing the job of all the engineers that have left. They were not replaced for unknown reasons. It was absolutely madness. The pace was unsustainable.</li> Finally, the crack. The CEO continued to change the roadmap, to take bad decisions, to not recognize all the efforts we were doing to save/grow the company. It was my turn to be declared in a severe burn-out by my physician. The last engineer to fall.</li> </ul> Another point: Syrus Akbary also has made many public errors that have created hostility against the company. Hopefully people were smart and kind enough to make the difference between the employees and the CEO (I won't name the people but they will recognize themselves: Thank you). I tried to fix that situation. Discussing with dozens of person to restore empathy and forgiveness, to create better collaboration, to cure and move forward. It was exhausting. I know people have appreciated what I did, but my mental health was ruined. Considering all the time I've devoted to the company, the very few consideration I got in return, the countless work hours (4 days per week, but frequently closing the computer at 1am due to very late meetings, I was working like hell), the precarious contract I had (did you ever see a co-founder with a freelancer contract?), the toxic working environment, the constant pressure etc., my passion was intact but my motivation was seriously damaged. Doing overtime was never recorded and was happening more than frequently, but taking half a day to take care of a sick child was immediately counted as holidays; the balance was broken. Criticisms. Micro-management. Disapprovals. Rewriting the facts and the reality to criticize what you're doing, flipping things against you, avoiding discussion when things get stormy. We even had a meeting titled “Why you're not productive enough” whilst everyone was working as hell, right after the rewrite of the entire runtime to release Wasmer 1.0, a period we all affectionately called “The Rewrite of Hell”. The team deserved vacations, congratulations, attentions, gratitude, … not such a shitty meeting. Well, you get the idea. When I've been declared in a severe burn-out, I had to take a break. The reaction from the CEO was… unexpected: Zero empathy, asking to never ever being sick again (otherwise I will be fired), dividing my equities, asking me to work more, saying I've never been involved in the company etc. That was the final straw to me. That's the wrong way to treat an employee, a collaborator, a contributor, the co-founder. What's next?#</a> </h2> I need to recover. As you can imagine, working 2.5 years at this pace leaves sequelae. Hopefully a couple of months should suffice. I'm still in love with Wasmer, the runtime, the open source projects we have created. It has a bright future. More companies are contributing to it, more individual contributors are bringing their stones to the monument. The project is owned by the public, by the users, by the contributors, they are doing most of the work today. It's well tested, well documented, it's easy to contribute to it. It's a fabulous open source piece of technology. I strongly hope Wasmer, the company, will change. The products that are going to be released are absolutely fantastic. It's a technology shift. It will enable the Decentralized Web, how we compile and distribute programs, how we will even consume programs. Wasmer has a solid bright future. I really hope things will change, and I wish the best to and am passing on the torch to the adventurers that will continue to move the company forward. I'm just too skeptical that things can improve or even slightly change. We have built something great. Please take a great care of it. As I said, I'm available for a new adventure, you can contact me at [email protected]">[email protected]</a>, @mnt_io</a>, ivan-enderlin</a> (LinkedIn). Discussions on Twitter</a> and on HackerNews</a>. Bye bye WhatsApp, hello ? 2021-01-19T00:00:00+00:00 Ce billet est en français, essentiellement à destination de mes ami(e)s et familles, il sert à vulgariser très rapidement les enjeux autour de WhatsApp</a>, Signal</a>, Telegram</a> et Matrix</a> (spoiler, c'est le gagnant). Tout le monde me pose la même question, alors voici une réponse rapidement en brouillon qui va m'éviter du copier/coller ! Je ne rentre volontairement pas dans les détails. Il faut que ce document reste à la portée de tous, sans aucune connaissance en réseau, chiffrement, sécurité etc. Ceux qui ont ces connaissances savent déjà que Matrix est le réseau vers lequel aller ;-). Les bases#</a> </h2> Quand on parle de messageries, il y a 2 choses primordiales plus 1 bonus : le chiffrement ;</li> la topologie du réseau (c'est facile, n'ayez pas peur) ;</li> l'accès libre et gratuit sans restriction au code source (open source).</li> </ul> Nous pouvons aussi parler du modèle économique du réseau rapidement, voir le tableau comparatif. Le chiffrement#</a> </h3> Pour respecter la vie privée et éviter l'espionnage et le vol des données, il faut que le chiffrement se fasse de bout en bout (end to end encryption). Ça veut dire que vous seul avez la clé pour chiffrer et/ou déchiffrer vos messages, et personne d'autre. Par message, j'entends message texte, audio, image, vidéo, appels audios-vidéos, tout. Vos données sont à vous, et uniquement vous, et personne ne peut les utiliser, à part la personne avec qui vous les partagez (qui elle, normalement à une clé de déchiffrement par exemple). Les clés servent aussi à identifier la personne avec qui vous parlez, ça permet d'éviter le vol d'identité. La topologie#</a> </h3> La plupart des réseaux sont centralisés : ça veut dire qu'on a un gros silot, un énorme ordinateur/serveur, et que tout le monde est dessus. Ça pose plein de problèmes : impossible d'avoir le contrôle dessus ;</li> impossible de faire confiance ;</li> faille unique.</li> </ul> Je prends l'exemple de WhatsApp pour illustrer tout ça parce que ça parle à tout le monde : Facebook décide unilatéralement de déchiffrer le réseau, personne n'a le contrôle dessus, c'est une violation grave de la vie privée de milliards de personnes et on ne peut rien faire (à part quitter le réseau). Avions-nous confiance dans ce que faisait Facebook avec nos données WhatsApp avant ? Non, aucunement. Ils disaient que c'était chiffré, l'était-ce vraiment ? J'accorde plus de confiance dans ceux qui ont créé et chiffré le réseau avant qu'il ne soit racheté par Facebook, donc j'ai envie d'y croire, mais… je ne peux pas le vérifier ! Pourquoi ? Parce que personne (en dehors de quelques employés chez Facebook) n'a accès au code source, aux programmes, qui font tourner WhatsApp. Et pour le côté faille unique, si Facebook est attaqué, c'est l'entièreté du réseau qui s'effondre, c'est une faille unique, un single point of failure comme on dit dans le métier. Pareil si le réseau est hacké, c'est un accès illimité à tout le réseau. Aucune transparence = aucune confiance. </blockquote> Mais il existe une alternative majeure bien sûr ! Les réseaux décentralisés. Au lieu d'avoir un serveur, il y a en des centaines, des milliers. Il n'y a plus de contrôle possible. Il n'y a plus de single point of failure. Un hacker ne peut accéder au pire qu'aux données d'un seul serveur, pas de tous les serveurs (il existe pleins d'exceptions mais je vulgarise, hein). Nous pouvons créer autant de serveurs que nous le souhaitons. Souvent, ce sont des réseaux open source, donc nous pouvons lire le code des programmes, vérifier qu'ils font bien ce qu'ils proclament faire. Tableau comparatif#</a> </h2> Comparons les services populaires avec ces critères de bases. Service</td> Chiffrement</td> Topologie</td> Open source</td> </tr> WhatsApp</td> bout en bout (pour le moment)</td> centralisé (US)</td> non</td> </tr> Telegram</td> bout en bout (pour le moment)</td> centralisé (Dubaï, US)</td> non</td> </tr> Signal</td> bout en bout</td> centralisé (US)</td> oui mais…</td> </tr> Matrix</td> bout en bout</td> décentralisé</td> oui</td> </tr> </tbody> </table> Comparons la base ! </figcaption> </figure> Signal est open source, mais nous ne pouvons pas vérifier ce qui est installé sur les serveurs, parce que le serveur est privé. De plus, le serveur open source n'a pas été mis à jour depuis avril 2020</a>, en année Informatique, c'est très long. Ça cache quelque chose ? Aucune idée, je ne peux pas le savoir, car je n'ai pas d'éléments pour prendre une décision. Est-ce que je veux déposer mes données privées sur un service dans lequel je n'ai pas confiance ? En plus, Signal comme WhatsApp sont hébergés/situés aux US, avec les lois liberticides qu'on leur connaît bien (comme le Cloud Act). Signal limite la casse grâce au chiffrement de bout en bout, mais peut être qu'une backdoor est présente et qu'on ne le saura jamais. Aucune transparence = aucune confiance </blockquote> Les réseaux décentralisés sont supérieurs à tous les niveaux (pas de contrôle, pas de hack massif etc.). Les réseaux open source sont ceux en qui nous pouvons avoir confiance. Donc le choix est vite fait, le gagnant ici est Matrix. Comparons maintenant comment les services sont financés, parce que c'est important. Si un service n'est pas rentable, il pourrait avoir de l'appétit pour les données de ses utilisateurs, et là c'est dangereux (c'est exactement ce qu'il se passe avec Facebook et WhatsApp). Service</td> Revenues</td> </tr> WhatsApp</td> Facebook veut utiliser les données privées pour vendre de la publicité ciblée.</td> </tr> Telegram</td> Les fondateurs sont millionnaires et injectent de l'argent. Dans peu de temps, financement via pubs et comptes premiums.</td> </tr> Signal</td> Organisation à but non-lucratif qui opère via des dons.</td> </tr> Matrix</td> Matrix développe, offre ou vend des services autour du réseau, mais pas autour des données !</td> </tr> </tbody> </table> Comment sont financés les services ? </figcaption> </figure> Les gagnants ici sont Signal et Matrix. Conclusion : Matrix gagnant#</a> </h2> Dans le cas des réseaux centralisés, Signal est une meilleure alternative à WhatsApp et Telegram de part son mode de financement (donc son appétit pour les données des utilisateurs), mais ils sont tous sujets aux même problèmes : aucune confiance car pas de transparence, hébergés aux US etc. Mais les réseaux décentralisés sont supérieurs car ils résolvent tous ces problèmes ! Matrix est décentralisé, est financé par des services autour du réseau mais pas par les données du réseau (qui sont inaccessibles de toute façon, elles n'existent que sur vos téléphones et ordinateurs, nul part ailleurs). J'utilise Matrix. Je vous conseille d'utiliser Matrix. Partir sur Signal, c'est sortir de la gueule d'un loup pour aller dans celle d'un autre. Je suis admiratif du travail des développeurs de chez Signal, ils sont vraiment bons, leur protocole de chiffrement est magnifique, mais je n'ai pas confiance dans leur service parce que je ne peux pas. Et personne ne le peut. J'utilise aussi WhatsApp et Signal pour rester en contact avec mes amis et ma famille, et leur dire d'utiliser Matrix, mais je n'y publierai jamais de données personnelles, photos ou quoi que ce soit, je n'ai aucune confiance. Libre à vous aussi d'utiliser plusieurs réseaux, après tout nous jonglons déjà avec plusieurs réseaux (mail, SMS, WhatsApp, Matrix, Twitter, Mastodon</a> etc.), ça n'est pas un problème ! Premier pas avec Matrix#</a> </h2> C'est parti, petit tuto Matrix. Le réseau est exceptionnel, mais le client officiel (Element</a>) est encore un peu « brut » à utiliser comparé à Signal ou WhatsApp. Notez que ça évolue très très vite (je compte 616 contributeurs qui travaillent dessus bénévolement, encore une grande force de l'open source !). Ce qui va vous titiller le plus c'est : vous ne pouvez pas toujours identifier vos contacts par numéro de téléphone (seulement s'ils sont enregistrés sur un serveur d'identité). Pourquoi ? Parce que votre compte à un identifiant, comme une adresse email. Le mien est @mnt_io:matrix.org</code> (le format est @identifiant:serveur</code>). C'est bien meilleur pour la vie privée. Et pis, ça n'est pas différent de MSN ou de tout autre réseau de l'époque, c'est vraiment WhatsApp qui a imposé la « découvertabilité » par le numéro de téléphone. Bien que très pratique, c'est dangereux pour la vie privée. Donc, go, on installe le client : sur iOS, macOS etc</a>.,</li> sur Android</a>,</li> sur votre bureau ou votre navigateur</a>.</li> </ul> Puis on crée un compte, et ajoutez moi. C'est parti ! Matrix/Element est basé sur les groupes. Les chats « directs » (1:1) sont des groupes aussi. Vous pouvez même rejoindre des rooms (gros groupes, des communautés) avec des centaines voire des milliers de personnes dedans. C'est très flexible. Parce que c'est open source, n'importe qui peut écrire son propre client (programme qui se connecte au réseau). Il existe des clients alternatives, comme Nio</a> ou FluffyChat</a>, ou même Ditto Chat</a>. Tous ces clients sont encore en beta, mais ça montre un futur très excitant pour Matrix avec des clients de plus en plus aboutis ! Matrix, Element, Vector, hein ?#</a> </h3> Element c'est le nom de l'entreprise qui travaille/développe le réseau, les serveurs et le client ;</li> Matrix c'est le nom du réseau ;</li> Vector, c'est l'ancien nom d'Element.</li> </ul> On parle souvent de façon indifférentiée de Matrix ou Element, c'est un abus de langage. Before: Mark as read. Now: Mark has read. </blockquote> Et par pitié, quittez Facebook… Announcing the first Java library to run WebAssembly: Wasmer JNI 2020-05-13T00:00:00+00:00 This is a copy of an article I wrote for Wasmer</a>. WebAssembly</a> is a portable binary format. That means the same file can run anywhere. To uphold this bold statement, each language, platform and system must be able to run WebAssembly — as fast and safely as possible. </blockquote> People who are familiar with Wasmer are used to this kind of announcement! Wasmer is written in Rust, and comes with an additional native C API. But you can use it in a lot of other languages. After having announced libraries to use Wasmer, and thus WebAssembly, in: PHP with the ext/wasm</code> extension</a>,</li> Python with the wasmer</code> library</a>,</li> Ruby with the wasmer</code> library</a>,</li> Go with the wasmer</code> library</a> (see Announcing the fastest WebAssembly runtime for Go: wasmer</code></a>), and even</li> Postgres with the wasmer</code> library</a> (see Announcing the first Postgres extension to run WebAssembly</a>),</li> and many other contributions in .NET/C#</a>, R</a> and Elixir</a>…</li> </ul> …we are jazzed to announce that Wasmer has now landed in Java</a>! Let’s discover the Wasmer JNI library together. Installation#</a> </h2> The Wasmer JNI (Java Native Interface) library is based on the Wasmer runtime</a>, which is written in Rust</a>, and is compiled to a shared library. For your convenience, we produce one JAR (Java Archive) per architecture and platform. By now, the following are supported, consistently tested, and pre-packaged (available in Bintray</a> and Github Releases</a>): amd64-darwin</code> for macOS, x86 64bits, </li> amd64-linux</code> for Linux, x86 64 bits, </li> amd64-windows</code> for Windows, x86 64 bits. </li> </ul> More architectures and more platforms will be added in the near future. If you need a specific one, feel free to ask</a>! However, it is possible to produce your own JAR for your own platform and architecture</a>. The JAR files are named as follows: wasmer-jni-$(architecture)-$(os)-$(version).jar</code>. Thus, to include Wasmer JNI as a dependency of your project (assuming you use Gradle</a>), write for instance: dependencies { implementation "org.wasmer:wasmer-jni-amd64-linux:0.2.0" }</code></pre> JAR are hosted on the Bintray/JCenter repository under the [wasmer-jni](https://bintray.com/wasmer/wasmer-jni/wasmer-jni)</code> project. They are also attached to our Github releases as assets</a>. Calling a WebAssembly function from Java#</a> </h2> As usual, let’s start with a simple Rust program that we will compile to WebAssembly, and then execute from Java. #[no_mangle] pub extern fn sum(x: i32, y: i32) -> i32 { x + y }</code></pre> After compilation to WebAssembly, we get a file like this one</a>, named simple.wasm</code>. The following Java program executes the sum</code> exported function by passing 5</code> and 37</code> as arguments: import org.wasmer.Instance; import java.io.IOException; import java.nio.file.Files; import java.nio.file.Paths; class SimpleExample { public static void main(String[] args) throws IOException { // Read the WebAssembly bytes. byte[] bytes = Files.readAllBytes(Paths.get("simple.wasm")); // Instantiate the WebAssembly module. Instance instance = new Instance(bytes); // Get the `sum` exported function, call it by passing 5 and 37, and get the result. Integer result = (Integer) instance.exports.getFunction("sum").apply(5, 37)[0]; assert result == 42; instance.close(); } }</code></pre> Great! We have successfully executed a Rust program, compiled to WebAssembly, in Java. As you can see, it is pretty straightforward. The API is very similar to the standard JavaScript API, or the other API we have designed for PHP, Python, Go, Ruby etc. The assiduous reader might have noticed the [0]</code> in .apply(5, 37)[0]</code> pattern. A WebAssembly function can return zero to many values, and in this case, we are reading the first one. Note: Java values passed to WebAssembly exported functions are automatically downcasted to WebAssembly values. Types are inferred at runtime, and casting is done automatically. Thus, a WebAssembly function acts as any regular Java function. </blockquote> Technically, an exported function is a functional interface as defined by the Java Language Specification (i.e. it is a [FunctionalInterface](https://docs.oracle.com/javase/8/docs/api/java/lang/FunctionalInterface.html)</code>). Thus, it is possible to write the following code where sum</code> is an actual function (of kind org.wasmer.exports.Function</code>): import org.wasmer.Instance; import org.wasmer.exports.Function; import java.io.IOException; import java.nio.file.Files; import java.nio.file.Paths; class SimpleExample { public static void main(String[] args) throws IOException { // Read the WebAssembly bytes. byte[] bytes = Files.readAllBytes(Paths.get("simple.wasm")); // Instantiate the WebAssembly module. Instance instance = new Instance(bytes); // Declare the `sum` function, as a regular Java function. Function sum = instance.exports.getFunction("sum"); // Call `sum`. Integer result = (Integer) sum.apply(1, 2)[0]; assert result == 3; instance.close(); } }</code></pre> But a WebAssembly module not only exports functions, it also exports memory. Reading the memory#</a> </h2> A WebAssembly instance has one or more linear memories, a contiguous and byte-addressable range of memory spanning from offset 0 and extending up to a varying memory size, represented by the org.wasmer.Memory</code> class. Let’s see how to use it. Consider the following Rust program: #[no_mangle] pub extern fn return_hello() -> *const u8 { b"Hello, World!\0".as_ptr() }</code></pre> The return_hello</code> function returns a pointer to the statically allocated string. The string exists in the linear memory of the WebAssembly module. It is then possible to read it in Java: import org.wasmer.Instance; import org.wasmer.Memory; import java.io.IOException; import java.nio.ByteBuffer; import java.nio.file.Files; import java.nio.file.Paths; class MemoryExample { public static void main(String[] args) throws IOException { // Read the WebAssembly bytes. byte[] bytes = Files.readAllBytes(Paths.get("memory.wasm")); // Instantiate the WebAssembly module. Instance instance = new Instance(bytes); // Get a pointer to the statically allocated string returned by `return_hello`. Integer pointer = (Integer) instance.exports.getFunction("return_hello").apply()[0]; // Get the exported memory named `memory`. Memory memory = instance.exports.getMemory("memory"); // Get a direct byte buffer view of the WebAssembly memory. ByteBuffer memoryBuffer = memory.buffer(); // Prepare the byte array that will hold the data. byte[] data = new byte[13]; // Let's position the cursor, and… memoryBuffer.position(pointer); // … read! memoryBuffer.get(data); // Let's encode back to a Java string. String result = new String(data); // Hello! assert result.equals("Hello, World!"); instance.close(); } }</code></pre> As we can see, the Memory</code> API provides a buffer</code> method. It returns a direct byte buffer</a> (of kind java.nio.ByteBuffer</code>) view of the memory. It’s a standard API for any Java developer. We think it’s best to not reinvent the wheel and use standard API as much as possible. The WebAssembly memory is dissociated from the JVM memory, and thus from the garbage collector. You can read the Greet Example</a> to see a more in-depth usage of the Memory</code> API. </blockquote> More documentation#</a> </h2> The project comes with a Makefile</code>. The make javadoc</code> command will generate a traditional local Javadoc for you, in the build/docs/javadoc/index.html</code> file. In addition, the project’s README.md</code> file has an API of the wasmer</code> library Section</a>. Finally, the project comes with a set of examples</a>. Use the make run-example EXAMPLE=Simple</code> to run the SimpleExample.java</code> example for instance. Performance#</a> </h2> WebAssembly aims at being safe, but also fast. Since Wasmer JNI is the first Java library to execute WebAssembly, we can’t compare to prior works in the Java ecosystem. However, you might know that Wasmer comes with 3 backends: Singlepass, Cranelift and LLVM. We’ve even written an article about it: A WebAssembly Compiler tale</a>. The Wasmer JNI library uses the Cranelift backend for the moment, which offers the best compromise between compilation-time and execution-time. Credits#</a> </h2> Asami (d0iasm</a> on Twitter) has improved this project during its internship at Wasmer under my guidance. She finished the internship before the release of the Wasmer JNI project, but she deserves credits for pushing the project forward! Good work Asami! This is an opportunity to remind everyone that we hire anywhere in the world. Asami was working from Japan while I am working from Switzerland, and the rest of the team is from US, Spain, China etc. Feel free to contact me (@mnt_io</a> or @syrusakbary</a> on Twitter) if you want to join us on this big adventure! Conclusion#</a> </h2> Wasmer JNI is a library to execute WebAssembly directly in Java. It embeds the WebAssembly runtime Wasmer</a>. The first releases provide the core API with Module</code>, Instance</code>, and Memory</code>. It comes pre-packaged as a JAR, one per architecture and per platform. The source code is open and hosted on Github at wasmerio/java-ext-wasm</a>. We are constantly improving the project, so if you have feedback, issues, or feature requests please open an issue in the repository, or reach us on Twitter at @wasmerio</a> or @mnt_io</a>. We look forward to see what you build with this! Announcing the first Postgres extension to run WebAssembly 2019-08-29T00:00:00+00:00 This is a copy of an article I wrote for Wasmer</a>. WebAssembly is a portable binary format. That means the same program can run anywhere. To uphold this bold statement, each language, platform and system must be able to run WebAssembly — as fast and safely as possible. </blockquote> Let’s say it again. Wasmer</a> is a WebAssembly runtime. We have successfully embedded the runtime in other languages: In Rust, as it is written in Rust</li> Using C and C++ bindings</a></li> In PHP, using php-ext-wasm</code></a></li> In Python, using python-ext-wasm</code></a> — wasmer package on PyPI</a></li> In Ruby, using ruby-ext-wasm</code></a> — wasmer gem on RubyGems</a></li> In Go, using go-ext-wasm</code></a> — see the announcement</a>.</li> </ul> The community has also embedded Wasmer in awesome projects: .NET/C#, using WasmerSharp</a></li> R, using Wasmr</a>.</li> </ul> It is now time to continue the story and to hang around… Postgres</a>! We are so happy to announce a newcrazy idea: WebAssembly on Postgres. Yes, you read that correctly. On Postgres</a>. Calling a WebAssembly function from Postgres#</a> </h2> As usual, we have to go through the installation process. There is no package manager for Postgres, so it’s a manual step. The Installation Section of the documentation</a> explains the details; here is a summary: $ # Build the shared library. $ just build $ # Install the extension in the Postgres tree. $ just install $ # Activate the extension. $ echo 'CREATE EXTENSION wasm;' | \ psql -h $host -d $database $ # Initialize the extension. $ echo "SELECT wasm_init('$(pwd)/target/release/libpg_ext_wasm.dylib');" | \ psql -h $host -d $database</code></pre> Once the extension is installed, activated and initialized, we can start having fun! The current API is rather small, however basic features are available. The goal is to gather a community and to design a pragmatic API together, discover the expectations, how developers would use this new technology inside a database engine. Let’s see how it works. To instantiate a WebAssembly module, we use the wasm_new_instance</code> function. It takes 2 arguments: The absolute path to the WebAssembly module, and a prefix for the module exported functions. Indeed, if a module exports a function named sum</code>, then a Postgres function named prefix_sum</code> calling the sum</code> function will be created dynamically. Let’s see it in action. Let’s start by editing a Rust program that compiles to WebAssembly: #[no_mangle] pub extern fn sum(x: i32, y: i32) -> i32 { x + y }</code></pre> Once this file compiled to simple.wasm</code>, we can instantiate the module, and call the exported sum</code> function: -- New instance of the `simple.wasm` WebAssembly module. SELECT wasm_new_instance('/absolute/path/to/simple.wasm', 'ns'); -- Call a WebAssembly exported function! SELECT ns_sum(1, 2); -- ns_sum -- -------- -- 3 -- (1 row)</code></pre> Et voilà ! The ns_sum</code> function calls the Rust sum</code> function through WebAssembly! How fun is that 😄? Inspect a WebAssembly instance#</a> </h2> This section shows how to inspect a WebAssembly instance. At the same time, it quickly explains how the extension works under the hood. The extension provides two foreign data wrappers, gathered together in the wasm</code> foreign schema: wasm.instances</code> is a table with the id</code> and wasm_file</code> columns, respectively for the unique instance ID, and the path of the WebAssembly module,</li> wasm.exported_functions</code> is a table with the instance_id</code>, name</code>, inputs</code>, and outputs</code> columns, respectively for the instance ID of the exported function, its name, its input types (already formatted for Postgres), and its output types (already formatted for Postgres).</li> </ul> Let’s see: -- Select all WebAssembly instances. SELECT * FROM wasm.instances; -- id | wasm_file -- -------------------------------------+------------------------------- -- 426e17af-c32f-5027-ad73-239e5450dd91 | /absolute/path/to/simple.wasm -- (1 row) -- Select all exported functions for a specific instance. SELECT name, inputs, outputs FROM wasm.exported_functions WHERE instance_id = '426e17af-c32f-5027-ad73-239e5450dd91'; -- name | inputs | outputs -- -------+-----------------+--------- -- ns_sum | integer,integer | integer -- (1 row)</code></pre> Based on these information, the wasm</code> Postgres extension is able to generate the SQL function to call the WebAssembly exported functions. It sounds simplistic, and… to be honest, it is! The trick is to use foreign data wrappers</a>, which is an awesome feature of Postgres. How fast is it, or: Is it an interesting alternative to PL/pgSQL?#</a> </h2> As we said, the extension API is rather small for now. The idea is to explore, to experiment, to have fun with WebAssembly inside a database. It is particularly interesting in two cases: To write extensions or procedures with any languages that compile to WebAssembly in place of PL/pgSQL</a>,</li> To remove a potential performance bottleneck where speed is involved.</li> </ol> Thus we run a basic benchmark. Like most of the benchmarks out there, it must be taken with a grain of salt. The goal is to compare the execution time between WebAssembly and PL/pgSQL, and see how both approaches scale. </blockquote> The Postgres WebAssembly extension uses Wasmer</a> as the runtime, compiled with the Cranelift backend (learn more about the different backends</a>). We run the benchmark with Postgres 10, on a MacBook Pro 15" from 2016, 2.9Ghz Core i7 with 16Gb of memory. The methodology is the following: Load both the plpgsql_fibonacci</code> and the wasm_fibonacci</code> functions,</li> Run them with a query like SELECT *_fibonacci(n) FROM generate_series(1, 1000)</code> where n</code> has the following values: 50, 500, and 5000, so that we can observe how both approaches scale,</li> Write the timings down,</li> Run this methodology multiple times, and compute the median of the results.</li> </ul> Here come the results. The lower, the better. Comparing WebAssembly vs. PL/pgSQL when computing the Fibonacci sequence with n=50, 500 and 5000. </figcaption> </figure> We notice that the Postgres WebAssembly extension is faster to run numeric computations. The WebAssembly approach scales pretty well compared to the PL/pgSQL approach, in this situation. When to use the WebAssembly extension?#</a> </h3> So far, the extension only supports integers (on 32- and 64-bits). The extension doesn’t support strings yet. It also doesn’t support records, views or other Postgres types. Keep in mind this is the very first step. Hence, it is too soon to tell whether WebAssembly can be an alternative to PL/pgSQL. But regarding the benchmark results above, we are sure they can live side-by-side, WebAssembly has clearly a place in the ecosystem! And we want to continue to pursue this exploration. Conclusion#</a> </h2> We are already talking with people that are interested in using WebAssembly inside databases. If you have any particular use cases, please reach us at wasmer.io</a>, or on Twitter at @wasmerio</a> directly or me @mnt_io</a>. Everything is open source, as usual! Happy hacking. Announcing the fastest WebAssembly runtime for Go: wasmer 2019-05-29T00:00:00+00:00 This is a copy of an article I wrote for Wasmer</a>. WebAssembly is a portable binary format. That means the same file can run anywhere. To uphold this bold statement, each language, platform and system must be able to run WebAssembly — as fast and safely as possible. </blockquote> Wasmer</a> is a WebAssembly runtime written in Rust</a>. It goes without saying that the runtime can be used in any Rust application. We have also successfully embedded the runtime in other languages: Using C and C++ bindings</a></li> In PHP, using php-ext-wasm</code></a></li> In Python, using python-ext-wasm</code></a> — wasmer package on PyPI</a></li> In Ruby, using ruby-ext-wasm</code></a> — wasmer gem on RubyGems</a></li> It is now time to hang around Go</a> 🐹!</li> </ul> We are super happy to announce github.com/wasmerio/go-ext-wasm/wasmer</code>, a Go library to run WebAssembly binaries, fast</a>. Calling a WebAssembly function from Go#</a> </h2> First, let’s install wasmer</code> in your go environment (with cgo support). $ export CGO_ENABLED=1; export CC=gcc; go install github.com/wasmerio/go-ext-wasm/wasmer</code></pre> Let’s jump immediately into some examples.github.com/wasmerio/go-ext-wasm/wasmer</code> is a regular Go library. The installation is automated with import "github.com/wasmerio/go-ext-wasm/wasmer"</code>. Let’s get our hands dirty. We will write a program that compiles to WebAssembly easily, using Rust for instance: #[no_mangle] pub extern fn sum(x: i32, y: i32) -> i32 { x + y }</code></pre> After compilation to WebAssembly, we get a file like this one</a>, named simple.wasm</code>. The following Go program executes the sum</code> function by passing 5</code> and 37</code> as arguments: package main import ( "fmt" wasm "github.com/wasmerio/go-ext-wasm/wasmer" ) func main() { // Reads the WebAssembly module as bytes. bytes, _ := wasm.ReadBytes("simple.wasm") // Instantiates the WebAssembly module. instance, _ := wasm.NewInstance(bytes) defer instance.Close() // Gets the `sum` exported function from the WebAssembly instance. sum := instance.Exports["sum"] // Calls that exported function with Go standard values. The WebAssembly // types are inferred and values are casted automatically. result, _ := sum(5, 37) fmt.Println(result) // 42! }</code></pre> Great! We have successfully executed a WebAssembly file inside Go. Note: Go values passed to the WebAssembly exported function are automatically cast to WebAssembly values. Types are inferred and casting is done automatically. Thus, a WebAssembly function acts as any regular Go function. </blockquote> WebAssembly calling Go funtions#</a> </h2> A WebAssembly module exports some functions, so that they can be called from the outside world. This is the entry point to execute WebAssembly. Nonetheless, a WebAssembly module can also have imported functions. Let’s consider the following Rust program: extern { fn sum(x: i32, y: i32) -> i32; } #[no_mangle] pub extern fn add1(x: i32, y: i32) -> i32 { unsafe { sum(x, y) } + 1 }</code></pre> The exported function add1</code> calls the sum</code> function. Its implementation is absent, only its signature is defined. This is an “extern function”, and for WebAssembly, this is an imported function, because its implementation must be imported. Let’s implement the sum</code> function in Go! To do so, need to use cgo</a>: The sum</code> function signature is defined in C (see the comment above import "C"</code>),</li> The sum</code> implementation is defined in Go. Notice the //export</code> which is the way cgo uses to map Go code to C code,</li> NewImports</code> is an API used to create WebAssembly imports. In this code "sum"</code> is the WebAssembly imported function name, sum</code> is the Go function pointer, and C.sum</code> is the cgo function pointer,</li> Finally, NewInstanceWithImports</code> is the constructor to use to instantiate the WebAssembly module with imports. That’s it.</li> </ol> Let’s see the complete program: package main // // 1️⃣ Declare the `sum` function signature (see cgo). // // #include <stdlib.h> // // extern int32_t sum(void *context, int32_t x, int32_t y); import "C" import ( "fmt" wasm "github.com/wasmerio/go-ext-wasm/wasmer" "unsafe" ) // 2️⃣ Write the implementation of the `sum` function, and export it (for cgo). //export sum func sum(context unsafe.Pointer, x int32, y int32) int32 { return x + y } func main() { // Reads the WebAssembly module as bytes. bytes, _ := wasm.ReadBytes("import.wasm") // 3️⃣ Declares the imported functions for WebAssembly. imports, _ := wasm.NewImports().Append("sum", sum, C.sum) // 4️⃣ Instantiates the WebAssembly module with imports. instance, _ := wasm.NewInstanceWithImports(bytes, imports) // Close the WebAssembly instance later. defer instance.Close() // Gets the `add1` exported function from the WebAssembly instance. add1 := instance.Exports["add1"] // Calls that exported function. result, _ := add1(1, 2) fmt.Println(result) // add1(1, 2) // = sum(1 + 2) + 1 // = 1 + 2 + 1 // = 4 // QED }</code></pre>Reading the memory#</a> </h2> A WebAssembly instance has a linear memory. Let’s see how to read it. Consider the following Rust program: #[no_mangle] pub extern fn return_hello() -> *const u8 { b"Hello, World!\0".as_ptr() }</code></pre> The return_hello</code> function returns a pointer to a string. The string terminates by a null byte, à la C. Let’s jump on the Go side: bytes, _ := wasm.ReadBytes("memory.wasm") instance, _ := wasm.NewInstance(bytes) defer instance.Close() // Calls the `return_hello` exported function. // This function returns a pointer to a string. result, _ := instance.Exports["return_hello"]() // Gets the pointer value as an integer. pointer := result.ToI32() // Reads the memory. memory := instance.Memory.Data() fmt.Println(string(memory[pointer : pointer+13])) // Hello, World!</code></pre> The return_hello</code> function returns a pointer as an i32</code> value. We get its value by calling ToI32</code>. Then, we fetch the memory data with instance.Memory.Data()</code>. This function returns a slice over the WebAssembly instance memory. It can be used as any regular Go slice. Fortunately for us, we already know the length of the string we want to read, so memory[pointer : pointer+13]</code> is enough to read the bytes, that are then cast to a string. Et voilà ! You can read the Greet Example</a> to see a more advanced usage of the memory API. </blockquote> Benchmarks#</a> </h2> So far, github.com/wasmerio/go-ext-wasm/wasmer</code> has a nice API, but …is it fast? Contrary to PHP or Ruby, there are already existing runtimes in the Go world to execute WebAssembly. The main candidates are: Life</a>, from Perlin Network, a WebAssembly interpreter</li> Wagon</a>, from Go Interpreter, a WebAssembly interpreter and toolkit.</li> </ul> In our blog post about the PHP extension</a>, we have used the n-body algorithm</a> to benchmark the performance. Life provides more benchmarks: the Fibonacci algorithm</a> (the recursive version), the Pollard’s rho algorithm</a>, and the Snappy Compress operation. The latter works successfully with github.com/wasmerio/go-ext-wasm/wasmer</code> but not with Life or Wagon. We have removed it from the benchmark suites. Benchmark sources</a> are online. We use Life 20190521143330–57f3819c2df0, and Wagon 0.4.0, i.e. the latest versions to date. The benchmark numbers represent the average result for 10 runs each. The computer that ran these benchmarks is a MacBook Pro 15" from 2016, 2.9Ghz Core i7 with 16Gb of memory. Results are grouped by benchmark algorithm on the X axis. The Y axis represents the time used to run the algorithm, expressed in milliseconds. The lower, the better. Speed comparison between Wasmer, Wagon and Life. Benchmark suites are the n-body, Fibonacci, and Pollard’s rho algorithms. Speed is expressed in ms. Lower is better. </figcaption> </figure> While both Life and Wagon provide on average the same speed, Wasmer (github.com/wasmerio/go-ext/wasmer</code>) is on average 72 times faster 🎉. It is important to know that Wasmer comes with 3 backends: Singlepass</a>, Cranelift</a>, and LLVM</a>. The default backend that is used by the Go library is Cranelift (learn more about Cranelift</a>). Using LLVM will provide performance close to native, but we decided to start with Cranelift as it offers the best tradeoff between compilation-time and execution-time (learn more about the different backends</a>, when to use them, pros and cons etc.). Conclusion#</a> </h2> [github.com/wasmerio/go-ext-wasm/wasmer](https://github.com/wasmerio/go-ext-wasm)</code> is a new Go library to execute WebAssembly binaries. It embeds the Wasmer</a> runtime. The first version supports all the required API for the most common usages. The current benchmarks (a mix from our benchmark suites and from Life suites) show that Wasmer is — on average — 72 times faster than Life and Wagon, the two major existing WebAssembly runtimes in the Go world. If you want to follow the development, take a look at @wasmerio</a> and @mnt_io</a> on Twitter, or @[email protected]">[email protected]</a></a> on Mastodon. And of course, everything is open source at wasmerio/go-ext-wasm</a>. Thank you for your time, we can’t wait to see what you build with us! 🐘+🦀+🕸 php-ext-wasm: Migrating from wasmi to Wasmer 2019-04-03T00:00:00+00:00 This is a copy of an article I wrote for Wasmer</a>. First as a joke, now as a real product, I started to develop php-ext-wasm</a>: a PHP</a> extension allowing to execute WebAssembly</a> binaries. The PHP virtual machine (VM) is Zend Engine</a>. To write an extension, one needs to develop in C or C++. The extension was simple C bindings to a Rust library I also wrote. At that time, this Rust library was using wasmi</code></a> for the WebAssembly VM. I knew that wasmi</code> wasn’t the fastest WebAssembly VM in the game, but the API is solid, well-tested, it compiles quickly, and is easy to hack. All the requirements to start a project! After 6 hours of development, I got something working. I was able to run the following PHP program: <?php $instance = new Wasm\Instance('simple.wasm'); $result = $instance->sum(1, 2); var_dump($result); // int(3)</code></pre> The API is straightforward: create an instance (here of simple.wasm</code>), then call functions on it (here sum</code> with 1 and 2 as arguments). PHP values are transformed into WebAssembly values automatically. For the record, here is the simple.rs</code> Rust program that is compiled to a WebAssembly binary: #[no_mangle] pub extern fn sum(x: i32, y: i32) -> i32 { x + y }</code></pre> It was great! 6 hours is a relatively small number of hours to go that far according to me. However, I quickly noticed that wasmi</code> is… slow. One of the promise of WebAssembly</a> is: WebAssembly aims to execute at native speed by taking advantage of common hardware capabilities</a> available on a wide range of platforms. </blockquote> And clearly, my extension wasn’t fulfilling this promise. Let’s see a basic comparison with a benchmark. I chose the n-body algorithm</a> from the Computer Language Benchmarks Game</a> from Debian, mostly because it’s relatively CPU intensive. Also, the algorithm has a simple interface: based on an integer, it returns a floating-point number; this API doesn’t involve any advanced instance memory API, which is perfect to test a proof-of-concept. As a baseline, I’ve run the n-body algorithm written in Rust</a>, let’s call it rust-baseline</code>. The same algorithm has been written in PHP</a>, let’s call it php</code>. Finally, the algorithm has been compiled from Rust to WebAssembly, and executed with the php-ext-wasm</code> extension, let’s call that case php+wasmi</code>. All results are for nbody(5000000)</code>: rust-baseline</code>: 287ms,</li> php</code>: 19,761ms,</li> php+wasmi</code>: 67,622ms.</li> </ul> OK, so… php-ext-wasm</code> with wasmi</code> is 3.4 times slower than PHP itself, it is pointless to use WebAssembly in such conditions! It confirms my first intuition though: In our case, wasmi</code> is really great to mock something up, but it’s not fast enough for our expectations. Faster, faster, faster…#</a> </h2> I wanted to use Cranelift</a> since the beginning. It’s a code generator, à la LLVM</a> (excuse the brutal shortcut, the goal isn’t to explain what Cranelift is in details, but that’s a really awesome project!). To quote the project itself: Cranelift is a low-level retargetable code generator. It translates a target-independent intermediate representation</a> into executable machine code. </blockquote> It basically means that the Cranelift API can be used to generate executable code. It’s perfect! I can replace wasmi</code> by Cranelift, and boom, profit. But… there is other ways to get even faster code execution — at the cost of a longer code compilation though. For instance, LLVM can provide a very fast code execution, almost at native speed. Or we can generate assembly code dynamically. Well, there is multiple ways to achieve that. What if a project could provide a WebAssembly virtual machine with multiple backends? Enter Wasmer#</a> </h2> And it was at that specific time that I’ve been hired by Wasmer</a>. To be totally honest, I was looking at Wasmer a few weeks before. It was a surprise and a great opportunity for me. Well, the universe really wants this rewrite from wasmi</code> to Wasmer, right 😅? Wasmer is organized as a set of Rust libraries (called crates). There is even a wasmer-runtime-c-api</code> crate which is a C and a C++ API on top of the wasmer-runtime</code> crate and the wasmer-runtime-core</code> crate, i.e. it allows running the WebAssembly virtual machine as you want, with the backend of your choice: Cranelift, LLVM, or Dynasm (at the time of writing). That’s perfect, it removes my Rust library between the PHP extension and wasmi</code>. Then php-ext-wasm</code> is reduced to a PHP extension without any Rust code, everything goes to wasmer-runtime-c-api</code>. That’s sad to remove Rust from this project, but it relies on more Rust code! Counting the time to make some patches on wasmer-runtime-c-api</code>, I’ve been able to migrate php-ext-wasm</code> to Wasmer in 5 days. By default, php-ext-wasm</code> uses Wasmer with the Cranelift backend, it does a great balance between compilation and execution time. It is really good. Let’s run the benchmark, with the addition of php+wasmer(cranelift)</code>: rust-baseline</code>: 287ms,</li> php</code>: 19,761ms,</li> php+wasmi</code>: 67,622ms,</li> php+wasmer(cranelift)</code>: 2,365ms 🎉.</li> </ul> Finally, the PHP extension provides a faster execution than PHP itself! php+wasmer(cranelift)</code> is 8.6 times faster than php</code> to be exact. And it is 28.6 times faster than php+wasmi</code>. Can we reach the native speed (represented by rust-baseline</code> here)? It’s very likely with LLVM. That’s for another article. I’m super happy with Cranelift for the moment. (See our previous blog post to learn how we benchmark different backends in Wasmer, and other WebAssembly runtimes</a>). More Optimizations#</a> </h2> Wasmer provides more features, like module caching. Those features are now included in the PHP extension. When booting the nbody.wasm</code> file (19kb), it took 4.2ms. By booting, I mean: reading the WebAssembly binary from a file, parsing it, validating it, compiling it to executable code and a WebAssembly module structure. PHP execution model is: starts, runs, dies. Memory is freed for each request. If one wants to use php-ext-wasm</code>, you don’t really want to pay that “booting cost” every time. Hopefully, wasmer-runtime-c-api</code> now provides a module serialization API, which is integrated into the PHP extension itself. It saves the “booting cost”, but it adds a “deserialization cost”. That second cost is smaller, but still, we need to know it exists. Hopefully again, Zend Engine has an API to get persistent in-memory data between PHP executions. php-ext-wasm</code> supports that API to get persistent modules, et voilà. Now it takes 4.2ms for the first boot of nbody.wasm</code> and 0.005ms for all the next boots. It’s 840 times faster! Conclusion#</a> </h2> Wasmer is a young — but mature — framework to build WebAssembly runtimes on top of. The default backend is Cranelift, and it shows its promises: It brings a correct balance between compilation time and execution time. wasmi</code> has been a good companion to develop a Proof-Of-Concept. This library has its place in other usages though, like very short-living WebAssembly binaries (I’m thinking of Ethereum contracts that compile to WebAssembly for instance, which is one of the actual use cases). It’s important to understand that no runtime is better than another, it depends on the use case. The next step is to stabilize php-ext-wasm</code> to release a 1.0.0 version. See you there! If you want to follow the development, take a look at @wasmerio</a> and @mnt_io</a> on Twitter. Bye bye Automattic, hello Wasmer 2019-03-04T00:00:00+00:00 Today is my first day at Wasmer</a>. It's with a lot of regrets that I leave Automattic. To be clear, I'm not leaving because something negative happened, I'm leaving because I've received the same job offer, 3 times in 10 days, from Wasmer, Google and Mozilla. Namely to work with Rust or C++ to build a WebAssembly runtime. This is an offer I can barely decline. It's an opportunity and a dream for me. And I was lucky enough to get a choice between 3 excellent companies! I can only encourage you to work with Automattic</a>. It's definitely the best company I've ever work with; stealing the 1st place to Mozilla. Automattic is not only about WordPress.com and other services: It's a way of living. The culture, the spirit, the interactions between people, the mission, everything is exceptional. It has been a super great experience. I could write 100 pages about my team. They have all been remarkable in many ways. I'm closer to them although they live at 10'000km, rather than colleagues I met everyday in person in the past. Congrats to Matt</a> for this incredible project. Now it's time to work on Wasmer</a>. It's a WebAssembly</a> runtime written in Rust</a>: My two current passions. It's powerful, modular, well-designed, and it comes with great ambitions. I'm really exciting. I work with an extraordinary team: Syrus Akbary</a> (the author of Graphene</a>, a GraphQL framework in Python), Lachlan Sneff</a> (the author of Nebulet</a>, a microkernel that implements a WebAssembly "usermode" that runs in Ring 0), Brandon Fish</a> (a great contributor of Truffleruby</a>, a high performance implementation of Ruby with GraalVM), Mackenzie Clark</a>, and soon more. My job will consist to work on the runtime of course, and also to integrate/embed the runtime into different languages, such as PHP —like I did with [php-ext-wasm](https://github.com/Hywan/php-ext-wasm)</code>, more to come on this blog—. More secret projects coming. Let's turn them into realities 🎉! The PHP galaxy 2018-10-29T00:00:00+00:00 The galaxy we will explore today is the PHP galaxy. This post will explain what PHP is, how to compile any Rust program to C and then to a PHP native extension. What is PHP, and why?#</a> </h2> PHP</a> is a: popular general-purpose scripting language that is especially suited to Web development. Fast, flexible, and pragmatic, PHP powers everything from your blog to the most popular websites in the world. </blockquote> PHP has sadly acquired a bad reputation along the years, but recent releases (since PHP 7.0 mostly) have introduced neat language features, and many cleanups, which are excessively ignored by haters. PHP is also a fast scripting language, and is very flexible. PHP now has declared types, traits, variadic arguments, closures (with explicit scopes!), generators, and a huge backward compatibility. The development of PHP is led by RFCs</a>, which is an open and democratic process. The Gutenberg project is a new editor for WordPress. The latter is written in PHP. This is naturally that we want a native extension for PHP to parse the Gutenberg post format. PHP is a language with a specification</a>. The most popular virtual machine is Zend Engine</a>. Other virtual machines exist, like HHVM</a> (but the PHP support has been dropped recently in favor of their own PHP fork, called Hack), Peachpie</a>, or Tagua VM</a> (under development). In this post, we will create an extension for Zend Engine. This virtual machine is written in C. Great, we have visited the C galaxy in the previous episode</a>! Rust 🚀 C 🚀 PHP#</a> </h2> </figure> To port our Rust parser into PHP, we first need to port it to C. It's been done in the previous episode. Two files result from this port to C: libgutenberg_post_parser.a</code> and gutenberg_post_parser.h</code>, respectively a static library, and the header file. Bootstrap with a skeleton#</a> </h3> PHP comes with a script to create an extension skeleton</a>/template, called ext_skel.php</code></a>. This script is accessible from the source of the Zend Engine virtual machine (which we will refer to as php-src</code>). One can invoke the script like this: $ cd php-src/ext/ $ ./ext_skel.php \ --ext gutenberg_post_parser \ --author 'Ivan Enderlin' \ --dir /path/to/extension \ --onlyunix $ cd /path/to/extension $ ls gutenberg_post_parser tests/ .gitignore CREDITS config.m4 gutenberg_post_parser.c php_gutenberg_post_parser.h</code></pre> The ext_skel.php</code> script recommends to go through the following steps: Rebuild the configuration of the PHP source (run ./buildconf</code> at the root of the php-src</code> directory),</li> Reconfigure the build system to enable the extension, like ./configure --enable-gutenberg_post_parser</code>,</li> Build with make</code>,</li> Done.</li> </ul> But our extension is very likely to live outside the php-src</code> tree. So we will use phpize</code> instead. phpize</code> is an executable that comes with php</code>, php-cgi</code>, phpdbg</code>, php-config</code> etc. It allows to compile extensions against an already compiled php</code> binary, which is perfect in our case! We will use it like this : $ cd /path/to/extension/gutenberg_post_parser $ # Get the bin directory for PHP utilities. $ PHP_PREFIX_BIN=$(php-config --prefix)/bin $ # Clean (except if it is the first run). $ $PHP_PREFIX_BIN/phpize --clean $ # “phpize” the extension. $ $PHP_PREFIX_BIN/phpize $ # Configure the extension for a particular PHP version. $ ./configure --with-php-config=$PHP_PREFIX_BIN/php-config $ # Compile. $ make install</code></pre> In this post, we will not show all the edits we have done, but we will rather focus on the extension binding. All the sources can be found here</a>. Shortly, here is the config.m4</code> file: PHP_ARG_ENABLE(gutenberg_post_parser, whether to enable gutenberg_post_parser support, [ --with-gutenberg_post_parser Include gutenberg_post_parser support], no) if test "$PHP_GUTENBERG_POST_PARSER" != "no"; then PHP_SUBST(GUTENBERG_POST_PARSER_SHARED_LIBADD) PHP_ADD_LIBRARY_WITH_PATH(gutenberg_post_parser, ., GUTENBERG_POST_PARSER_SHARED_LIBADD) PHP_NEW_EXTENSION(gutenberg_post_parser, gutenberg_post_parser.c, $ext_shared) fi</code></pre> What it does is basically the following: Register the --with-gutenberg_post_parser</code> option in the build system, and</li> Declare the static library to compile with, and the source of the extension itself.</li> </ul> We must add the libgutenberg_post_parser.a</code> and gutenberg_post_parser.h</code> files in the same directory (a symlink is perfect), to get a structure such as: $ ls gutenberg_post_parser tests/ # from ext_skel .gitignore # from ext_skel CREDITS # from ext_skel config.m4 # from ext_skel (edited) gutenberg_post_parser.c # from ext_skel (will be edited) gutenberg_post_parser.h # from Rust libgutenberg_post_parser.a # from Rust php_gutenberg_post_parser.h # from ext_skel</code></pre> The core of the extension is the gutenberg_post_parser.c</code> file. This file is responsible to create the module, and to bind our Rust code to PHP. The module, aka the extension#</a> </h3> As said, we will work in the gutenberg_post_parser.c</code> file. First, let's include everything we need: #include "php.h" #include "ext/standard/info.h" #include "php_gutenberg_post_parser.h" #include "gutenberg_post_parser.h"</code></pre> The last line includes the gutenberg_post_parser.h</code> file generated by Rust (more precisely, by cbindgen</code>, if you don't remember, take a look at the previous episode</a>). Then, we have to decide what API we want to expose into PHP? As a reminder, the Rust parser produces an AST defined as: pub enum Node<'a> { Block { name: (Input<'a>, Input<'a>), attributes: Option<Input<'a>>, children: Vec<Node<'a>> }, Phrase(Input<'a>) }</code></pre> The C variant of the AST is very similar (with more structures, but the idea is almost identical). So in PHP, the following structure has been selected: <?php class Gutenberg_Parser_Block { public string $namespace; public string $name; public string $attributes; public array $children; } class Gutenberg_Parser_Phrase { public string $content; } function gutenberg_post_parse(string $gutenberg_post): array;</code></pre> The gutenberg_post_parse</code> function will output an array of objects of kind Gutenberg_Parser_Block</code> or Gutenberg_Parser_Phrase</code>, i.e. our AST. So, let's declare those classes! Declare the classes#</a> </h3> Note: The next 4 code blocks are not the core of the post, it is just code that needs to be written, you can skip it if you are not about to write a PHP extension. zend_class_entry *gutenberg_parser_block_class_entry; zend_class_entry *gutenberg_parser_phrase_class_entry; zend_object_handlers gutenberg_parser_node_class_entry_handlers; typedef struct _gutenberg_parser_node { zend_object zobj; } gutenberg_parser_node;</code></pre> A class entry represents a specific class type. A handler is associated to a class entry. The logic is somewhat complicated. If you need more details, I recommend to read the PHP Internals Book</a>. Then, let's create a function to instanciate those objects: static zend_object *create_parser_node_object(zend_class_entry *class_entry) { gutenberg_parser_node *gutenberg_parser_node_object; gutenberg_parser_node_object = ecalloc(1, sizeof(*gutenberg_parser_node_object) + zend_object_properties_size(class_entry)); zend_object_std_init(&gutenberg_parser_node_object->zobj, class_entry); object_properties_init(&gutenberg_parser_node_object->zobj, class_entry); gutenberg_parser_node_object->zobj.handlers = &gutenberg_parser_node_class_entry_handlers; return &gutenberg_parser_node_object->zobj; }</code></pre> Then, let's create a function to free those objects. It works in two steps: Destruct the object by calling its destructor (in the user-land), then free it for real (in the VM-land): static void destroy_parser_node_object(zend_object *gutenberg_parser_node_object) { zend_objects_destroy_object(gutenberg_parser_node_object); } static void free_parser_node_object(zend_object *gutenberg_parser_node_object) { zend_object_std_dtor(gutenberg_parser_node_object); }</code></pre> Then, let's initialize the “module”, i.e. the extension. During the initialisation, we will create the classes in the user-land, declare their attributes etc. PHP_MINIT_FUNCTION(gutenberg_post_parser) { zend_class_entry class_entry; // Declare Gutenberg_Parser_Block. INIT_CLASS_ENTRY(class_entry, "Gutenberg_Parser_Block", NULL); gutenberg_parser_block_class_entry = zend_register_internal_class(&class_entry TSRMLS_CC); // Declare the create handler. gutenberg_parser_block_class_entry->create_object = create_parser_node_object; // The class is final. gutenberg_parser_block_class_entry->ce_flags |= ZEND_ACC_FINAL; // Declare the `namespace` public attribute, // with an empty string for the default value. zend_declare_property_string(gutenberg_parser_block_class_entry, "namespace", sizeof("namespace") - 1, "", ZEND_ACC_PUBLIC); // Declare the `name` public attribute, // with an empty string for the default value. zend_declare_property_string(gutenberg_parser_block_class_entry, "name", sizeof("name") - 1, "", ZEND_ACC_PUBLIC); // Declare the `attributes` public attribute, // with `NULL` for the default value. zend_declare_property_null(gutenberg_parser_block_class_entry, "attributes", sizeof("attributes") - 1, ZEND_ACC_PUBLIC); // Declare the `children` public attribute, // with `NULL` for the default value. zend_declare_property_null(gutenberg_parser_block_class_entry, "children", sizeof("children") - 1, ZEND_ACC_PUBLIC); // Declare the Gutenberg_Parser_Block. … skip … // Declare Gutenberg parser node object handlers. memcpy(&gutenberg_parser_node_class_entry_handlers, zend_get_std_object_handlers(), sizeof(gutenberg_parser_node_class_entry_handlers)); gutenberg_parser_node_class_entry_handlers.offset = XtOffsetOf(gutenberg_parser_node, zobj); gutenberg_parser_node_class_entry_handlers.dtor_obj = destroy_parser_node_object; gutenberg_parser_node_class_entry_handlers.free_obj = free_parser_node_object; return SUCCESS; }</code></pre> If you are still reading, first: Thank you, and second: Congrats! Then, there is a PHP_RINIT_FUNCTION</code> and a PHP_MINFO_FUNCTION</code> functions that are already generated by the ext_skel.php</code> script. Same for the module entry definition and other module configuration details. The gutenberg_post_parse</code> function#</a> </h3> We will now focus on the gutenberg_post_parse</code> PHP function. This function takes a string as a single argument and returns either false</code> if the parsing failed, or an array of objects of kind Gutenberg_Parser_Block</code> or Gutenberg_Parser_Phrase</code> otherwise. Let's write it! Notice that it is declared with the PHP_FUNCTION</code> macro</a>. PHP_FUNCTION(gutenberg_post_parse) { char *input; size_t input_len; // Read the input as a string. if (zend_parse_parameters(ZEND_NUM_ARGS() TSRMLS_CC, "s", &input, &input_len) == FAILURE) { return; }</code></pre> At this step, the argument has been declared and typed as a string ("s"</code>). The string value is in input</code> and the string length is in input_len</code>. The next step is to parse the input</code>. (The length of the string is not needed). This is where we are going to call our Rust code! Let's do that: // Parse the input. Result parser_result = parse(input); // If parsing failed, then return false. if (parser_result.tag == Err) { RETURN_FALSE; } // Else map the Rust AST into a PHP array. const Vector_Node nodes = parse_result.ok._0;</code></pre> The Result</code> type and the parse</code> function come from Rust. If you don't remember those types, please read the previous episode about the C galaxy</a>. Zend Engine has a macro called RETURN_FALSE</code> to return… false</code>! Handy isn't it? Finally, if everything went well, we get back a collection of node as a Vector_Node</code> type. The next step is to map those Rust/C types into PHP types, i.e. an array of the Gutenberg classes. Let's go: // Note: return_value is a “magic” variable that holds the value to be returned. // // Allocate an array. array_init_size(return_value, nodes.length); // Map the Rust AST. into_php_objects(return_value, &nodes); }</code></pre> Done 😁! Oh wait… the into_php_objects</code> function need to be written! The into_php_objects</code> function#</a> </h3> This function is not terribly complex: It's just full of Zend Engine specific API as expected. We are going to explain how to map a Block</code> into a Gutenberg_Parser_Block</code> object, and to let the Phrase</code> mapping to Gutenberg_Parser_Phrase</code> for the assiduous readers. And there we go: void into_php_objects(zval *php_array, const Vector_Node *nodes) { const uintptr_t number_of_nodes = nodes->length; if (number_of_nodes == 0) { return; } // Iterate over all nodes. for (uintptr_t nth = 0; nth < number_of_nodes; ++nth) { const Node node = nodes->buffer[nth]; if (node.tag == Block) { // Map Block into Gutenberg_Parser_Block. } else if (node.tag == Phrase) { // Map Phrase into Gutenberg_Parser_Phrase. } } }</code></pre> Now let's map a block. The process is the following: Allocate PHP strings for the block namespace, and for the block name,</li> Allocate an object,</li> Set the block namespace and the block name to their respective object properties,</li> Allocate a PHP string for the block attributes if any,</li> Set the block attributes to its respective object property,</li> If any children, initialise a new array, and call into_php_objects</code> with the child nodes and the new array,</li> Set the children to its respective object property,</li> Finally, add the block object inside the array to be returned.</li> </ol> const Block_Body block = node.block; zval php_block, php_block_namespace, php_block_name; // 1. Prepare the PHP strings. ZVAL_STRINGL(&php_block_namespace, block.namespace.pointer, block.namespace.length); ZVAL_STRINGL(&php_block_name, block.name.pointer, block.name.length);</code></pre> Do you remember that namespace, name and other similar data are of type Slice_c_char</code>? It's just a structure with a pointer and a length. The pointer points to the original input string, so that there is no copy (and this is the definition of a slice actually). Well, Zend Engine has a ZVAL_STRINGL</code> macro</a> that allows to create a string from a pointer and a length, great! Unfortunately for us, Zend Engine does a copy behind the scene</a>… There is no way to keep the pointer and the length only, but it keeps the number of copies small. I think it is to take the full ownership of the data, which is required for the garbage collector. // 2. Create the Gutenberg_Parser_Block object. object_init_ex(&php_block, gutenberg_parser_block_class_entry);</code></pre> The object has been instanciated with a class represented by the gutenberg_parser_block_class_entry</code>. // 3. Set the namespace and the name. add_property_zval(&php_block, "namespace", &php_block_namespace); add_property_zval(&php_block, "name", &php_block_name); zval_ptr_dtor(&php_block_namespace); zval_ptr_dtor(&php_block_name);</code></pre> The zval_ptr_dtor</code> adds 1 to the reference counter. This is required for the garbage collector. // 4. Deal with block attributes if some. if (block.attributes.tag == Some) { Slice_c_char attributes = block.attributes.some._0; zval php_block_attributes; ZVAL_STRINGL(&php_block_attributes, attributes.pointer, attributes.length); // 5. Set the attributes. add_property_zval(&php_block, "attributes", &php_block_attributes); zval_ptr_dtor(&php_block_attributes); }</code></pre> It is similar to what has been done for namespace</code> and name</code>. Now let's continue with children. // 6. Handle children. const Vector_Node *children = (const Vector_Node*) (block.children); if (children->length > 0) { zval php_children_array; array_init_size(&php_children_array, children->length); // Recursion. into_php_objects(&php_children_array, children); // 7. Set the children. add_property_zval(&php_block, "children", &php_children_array); Z_DELREF(php_children_array); } free((void*) children);</code></pre> Finally, add the block instance into the array to be returned: // 8. Insert the object in the collection. add_next_index_zval(php_array, &php_block);</code></pre> The entire code lands here</a>. PHP extension 🚀 PHP userland#</a> </h2> Now the extension is written, we have to compile it. That's the repetitive set of commands we have shown above with phpize</code>. Once the extension is compiled, the generated gutenberg_post_parser.so</code> file must be located in the extension directory. This directory can be found with the following command: $ php-config --extension-dir</code></pre> For instance, in my computer, the extension directory is /usr/local/Cellar/php/7.2.11/pecl/20170718</code>. Then, to enable the extension for a given execution, you must write: $ php -d extension=gutenberg_post_parser -m | \ grep gutenberg_post_parser</code></pre> Or, to enable the extension for all executions, locate the php.ini</code> file with php --ini</code> and edit it to add: extension=gutenberg_post_parser</code></pre> Done! Now, let's use some reflection to check the extension is correctly loaded and handled by PHP: $ php --re gutenberg_post_parser Extension [ <persistent> extension #64 gutenberg_post_parser version 0.1.0 ] { - Functions { Function [ <internal:gutenberg_post_parser> function gutenberg_post_parse ] { - Parameters [1] { Parameter #0 [ <required> $gutenberg_post_as_string ] } } } - Classes [2] { Class [ <internal:gutenberg_post_parser> final class Gutenberg_Parser_Block ] { - Constants [0] { } - Static properties [0] { } - Static methods [0] { } - Properties [4] { Property [ <default> public $namespace ] Property [ <default> public $name ] Property [ <default> public $attributes ] Property [ <default> public $children ] } - Methods [0] { } } Class [ <internal:gutenberg_post_parser> final class Gutenberg_Parser_Phrase ] { - Constants [0] { } - Static properties [0] { } - Static methods [0] { } - Properties [1] { Property [ <default> public $content ] } - Methods [0] { } } } }</code></pre> Everything looks good: There is one function and two classes that are defined as expected. Now, let's write some PHP code for the first time in this blog post! <?php var_dump( gutenberg_post_parse( 'barqux' ) ); /** * Will output: * array(3) { * [0]=> * object(Gutenberg_Parser_Block)#1 (4) { * ["namespace"]=> * string(4) "core" * ["name"]=> * string(3) "foo" * ["attributes"]=> * NULL * ["children"]=> * NULL * } * [1]=> * object(Gutenberg_Parser_Phrase)#2 (1) { * ["content"]=> * string(3) "bar" * } * [2]=> * object(Gutenberg_Parser_Block)#3 (4) { * ["namespace"]=> * string(4) "core" * ["name"]=> * string(3) "baz" * ["attributes"]=> * NULL * ["children"]=> * array(1) { * [0]=> * object(Gutenberg_Parser_Phrase)#4 (1) { * ["content"]=> * string(3) "qux" * } * } * } * } */</code></pre> It works very well! Conclusion#</a> </h2> The journey is: A string written in PHP,</li> Allocated by the Zend Engine from the Gutenberg extension,</li> Passed to Rust through FFI (static library + header),</li> Back to Zend Engine in the Gutenberg extension,</li> To generate PHP objects,</li> That are read by PHP.</li> </ul> Rust fits really everywhere! We have seen in details how to write a real world parser in Rust, how to bind it to C and compile it to a static library in addition to C headers, how to create a PHP extension exposing one function and two objects, how to integrate the C binding into PHP, and how to use this extension in PHP. As a reminder, the C binding is about 150 lines of code. The PHP extension is about 300 lines of code, but substracting “decorations” (the boilerplate to declare and manage the extension) that are automatically generated, the PHP extension reduces to about 200 lines of code. Once again, I find this is a small surface of code to review considering the fact that the parser is still written in Rust, and modifying the parser will not impact the bindings (except if the AST is updated obviously)! PHP is a language with a garbage collector. It explains why all strings are copied, so that they are owned by PHP itself. However, the fact that Rust does not copy any data saves memory allocations and deallocations, which is the biggest cost most of the time. Rust also provides safety. This property can be questionned considering the number of binding we are going through: Rust to C to PHP: Does it still hold? From the Rust perspective, yes, but everything that happens inside C or PHP must be considered unsafe. A special care must be put in the C binding to handle all situations. Is it still fast? Well, let's benchmark. I would like to remind that the first goal of this experiment was to tackle the bad performance of the original PEG.js parser. On the JavaScript ground, WASM and ASM.js have shown to be very much faster (see the WebAssembly galaxy</a>, and the ASM.js galaxy</a>). For PHP, phpegjs</code> is used</a>: It reads the grammar written for PEG.js and compiles it to PHP. Let's see how they compare: Document</th> PEG PHP parser (ms)</th> Rust parser as a PHP extension (ms)</th> speedup</th></tr></thead> demo-post.html</code></a></td> 30.409</td> 0.0012</td> × 25341</td></tr> shortcode-shortcomings.html</code></a></td> 76.39</td> 0.096</td> × 796</td></tr> redesigning-chrome-desktop.html</code></a></td> 225.824</td> 0.399</td> × 566</td></tr> web-at-maximum-fps.html</code></a></td> 173.495</td> 0.275</td> × 631</td></tr> early-adopting-the-future.html</code></a></td> 280.433</td> 0.298</td> × 941</td></tr> pygmalian-raw-html.html</code></a></td> 377.392</td> 0.052</td> × 7258</td></tr> moby-dick-parsed.html</code></a></td> 5,437.630</td> 5.037</td> × 1080</td></tr> </tbody></table> Benchmarks between PEG PHP parser and Rust parser as a PHP extension. </figcaption> </figure> The PHP extension of the Rust parser is in average 5230 times faster than the actual PEG PHP implementation. The median of the speedup is 941. Another huge issue was that the PEG parser was not able to handle many Gutenberg documents because of a memory limit. Of course, it is possible to grow the size of the memory, but it is not ideal. With the Rust parser as a PHP extension, memory stays constant and close to the size of the parsed document. I reckon we can optimise the extension further to generate an iterator instead of an array. This is something I want to explore and analyse the impact on the performance. The PHP Internals Book has a chapter about Iterators</a>. Thanks for reading! The C galaxy 2018-09-11T00:00:00+00:00 The galaxy we will explore today is the C galaxy. This post will explain what C is (shortly), how to compile any Rust program in C in theory, and how to do that practically with our Rust parser from the Rust side and the C side. We will also see how to test such a binding. What is C, and why?#</a> </h2> C</a> is probably the most used and known programming language in the world. Quoting Wikipedia: C […] is a general-purpose, imperative computer programming language, supporting structured programming, lexical variable scope and recursion, while a static type system prevents many unintended operations. By design, C provides constructs that map efficiently to typical machine instructions, and therefore it has found lasting use in applications that had formerly been coded in assembly language, including operating systems, as well as various application software for computers ranging from supercomputers to embedded systems. </blockquote> Dennis Ritchie, the inventor of the C language. </figcaption> </figure> The impact of C is probably without precedent on the progamming language world. Almost everything is written in C, starting with operating systems. Today, it is one of the few common denominator between any programs on any systems on any machines in the world. In other words, being compatible with C opens a large door to everything. Your program will be able to talk directly to any program easily. Because languages like PHP or Python are written in C, in our particular Gutenberg parser usecase, it means that the parser can be embedded and used by PHP or Python directly, with almost no overhead. Neat! Rust 🚀 C#</a> </h2> </figure> In order to use Rust from C, one may need 2 elements: A static library (.a</code> file),</li> A header file (.h</code> file).</li> </ol> The theory#</a> </h3> To compile a Rust project into a static library, the crate-type</code> property must contain the staticlib</code> value. Let's edit the Cargo.toml</code> file such as: [lib] name = "gutenberg_post_parser" crate-type = ["staticlib"]</code></pre> Once cargo build --release</code> is run, a libgutenberg_post_parser.a</code> file is created in target/release/</code>. Done. cargo</code> and rustc</code> make this step really a doddle. Now the header file. It can be written manually, but it's tedious and it gets easily outdated. The goal is to automatically generate it. Enter cbindgen</code></a>: cbindgen</code> can be used to generate C bindings for Rust code. It is currently being developed to support creating bindings for WebRender</a>, but has been designed to support any project. </blockquote> To install cbindgen</code>, edit your Cargo.toml</code> file, such as: [package] build = "build.rs" [build-dependencies] cbindgen = "^0.6.0"</code></pre> Actually, cbindgen</code> comes in 2 flavors: CLI executable, or a library. I prefer to use the library approach, which makes installation easier. Note that Cargo has been instructed to use the build.rs</code> file to build the project. This file is an appropriate place to generate the C headers file with cbindgen</code>. Let's write it! extern crate cbindgen; fn main() { let crate_dir = std::env::var("CARGO_MANIFEST_DIR").unwrap(); cbindgen::generate(crate_dir) .expect("Unable to generate C bindings.") .write_to_file("dist/gutenberg_post_parser.h"); }</code></pre> With those information, cbindgen</code> will scan the source code of the project and will generate C headers automatically in the dist/gutenberg_post_parser.h</code> header file. Scanning will be detailed in a moment, but before that, let's quickly see how to control the content of the header file. With the code snippet above, cbindgen</code> will look for a cbindgen.toml</code> configuration file in the CARGO_MANIFEST_DIR</code> directory, i.e. the root of your crate. Mine looks like this: header = """ /* Gutengerg Post Parser, the C bindings. Warning, this file is autogenerated by `cbindgen`. Do not modify this manually. */""" tab_width = 4 language = "C"</code></pre> It describes itself quite easily. The documentation details the configuration</a> very well. cbindgen</code> will scan the code and will stop on struct</code>s or enum</code>s that have the decorator #[repr(C)]</code>, #[repr(_size_)]</code> or #[repr(transparent)]</code>, or functions that are marked as extern "C"</code> and are public. So when one writes: #[repr(C)] pub struct Slice { pointer: *const c_char, length: usize } #[repr(C)] pub enum Option { Some(Slice), None } #[no_mangle] pub extern "C" parse(pointer: *const c_char) -> c_void { … }</code></pre> Then cbindgen</code> will generate this: // … header comment … typedef struct { const char *pointer; uintptr_t length; } Slice; typedef enum { Some, None, } Option_Tag; typedef struct { Slice _0; } Some_Body; typedef struct { Option_Tag tag; union { Some_Body some; }; } Option; void parse(const char *pointer);</code></pre> It works; Great! Note the #[no_mangle]</code> that decorates the Rust parse</code> function. It instructs the compiler to not rename the function, so that the function has the same name from the perspective of C. OK, that's all for the theory. Let's practise now, we have a parser to bind to C! Practise#</a> </h3> We want to bind a function named parse</code>. The function outputs an AST representing the language being analysed. For the recall</a>, the original AST looks like this: pub enum Node<'a> { Block { name: (Input<'a>, Input<'a>), attributes: Option<Input<'a>>, children: Vec<Node<'a>> }, Phase(Input<'a>) }</code></pre> This AST is defined in the Rust parser. The Rust binding to C will transform this AST into another set of structs and enums for C. It is mandatory only for types that are directly exposed to C, not internal types that Rust uses. Let's start by defining Node</code>: #[repr(C)] pub enum Node { Block { namespace: Slice_c_char, name: Slice_c_char, attributes: Option_c_char, children: *const c_void }, Phrase(Slice_c_char) }</code></pre> Some immediate thoughts: The structure Slice_c_char</code> emulates Rust slices (see below),</li> The enum Option_c_char</code> emulates Option</code> (see below),</li> The field children</code> has type *const c_void</code>. It should be *const Vector_Node</code> (our definition of Vector</code>), but the definition of Node</code> is based on Vector_Node</code> and vice versa. This cyclical definition case is unsupported by cbindgen</code> so far</a>. So… yes, it is defined as a void</code> pointer, and will be casted later in C,</li> The fields namespace</code> and name</code> are originally a tuple in Rust. Tuples have no equivalent in C with cbindgen</code>, so two fields are used instead.</li> </ul> Let's define Slice_c_char</code>: #[repr(C)] pub struct Slice_c_char { pointer: *const c_char, length: usize }</code></pre> This definition borrows the semantics of Rust' slices</a>. The major benefit is that there is no copy when binding a Rust slice to this structure. Let's define Option_c_char</code>: #[repr(C)] pub enum Option_c_char { Some(Slice_c_char), None }</code></pre> Finally, we need to define Vector_Node</code> and our own Result</code> for C. They mimic the Rust semantics closely: #[repr(C)] pub struct Vector_Node { buffer: *const Node, length: usize } #[repr(C)] pub enum Result { Ok(Vector_Node), Err }</code></pre> Alright, all types are declared! It's time to write the parse</code> function: #[no_mangle] pub extern "C" fn parse(pointer: *const c_char) -> Result { … }</code></pre> The function takes a pointer from C. It means that the data to analyse (i.e. the Gutenberg blog post) is allocated and owned by C: The memory is allocated on the C side, and Rust is only responsible of the parsing. This is where Rust shines: No copy, no clone, no memory mess, only pointers to this data will be returned to C as slices and vectors. The workflow will be the following: First thing to do when we deal with C: Check that the pointer is not null,</li> Reconstitute an input from the pointer with CStr</code></a>. This standard API is useful to abstract C strings from the Rust point of view. The difference is that a C string terminates by a NULL</code> byte and has no length, while in Rust a string has a length and does not terminate with a NULL</code> byte,</li> Run the parser, then transform the AST into the “C AST”.</li> </ul> Let's do that! pub extern "C" fn parse(pointer: *const c_char) -> Result { if pointer.is_null() { return Result::Err; } let input = unsafe { CStr::from_ptr(pointer).to_bytes() }; if let Ok((_remaining, nodes)) = gutenberg_post_parser::root(input) { let output: Vec = nodes .into_iter() .map(|node| into_c(&node)) .collect(); let vector_node = Vector_Node { buffer: output.as_slice().as_ptr(), length: output.len() }; mem::forget(output); Result::Ok(vector_node); } else { Result::Err } }</code></pre> Only pointers are used in Vector_Node</code>: Pointer to the output, and the length of the output. The conversion is light. Now let's see the into_c</code> function. Some parts will not be detailed; Not because they are difficult but because they are repetitive. The entire code lands here</a>. fn into_c<'a>(node: &ast::Node<'a>) -> Node { match *node { ast::Node::Block { name, attributes, ref children } => { Node::Block { namespace: …, name: …, attributes: …, children: … } }, ast::Node::Phrase(input) => { Node::Phrase(…) } } }</code></pre> I want to show namespace</code> for the warm-up (name</code>, attributes</code> and Phrase</code> are very similar), and children</code> because it deals with void</code>. Let's convert ast::Node::Block.name.0</code> into Node::Block.namespace</code>: ast::Node::Block { name, …, … } => { Node::Block { namespace: Slice_c_char { pointer: name.0.as_ptr() as *const c_char, length: name.0.len() }, …</code></pre> Pretty straightforward so far. namespace</code> is a Slice_c_char</code>. The pointer</code> is the pointer of the name.0</code> slice, and the length</code> is the length of the same name.0</code>. This is the same process for other Rust slices. children</code> is different though. It works in three steps: Collect all children as C AST nodes in a Rust vector,</li> Transform the Rust vector into a valid Vector_Node</code>,</li> Transform the Vector_Node</code> into a *const c_void</code> pointer.</li> </ol> ast::Node::Block { …, …, ref children } => { Node::Block { … children: { // 1. Collect all children as C AST nodes. let output: Vec = children .into_iter() .map(|node| into_c(&node)) .collect(); // 2. Transform the vector into a Vector_Node. let vector_node = if output.is_empty() { Box::new( Vector_Node { buffer: ptr::null(), length: 0 } ) } else { Box::new( Vector_Node { buffer: output.as_slice().as_ptr(), length: output.len() } ) } // 3. Transform Vector_Node into a *const c_void pointer. let vector_node_pointer = Box::into_raw(vector_node) as *const c_void; mem::forget(output); vector_node_pointer }</code></pre> Step 1 is straightforward. Step 2 defines what is the behavior when there is no node. In other words, it defines what an empty Vector_Node</code> is. The buffer</code> must contain a NULL</code> raw pointer, and the length is obviously 0. Without this behavior I got various segmentation fault in my code, even if I checked the length</code> before the buffer</code>. Note that Vector_Node</code> is allocated on the heap with Box::new</code> so that the pointer can be easily shared with C. Step 3 uses the Box::into_raw</code> function</a> to consume the box and to return the wrapped raw pointer of the data it owns. Rust will not free anything here, it's our responsability (or the responsability of C to be pedantic). Then the *mut Vector_Node</code> returned by Box::into_raw</code> can be freely casted into *const c_void</code>. Finally, we instruct the compiler to not drop output</code> when it goes out of scope with mem::forget</code> (at this step of the series, you are very likely to know what it does). Personally, I spent few hours to understand why my pointers got random addresses, or were pointing to a NULL</code> data. The resulting code is simple and kind of clear to read, but it wasn't obvious for me what to do beforehand. And that's all for the Rust part! The next section will present the C code that calls Rust, and how to compile everything all together. C 🚀 executable#</a> </h2> </figure> Now the Rust part is ready, the C part must be written to call it. Minimal Working Example#</a> </h3> Let's do something very quick to see if it links and compiles: #include <stdlib.h> #include <stdio.h> #include <string.h> #include "gutenberg_post_parser.h" int main(int argc, char **argv) { FILE* file = fopen(argv[1], "rb"); fseek(file, 0, SEEK_END); long file_size = ftell(file); rewind(file); char* file_content = (char*) malloc(file_size * sizeof(char)); fread(file_content, 1, file_size, file); // Let's call Rust! Result output = parse(file_content); if (output.tag == Err) { printf("Error while parsing.\n"); return 1; } const Vector_Node nodes = output.ok._0; // Do something with nodes. free(file_content); fclose(file); return 0; }</code></pre> To keep the code concise, I left all the error handlers out of the example. The entire code lands here</a> if you're curious. What happens in this code? The first thing to notice is #include "gutenberg_post_parser.h"</code> which is the header file that is automatically generated by cbindgen</code>. Then a filename from argv[1]</code> is used to read a blog post to parse. The parse</code> function is from Rust, just like the Result</code> and Vector_Node</code> types. The Rust enum Result { Ok(Vector_Node), Err }</code> is compiled to C as: typedef enum { Ok, Err, } Result_Tag; typedef struct { Vector_Node _0; } Ok_Body; typedef struct { Result_Tag tag; union { Ok_Body ok; }; } Result;</code></pre> No need to say that the Rust version is easier and more compact to read, but this isn't the point. To check if Result</code> contains an Ok</code> value or an Err</code>or, one has to check the tag</code> field, like we did with output.tag == Err</code>. To get the content of the Ok</code>, we did output.ok._0</code> (_0</code> is a field from Ok_Body</code>). Let's compile this with clang</code></a>! We assume that this code above is located in the same directory than the gutenberg_post_parser.h</code> file, i.e. in a dist/</code> directory. Thus: $ cd dist $ clang \ # Enable all warnings. \ -Wall \ # Output executable name. \ -o gutenberg-post-parser \ # Input source file. \ gutenberg_post_parser.c \ # Directory where to find the static library (*.a). \ -L ../target/release/ \ # Link with the gutenberg_post_parser.h file. \ -l gutenberg_post_parser \ # Other libraries to link with. -l System \ -l pthread \ -l c \ -l m</code></pre> And that's all! We end up with a gutenberg-post-parser</code> executable that runs C and Rust. More details#</a> </h3> In the original source code</a>, a recursive function that prints the entire AST on stdout</code> can be found, namely print</code> (original, isn't it?). Here is some side-by-side comparisons between Rust syntax and C syntax. The Vector_Node</code> struct in Rust: pub struct Vector_Node { buffer: *const Node, length: usize }</code></pre> The Vector_Node</code> struct in C: typedef struct { const Node *buffer; uintptr_t length; } Vector_Node;</code></pre> So to respectivelly read the number of nodes (length of the vector) and the nodes in C, one has to write: const uintptr_t number_of_nodes = nodes->length; for (uintptr_t nth = 0; nth < number_of_nodes; ++nth) { const Node node = nodes->buffer[nth]; }</code></pre> This is almost idiomatic C code! A Node</code> is defined in C as: typedef enum { Block, Phrase, } Node_Tag; typedef struct { Slice_c_char namespace; Slice_c_char name; Option_c_char attributes; const void* children; } Block_Body; typedef struct { Slice_c_char _0; } Phrase_Body; typedef struct { Node_Tag tag; union { Block_Body block; Phrase_Body phrase; }; } Node;</code></pre> So once a node is fetched, one can write the following code to detect its kind: if (node.tag == Block) { // … } else if (node.tag == Phrase) { // … }</code></pre> Let's focus on Block</code> for a second, and let's print the namespace and the name of the block separated by a slash (/</code>): const Block_Body block = node.block; const Slice_c_char namespace = block.namespace; const Slice_c_char name = block.name; printf( "%.*s/%.s\n", (int) namespace.length, namespace.pointer, (int) name.length, name.pointer );</code></pre> The special %.*s</code> form in printf</code> allows to print a string based on its length and its pointer. I think it is interesting to see the cast from void to Vector_Node</code> for children</code>. It's a single line: const Vector_Node* children = (const Vector_Node*) (block.children);</code></pre> I think that's all for the details! Testing#</a> </h3> I reckon it is also interesting to see how to unit test C bindings directly with Rust. To emulate a C binding, first, the inputs must be in “C form”, so strings must be C strings. I prefer to write a macro for that: macro_rules! str_to_c_char { ($input:expr) => ( { ::std::ffi::CString::new($input).unwrap() } ) }</code></pre> And second, the opposite: The parse</code> function returns data for C, so they need to be “converted back” to Rust. Again, I prefer to write a macro for that: macro_rules! slice_c_char_to_str { ($input:ident) => ( unsafe { ::std::ffi::CStr::from_bytes_with_nul_unchecked( ::std::slice::from_raw_parts( $input.pointer as *const u8, $input.length + 1 ).to_str().unwrap() ) } ) }</code></pre> All right! The final step is to write a unit test. As an example, a Phrase</code> will be tested; The idea remains the same for Block</code> but the code is more concise for the former. #[test] fn test_root_with_a_phrase() { let input = str_to_c_char!("foo"); let output = parse(input.as_ptr()); match output { Result::Ok(result) => match result { Vector_Node { buffer, length } if length == 1 => match unsafe { &*buffer } { Node::Phrase(phrase) => { assert_eq!(slice_c_char_to_str!(phrase), "foo"); }, _ => assert!(false) }, _ => assert!(false) }, _ => assert!(false) } }</code></pre> What happens here? The input</code> and output</code> have been prepared. The former is the C string "foo"</code>. The latter is the result of parse</code>. Then there is a match</code> to validate the form of the AST. Rust is very expressive, and this test is a good illustration. The Vector_Node</code> branch is activated if and only if the length of the vector is 1, which is expressed with the guard if length == 1</code>. Then the content of the phrase is transformed into a Rust string and compared with a regular assert_eq!</code> macro. Note that —in this case— buffer</code> is of type *const Node</code>, so it represents the first element of the vector. If we want to access the next elements, we would need to use the Vec::from_raw_parts</code> function</a> to get a proper Rust API to manipulate this vector. Conclusion#</a> </h2> We have seen that Rust can be embedded in C very easily. In this example, Rust has been compiled to a static library, and a header file; the former is native with Rust tooling, the latter is automatically generated with cbindgen</code>. The parser written in Rust manipulates a string allocated and owned by C. Rust only returns pointers (as slices) to this string back to C. Then C has no difficulties to read those pointers. The only tricky part is that Rust allocates some data (like vectors of nodes) on the heap that C must free. The “free” part has been omitted from the article though: It does not represent a big challenge, and a C developer is likely to be used to this kind of situation. The fact that Rust does not use a garbage collector makes it a perfect candidate for these usecases. The story behind these bindings is actually all about memory: Who allocates what, and What is the form of the data in memory. Rust has a #[repr(C)]</code> decorator to instruct the compiler to use a C memory layout, which makes C bindings extremely simple for the developer. We have also seen that the C bindings can be unit tested within Rust itself, and run with cargo test</code>. cbindgen</code> is a precious companion in this adventure, by automating the header file generation, it reduces the update and the maintenance of the code to a build.rs</code> script. In terms of performance, C should have similar results than Rust, i.e. extremely fast. I didn't run a benchmark to verify this statement, it's purely theoretical. It can be a subject for a next post! Now that we have successfully embedded Rust in C, a whole new world opens up to us! The next episode will push Rust in the PHP world as a native extension (written in C). Let's go! The ASM.js galaxy 2018-08-28T00:00:00+00:00 The second galaxy that our Rust parser will explore is the ASM.js galaxy. This post will explain what ASM.js is, how to compile the parser into ASM.js, and how to use the ASM.js module with Javascript in a browser. The goal is to use ASM.js as a fallback to WebAssembly when it is not available. I highly recommend to read the previous episode</a> about WebAssembly since they have a lot in common. What is ASM.js, and why?#</a> </h2> The main programming language on the Web is Javascript. Applications that want to exist on the Web had to compile to Javascript, like for example games. But a problem occurs: The resulting file is heavy (hence WebAssembly) and Javascript virtual machines have difficulties to optimise this particular code, resulting in slow or inefficient executions (considering the example of games). Also —in this context— Javascript is a compilation target, and as such, some language constructions are useless (like eval</code>). So what if a “new” language can be a compilation target and still be executed by Javascript virtual machines? This is WebAssembly today, but in 2013, the solution was ASM.js</a>: asm.js, a strict subset of Javascript that can be used as a low-level, efficient target language for compilers. This sublanguage effectively describes a sandboxed virtual machine for memory-unsafe languages like C or C++. A combination of static and dynamic validation allows Javascript engines to employ an ahead-of-time (AOT) optimizing compilation strategy for valid asm.js code. </blockquote> So an ASM.js program is a regular Javascript program. It is not a new language but a subset of it. It can be executed by any Javascript virtual machines. However, the specific usage of the magic statement 'use asm';</code> instructs the virtual machine to optimise the program with an ASM.js “engine”. ASM.js introduces types by using arithmetical operators as an annotation system. For instance, x | 0</code> annotes x</code> to be an integer, +x</code> annotates x</code> to be a double, and fround(x)</code> annotates x</code> to be a float. The following example declares a function fn increment(x: u32) -> u32</code>: function increment(x) { x = x | 0; return (x + 1) | 0; }</code></pre> Another important difference is that ASM.js works by module in order to isolate them from Javascript. A module is a function that takes 3 arguments: stdlib</code>, an object with references to standard library APIs,</li> foreign</code>, an object with user-defined functionalities (such as sending something over a WebSocket),</li> heap</code>, an array buffer representing the memory (because memory is manually managed).</li> </ol> But it's still Javascript. So the good news is that if your virtual machine has no specific optimisations for ASM.js, it is executed as any regular Javascript program. And if it does, then you get a pleasant boost. A graph showing 3 benchmarks running against different Javascript engines: Firefox, Firefox + asm.js, Google, and native. </figcaption> </figure> Remember that ASM.js has been designed to be a compilation target. So normally you don't have to care about that because it is the role of the compiler. The typical compilation and execution pipeline from C or C++ to the Web looks like this: Classical ASM.js compilation and execution pipeline from C or C++ to the Web. </figcaption> </figure> Emscripten</a>, as seen in the schema above, is a very important project in this whole evolution of the Web platform. Emscripten is: a toolchain for compiling to asm.js and WebAssembly, built using LLVM, that lets you run C and C++ on the web at near-native speed without plugins. </blockquote> You are very likely to see this name one day or another if you work with ASM.js or WebAssembly. I will not explain deeply what ASM.js is with a lot of examples. I recommend instead to read Asm.js: The Javascript Compile Target</a> by John Resig, or Big Web app? Compile it!</a> by Alon Zakai. Our process will be different though. We will not compile our Rust code directly to ASM.js, but instead, we will compile it to WebAssembly, which in turn will be compiled into ASM.js. Rust 🚀 ASM.js#</a> </h2> </figure> This episode will be very short, and somehow the most easiest one. To compile Rust to ASM.js, you need to first compile it to WebAssembly (see the previous episode</a>), and then compile the WebAssembly binary into ASM.js. Actually, ASM.js is mostly required when the browser does not support WebAssembly, like Internet Explorer. It is essentially a fallback to run our program on the Web. The workflow is the following: Compile your Rust project into WebAssembly,</li> Compile your WebAssembly binary into an ASM.js module,</li> Optimise and shrink the ASM.js module.</li> </ol> The wasm2js tool</a> will be your best companion to compile the WebAssembly binary into an ASM.js module. It is part of Binaryen project. Then, assuming we have the WebAssembly binary of our program, all we have to do is: $ wasm2js --pedantic --output gutenberg_post_parser.asm.js gutenberg_post_parser.wasm</code></pre> At this step, the gutenberg_post_parser.asm.js</code> weights 212kb. The file contains ECMAScript 6 code. And remember that old browsers are considered, like Internet Explorer, so the code needs to be transformed a little bit. To optimise and shrink the ASM.js module, we will use the uglify-es</code> tool</a>, like this: $ # Transform code, and embed in a function. $ sed -i '' '1s/^/function GUTENBERG_POST_PARSER_ASM_MODULE() {/; s/export //' gutenberg_post_parser.asm.js $ echo 'return { root, alloc, dealloc, memory }; }' >> gutenberg_post_parser.asm.js $ # Shrink the code. $ uglifyjs --compress --mangle --output .temp.asm.js gutenberg_post_parser.asm.js $ mv .temp.asm.js gutenberg_post_parser.asm.js</code></pre> Just like we did for the WebAssembly binary, we can compress the resulting files with gzip</code></a> and brotli</code></a>: $ # Compress. $ gzip --best --stdout gutenberg_post_parser.asm.js > gutenberg_post_parser.asm.js.gz $ brotli --best --stdout --lgwin=24 gutenberg_post_parser.asm.js > gutenberg_post_parser.asm.js.br</code></pre> We end up with the following file sizes: .asm.js</code>: 54kb,</li> .asm.js.gz</code>: 13kb,</li> .asm.js.br</code>: 11kb.</li> </ul> That's again pretty small! When you think about it, this is a lot of transformations: From Rust to WebAssembly to Javascript/ASM.js… The amount of tools is rather small compared to the amount of work. It shows a well-designed pipeline and a collaboration between many groups of people. Aside: If you are reading this post, I assume you are developers. And as such, I'm sure you can spend hours looking at a source code like if it is a master painting. Did you ever wonder what a Rust program looks like once compiled to Javascript? See bellow: A Rust program compiled as WebAssembly compiled as ASM.js. </figcaption> </figure> I like it probably too much. ASM.js 🚀 Javascript#</a> </h2> The resulting gutenberg_post_parser.asm.js</code> file contains a single function named GUTENBERG_POST_PARSER_ASM_MODULE</code> which returns an object pointing to 4 private functions: root</code>, the axiom of our grammar,</li> alloc</code>, to allocate memory,</li> dealloc</code>, to deallocate memory, and</li> memory</code>, the memory buffer.</li> </ol> It sounds familiar if you have read the previous episode with WebAssembly</a>. Don't expect root</code> to return a full AST: It will return a pointer to the memory, and the data need to be encoded and decoded, and to write into and to read from the memory the same way. Yes, the same way. The exact same way. So the code of the boundary layer is strictly the same. Do you remember the Module</code> object in our WebAssembly Javascript boundary? This is exactly what the GUTENBERG_POST_PARSER_ASM_MODULE</code> function returns. You can replace Module</code> by the returned object, et voilà! The entired code lands here</a>. It completely reuses the Javascript boundary layer for WebAssembly. It just sets the Module</code> differently, and it does not load the WebAssembly binary. Consequently, the ASM.js boundary layer is made of 34 lines of code, only 🙃. It compresses to 218 bytes. Conclusion#</a> </h2> We have seen that ASM.js can be fallback to WebAssembly in environments that only support Javascript (like Internet Explorer), with or without ASM.js optimisations. The resulting ASM.js file and its boundary layer are quite small. By design, the ASM.js boundary layer reuses almost the entire WebAssembly boundary layer. Therefore there is again a tiny surface of code to review and to maintain, which is helpful. We have seen in the previous episode that Rust is very fast. We have been able to observe the same statement for WebAssembly compared to the actual Javascript parser for the Gutenberg project. However, is it still true for the ASM.js module? In this case, ASM.js is a fallback, and like all fallbacks, they are notably slower than the targeted implementations. Let's run the same benchmark but use the Rust parser as an ASM.js module: Document</th> Javascript parser (ms)</th> Rust parser as an ASM.js module (ms)</th> speedup</th></tr></thead> demo-post.html</code></a></td> 15.368</td> 2.718</td> × 6</td></tr> shortcode-shortcomings.html</code></a></td> 31.022</td> 8.004</td> × 4</td></tr> redesigning-chrome-desktop.html</code></a></td> 106.416</td> 19.223</td> × 6</td></tr> web-at-maximum-fps.html</code></a></td> 82.92</td> 27.197</td> × 3</td></tr> early-adopting-the-future.html</code></a></td> 119.880</td> 38.321</td> × 3</td></tr> pygmalian-raw-html.html</code></a></td> 349.075</td> 23.656</td> × 15</td></tr> moby-dick-parsed.html</code></a></td> 2,543.75</td> 361.423</td> × 7</td></tr> </tbody></table> Benchmark between Javascript parser and Rust parser as an ASM.js module. </figcaption> </figure> The ASM.js module of the Rust parser is in average 6 times faster than the actual Javascript implementation. The median speedup is 6. That's far from the WebAssembly results, but this is a fallback, and in average, it is 6 times faster, which is really great! So not only the whole pipeline is safer because it starts from Rust, but it ends to be faster than Javascript. We will see in the next episodes of this series that Rust can reach a lot of galaxies, and the more it travels, the more it gets interesting. Thanks for reading! The WebAssembly galaxy 2018-08-22T00:00:00+00:00 The first galaxy that our Rust parser will explore is the WebAssembly (Wasm) galaxy. This post will explain what WebAssembly is, how to compile the parser into WebAssembly, and how to use the WebAssembly binary with Javascript in a browser and with NodeJS. What is WebAssembly, and why?#</a> </h2> If you already know WebAssembly, you can skip this section. WebAssembly</a> defines itself as: WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable target for compilation of high-level languages like C/C++/Rust, enabling deployment on the web for client and server applications. </blockquote> Should I say more? Probably, yes… WebAssembly is a new portable binary format. Languages like C, C++, or Rust already compiles to this target. It is the spirit successor of ASM.js</a>. By spirit successor, I mean it is the same people trying to extend the Web platform and to make the Web fast that are working on both technologies. They share some design concepts too, but that's not really important right now. Before WebAssembly, programs had to compile to Javascript in order to run on the Web platform. The resulting files were most of the time large. And because the Web is a network, the files had to be downloaded, and it took time. WebAssembly is designed to be encoded in a size- and load-time efficient binary format</a>. WebAssembly is also faster than Javascript for many reasons. Despites all the crazy optimisations engineers put in the Javascript virtual machines, Javascript is a weakly and dynamically typed language, which requires to be interpreted. WebAssembly aims to execute at native speed by taking advantage of common hardware capabilities</a>. WebAssembly also loads faster than Javascript</a> because parsing and compiling happen while the binary is streamed from the network. So once the binary is entirely fetched, it is ready to run: No need to wait on the parser and the compiler before running the program. Today, and our blog series is a perfect example of that, it is possible to write a Rust program, and to compile it to run on the Web platform. Why? Because WebAssembly is implemented by all major browsers</a>, and because it has been designed for the Web: To live and run on the Web platform (like a browser). But its portable aspect and its safe and sandboxed memory design</a> make it a good candidate to run outside of the Web platform (see a serverless Wasm framework</a>, or an application container built for Wasm</a>). I think it is important to remind that WebAssembly is not here to replace Javascript. It is just another technology which solves many problems we can meet today, like load-time, safety, or speed. Rust 🚀 WebAssembly#</a> </h2> </figure> The Rust Wasm team</a> is a group of people leading the effort of pushing Rust into WebAssembly with a set of tools and integrations. There is a book</a> explaining how to write a WebAssembly program with Rust. With the Gutenberg Rust parser, I didn't use tools like wasm-bindgen</code></a> (which is a pure gem) when I started the project few months ago because I hit some limitations. Note that some of them have been addressed since then! Anyway, we will do most of the work by hand, and I think this is an excellent way to understand how things work in the background. When you are familiar with WebAssembly interactions, then wasm-bindgen</code> is an excellent tool to have within easy reach, because it abstracts all the interactions and let you focus on your code logic instead. I would like to remind the reader that the Gutenberg Rust parser exposes one AST, and one root</code> function (the axiom of the grammar), respectively defined as: pub enum Node<'a> { Block { name: (Input<'a>, Input<'a>), attributes: Option<Input<'a>>, children: Vec<Node<'a>> }, Phrase(Input<'a>) }</code></pre> and pub fn root( input: Input ) -> Result<(Input, Vec<ast::Node>), nom::Err<Input>>;</code></pre> Knowing that, let's go! General design#</a> </h3> Here is our general design or workflow: Javascript (for instance) writes the blog post to parse into the WebAssembly module memory,</li> Javascript runs the root</code> function by passing a pointer to the memory, and the length of the blog post,</li> Rust reads the blog post from the memory, runs the Gutenberg parser, compiles the resulting AST into a sequence of bytes, and returns the pointer to this sequence of bytes to Javascript,</li> Javascript reads the memory from the received pointer, and decodes the sequence of bytes as Javascript objects in order to recreate an AST with a friendly API.</li> </ol> Why a sequence of bytes? Because WebAssembly only supports integers and floats, not strings or vectors, and also because our Rust parser takes a slice of bytes as input, so this is handy. We use the term boundary layer to refer to this Javascript piece of code responsible to read from and write into the WebAssembly module memory, and responsible of exposing a friendly API. Now, we will focus on the Rust code. It consists of only 4 functions: alloc</code> to allocate memory (exported),</li> dealloc</code> to deallocate memory (exported),</li> root</code> to run the parser (exported),</li> into_bytes</code> to transform the AST into a sequence of bytes.</li> </ul> The entire code lands here</a>. It is approximately 150 lines of code. We explain it. Memory allocation#</a> </h3> Let's start by the memory allocator. I choose to use wee_alloc</code> for the memory allocator</a>. It is specifically designed for WebAssembly by being very small (less than a kilobyte) and efficient. The following piece of code describes the memory allocator setup and the “prelude” for our code (enabling some compiler features, like alloc</code>, declaring external crates, some aliases, and declaring required function like panic</code>, oom</code> etc.). This can be considered as a boilerplate: #![no_std] #![feature( alloc, alloc_error_handler, core_intrinsics, lang_items )] extern crate gutenberg_post_parser; extern crate wee_alloc; #[macro_use] extern crate alloc; use gutenberg_post_parser::ast::Node; use alloc::vec::Vec; use core::{mem, slice}; #[global_allocator] static ALLOC: wee_alloc::WeeAlloc = wee_alloc::WeeAlloc::INIT; #[panic_handler] fn panic(_info: &core::panic::PanicInfo) -> ! { unsafe { core::intrinsics::abort(); } } #[alloc_error_handler] fn oom(_: core::alloc::Layout) -> ! { unsafe { core::intrinsics::abort(); } } // This is the definition of `std::ffi::c_void`, but Wasm runs without std in our case. #[repr(u8)] #[allow(non_camel_case_types)] pub enum c_void { #[doc(hidden)] __variant1, #[doc(hidden)] __variant2 }</code></pre> The Rust memory is the WebAssembly memory. Rust will allocate and deallocate memory on its own, but Javascript for instance needs to allocate and deallocate WebAssembly memory in order to communicate/exchange data. So we need to export one function to allocate memory and one function to deallocate memory. Once again, this is almost a boilerplate. The alloc</code> function creates an empty vector of a specific capacity (because it is a linear segment of memory), and returns a pointer to this empty vector: #[no_mangle] pub extern "C" fn alloc(capacity: usize) -> *mut c_void { let mut buffer = Vec::with_capacity(capacity); let pointer = buffer.as_mut_ptr(); mem::forget(buffer); pointer as *mut c_void }</code></pre> Note the #[no_mangle]</code> attribute that instructs the Rust compiler to not mangle the function name, i.e. to not rename it. And extern "C"</code> to export the function in the WebAssembly module, so it is “public” from outside the WebAssembly binary. The code is pretty straightforward and matches what we announced earlier: A Vec</code> is allocated with a specific capacity, and the pointer to this vector is returned. The important part is mem::forget(buffer)</code>. It is required so that Rust will not deallocate the vector once it goes out of scope. Indeed, Rust enforces Resource Acquisition Is Initialization (RAII)</a>, so whenever an object goes out of scope, its destructor is called and its owned resources are freed. This behavior shields against resource leaks bugs, and this is why we will never have to manually free memory or worry about memory leaks in Rust (see some RAII examples</a>). In this case, we want to allocate and keep the allocation after the function execution, hence the mem::forget</code> call</a>. Let's jump on the dealloc</code> function. The goal is to recreate a vector based on a pointer and a capacity, and to let Rust drops it: #[no_mangle] pub extern "C" fn dealloc(pointer: *mut c_void, capacity: usize) { unsafe { let _ = Vec::from_raw_parts(pointer, 0, capacity); } }</code></pre> The Vec::from_raw_parts</code> function</a> is marked as unsafe, so we need to delimit it in an unsafe</code> block so that the dealloc</code> function is considered as safe. The variable _</code> contains our data to deallocate, and it goes out of scope immediately, so Rust drops it. From input to a flat AST#</a> </h3> Now the core of the binding! The root</code> function reads the blog post to parse based on a pointer and a length, then it parses it. If the result is OK, it serializes the AST into a sequence of bytes, i.e. it flatten it, otherwise it returns an empty sequence of bytes. The image illustrates the flow of the data: first off there is a blog post; second there is the AST of the blog post; finally there is a linear byte-encoded representation of the AST of the blog post. </figcaption> </figure> The logic flow of the parser: The input on the left is parsed into an AST, which is serialized into a flat sequence of bytes on the right. #[no_mangle] pub extern "C" fn root(pointer: *mut u8, length: usize) -> *mut u8 { let input = unsafe { slice::from_raw_parts(pointer, length) }; let mut output = vec![]; if let Ok((_remaining, nodes)) = gutenberg_post_parser::root(input) { // Compile the AST (nodes) into a sequence of bytes. } let pointer = output.as_mut_ptr(); mem::forget(output); pointer }</code></pre> The variable input</code> contains the blog post. It is fetched from memory with a pointer and a length. The variable output</code> is the sequence of bytes the function will return. gutenberg_post_parser::root(input)</code> runs the parser. If parsing is OK, then the nodes</code> are compiled into a sequence of bytes (omitted for now). Then the pointer to the sequence of bytes is grabbed, the Rust compiler is instructed to not drop it, and finally the pointer is returned. The logic is again pretty straightforward. Now, let's focus on the AST to the sequence of bytes (u8</code>) compilation. All data the AST hold are already bytes, which makes the process easier. The goal is only to flatten the AST: The first 4 bytes represent the number of nodes at the first level (4 × u8</code> represents u32</code>) ,</li> Next, if the node is Block</code>: The first byte is the node type: 1u8</code> for a block,</li> The second byte is the size of the block name,</li> The third to the sixth bytes are the size of the attributes,</li> The seventh byte is the number of node children the block has,</li> Next bytes are the block name,</li> Next bytes are the attributes (&b"null"[..]</code> if none),</li> Next bytes are node children as a sequence of bytes,</li> </ul> </li> Next, if the node is Phrase</code>: The first byte is the node type: 2u8</code> for a phrase,</li> The second to the fifth bytes are the size of the phrase,</li> Next bytes are the phrase.</li> </ul> </li> </ul> Here is the missing part of the root</code> function: if let Ok((_remaining, nodes)) = gutenberg_post_parser::root(input) { let nodes_length = u32_to_u8s(nodes.len() as u32); output.push(nodes_length.0); output.push(nodes_length.1); output.push(nodes_length.2); output.push(nodes_length.3); for node in nodes { into_bytes(&node, &mut output); } }</code></pre> And here is the into_bytes</code> function: fn into_bytes<'a>(node: &Node<'a>, output: &mut Vec<u8>) { match *node { Node::Block { name, attributes, ref children } => { let node_type = 1u8; let name_length = name.0.len() + name.1.len() + 1; let attributes_length = match attributes { Some(attributes) => attributes.len(), None => 4 }; let attributes_length_as_u8s = u32_to_u8s(attributes_length as u32); let number_of_children = children.len(); output.push(node_type); output.push(name_length as u8); output.push(attributes_length_as_u8s.0); output.push(attributes_length_as_u8s.1); output.push(attributes_length_as_u8s.2); output.push(attributes_length_as_u8s.3); output.push(number_of_children as u8); output.extend(name.0); output.push(b'/'); output.extend(name.1); if let Some(attributes) = attributes { output.extend(attributes); } else { output.extend(&b"null"[..]); } for child in children { into_bytes(&child, output); } }, Node::Phrase(phrase) => { let node_type = 2u8; let phrase_length = phrase.len(); output.push(node_type); let phrase_length_as_u8s = u32_to_u8s(phrase_length as u32); output.push(phrase_length_as_u8s.0); output.push(phrase_length_as_u8s.1); output.push(phrase_length_as_u8s.2); output.push(phrase_length_as_u8s.3); output.extend(phrase); } } }</code></pre> What I find interesting with this code is it reads just like the bullet list above the code. For the most curious, here is the u32_to_u8s</code> function: fn u32_to_u8s(x: u32) -> (u8, u8, u8, u8) { ( ((x >> 24) & 0xff) as u8, ((x >> 16) & 0xff) as u8, ((x >> 8) & 0xff) as u8, ( x & 0xff) as u8 ) }</code></pre> Here we are. alloc</code>, dealloc</code>, root</code>, and into_bytes</code>. Four functions, and everything is done. Producing and optimising the WebAssembly binary#</a> </h3> To get a WebAssembly binary, the project has to be compiled to the wasm32-unknown-unknown</code> target. For now (and it will change in a near future), the nightly toolchain is needed to compile the project, so make sure you have the latest nightly version of rustc</code> & co. installed with rustup update nightly</code>. Let's run cargo</code>: $ RUSTFLAGS='-g' cargo +nightly build --target wasm32-unknown-unknown --release</code></pre> The WebAssembly binary weights 22kb. Our goal is to reduce the file size. For that, the following tools will be required: wasm-gc</code></a> to garbage-collect unused imports, internal functions, types etc.,</li> wasm-snip</code></a> to mark some functions as unreachable, this is useful when the binary includes unused code that the linker were not able to remove,</li> wasm-opt</code> from the Binaryen project</a>, to optimise the binary,</li> gzip</code></a> and brotli</code></a> to compress the binary.</li> </ul> Basically, what we do is the following: $ # Garbage-collect unused data. $ wasm-gc gutenberg_post_parser.wasm $ # Mark fmt and panicking as unreachable. $ wasm-snip --snip-rust-fmt-code --snip-rust-panicking-code gutenberg_post_parser.wasm -o gutenberg_post_parser_snipped.wasm $ mv gutenberg_post_parser_snipped.wasm gutenberg_post_parser.wasm $ # Garbage-collect unreachable data. $ wasm-gc gutenberg_post_parser.wasm $ # Optimise for small size. $ wasm-opt -Oz -o gutenberg_post_parser_opt.wasm gutenberg_post_parser.wasm $ mv gutenberg_post_parser_opt.wasm gutenberg_post_parser.wasm $ # Compress. $ gzip --best --stdout gutenberg_post_parser.wasm > gutenberg_post_parser.wasm.gz $ brotli --best --stdout --lgwin=24 gutenberg_post_parser.wasm > gutenberg_post_parser.wasm.br </code></pre> We end up with the following file sizes: .wasm</code>: 16kb,</li> .wasm.gz</code>: 7.3kb,</li> .wasm.br</code>: 6.2kb.</li> </ul> Neat! Brotli is implemented by most browsers</a>, so when the client sends Accept-Encoding: br</code>, the server can response with the .wasm.br</code> file. To give you a feeling of what 6.2kb represent, the following image also weights 6.2kb: An image that is as weight as our compressed WebAssembly module. </figcaption> </figure> The WebAssembly binary is ready to run! WebAssembly 🚀 Javascript#</a> </h2> </figure> In this section, we assume Javascript runs in a browser. Thus, what we need to do is the following: Load/stream and instanciate the WebAssembly binary,</li> Write the blog post to parse in the WebAssembly module memory,</li> Call the root</code> function on the parser,</li> Read the WebAssembly module memory to load the flat AST (a sequence of bytes) and decode it to build a “Javascript AST” (with our own objects).</li> </ol> The entire code lands here</a>. It is approximately 150 lines of code too. I won't explain the whole code since some parts of it is the “friendly API” that is exposed to the user. So I will rather explain the major pieces. Loading/streaming and instanciating#</a> </h3> The WebAssembly</code> API</a> exposes multiple ways to load a WebAssembly binary. The best you can use is the WebAssembly.instanciateStreaming</code> function</a>: It streams the binary and compiles it in the same time, nothing is blocking. This API relies on the Fetch</code> API</a>. You might have guessed it: It is asynchronous (it returns a promise). WebAssembly itself is not asynchronous (except if you use thread), but the instanciation step is. It is possible to avoid that, but this is tricky, and Google Chrome has a strong limit of 4kb for the binary size which will make you give up quickly. To be able to stream the WebAssembly binary, the server must send the application/wasm</code> MIME type (with the Content-Type</code> header). Let's instanciate our WebAssembly: const url = '/gutenberg_post_parser.wasm'; const wasm = WebAssembly. instantiateStreaming(fetch(url), {}). then(object => object.instance). then(instance => { /* step 2 */ });</code></pre> The WebAssembly binary has been instanciated! Now we can move to the next step. Last polish before running the parser#</a> </h3> Remember that the WebAssembly binary exports 3 functions: alloc</code>, dealloc</code>, and root</code>. They can be found on the exports</code> property, along with the memory. Let's write that: then(instance => { const Module = { alloc: instance.exports.alloc, dealloc: instance.exports.dealloc, root: instance.exports.root, memory: instance.exports.memory }; runParser(Module, 'xyz'); });</code></pre> Great, everything is ready to write the runParser</code> function! The parser runner#</a> </h3> As a reminder, this function has to: Write the input</code> (the blog post to parse) in the WebAssembly module memory (Module.memory</code>), to call the root</code> function (Module.root</code>), and to read the result from the WebAssembly module memory. Let's do that: function runParser(Module, raw_input) { const input = new TextEncoder().encode(raw_input); const input_pointer = writeBuffer(Module, input); const output_pointer = Module.root(input_pointer, input.length); const result = readNodes(Module, output_pointer); Module.dealloc(input_pointer, input.length); return result; }</code></pre> In details: The raw_input</code> is encoded into a sequence of bytes with the TextEncoder</code>API</a>, in input</code>,</li> The input is written into the WebAssembly memory module with writeBuffer</code> and its pointer is returned,</li> Then the root</code> function is called with the pointer to the input and the length of the input as expected, and the pointer to the output is returned,</li> Then the output is decoded,</li> And finally, the input is deallocated. The output of the parser will be deallocated in the readNodes</code> function because its length is unknown at this step.</li> </ul> Great! So we have 2 functions to write right now: writeBuffer</code> and readNodes</code>. Writing the data in memory#</a> </h3> Let's go with the first one, writeBuffer</code>: function writeBuffer(Module, buffer) { const buffer_length = buffer.length; const pointer = Module.alloc(buffer_length); const memory = new Uint8Array(Module.memory.buffer); for (let i = 0; i < buffer_length; ++i) { memory[pointer + i] = buffer[i]; } return pointer; }</code></pre> In details: The length of the buffer is read in buffer_length</code>,</li> A space in memory is allocated to write the buffer,</li> Then a uint8</code> view of the buffer is instanciated, which means that the buffer will be viewed as a sequence of u8</code>, exactly what Rust expects,</li> Finally the buffer is copied into the memory with a loop, that's very basic, and return the pointer.</li> </ul> Note that, unlike C strings, adding a NUL</code> byte at the end is not mandatory. This is just the raw data (on the Rust side, we read it with slice::from_raw_parts</code>, slice is a very simple structure). Reading the output of the parser#</a> </h3> So at this step, the input has been written in memory, and the root</code> function has been called so it means the parser has run. It has returned a pointer to the output (the result) and we now have to read it and decode it. Remind that the first 4 bytes encodes the number of nodes we have to read. Let's go! function readNodes(Module, start_pointer) { const buffer = new Uint8Array(Module.memory.buffer.slice(start_pointer)); const number_of_nodes = u8s_to_u32(buffer[0], buffer[1], buffer[2], buffer[3]); if (0 >= number_of_nodes) { return null; } const nodes = []; let offset = 4; let end_offset; for (let i = 0; i < number_of_nodes; ++i) { const last_offset = readNode(buffer, offset, nodes); offset = end_offset = last_offset; } Module.dealloc(start_pointer, start_pointer + end_offset); return nodes; }</code></pre> In details: A uint8</code> view of the memory is instanciated… more precisely: A slice of the memory starting at start_pointer</code>,</li> The number of nodes is read, then all nodes are read,</li> And finally, the output of the parser is deallocated.</li> </ul> For the record, here is the u8s_to_u32</code> function, this is the exact opposite of u32_to_u8s</code>: function u8s_to_u32(o, p, q, r) { return (o << 24) | (p << 16) | (q << 8) | r; }</code></pre> And I will also share the readNode</code> function, but I won't explain the details. This is just the decoding part of the output from the parser. function readNode(buffer, offset, nodes) { const node_type = buffer[offset]; // Block. if (1 === node_type) { const name_length = buffer[offset + 1]; const attributes_length = u8s_to_u32(buffer[offset + 2], buffer[offset + 3], buffer[offset + 4], buffer[offset + 5]); const number_of_children = buffer[offset + 6]; let payload_offset = offset + 7; let next_payload_offset = payload_offset + name_length; const name = new TextDecoder().decode(buffer.slice(payload_offset, next_payload_offset)); payload_offset = next_payload_offset; next_payload_offset += attributes_length; const attributes = JSON.parse(new TextDecoder().decode(buffer.slice(payload_offset, next_payload_offset))); payload_offset = next_payload_offset; let end_offset = payload_offset; const children = []; for (let i = 0; i < number_of_children; ++i) { const last_offset = readNode(buffer, payload_offset, children); payload_offset = end_offset = last_offset; } nodes.push(new Block(name, attributes, children)); return end_offset; } // Phrase. else if (2 === node_type) { const phrase_length = u8s_to_u32(buffer[offset + 1], buffer[offset + 2], buffer[offset + 3], buffer[offset + 4]); const phrase_offset = offset + 5; const phrase = new TextDecoder().decode(buffer.slice(phrase_offset, phrase_offset + phrase_length)); nodes.push(new Phrase(phrase)); return phrase_offset + phrase_length; } else { console.error('unknown node type', node_type); } }</code></pre> Note that this code is pretty simple and easy to optimise by the Javascript virtual machine. It is almost important to note that this is not the original code. The original version is a little more optimised here and there, but they are very close. And that's all! We have successfully read and decoded the output of the parser! We just need to write the Block</code> and Phrase</code> classes like this: class Block { constructor(name, attributes, children) { this.name = name; this.attributes = attributes; this.children = children; } } class Phrase { constructor(phrase) { this.phrase = phrase; } }</code></pre> The final output will be an array of those objects. Easy! WebAssembly 🚀 NodeJS#</a> </h2> </figure> The differences between the Javascript version and the NodeJS version are few: The Fetch</code> API does not exist in NodeJS, so the WebAssembly binary has to be instanciated with a buffer directly, like this: WebAssembly.instantiate(fs.readFileSync(url), {})</code>,</li> The TextEncoder</code> and TextDecoder</code> objects do not exist as global objects, they are in util.TextEncoder</code> and util.TextDecoder</code>.</li> </ul> In order to share the code between both environments, it is possible to write the boundary layer (the Javascript code we wrote) in a .mjs</code> file, aka ECMAScript Module. It allows to write something like import { Gutenberg_Post_Parser } from './gutenberg_post_parser.mjs'</code> for example (considering the whole code we wrote before is a class). On the browser side, the script must be loaded with<script type="module" src="…" /></code>, and on the NodeJS side, node</code> must run with the --experimental-modules</code> flag. I can recommend you this talk Please wait… loading: a tale of two loaders by Myles Borins</a> at the JSConf EU 2018 to understand all the story about that. The entire code lands here</a>. Conclusion#</a> </h2> We have seen in details how to write a real world parser in Rust, how to compile it into a WebAssembly binary, and how to use it with Javascript and with NodeJS. The parser can be used in a browser with regular Javascript code, or as a CLI with NodeJS, or on any platforms NodeJS supports. The Rust part for WebAssembly plus the Javascript part totals 313 lines of code. This is a tiny surface of code to review and to maintain compared to writing a Javascript parser from scratch. Another argument is the safety and performance. Rust is memory safe, we know that. It is also performant, but is it still true for the WebAssembly target? The following table shows the benchmark results of the actual Javascript parser for the Gutenberg project (implemented with PEG.js</a>), against this project: The Rust parser as a WebAssembly binary. Document</th> Javascript parser (ms)</th> Rust parser as a WebAssembly binary (ms)</th> speedup</th></tr></thead> demo-post.html</code></a></td> 13.167</td> 0.252</td> × 52</td></tr> shortcode-shortcomings.html</code></a></td> 26.784</td> 0.271</td> × 98</td></tr> redesigning-chrome-desktop.html</code></a></td> 75.500</td> 0.918</td> × 82</td></tr> web-at-maximum-fps.html</code></a></td> 88.118</td> 0.901</td> × 98</td></tr> early-adopting-the-future.html</code></a></td> 201.011</td> 3.329</td> × 60</td></tr> pygmalian-raw-html.html</code></a></td> 311.416</td> 2.692</td> × 116</td></tr> moby-dick-parsed.html</code></a></td> 2,466.533</td> 25.14</td> × 98</td></tr> </tbody></table> Benchmarks between Javascript parser and Rust parser as a WebAssembly binary. </figcaption> </figure> The WebAssembly binary is in average 86 times faster than the actual Javascript implementation. The median of the speedup is 98. Some edge cases are very interesting, like moby-dick-parsed.html</code> where it takes 2.5s with the Javascript parser against 25ms with WebAssembly. So not only it is safer, but it is faster than Javascript in this case. And it is only 300 lines of code. Note that WebAssembly does not support SIMD yet: It is still a proposal</a>. Rust is gently supporting it (example with PR #549</a>). It will dramatically improve the performances! We will see in the next episodes of this series that Rust can reach a lot of galaxies, and the more it travels, the more it gets interesting. Thanks for reading! Prelude 2018-08-21T00:00:00+00:00 At my work</a>, I had an opportunity to start an experiment: Writing a single parser implementation in Rust for the new Gutenberg post format</a>, bound to many platforms and environments. Gutenberg's logo. </figcaption> </figure> This series of posts is about those bindings, and explains how to send Rust beyond earth, into many different galaxies. The Gutenberg post format#</a> </h2> Let's introduce quickly what Gutenberg is, and why a new post format. If you want an in-depth presentation, I highly recommend to read The Language of Gutenberg</a>. Note that this is not required for the reader to understand the Gutenberg post format. Gutenberg</a> is the next WordPress editor. It is a little revolution on its own. The features it unlocks are very powerful. The editor will create a new page- and post-building experience that makes writing rich posts effortless, and has “blocks” to make it easy what today might take shortcodes, custom HTML, or “mystery meat” embed discovery. — Matt Mullenweg </blockquote> The format of a blog post was HTML. And it continues to be. However, another semantics layer is added through annotations. Annotations are written in comments and borrow the XML syntax, e.g.:  <p>phrase</p> </code></pre> The Gutenberg format provides 2 constructions: Block, and Phrase. The example above contains both: There is a block wrapping a phrase. A phrase is basically anything that is not a block. Let's describe the example: It starts with an annotation (</code>),</li> The wp:</code> is mandatory to represent a Gutenberg block,</li> It is followed by a fully qualified block name, which is a pair of an optional namespace (here sets to ns</code> , defaults to core</code>) and a block name (here sets to block-name</code>), separated by a slash,</li> A block has optional attributes encoded as a JSON object (see RFC 7159, Section 4, Objects</a>),</li> Finally, a block has optional children, i.e. a heterogeneous collection of blocks or phrases. In the example above, there is one child that is the phrase phrase</code>. And the following example below shows a block with no child:</li> </ul> </code></pre> The complete grammar can be found in the parser's documentation</a>. Finally, the parser is used on the editor side, not on the rendering side. Once rendered, the blog post is a regular HTML file. Some blocks are dynamics though, but this is another topic. The logic flow of the editor (How Little Blocks Work</a>). </figcaption> </figure> The grammar is relatively small. The challenges are however to be as much performant and memory efficient as possible on many platforms. Some posts can reach megabytes, and we don't want the parser to be the bottleneck. Even if it is used when creating the post state (cf. the schema above), we have measured several seconds to load some posts. Time during which the user is blocked, and waits, or see an error. In other scenarii, we have hit memory limit of the language's virtual machines. Hence this experimental project! The current parsers are written in JavaScript (with PEG.js</a>) and in PHP (with phpegjs</code></a>). This Rust project proposes a parser written in Rust, that can run in the JavaScript and in the PHP virtual machines, and on many other platforms. Let's try to be very performant and memory efficient! Why Rust?#</a> </h2> That's an excellent question! Thanks for asking. I can summarize my choice with a bullet list: It is fast, and we need speed,</li> It is memory safe, and also memory efficient,</li> No garbage collector, which simplifies memory management across environments,</li> It can expose a C API (with Foreign Function Interface, FFI</a>), which eases the integration into multiple environments,</li> It compiles to many targets</a>,</li> Because I love it.</li> </ul> One of the goal of the experimentation is to maintain a single implementation (maybe the future reference implementation) with multiple bindings. The parser#</a> </h2> The parser is written in Rust. It relies on the fabulous nom library</a>. nom will happily take a byte out of your files 🙂 </figcaption> </figure> The source code is available in the src/</code> directory in the repository</a>. It is very small and fun to read. The parser produces an Abstract Syntax Tree (AST) of the grammar, where nodes of the tree are defined as: pub enum Node<'a> { Block { name: (Input<'a>, Input<'a>), attributes: Option<Input<'a>>, children: Vec<Node<'a>> }, Phrase(Input<'a>) }</code></pre> That's all! We find again the block name, the attributes and the children, and the phrase. Block children are defined as a collection of node, this is recursive. Input<'a></code> is defined as &'a [u8]</code>, i.e. a slice of bytes. The main parser entry is the root</code> function</a>. It represents the axiom of the grammar, and is defined as: pub fn root( input: Input ) -> Result<(Input, Vec<ast::Node>), nom::Err<Input>>;</code></pre> So the parser returns a collection of nodes in the best case. Here is an simple example: use gutenberg_post_parser::{root, ast::Node}; let input = &b""[..]; let output = Ok( ( // The remaining data. &b""[..], // The Abstract Syntax Tree. vec![ Node::Block { name: (&b"core"[..], &b"foo"[..]), attributes: Some(&b"{\"bar\": true}"[..]), children: vec![] } ] ) ); assert_eq!(root(input), output);</code></pre> The root</code> function and the AST will be the items we are going to use and manipulate in the bindings. The internal items of the parser will stay private. Bindings#</a> </h2> </figure> From now, our goal is to expose the root</code> function and the Node</code> enum in different platforms or environments. Ready? 3… 2… 1… lift-off! How Automattic (WordPress.com & co.) partly moved away from PHPUnit to atoum? 2018-02-26T00:00:00+00:00 Hello fellow developers and testers, Few months ago at Automattic</a>, my team and I started a new project: Having better tests for the payment system. The payment system is used by all the services at Automattic, i.e. WordPress</a>, VaultPress</a>, Jetpack</a>, Akismet</a>, PollDaddy</a> etc. It's a big challenge! Cherry on the cake: Our experiment could define the future of the testing practices for the entire company. No pressure. This post is a summary about what have been accomplished so far, the achievements, the failures, and the future, focused around manual tests. As the title of this post suggests, we are going to talk about PHPUnit</a> and atoum</a>, which are two PHP test frameworks. This is not a PHPUnit vs. atoum fight. These are observations made for our software, in our context, with our requirements, and our expectations. I think the discussion can be useful for many projects outside Automattic. I would like to apologize in advance if some parts sound too abstract, I hope you understand I can't reveal any details about the payment system for obvious reasons. Where we were, and where to go#</a> </h2> For historical reasons, WordPress, VaultPress, Jetpack & siblings use PHPUnit</a> for server-side manual tests. There are unit, integration, and system manual tests. There are also end-to-end tests or benchmarks, but we are not interested in them now. When those products were built, PHPUnit was the main test framework in town. Since then, the test landscape has considerably changed in PHP. New competitors, like atoum</a> or Behat</a>, have a good position in the game. Those tests exist for many years. Some of them grew organically. PHPUnit does not require any form of structure, which is —despite being questionable according to me— a reason for its success. It is a requirement that the code does not need to be well-designed to be tested, but too much freedom on the test side comes with a cost in the long term if there is not enough attention. Our situation is the following. The code is complex for justified reasons, and the testability is sometimes lessened. Testing across many services is indubitably difficult. Some parts of the code are really old, mixed with others that are new, shiny, and well-done. In this context, it is really difficult to change something, especially moving to another test framework. The amount of work it represents is colossal. Any new test framework does not worth the price for this huge refactoring. But maybe the new test frameworks can help us to better test our code? I'm a long term contributor of atoum</a> (top 3 contributors). And at the time of writing, I'm a core member. You have to believe me when I say that, at each step of the discussions or the processes, I have been neutral, arguing in favor or against atoum. The idea to switch to atoum partly came from me actually, but my knowledge about atoum is definitively a plus. I am in a good position to know the pros and the cons of the tool, and I'm perfectly aware of how it could solve issues we have. So after many debates and discussions, we decided to try to move to atoum. A survey and a meeting were scheduled 2 months later to decide whether we should continue or not. Spoiler: We will partly continue with it. Our needs and requirements#</a> </h2> Our code is difficult to test. In other words, the testability is low for some parts of the code. atoum has features to help increase the testability. I will try to summarize those features in the following short sections. atoum/phpunit-extension</code>#</a> </h3> As I said, it's not possible to rewrite/migrate all the existing tests. This is a colossal effort with a non-neglieable cost. Then, enter atoum/phpunit-extension</code></a>. As far as I know, atoum is the only PHP framework that is able to run tests that have been written for another framework. The atoum/phpunit-extension</code> does exactly that. It runs tests written with the PHPUnit API with the atoum engines. This is fabulous! PHPUnit is not required at all. With this extension, we have been able to run our “legacy” (aka PHPUnit) tests with atoum. The following scenarios can be fulfilled: Existing test suites written with the PHPUnit API can be run seamlessly by atoum, no need to rewrite them,</li> Of course, new test suites are written with the atoum API,</li> In case of a test suite migration from PHPUnit to atoum, there are two solutions: Rewrite the test suite entirely from scratch by logically using the atoum API, or</li> Only change the parent class from PHPUnit\Framework\TestCase</code> to atoum\phpunit\test</code>, and suddenly it is possible to use both API at the same time (and thus migrate one test case after the other for instance).</li> </ol> </li> </ul> This is a very valuable tool for an adventure like ours. atoum/phpunit-extension</code> is not perfect though. Some PHPUnit APIs are missing. And while the test verdict is strictly the same, error messages can be different, some PHPUnit extensions may not work properly etc. Fortunately, our usage of PHPUnit is pretty raw: No extensions except home-made ones, few hacks… Everything went well. We also have been able to contribute easily to the extension. Mock engines (plural)#</a> </h3> atoum comes with 3 mock engines</a>: Class-like mock engine for classes and interfaces,</li> Function mock engine,</li> Constant mock engine.</li> </ul> Being able to mock global functions or global constants is an important feature for us. It suddenly increases the testability of our code! The following example is fictional, but it's a good illustration. WordPress is full of global functions, but it is possible to mock them with atoum like this: <?php public function test_foo() { $this->function->get_userdata = (object) [ 'user_login' => …, 'user_pass' => …, … ]; }</code></pre> In one line of code, it was possible to mock the get_userdata</code></a> function. Runner engines#</a> </h3> Being able to isolate test execution is a necessity to avoid flakey tests, and to increase the trust we put in the test verdicts. atoum comes with de facto 3 runner engines: Inline, one test case after another in the same process,</li> Isolate, one test case after another but each time in a new process (full isolation),</li> Concurrent, like isolate but tests run concurrently (“at the same time”).</li> </ul> I'm not saying PHPUnit doesn't have those features. It is possible to run tests in a different process each time —with the isolate engine—, but test execution time blows up, and the isolation is not strict. We don't use it. The concurrent runner engine in atoum tends to reduce the execution time to be close to the inline engine, while still ensuring a strict isolation. Fun fact: By using atoum and the atoum/phpunit-extension</code>, we are able to run PHPUnit tests concurrently with a strict isolation! Code coverage reports#</a> </h3> At the time of writing, PHPUnit is not able to generate code coverage reports containing the Branch- or Path Coverage Criteria data. atoum supports them natively with the atoum/reports-extension</code></a> (including nice graphs, see the demonstration</a>). And we need those data. The difficulties#</a> </h2> On paper, most of the pain points sound addressable. It was time to experiment. Integration to the Continuous Integration server#</a> </h3> Our CI does not natively support standard test execution report formats. Thus we had to create the atoum/teamcity-extension</code></a>. Learn more</a> by reading a blog post I wrote recently. The TeamCity support is native inside PHPUnit (see the --log-teamcity</code> option</a>). Bootstrap test environments#</a> </h3> Our bootstrap files are… challenging. It's expected though. Setting up a functional test environment for a software like WordPress.com is not a task one can accomplish in 2 minutes. Fortunately, we have been able to re-use most of the PHPUnit parts. Today, our unit tests run in complete isolation and concurrently. Our integration tests, and system tests run in complete isolation but not concurrently, due to MySQL limitations. We have solutions, but time needs to be invested. Generally, even if it works now, it took time to re-organize the bootstrap so that some parts can be shared between the test runners (because we didn't switch the whole company to atoum yet, it was an experiment). Documentation and help#</a> </h3> Here is an interesting paradox. The majority of the team recognized that atoum's documentation is better than PHPUnit's, even if some parts must be rewritten or reworked. But developers already know PHPUnit, so they don't look at the documentation. If they have to, they will instead find their answers on StackOverflow, or by talking to someone else in the company, but not by checking the official documentation. atoum does not have many StackOverflow threads, and few people are atoum users within the company. What we have also observed is that when people create a new test, it's a copy-paste from an existing one. Let's admit this is a common and natural practice. When a difficulty is met, it's legit to look at somewhere else in the test repository to check if a similar situation has been resolved. In our context, that information lacked a little bit. We tried to write more and more tests, but not fast enough. It should not be an issue if you have time to try, but in our context, we unfortunately didn't have this time. The team faced many challenges in the same period, and the tests we are building are not simple _Hello, World!_s as you might think, so it increases the effort. To be honest, this was not the biggest difficulty, but still, it is important to notice. Concurrent integration test executions#</a> </h3> Due to some MySQL limitations combined with the complexity of our code, we are not able to run integration (and system) tests concurrently yet. Therefore it takes time to run them, probably too much in our development environments. Even if atoum has friendly options to reduce the debug loop (e.g. see the --loop</code> option</a>), the execution is still slow. The problem can be solved but it requires time, and deep modifications of our code. Note that with our PHPUnit tests, no isolation is used. This is wrong. And thus we have a smaller trust in the test verdict than with atoum. Almost everyone in the team prefers to have slow test execution but isolation, rather than fast test execution but no confidence in the test verdict. So that's partly a difficulty. It's a mix of a positive feature and a needle in the foot, and a needle we can live with. atoum is not responsible of this latency: The state of our code is. The results#</a> </h2> First, let's start by the positive impacts: In 2 months, we have observed that the testability of our code has been increased by using atoum,</li> We have been able to find bugs in our code that were not detected by PHPUnit, mostly because atoum checks the type of the data,</li> We have been able to migrate “legacy tests” (aka PHPUnit tests) to atoum by just moving the files from one directory to another: What a smooth migration!</li> The trust we put in our test verdict has increased thanks to a strict test execution isolation.</li> </ul> Now, the negative impacts: Even if the testability has been increased, it's not enough. Right now, we are looking at refactoring our code. Introducing atoum right now was probably too early. Let's refactor first, then use a better test toolchain later when things will be cleaner,</li> Moving the whole company at once is hard. There are thousands of manual tests. The atoum/phpunit-extension</code> is not magical. We have to come with more solid results, stuff to blow minds. It is necessary to set the institutional inertia in motion. For instance, not being able to run integration and system tests concurrently slows down the builds on the CI; it increases the trust we put in the test verdict, but this latency is not acceptable at the company scale,</li> All the issues we faced can be addressed, but it needs time. The experiment time frame was 2 months. We need 1 or 2 other months to solve the majority of the remaining issues. Note that I was kind of in-charge of this project, but not full time.</li> </ul> We stop using atoum for manual tests. It's likely to be a pause though. The experiment has shown we need to refactor and clean our code, then there will be a good chance for atoum to come back. The experiment has also shown how to increase the testability of our code: Not everything can be addressed by using another test framework even if it largely participates. We can focus on those points specifically, because we know where they are now. Finally, I reckon it has participated in moving the test infrastructure inside Automattic by showing that something else exists, and that we can go further. I said we stopped using atoum “for manual tests”. Yes. Because we also have automatically generated tests. The experiment was not only about switching to atoum. Many other aspects of the experiment are still running! For instance, Kitab</a> is used for our code documentation. Kitab is able to (i) render the documentation, and (ii) test the examples written inside the documentation. That way the documentation is ensured to be always up-to-date and working. Kitab generates tests for- and executes tests with atoum. It was easy to set up: We just had to use the existing test bootstraps designed for atoum. We also have another tool to compile HTTP API Blueprint specifications into executable tests</a>. So far, everyone is happy with those tools, no need to go back, everything is automat(t)ic. Other tools are likely to be introduced in the future to automatically generate tests. I want to detail this particular topic in another blog post. Conclusion#</a> </h2> Moving to another test framework is a huge decision with many factors. The fact atoum has atoum/phpunit-extension</code> is a time saver. Nonetheless a new test framework does not mean it will fix all the testability issues of the code. The benefits of the new test framework must largely overtake the costs. In our current context, it was not the case. atoum solves issues that are not our priorities. So yes, atoum can help us to solve important issues, but since these issues are not priorities, then the move to atoum was too early. During the project, we gained new automatic test tools, like Kitab</a>. The experiment is not a failure. Will we retry atoum? It's very likely. When? I hope in a year. One conference per day, for one year (2017) 2018-01-25T00:00:00+00:00 My self-assigned challenge for 2017 was to watch at least one conference per day, for one year. That's the first time I try this challenge. Let's dive in for a recap. 267 conferences#</a> </h2> In some way, I failed the challenge because I've been able to watch only 267 conferences. With an average of 34 minutes per conference, I've watched 9078 minutes, or 151 hours of freely available conferences online. Why did I fail to watch 365 of them? Because my first kid was 1.5 years in January 2017, a new little lady came in December 2017, I got a new job</a>, I travelled for my job</a>, I gave talks</a>, I maintain important open source projects requiring lot of time, I'm building my own self-sufficient ecological house, the vegetable garden requires many hours, I watch other videos, and because I'm lazy sometimes. Most of the time, I was able to watch 2 or 3 conferences in a row. Where to find the resources?#</a> </h2> All these conferences are freely available online, on YouTube, or on Vimeo, for most of them. The channel I mostly watch are the following: Rust</a>,</li> Confreaks</a>,</li> Devoxx</a>,</li> elm-conf</a>,</li> BoostCon</a>,</li> JSConf</a>,</li> LLVM</a>,</li> CppCon</a>,</li> Strange Loop</a>.</li> </ul> It's very Computer Science centric as you might have noticed, and it targets Rust, C++, Elm, LLVM, or Web technologies (JS, CSS…), but not only, you can find Haskell or Clojure sometimes. My best-of list#</a> </h2> In March 2017, more and more people were questionning me, and asked for sharing. I then decided to start a playlist of my “best-of” conferences</a>. I've added 78 conferences in 2017, and 3 new conferences have been added since then. </iframe> My best-of talks playlist. </figcaption> </figure> Thoughts and conclusion#</a> </h2> The challenge was sometimes easy and relaxing, or it was very hard to understand everything especially at 2am after a long day (looking at you CppCon). But it has been a very enjoyable way to learn a lot in a very short period of time. Many speakers are talented, and listening to them is a real pleasure. Some others are just… let's say unprepared, and it's good to stop and jump onto another talk. It's also a good way to get inspired by technologies you don't necessarily know (for instance, I'm not a big fan of Clojure, but some projects are really inspiring, like Proto REPL</a>). Sometimes I tweeted</a> about the talk I watched, and it was quite appreciated too. I reckon because it's a fun and an easy way to learn, especially with the help of video platforms like Youtube. Am I going to continue this challenge in 2018? Yes! But maybe not at this frequency. It's now part of my routine to watch conferences many times per week. I like it. I don't want to stop. As a closing note, I would like to thank every speakers, and more importantly, every conference organizer. You are doing an amazing job: From the program, to the event, to the final sharing on Internet with everyone. Most of you are volunteers. I know the work it represents. You are producing extremely valuable resources. Thank you! Random thoughts about `::class` in PHP 2018-01-24T00:00:00+00:00 The special ::class</code> constant allows for fully qualified class name resolution at compile, this is useful for namespaced classes. </blockquote> I'm quoting the PHP manual</a>. But things can be funny sometimes. Let's go through some examples. <?php use A\B as C; $_ = C::class; ```  resolves to `A\B`, which is perfect 🙂 </code></pre></li> <?php class C { public function f() { $_ = self::class; } }</code></pre> resolves to C</code>, which is perfect 😀 </li> <?php class C {} class D extends C { public function f() { $_ = parent::class; } } ```  resolves to `C`, which is perfect 😄 </code></pre></li> <?php class C { public static function f() { $_ = static::class; } } class D extends C {} D::f();</code></pre> resolves to D</code>, which is perfect 😍 </li> <?php 'foo'::class ```  resolves to `'foo'`, which is… huh? 🤨 </code></pre></li> <?php "foo"::class</code></pre> resolves to 'foo'</code>, which is… expected somehow 😕 </li> <?php $a = 'oo'; "f{$a}"::class ```  generates a parse error 🙃 </code></pre></li> <?php PHP_VERSION::class</code></pre> resolves to 'PHP_VERSION'</code>, which is… strange: It resolves to the fully qualified name of the constant, not the class 🤐 </li> </ul> ::class</code> is very useful to get rid of of the get_class</code> or the get_called_class</code> functions, or even the get_class($this)</code> trick. This is something truly useful in PHP where entities are referenced as strings, not as symbols. ::class</code> on constants makes sense, but the name is no longer relevant. And finally, ::class</code> on single quote strings is absolutely useless; on double quotes strings it is a source of error because the value can be dynamic (and remember, ::class</code> is resolved at compile time, not at run time). Automattic, Grand Meetup 2017 2017-11-26T00:00:00+00:00 Awesome company, awesome teams, awesome people. Thanks everyone for this moment! All the people! </figcaption> </figure> atoum supports TeamCity 2017-11-06T00:00:00+00:00 atoum</a> is a popular PHP test framework. TeamCity</a> is a Continuous Integration and Continuous Delivery software developed by Jetbrains. Despites atoum supports many industry standards</a> to report test execution verdicts, TeamCity uses its own non-standard report</a>, and thus atoum is not compatible with TeamCity… until now. The atoum/teamcity-extension</code> provides TeamCity support inside atoum. When executing tests, the reported verdicts are understandable by TeamCity, and activate all its UI features. Install#</a> </h2> If you have Composer, just run: $ composer require atoum/teamcity-extension '~1.0'</code></pre> From this point, you need to enable the extension in your .atoum.php</code> configuration file. The following example forces to enable the extension for every test execution: <?php $extension = new atoum\teamcity\extension($script); $extension->addToRunner($runner);</code></pre> The following example enables the extension only within a TeamCity environment: <?php $extension = new atoum\teamcity\extension($script); $extension->addToRunnerWithinTeamCityEnvironment($runner);</code></pre> This latter installation is recommended. That's it 🙂. Glance#</a> </h2> The default CLI report looks like this: The default CLI report is the default one from atoum. </figcaption> </figure> The TeamCity report looks like this in your terminal (note the TEAMCITY_VERSION</code> variable as a way to emulate a TeamCity environment): The TeamCity report is text-based, but it is aimed at being consumed by a formatter to produce HTML. </figcaption> </figure> Which is less easy to read. However, when it comes into TeamCity UI, we will have the following result: The final rendering, at an HTML document inside TeamCity itself. </figcaption> </figure> We are using it at Automattic</a>. Hope it is useful for someone else! If you find any bugs, or would like any other features, please use Github at the following repository: https://github.com/Hywan/atoum-teamcity-extension/</a>. Export functions in PHP à la Javascript 2017-10-30T00:00:00+00:00 Warning: This post is totally useless. It is the result of a fun private company thread. Export functions in Javascript#</a> </h2> In Javascript, a file can export functions like this: export function times2(x) { return x * 2; }</code></pre> And then we can import this function in another file like this: import {times2} from 'foo'; console.log(times2(21)); // 42</code></pre> Is it possible with PHP? Export functions in PHP#</a> </h2> Every entity is public in PHP: Constant, function, class, interface, or trait. They can live in a namespace. So exporting functions in PHP is absolutely useless, but just for the fun, let's keep going. A PHP file can return an integer, a real, an array, an anonymous function, anything. Let's try this: <?php return function (int $x): int { return $x * 2; };</code></pre> And then in another file: <?php $times2 = require 'foo.php'; var_dump($times2(21)); // int(42)</code></pre> Great, it works. What if our file returns more than one function? Let's use an array (which has most hashmap properties): <?php return [ 'times2' => function (int $x): int { return $x * 2; }, 'answer' => function (): int { return 42; } ];</code></pre> To choose what to import, let's use the list</code> intrinsic</a>. It has several forms: With or without key matching, long (list(…)</code>) and short syntax ([…]</code>). Because we are modern, we will use the short syntax with key matching to selectively import functions: <?php ['times2' => $mul] = require 'foo.php'; var_dump($mul(21)); // int(42)</code></pre> Notice that times2</code> has been aliased to $mul</code>. What a feature! Is it useful? Absolutely not. Is it fun? For me it is. Finite-State Machine as a Type System illustrated with a store product 2017-08-09T00:00:00+00:00 Hello fellow coders! In this article, I would like to talk about how to implement a Finite-State Machine</a> (FSM) with the PHP type system. The example is a store product (in an e-commerce solution for instance), something we are likely to meet once in our lifetime. Our goal is to simply avoid impossible states and transitions. I am in deep love with Type theory</a>, however I will try to keep the formulas away from this article to focus on the code. Moreover, you might be aware that the PHP runtime type system is somewhat very permissive and “poor” (this is not a formal definition), hopefully some tricks can help us to express nice constraints. The Product FSM#</a> </h2> A product in a store might have the following states: Active: Can be purchased,</li> Inactive: Has been cancelled or discontinued (a discontinued product can no longer be purchased),</li> Purchased and renewable,</li> Purchased and not renewable,</li> Purchased and cancellable.</li> </ul> The transitions between these states can be viewed as a Finite-State Machine</a> (FSM). We read this graph as: A product is in the state A</code>. If the purchase</code> action is called, then it transitions to the state B</code>. If the once-off purchase</code> action is called, then it transitions to the state C</code>. From the state B</code>, if the renew</code> action is called, it remains in the same state. If the cancel</code> action is called, it transitions to the D</code> state. Same for the C</code> to D</code> states. </figcaption> </figure> Our goal is to respect this FSM. Invalid actions must be impossible to do. Finite-State Machine as a Type System#</a> </h2> Having a FSM is a good thing to define the states and the transitions between them: It is formal and clear. However, it is tested at runtime, not at compile-time, i.e. if</code> statements are required to test if the state of a product can transition into another state, or else throw an exception, and this is decided at runtime. Note that PHP does not really have a compile-time because it is an online compiler (learn more by reading Tagua VM, a safe PHP virtual machine</a>, at slide 29). Our goal is to prevent illegal/invalid states at parse-/compile-time so that the PHP virtual machine, IDE or static analysis tools can prove the state of a product without executing PHP code. Why is this important? Imagine that we decide to change a product to be once-off purchasable instead of purchasable, then we can no longer renew it. We replace an interface on this product, and boom, the IDE tells us that the code is broken in x places. It detects impossible scenarios ahead of code execution. No more talking. Here is the code. The mighty product#</a> </h3> <?php /** * A product. */ interface Product {}</code></pre> A product is a class implementing the Product</code> interface. It allows to type a generic product, with no regards about its state. Active and inactive#</a> </h3> <?php /** * A product that is active. */ interface Active extends Product { public function getProduct(): self; } /** * A product that has been cancelled, or not in stock. */ interface Inactive extends Product { public function getProduct(): self; }</code></pre> The Active</code> and Inactive</code> interfaces are useful to create constraints such as: A product can be purchased only if it is active, and</li> A product is inactive if and only if it has been cancelled,</li> To finally conclude that an inactive product can no longer be purchased, nor renewed, nor cancelled.</li> </ul> Basically, it defines the axiom (initial state) and the final states of our FSM. The getProduct(): self</code> trick will make sense later. It helps to express the following constraint: “A valid product cannot be invalid, and vice-versa”, i.e. both interfaces cannot be implemented by the same value. Purchase, renew, and cancel#</a> </h3> <?php /** * A product that can be purchased. */ interface Purchasable extends Active { public function purchase(): Renewable; }</code></pre> Only an active product can be purchased. The action is purchase</code> and it generates a product that is renewable. purchase</code> transitions from the state A</code> to B</code> (regarding the graph above). <?php /** * A product that can be cancelled. */ interface Cancellable extends Active { public function cancel(): Inactive; }</code></pre> Only an active product can be cancelled. The action is cancel</code> and it generates an inactive product, so it transitions from the state B</code> to D</code>. <?php /** * A product that can be renewed. */ interface Renewable extends Cancellable { public function renew(): self; }</code></pre> A renewable product is also cancellable. The action is renew</code> and this is a reflexive transition from the state B</code> to B</code>. <?php /** * A product that can be once-off purchased, i.e. it can be purchased but not * renewed. */ interface PurchasableOnce extends Active { public function purchase(): Cancellable; }</code></pre> Finally, a once-off purchasable product has one action: purchase</code> that produces a Cancellable</code> product, and it transitions from the state A</code> to C</code>. Take a breath#</a> </h3> </figure> So far we have defined interfaces, but the FSM is not implemented yet. Interfaces only define constraints in our type system. An interface provides a constraint but also defines type capabilities: What operations can be performed on a value implementing a particular interface. SecretProduct#</a> </h3> Let's consider the SecretProduct</code> as a new super secret product that will revolutionise our store: <?php /** * The `SecretProduct` class is: * * * A product, * * Active, * * Purchasable. * * Note that in this implementation, the `SecretProduct` instance is mutable: Every * action happens on the same `SecretProduct` instance. It makes sense because * having 2 instances of the same product with different states might be error-prone * in most scenarios. */ class SecretProduct implements Active, Purchasable { public function getProduct(): Active { return $this; } /** * Purchase the product will return an active product that is renewable, * and also cancellable. */ public function purchase(): Renewable { return new class ($this->getProduct()) implements Renewable { protected $product; public function __construct(SecretProduct $product) { $this->product = $product; // Do the purchase. } public function getProduct(): Active { return $this->product; } public function renew(): Renewable { // Do the renew. return $this; } public function cancel(): Inactive { return new class ($this->getProduct()) implements Inactive { protected $product; public function __construct(SecretProduct $product) { $this->product = $product; // Do the cancel. } public function getProduct(): Inactive { return $this->product; } }; } }; } }</code></pre> The SecretProduct</code> is a product that is active and purchasable. PHP verifies that the Active::getProduct</code> method is implemented, and that the Purchasable::purchase</code> method is implemented too. When this latter is called, it returns an object implementing the Renewable</code> interface (which is also a cancellable active product). The object in this context is an instance of an anonymous class implementing the Renewable</code> interface. So the Active::getProduct</code>, Renewable::renew</code>, and Cancellable::cancel</code> methods must be implemented. Having an anonymous class is not required at all, this is just simpler for the example. A named class may even be better from the testing point of view. Note that: The real purchase action is performed in the constructor of the anonymous class: This is not a hard rule, this is just convenient; it can be done in the method before returning the new instance,</li> The real renew action is performed in the renew</code> method before returning $this</code> ,</li> And the real cancel action is performed in… we have to dig a little bit more (the principle is exactly the same though): The Cancellable::cancel</code> method must return an object implementing the Inactive</code> interface.</li> It generates an instance of an anonymous class implementing the Inactive</code> interface, and the real cancel action is done in the constructor.</li> </ul> </li> </ul> Assert possible and impossible actions#</a> </h3> Let's try some valid and invalid actions. Those followings are possible actions: <?php assert((new SecretProduct())->purchase() instanceof Product); assert((new SecretProduct())->purchase()->renew() instanceof Product); assert((new SecretProduct())->purchase()->cancel() instanceof Product); assert((new SecretProduct())->purchase()->renew()->renew()->cancel() instanceof Product);</code></pre> It is possible to purchase a product, then renew it zero or many times, and finally to cancel it. It matches the FSM! Those followings are impossible actions: <?php (new SecretProduct())->renew(); (new SecretProduct())->cancel(); (new SecretProduct())->purchase()->cancel()->purchase(); (new SecretProduct())->purchase()->cancel()->renew(); (new SecretProduct())->purchase()->purchase(); (new SecretProduct())->purchase()->cancel()->cancel();</code></pre> It is impossible: To renew or to cancel a product that has not been purchased,</li> To purchase or renew a product that has been cancelled,</li> To purchase a product more than once,</li> To cancel a product more than once.</li> </ul> Those followings are impossible implementations: <?php class SecretProduct implements Active, Purchasable, PurchasableOnce {}</code></pre> A product cannot be purchasable and once-off purchasable at the same time, because Purchasable::purchase</code> is not compatible with PurchasableOnce::purchase</code>. <?php class SecretProduct implements Inactive, Cancellable {}</code></pre> An inactive product cannot be purchased nor renewed nor cancelled because Active::getProduct</code> and Inactive::getProduct</code> are not compatible. Wow, that's great garantees isn't it? PHP will raise fatal errors for impossible actions or impossible states. No warnings or notices: Fatal errors. Most of them are correctly inferred by IDE, so… follow the red crosses in your IDE. Restoring a product#</a> </h2> One major thing is missing: The state of a product is stored in the database. When loading the product, we must be able to get an instance of a product at its previous state. To avoid repeating code, we will use traits. Rebuilding the state of a product is “just” (it really is) a composition of traits. Note: In these examples, we are using anonymous classes and traits. It is possible to achieve the same behavior with final named classes. Also we are using a repository, which is convenient for this article, but not necessarily the best solution. Repository#</a> </h3> The following ProductRepository\load</code> function is just here to give you an idea of how it works. <?php namespace ProductRepository; function load(int $id, string $state): Product { // Load the product from the database with `$id`. // // The states can be `Renewable`, `Cancellable`, or `Inactive` (check // the FSM to double-check). Products that have not been purchased // are not in the database. // Fake minimal active product. $product = new class implements Active { public function getProduct(): Active { return $this; } }; switch ($state) { // State B. case Renewable::class: return new class ($product) implements Renewable { use ActiveProduct; use RenewableProduct; use CancellableProduct; }; // State C. case Cancellable::class: return new class ($product) implements Cancellable { use ActiveProduct; use CancellableProduct; }; // State D. case Inactive::class: return new class ($product) implements Inactive { use InactiveProduct; }; // Invalid state. default: throw new RuntimeException('Invalid product state.'); } }</code></pre>Traits#</a> </h3> The code must look familiar because this is just a split from the SecretProduct</code> implementation. <?php trait ActiveProduct { protected $product; public function __construct(Product $product) { $this->product = $product; } public function getProduct(): Active { return $this->product; } } trait RenewableProduct { public function renew(): Renewable { // Do the renew. return $this; } } trait CancellableProduct { public function cancel(): Inactive { return new class ($this->getProduct()) implements Inactive { protected $product; public function __construct(Product $product) { $this->product = $product; // Do the cancel. } public function getProduct(): Inactive { return $this->product; } }; } } trait InactiveProduct { protected $product; public function __construct(Product $product) { $this->product = $product; } public function getProduct(): Inactive { return $this->product; } }</code></pre>Assert possible and impossible actions#</a> </h3> The possible actions are: <?php $product = ProductRepository\load(42, Renewable::class); assert($product instanceof Product); assert($product->renew() instanceof Product); assert($product->cancel() instanceof Product);</code></pre> Product 42 is assumed to be in the state B</code> (Renewable::class</code>), so we can renew and cancel it. Those followings are impossible actions: <?php $product = ProductRepository\load(42, Renewable::class); $product->purchase(); $product->cancel()->cancel();</code></pre> It is impossible to purchase the product 42 because it is in state B</code>, so it has already been purchased. It is impossible to cancel a product twice. Same garantees apply here! Conclusion#</a> </h2> It is possible to re-implement SecretProduct</code> with the traits we have defined for the ProductRepository</code>, or to use named classes. I let this as an easy wrap up exercise for the reader. The real conclusion is that we have successfully implemented the Finite-State Machine of a product with a Type System. It is impossible to have an invalid implementation that violates the constraints, such as an inactive renewable product. PHP detects it immediately at runtime. Invalid actions are also impossible, such as purchasing a product twice, or renewing a once-off purchased product. It is also detected by PHP. All violations take the form of PHP fatal errors. The product repository is an example of how to restore a product at a particular state, with the help of the defined interfaces, and new small and simple traits. One more thing#</a> </h2> It is possible to integrate product categories in this type system (like bundles). It is more complex, but possible. I would highly recommend these following readings: What to know before debating type systems</a> to have an overview of different systems,</li> Rust's Type System is Turing-Complete</a> to see how powerful a type system can be,</li> Fear Not the Machine of State!</a> to see how to integrate an FSM into an object without using a type system.</li> </ul> I would like to particularly emphasize a paragraph from the first article: So what is a type? The only true definition is this: a type is a label used by a type system to prove some property of the program's behavior. If the type checker can assign types to the whole program, then it succeeds in its proof; otherwise it fails and points out why it failed. </blockquote> Seeing types as labels is a very smart way of approaching them. I would like to thanks Marco Pivetta</a> for the reviews! Tagua VM, a safe PHP virtual machine 2017-06-19T00:00:00+00:00 </iframe> PHPTour Nantes 2017 (in French): PHP est un langage extrêment populaire. En 2015, PHP était utilisé par plus de 80% de tous les sites Web. Cependant, 500 vulnérabilités sévères sont répertoriées. Bien qu'inhérent à tous langages populaires, cela reste très dangereux. L'objectif du projet Tagua VM est de fournir une VM PHP qui garantie un haut niveau de sûreté et de qualité en supprimant des larges classes de vulnérabilités, grâce à des outils appropriés comme Rust et LLVM. Rust est un langage remarquable qui apporte des garanties fortes à propos de la sûreté de la mémoire. C'est aussi un langage très rapide qui rivalise avec C. LLVM est une infrastructure de compilateur célèbre qui apporte de la modernité, des algorithmes à la pointe, des performances, une suite d'outils pour développeur etc. Ce projet va résoudre trois problèmes en une fois : Fournir un niveau haut niveau de sûreté et de qualité en supprimant des larges classes de vulnérabilité, et ainsi éviter des coûts de bugs dramatiques ;</li> Fournir de la modernité, une nouvelle expérience développeur et des algorithmes à la pointe de la recherche, donc des performances ;</li> Fournir un ensemble de bibliothèques qui vont composer la VM et qui pourront être réutiliser en dehors du projet (comme le parseur, les analyseurs, les extensions etc.).</li> </ol> Durant cette conférence, nous présenterons les objectifs de ce projet, ainsi que son avancement. Nous expliquerons pourquoi il est crucial et pourquoi il reçoit le soutient d'une communauté grandissante et de développeurs notables (avec un rôle important dans le développement de PHP). </blockquote> View slides</a>. Faster find algorithms in nom 2017-05-23T00:00:00+00:00 Tagua VM</a> is an experimental PHP virtual machine written in Rust and LLVM. It is composed as a set of libraries. One of them that keeps me busy these days is tagua-parser</code></a>. It contains the lexical and syntactic analysers for the PHP language, in addition to the AST (Abstract Syntax Tree). If you would like to know more about this project, you can see this conference I gave at the PHPTour last week: Tagua VM, a safe PHP virtual machine</a>. The library tagua-parser</code> is built with parser combinators. Instead of having a classical grammar, compiled to a parser, we write pure functions acting as small parsers. We then combine them together. This post does not explain why this is a sane approach in our context, but keep in mind this is much easier to test, to maintain, and to optimise. Because this project is complex enought, we are delegating the parser combinator implementation to nom</a>. nom is a parser combinators library written in Rust. Its goal is to provide tools to build safe parsers without compromising the speed or memory consumption. To that end, it uses extensively Rust's strong typing, zero copy parsing, push streaming, pull streaming, and provides macros and traits to abstract most of the error prone plumbing. </blockquote> Recently, I have been working on optimisations in the FindToken</code> and FindSubstring</code> traits from nom itself. These traits provide methods to find a token (i.e. a lexeme), and to find a substring, crazy naming. However, this is not totally valid: FindToken</code> expects to find a single item (if implemented for u8</code>, it will look for a u8</code> in a &[u8]</code>), and FindSubstring</code> really is about finding a substring, so a token of any length. It appeared that these methods can be optimised in some cases. Both default implementations are using Rust iterators: Regular iterator for FindToken</code>, and window iterator</a> for FindSubstring</code>, i.e. an iterator over overlapping subslices of a given length. We have benchmarked big PHP comments, which are analysed by parsers actively using these two trait implementations. Here are the result, before and after our optimisations: test …::bench_span ... bench: 73,433 ns/iter (+/- 3,869) test …::bench_span ... bench: 15,986 ns/iter (+/- 3,068)</code></pre> A boost of 78%! Nice! The pull request has been merged</a> today, thank you Geoffroy Couprie! The new algorithms heavily rely on the memchr</code> crate</a>. So all the credits should really go to Andrew Gallant! This crate provides a safe interface libc</code>'s memchr</code> and memrchr</code>. It also provides fallback implementations when either function is unavailable. The new algorithms are only implemented for &[u8]</code> though. Fortunately, the implementation for &str</code> fallbacks to the former. This is small contribution, but it brings a very nice boost. Hope it will benefit to other projects! I am also blowing the dust off of Algorithms on Strings</a>, by M. Crochemore, C. Hancart, and T. Lecroq. I am pretty sure it should be useful for nom and tagua-parser</code>. If you haven't read this book yet, I can only encourage you to do so! Welcome to Chaos 2017-04-24T00:00:00+00:00 Recently, I joined Automattic</a>. This is a world-wide distributed company. The first three weeks you incarn a Happiness Engineer. This is part of the Happiness Rotation duty. This article explains why I loved it, and why I reckon you should do it too. Happiness Engineer, really?#</a> </h2> Does it sound mad as a Cheshire cat? Pretentious maybe? Actually, it's not at all. As a Happiness Engineer, I had to make the support. This is part of the Happiness Rotation: Once a year, almost everyone swaps its position to help our users. I will go back on this later. My role was to make our users happy. To achieve that, I had to: Meet our users, understand who they are, what they want to achieve,</li> Listen to and understand their issues,</li> Find a way to fix the issues.</li> </ul> Meet the users#</a> </h3> I need motivations in my job. Learning who our users are, and what they want to achieve, is a great motivation. After these three weeks, I know what my contributions will serve. It gives a meaning to each contribution, to each day I wake up. Especially in a distributed company on Internet, our users are world-wide, they speak almost all the languages on Earth, they are present on all continents. Their needs vary a lot, they use our software in ways I was not able to foresee. Listen to, understand, and fix their issues#</a> </h3> When you are chatting with a “support guy”, you cannot imagine this is a real engineer. This is not a random person filling a pre-defined vague form somewhere where it is cheap to hire her. You will chat with someone very competent. Someone that has no superior. Someone that has all the tools to make you happy. Personally, when I started, it was the first time I was using WordPress. I was more novice than the user I was talking to. So how to fix it on my end? I had to: Ask help to the right persons,</li> Therefore, meet Automatticians (people working with Automattic),</li> Discover all the interactions between them,</li> Understand the structure of the company,</li> How to ask help, how to formulate my questions, how to reformulate the issues of the users…</li> Discover all the internal tools,</li> Therefore, learn how the software work internally and together,</li> Discover the giant internal and public documentations,</li> When needed, create bug reports or feature requests to the appropriated teams,</li> Learn the culture of the company.</li> </ul> This is why it is called Welcome to Chaos. Yes, you have to learn a lot in three weeks, but it is extremely educative. This is like a speed training. Happiness#</a> </h3> I can ensure that when a user is grateful after you fixed its issue, the term Happiness Engineer makes a lot of sense. Automattic provides a lot of freedom to their Happiness Engineers to make people really happy, both in term of tooling or financial. This is the first time I see a company that is that much generous with its customers. Thanks buddy#</a> </h3> Of course, when embracing the chaos, you are not alone. Everyone is here to help you, and to answer your questions. After all, this is part of the Automattic's creed</a> (story of the creed</a>): I will never stop learning. I won’t just work on things that are assigned to me. I know there’s no such thing as a status quo. I will build our business sustainably through passionate and loyal customers. I will never pass up an opportunity to help out a colleague, and I’ll remember the days before I knew everything. I am more motivated by impact than money, and I know that Open Source is one of the most powerful ideas of our generation. I will communicate as much as possible, because it’s the oxygen of a distributed company. I am in a marathon, not a sprint, and no matter how far away the goal is, the only way to get there is by putting one foot in front of another every day. Given time, there is no problem that’s insurmountable. </blockquote> In addition to everyone willing to help, a buddy was assigned to me. A person that helps and teaches you every time. This is very helpful. Thank you Hannah! Happiness Rotation#</a> </h2> This experience is great. But after some time, you might forget it. So as a reminder, once a year, you incarn a Happiness Engineer again. This is part of the happiness rotation. As far as I understand, it implies almost everyone in the company. Note: Obviously, there is permanent happiness engineers. Conclusion#</a> </h2> I deeply think this approach has many advantages. Some of them are listed above. It helps to understand the company, and more importantly the users. The happiness rotation stresses the fact that users are central to Automattic, probably like any companies, but not with this care. Remember the creed: I will build our business sustainably through passionate and loyal customers. To have passionate and loyal users, you need to know them. For me, it was a great experience. It was chaotic at first, but it is worth it. Bye bye Liip, hello Automattic 2017-04-18T00:00:00+00:00 Since April 2017, I have left Liip</a> to join Automattic</a>. Bye bye Liip#</a> </h2> After almost 20 months at Liip, I am leaving. Liip was a great experience. It was my first industrial non-remote job. It was also my first job in the country I am currently living in. And I have discovered a new way of working. First industrial non-remote job#</a> </h3> Before working for Liip, I was working for fruux</a>. My situation was the following: A french citizen, living as a foreigner in Switzerland, working for a German company, with employees from Germany, Holland, and Canada. Everything happened on chat, mail, and Skype. When my son was born, I had to change my work to simplify my life. It was not the only reason, but one of them. And before fruux, I was working for INRIA</a>, a research institute in France. It was partially a remote job. Liip has several offices. I was based in Lausanne. So, yes, Liip was my first industrial non-remote job. And I liked it. Working in the train on the morning, walking in Lausanne, seeing real people, everything in my local language. Because yes, it was my first job in my native language too. Everything was simpler. And when you have your first baby, anything else that is simpler saves your life. Introducing Holacracy#</a> </h3> Giant discussions were happening to remove any form of hierarchy in Liip. Then we discovered Holacracy</a>, and we started moving to this system. This is a new governance system. If you are familiar with distributed network topologies in Computer Science, or data structures, it really looks like a Distributed Spanning Tree</a> [DahanPN09</a>]. Note: I am sure that the authors of Holacracy are not aware of DST, but, eh. So nothing new from a research point of view, but it is cool to see this algorithm coming alive in real life. And it worked: Less meetings, more self-organisation, more shared responsabilities, no more “boss” etc. This is not a tool for all companies, but I am sure that if you are reading my blog, then your company should give it a try. Open source projects#</a> </h3> Liip has been very generous with me regarding my open source engagements. I was involved in Hoa</a>, atoum</a>, and Pickle</a> when joining the company. Liip gave me a 5% budget, so roughly 1 hour per day to work on Hoa. Thank you for that! After that, I have started a new big project, called Tagua VM</a>. They gave me an additional 5% budget. So I got 2 hours per day to work on Hoa and Tagua VM. Again, thank you for that! Finally, I have started an in-house open project called The A11y Machine</a> (a11ym for short). I have written a case study for this tool on the Liip's blog: Accessibility: make your website barrier-free with a11ym!</a> The goal of a11ym is to automate the accessibility testing of any site by crawling and testing each page. A sweet report is generated, showing all errors, warnings, and notices, with all information needed by the developer to fix the issues as fast as possible. Dashboard of a11ym, showing the evolution of the accessibility of a site in time </figcaption> </figure> A typical a11ym report listing all errors, warnings, and notices for a given URL </figcaption> </figure> This project has received really good feedbacks from the accessibility community. It has been downloaded 7000 times so far, which is not bad considering the niche it targets. A new SaaS platform is being build around this software. I enjoyed working on it, and it was really tangible. Main customer, huge project#</a> </h3> Liip is a Web agency, so you have dozens of customers at the same time. However, I was in a special team for an important customer. The site is a luxury watches and jewellery e-commerce platform, located in several countries, in 10 languages, accessible from 16 domains, shared in 2 datacenters. This is not a casual site. I learned a lot about all the complexity such a site brings: Checkout rules (oh damned…), product catalogs in different formats for different countries with different references, all the business logic inherent to each country, different payment providers, crazy front end compatibilities etc. I have a hundred of crazy anecdotes to tell. This was clearly not a job for me at first glance: I am a researcher, I have an open source culture background, I am not tailored for this kind of project. But at the end of the story, I learned a lot. Really a lot. I have a better overview of the crazy things any customer can ask, or has to deal with, and the infrastructure craziness that can be set up. I learned how to make better things: How to transform a really crappy software into something understandable by everyone, how to not break a 10+ years old progam with no test etc. And it requires skills. I learned it the hard way, but I learned it. Why leaving?#</a> </h3> Because even if I learned during my time at Liip, the Web agency model was definitively not for me. I am very thankful to every Liiper, I had a great time, I love the Web, but not in an agency. My son is now 21 months old, and I need fresh air. I can take new challenges. Welcome Automattic#</a> </h2> Automattic</a> is the company behind WordPress.com</a>, WooCommerce</a>, Akismet</a>, Simplenote</a>, Cloudup</a>, Simperium</a>, Gravatar</a> and other giant services. I came to Automattic by coincidence. I was looking for a sponsor for Tagua VM, and someone pointed me out Automattic. After some researches about the company, it appears that it could be a really great place where to work. So I applied. The hiring process was 4 months long. It was exhausting because it happened at the same time than a big sprint at Liip (remember the SaaS platform for The A11y Machine?). But after 4 months, it appears I succeeded, and I am very glad of that fact! I am just starting my job at Automattic. I don't have anything strong and finite to say now, apart that everything is just awesome so far. In few weeks, I am likely to write about my start at Automattic I did, see Welcome to Chaos</a>. They have a very interesting way to get you on board. Time for a new adventure! DuckDuckGo in a Shell 2015-08-05T00:00:00+00:00 The tip#</a> </h2> When I go outside my terminal, I am kind of lost. I control everything from my terminal and I hate clicking. That's why I found a small tip today to open a search on DuckDuckGo directly from the terminal. It redirects me to my default browser in the background, which is the expected behavior. First, I create a function called duckduckgo</code>: function duckduckgo { query=`php -r 'echo urlencode($argv[1]);' "$1"` open 'https://duckduckgo.com/?q='$query }</code></pre> Note how I (avoid to) deal with quotes in $1</code>. Then, I just have to create an alias called ?</code>: alias '?'='duckduckgo'</code></pre> And here we (duckduck) go! $ ? "foo bar's baz"</code></pre> You can see the commit</a> that adds this to my “shell home framework”. Oh, and to open the default browser, I use open (1)</code></a>, like this: alias open='open -g'</code></pre> Hope it helps! sabre/katana 2015-07-13T00:00:00+00:00 Project's logo. </figcaption> </figure> What is it?#</a> </h2> sabre/katana</code> is a contact, calendar, task list and file server. What does it mean? Assuming nowadays you have multiple devices (PC, phones, tablets, TVs…). If you would like to get your address books, calendars, task lists and files synced between all these devices from everywhere, you need a server. All your devices are then considered as clients. But there is an issue with the server. Most of the time, you might choose Google</a> or maybe Apple</a>, but one may wonder: Can we trust these servers? Can we give them our private data, like all our contacts, our calendars, all our photos…? What if you are a company or an association and you have sensitive data that are really private or strategic? So, can you still trust them? Where the data are stored? Who can look at these data? More and more, there is a huge need for “personal” server. Moreover, servers like Google or Apple are often closed: You reach your data with specific clients and they are not available in all platforms. This is for strategic reasons of course. But with sabre/katana</code>, you are not limited. See the above schema: Firefox OS can talk to iOS or Android at the same time. sabre/katana</code> is this kind of server. You can install it on your machine and manage users in a minute. Each user will have a collection of address books, calendars, task lists and files. This server can talk to a loong list of devices</a>, mainly thanks to a scrupulous respect of industrial standards: macOS: OS X 10.10 (Yosemite),</li> OS X 10.9 (Mavericks),</li> OS X 10.8 (Mountain Lion),</li> OS X 10.7 (Lion),</li> OS X 10.6 (Snow Leopard),</li> OS X 10.5 (Leopard),</li> BusyCal,</li> BusyContacts,</li> Fantastical,</li> Rainlendar,</li> ReminderFox,</li> SoHo Organizer,</li> Spotlife,</li> Thunderbird ,</li> </ul> </li> Windows: eM Client,</li> Microsoft Outlook 2013,</li> Microsoft Outlook 2010,</li> Microsoft Outlook 2007,</li> Microsoft Outlook with Bynari WebDAV Collaborator,</li> Microsoft Outlook with iCal4OL,</li> Rainlendar,</li> ReminderFox,</li> Thunderbird,</li> </ul> </li> Linux: Evolution,</li> Rainlendar,</li> ReminderFox,</li> Thunderbird,</li> </ul> </li> Mobile: Android,</li> BlackBerry 10,</li> BlackBerry PlayBook,</li> Firefox OS,</li> iOS 8,</li> iOS 7,</li> iOS 6,</li> iOS 5,</li> iOS 4,</li> iOS 3,</li> Nokia N9,</li> Sailfish.</li> </ul> </li> </ul> Did you find your device in this list? Probably yes 😉. sabre/katana</code> sits in the middle of all your devices and synced all your data. Of course, it is free and open source. Go check the source</a>! List of features#</a> </h2> Here is a non-exhaustive list of features supported by sabre/katana</code>. Depending whether you are a user or a developer, the features that might interest you are radically not the same. I decided to show you a list from the user point of view. If you would like to get a list from the developer point of view, please see this exhaustive list of supported RFC</a> for more details. Contacts#</a> </h3> All usual fields are supported, like phone numbers, email addresses, URLs, birthday, ringtone, texttone, related names, postal addresses, notes, HD photos etc. Of course, groups of cards are also supported. My card inside the native Contact application of macOS. </figcaption> </figure> My card inside the native Contact application of Firefox OS. </figcaption> </figure> My photo is not in HD, I really have to update it! Cards can be encoded into several formats. The most usual format is VCF. sabre/katana</code> allows you to download the whole address book of a user as a single VCF file. You can also create, update and delete address books. Calendars#</a> </h3> A calendar is just a set of events. Each event has several properties, such as a title, a location, a date start, a date end, some notes, URLs, alarms etc. sabre/katana</code> also support recurring events (“each last Monday of the month, at 11am…”), in addition to scheduling (see bellow). My calendars inside the native Calendar application of macOS. </figcaption> </figure> My calendars inside the native Calendar application of Firefox OS. </figcaption> </figure> Few words about calendar scheduling. Let's say you are organizing an event, like New release (we always enjoy release day!). You would like to invite several people but you don't know if they could be present or not. In your event, all you have to do is to add attendees. How are they going to be notified about this event? Two situations: Either attendees are registered on your sabre/katana</code> server and they will receive an invite inside their calendar application (we call this iTIP),</li> Or they are not registered on your server and they will receive an email with the event as an attached file (we call this iMIP). All they have to do is to open this event in their calendar application.</li> </ol> Invite an attendee by email because she is not registered on your sabre/katana</code> server. </figcaption> </figure> Notice the gorgeous map embedded inside the email! Once they received the event, they can accept, decline or “don't know” (they will try to be present at) the event. Receive an invite to an event. Here: Gordon is inviting Hywan. Three choices for Hywan: </figcaption> </figure> Hywan has accepted the event. Here is what the event looks like. Hywan can see the response of each attendees. </figcaption> </figure> Gordon is even notified that Hywan has accepted the event. </figcaption> </figure> Of course, attendees will be notified too if the event has been moved, canceled, refreshed etc. Calendars can be encoded into several formats. The most usal format is ICS. sabre/katana</code> allows you to download the whole calendar of a user as a single ICS file. You can also create, update and delete calendars. Task lists#</a> </h3> A task list is exactly like a calendar (from a programmatically point of view). Instead of containg event objects, it contains todo objects. sabre/katana</code> supports group of tasks, reminder, progression etc. My task lists inside the native Reminder application of macOS. </figcaption> </figure> Just like calendars, task lists can be encoded into several formats, whose ICS. sabre/katana</code> allows you to download the whole task list of a user as a single ICS file. You can also create, update and delete task lists. Files#</a> </h3> Finally, sabre/katana</code> creates a home collection per user: A personal directory that can contain files and directories and… synced between all your devices (as usual 😄). sabre/katana</code> also creates a special directory called public/</code> which is a public directory. Every files and directories stored inside this directory are accessible to anyone that has the correct link. No listing is prompted to protect your public data. Just like contact, calendar and task list applications, you need a client application to connect to your home collection on sabre/katana</code>. Connect to a server with the Finder application of macOS. </figcaption> </figure> Then, your public directory on sabre/katana</code> will be a regular directory as every other. List of my files, right here in the Finder application of macOS. </figcaption> </figure> sabre/katana</code> is able to store any kind of files. Yes, any kinds. It's just files. However, it white-lists the kind of files that can be showed in the browser. Only images, audios, videos, texts, PDF and some vendor formats (like Microsoft Office) are considered as safe (for the server). This way, associations can share musics, videos or images, companies can share PDF or Microsoft Word documents etc. Maybe in the future sabre/katana</code> might white-list more formats. If a format is not white-listed, the file will be forced to download. How is sabre/katana</code> built?#</a> </h2> sabre/katana</code> is based on two big and solid projects: sabre/dav</code></a>,</li> Hoa</a>.</li> </ol> sabre/dav</code> is one of the most powerful CardDAV</a>, CalDAV</a> and WebDAV</a> framework in the planet. Trusted by the likes of Atmail</a>, Box</a>, fruux</a> and ownCloud</a>, it powers millions of users world-wide! It is written in PHP and is open source. Hoa is a modular, extensible and structured set of PHP libraries. Fun fact: Also open source, this project is also trusted by ownCloud</a>, in addition to Mozilla</a>, joliCode</a> etc. Recently, this project has recorded more than 600,000 downloads and the community is about to reach 1000 people. sabre/katana</code> is then a program based on sabre/dav</code> for the DAV part and Hoa for everything else, like the logic code inside the sabre/dav</code>'s plugins. The result is a ready-to-use server with a nice interface for the administration. To ensure code quality, we use atoum</a>, a popular and modern test framework for PHP. So far, sabre/dav</code> has more than 1000 assertions. Conclusion#</a> </h2> sabre/katana</code> is a server for contacts, calendars, task lists and files. Everything is synced, everytime and everywhere. It perfectly connects to a lot of devices on the market. Several features we need and use daily have been presented. This is the easiest and a secure way to host your own private data. Go download it</a>! RFCs should provide executable test suites 2015-02-27T00:00:00+00:00 Recently, I implemented xCal and xCard formats inside the sabre/dav</code> libraries. While testing the different RFCs against my implementation, several errata have been found. This article, first, quickly list them and, second, ask questions about how such errors can be present and how they can be easily revealed. If reading my dry humor about RFC errata is boring, the next sections are more interesting. The whole idea is: Why RFCs do not provide executable test suites? What is xCal and xCard?#</a> </h2> The Web is a read-only media. It is based on the HTTP protocol. However, there is the WebDAV</a> protocol, standing for Web Distributed Authoring and Versioning. This is an extension to HTTP. Et voilà ! The Web is a read and write media. WebDAV is standardized in RFC2518</a> and RFC4918</a>. Based on WebDAV, we have CalDAV</a> and CardDAV</a>, respectively for reading and writing calendars and addressbooks. They are standardized in RFC4791</a>, RFC6638</a> and RFC6352</a>. Good! But these protocols only explain how to read and write, not how to represent a real calendar or an addressbook. So let's leave protocols for formats. The iCalendar</a> format represents calendar events, like events (VEVENT</code>), tasks (VTODO</code>), journal entry (VJOURNAL</code>, very rare…), free/busy time (VFREEBUSY</code>) etc. The vCard</a> format represents cards. The formats are very similar and share a common ancestry: This is a horrible line-, colon- and semicolon-, randomly-escaped based format. For instance: BEGIN:VCALENDAR VERSION:2.0 CALSCALE:GREGORIAN PRODID:-//Example Inc.//Example Calendar//EN BEGIN:VEVENT DTSTAMP:20080205T191224Z DTSTART;VALUE=DATE:20081006 SUMMARY:Planning meeting UID:4088E990AD89CB3DBB484909 END:VEVENT END:VCALENDAR</code></pre> Horrible, yes. You were warned. These formats are standardized in several RFCs, to list some of them: RFC5545</a>, RFC2426</a> and RFC6350</a>. This format is impossible to read, even for a computer. That's why we have jCal and jCard, which are respectively another representation of iCalendar and vCard but in JSON</a>. JSON is quite popular in the Web today, especially because it eases the manipulation and exchange of data in Javascript. This is just a very simple, and —from my point of view— human readable, serialization format. jCal and jCard are respectively standardized in RFC7265</a> and RFC7095</a>. Thus, the equivalent of the previous iCalendar example in jCal is: [ "vcalendar", [ ["version", {}, "text", "2.0"], ["calscale", {}, "text", "GREGORIAN"], ["prodid", {}, "text", "-\/\/Example Inc.\/\/Example Calendar\/\/EN"] ], [ [ "vevent", [ ["dtstamp", {}, "date-time", "2008-02-05T19:12:24Z"], ["dtstart", {}, "date", "2008-10-06"], ["summary", {}, "text", "Planning meeting"], ["uid", {}, "text", "4088E990AD89CB3DBB484909"] ] ] ] ]</code></pre> Much better. But this is JSON, which is a rather loose format, so we also have xCal and xCard another representation of iCalendar and vCard but in XML</a>. They are standardized in RFC6321</a> and RFC6351</a>. The same example in xCal looks like this: <icalendar xmlns="urn:ietf:params:xml:ns:icalendar-2.0"> <vcalendar> <properties> <version> <text>2.0</text> </version> <calscale> <text>GREGORIAN</text> </calscale> <prodid> <text>-//Example Inc.//Example Calendar//EN</text> </prodid> </properties> <components> <vevent> <properties> <dtstamp> <date-time>2008-02-05T19:12:24Z</date-time> </dtstamp> <dtstart> <date>2008-10-06</date> </dtstart> <summary> <text>Planning meeting</text> </summary> <uid> <text>4088E990AD89CB3DBB484909</text> </uid> </properties> </vevent> </components> </vcalendar> </icalendar></code></pre> More semantics, more meaning, easier to read (from my point of view), namespaces… It is very easy to embed xCal and xCard inside other XML formats. Managing all these formats is an extremely laborious task. I suggest you to take a look at sabre/vobject</code></a> (see the Github repository of sabre/vobject</code></a>). This is a PHP library to manage all the weird formats. The following example shows how to read from iCalendar and write to jCal and xCal: <?php // Read iCalendar. $document = Sabre\VObject\Reader::read($icalendar); // Write jCal. echo Sabre\VObject\Writer::writeJson($document); // Write xCal. echo Sabre\VObject\Writer::writeXml($document);</code></pre> Magic when you know the complexity of these formats (in both term of parsing and validation)! List of errata#</a> </h2> Now, let's talk about all the errata I submited recently: 4241, in RFC6351</a> (xCard),</li> 4243, in RFC6351</a> (xCard),</li> 4246, in RFC6350</a> (vCard),</li> 4247, in RFC6351</a> (xCard),</li> 4245, in RFC6350</a> (vCard),</li> 4261, in RFC6350</a> (vCard).</li> </ul> The 2 last ones are reported, not yet verified. 4241, 4243 and 4246 are just typos in examples. “just” is a bit of an under-statement when you are reading RFCs for days straight, you have 10 of them opened in your browser and trying to figure out how everything fits together and if you are doing everything correctly. Finding typos at that point in your process can be very confusing… 4247 is more subtle. The RFC about xCard comes with an XML Schema</a>. That's great! It will help us to test our documents and see if they are valid or not! No? No. Most of the time, I try to relax and deal with the incoming problems. But the date and time format in iCalendar, vCard, jCal, jCard, xCal and xCard can make my blood boil in a second. In what world, exactly, --10</code> or ---28</code> is a conceivable date and time format? How long did I sleep? “Well” — was I saying to myself, “do not make a drama, we have the XML Schema!”. No. Because there is an error in the schema. More precisely, in a regular expression: value-time = element time { xsd:string { pattern = "(\d\d(\d\d(\d\d)?)?|-\d\d(\d\d?)|--\d\d)" ~ "(Z|[+\-]\d\d(\d\d)?)?" } }</code></pre> Did you find the error? (\d\d?)</code> is invalid, this is (\d\d)?</code>. Don't get me wrong: Everyone makes mistakes, but not this kind of error. I will explain why in the next section. 4245 is not an editorial error but a technical one, under review. 4261 is crazy. It deserves a whole sub-section. Welcome in the crazy world of date and time formats#</a> </h3> There are two major popular date and time format: RFC2822</a> and ISO.8601. Examples: Fri, 27 Feb 2015 16:06:58 +0100</code> and</li> 2015-02-27T16:07:16+01:00</code>.</li> </ul> The second one is a good candidate for a computer representation: no locale, only digits, all information are present… Maybe you noticed there is no link on ISO.8601. Why? Because ISO standards are not free and I don't want to pay 140€</a> to buy a standard… The date and time format adopted by iCalendar and vCard (and the rest of the family) is ISO.8601.2004. I cannot read it. However, since we said in xCard we have an XML Schema; we can read this (after having applied erratum 4247): # 4.3.1 value-date = element date { xsd:string { pattern = "\d{8}|\d{4}-\d\d|--\d\d(\d\d)?|---\d\d" } } # 4.3.2 value-time = element time { xsd:string { pattern = "(\d\d(\d\d(\d\d)?)?|-\d\d(\d\d)?|--\d\d)" ~ "(Z|[+\-]\d\d(\d\d)?)?" } } # 4.3.3 value-date-time = element date-time { xsd:string { pattern = "(\d{8}|--\d{4}|---\d\d)T\d\d(\d\d(\d\d)?)?" ~ "(Z|[+\-]\d\d(\d\d)?)?" } } # 4.3.4 value-date-and-or-time = value-date | value-date-time | value-time</code></pre> Question: --10</code> is October or 10 seconds? --10</code> can fit into value-date</code> and value-time</code>: From value-date</code>, the 3rd element in the disjunction is --\d\d(\d\d)?</code>, so it matches --10</code> ,</li> From value-time</code>, the last element in the first disjunction is --\d\d</code>, so it matches --10</code>.</li> </ul> If we have a date-and-or-time value, value-date</code> comes first, so --10</code> is always October. Nevertheless, if we have a time value, --10</code> is 10 seconds. Crazy now? Oh, and XML has its own date and time format, which is well-defined and standardized. Why should we drag this crazy format along? Oh, and I assume every format depending on ISO.8601.2004 has this bug. But I am not sure because ISO standards are not free. How can RFCs have such errors?#</a> </h2> So far, RFCs are textual standards. Great. But they are just text. Written by humans, and thus they are subject to errors or failures. It is even error-prone. I do not understand: Why an RFC does not come with an executable test suite? I am pretty sure every reader of an RFC will try to create a test suite on its own. I assume xCal and xCard formats are not yet very popular. Consequently, few people read the RFC and tried to write an implementation. This is my guess. However, it does not avoid the fact an executable test suite should (must?) be provided. How did I find them?#</a> </h2> This is how I found these errors. I wrote a test suite for xCal and xCard in sabre/vobject</code></a>. I would love to write a test suite agnostic of the implementation, but I ran out of time. This is basically format transformation: R:x→y where R can be a reflexive operator or not (depending of the versions of iCalendar and vCard we consider). For “simple“ errata, I found the errors by testing it manually. For errata 4247 and 4261 (with the regular expressions), I found the error by applying the algorithms presented in Generate strings based on regular expressions</a> . Conclusion#</a> </h2> sabre/vobject</code> supports xCal and xCard. Control the terminal, the right way 2015-01-04T00:00:00+00:00 Nowadays, there are plenty of terminal emulators in the wild. Each one has a specific way to handle controls. How many colours does it support? How to control the style of a character? How to control more than style, like the cursor or the window? In this article, we are going to explain and show in action the right ways to control your terminal with a portable and an easy to maintain API. We are going to talk about stat</code>, tput</code>, terminfo</code>, hoa/ console</code>… but do not be afraid, it's easy and fun! Introduction#</a> </h2> Terminals. They are the ancient interfaces, still not old fashioned yet. They are fast, efficient, work remotely with a low bandwidth, secured and very simple to use. A terminal is a canvas composed of columns and lines. Only one character fits at a position. According to the terminal, we have some features enabled; for instance, a character might be stylized with a colour, a decoration, a weight etc. Let's consider the former. A colour belongs to a palette, which contains either 2, 8, 256 or more colours. One may wonder: How many colours does a terminal support?</li> How to control the style of a character?</li> How to control more than style, like the cursor or the window?</li> </ul> Well, this article is going to explain how a terminal works and how we interact with it. We are going to talk about terminal capabilities, terminal information (stored in database) and Hoa\Console</code></a>, a PHP library that provides advanced terminal controls. The basis of a terminal#</a> </h2> A terminal, or a console, is an interface that allows to interact with the computer. This interface is textual. Like a graphical interface, there are inputs: The keyboard and the mouse, and ouputs: The screen or a file (a real file, a socket, a FIFO, something else…). There is a ton of terminals. The most famous ones are: xterm</a>,</li> iTerm2</a>,</li> urxvt</a>,</li> TeraTerm</a>.</li> </ul> Whatever the terminal you use, inputs are handled by programs (or processus) and outputs are produced by these latters. We said outputs can be the screen or a file. Actually, everything is a file, so the screen is also a file. However, the user is able to use redirections</a> to choose where the ouputs must go. Let's consider the echo</code> program that prints all its options/arguments on its output. Thus, in the following example, foobar</code> is printed on the screen: $ echo 'foobar'</code></pre> And in the following example, foobar</code> is redirected to a file called log</code>: $ echo 'foobar' > log</code></pre> We are also able to redirect the output to another program, like wc</code> that counts stuff: $ echo 'foobar' | wc -c 7</code></pre> Now we know there are 7 characters in foobar</code>… no! echo</code> automatically adds a new-line (\n</code>) after each line; so: $ echo -n 'foobar' | wc -c 6</code></pre> This is more correct! Detecting type of pipes#</a> </h2> Inputs and outputs are called pipes. Yes, trivial, this is nothing more than basic pipes! There are 3 standard pipes: STDIN</code>, standing for the standard input pipe,</li> STDOUT</code>, standing for the standard output pipe and</li> STDERR</code>, standing for the standard error pipe (also an output one).</li> </ul> If the output is attached to the screen, we say this is a “direct output”. Why is it important? Because if we stylize a text, this is only for the screen, not for a file. A file should receive regular text, not all the decorations and styles. Hopefully, the Hoa\Console\Console</code> class</a> provides the isDirect</code>, isPipe</code> and isRedirection</code> static methods to know whether the pipe is respectively direct, a pipe or a redirection (damn naming…!). Thus, let Type.php</code> be the following program: <?php echo 'is direct: '; var_dump(Hoa\Console\Console::isDirect(STDOUT)); echo 'is pipe: '; var_dump(Hoa\Console\Console::isPipe(STDOUT)); echo 'is redirection: '; var_dump(Hoa\Console\Console::isRedirection(STDOUT));</code></pre> Now, let's test our program: $ php Type.php is direct: bool(true) is pipe: bool(false) is redirection: bool(false) $ php Type.php | xargs -I@ echo @ is direct: bool(false) is pipe: bool(true) is redirection: bool(false) $ php Type.php > /tmp/foo; cat !!$ is direct: bool(false) is pipe: bool(false) is redirection: bool(true)</code></pre> The first execution is very classic. STDOUT</code>, the standard output, is direct. The second execution redirects the output to another program, then STDOUT</code> is of kind pipe. Finally, the last execution redirects the output to a file called /tmp/foo</code>, so STDOUT</code> is a redirection. How does it work? We use fstat</code></a> to read the mode</code> of the file. The underlying fstat</code> implementation is defined in C, so let's take a look at the documentation of fstat(2)</code></a>. stat</code> is a C structure that looks like: struct stat { dev_t st_dev; /* device inode resides on */ ino_t st_ino; /* inode's number */ mode_t st_mode; /* inode protection mode */ nlink_t st_nlink; /* number of hard links to the file */ uid_t st_uid; /* user-id of owner */ gid_t st_gid; /* group-id of owner */ dev_t st_rdev; /* device type, for special file inode */ struct timespec st_atimespec; /* time of last access */ struct timespec st_mtimespec; /* time of last data modification */ struct timespec st_ctimespec; /* time of last file status change */ off_t st_size; /* file size, in bytes */ quad_t st_blocks; /* blocks allocated for file */ u_long st_blksize; /* optimal file sys I/O ops blocksize */ u_long st_flags; /* user defined flags for file */ u_long st_gen; /* file generation number */ }</code></pre> The value of mode</code> returned by the PHP fstat</code> function is equal to st_mode</code> in this structure. And st_mode</code> has the following bits: #define S_IFMT 0170000 /* type of file mask */ #define S_IFIFO 0010000 /* named pipe (fifo) */ #define S_IFCHR 0020000 /* character special */ #define S_IFDIR 0040000 /* directory */ #define S_IFBLK 0060000 /* block special */ #define S_IFREG 0100000 /* regular */ #define S_IFLNK 0120000 /* symbolic link */ #define S_IFSOCK 0140000 /* socket */ #define S_IFWHT 0160000 /* whiteout */ #define S_ISUID 0004000 /* set user id on execution */ #define S_ISGID 0002000 /* set group id on execution */ #define S_ISVTX 0001000 /* save swapped text even after use */ #define S_IRWXU 0000700 /* RWX mask for owner */ #define S_IRUSR 0000400 /* read permission, owner */ #define S_IWUSR 0000200 /* write permission, owner */ #define S_IXUSR 0000100 /* execute/search permission, owner */ #define S_IRWXG 0000070 /* RWX mask for group */ #define S_IRGRP 0000040 /* read permission, group */ #define S_IWGRP 0000020 /* write permission, group */ #define S_IXGRP 0000010 /* execute/search permission, group */ #define S_IRWXO 0000007 /* RWX mask for other */ #define S_IROTH 0000004 /* read permission, other */ #define S_IWOTH 0000002 /* write permission, other */ #define S_IXOTH 0000001 /* execute/search permission, other */</code></pre> Awesome, we have everything we need! We mask mode</code> with S_IFMT</code> to get the file data. Then we just have to check whether it is a named pipe S_IFIFO</code>, a character special S_IFCHR</code> etc. Concretly: isDirect</code> checks that the mode is equal to S_IFCHR</code>, it means it is attached to the screen (in our case),</li> isPipe</code> checks that the mode is equal to S_IFIFO</code>: This is a special file that behaves like a FIFO stack (see the documentation of mkfifo(1)</code></a>), everything which is written is directly read just after and the reading order is defined by the writing order (first-in, first-out!),</li> isRedirection</code> checks that the mode is equal to S_IFREG</code> , S_IFDIR</code> , S_IFLNK</code> , S_IFSOCK</code> or S_IFBLK</code> , in other words: All kind of files on which we can apply a redirection. Why? Because the STDOUT</code> (or another STD_*_</code> pipe) of the current processus is defined as a file pointer to the redirection destination and it can be only a file, a directory, a link, a socket or a block file.</li> </ul> I encourage you to read the implementation of the Hoa\Console\Console::getMode</code> method</a>. So yes, this is useful to enable styles on text but also to define the default verbosity level. For instance, if a program outputs the result of a computation with some explanations around, the highest verbosity level would output everything (the result and the explanations) while the lowest level would output only the result. Let's try with the toUpperCase.php</code> program: <?php $verbose = Hoa\Console\Console::isDirect(STDOUT); $string = $argv[1]; $result = (new Hoa\String\String($string))->toUpperCase(); if(true === $verbose) { echo $string, ' becomes ', $result, ' in upper case!', "\n"; } else { echo $result, "\n"; }</code></pre> Then, let's execute this program: $ php toUpperCase.php 'Hello world!' Hello world! becomes HELLO WORLD! in upper case!</code></pre> And now, let's execute this program with a pipe: $ php toUpperCase.php 'Hello world!' | xargs -I@ echo @ HELLO WORLD!</code></pre> Useful and very simple, isn't it? Terminal capabilities#</a> </h2> We can control the terminal with the inputs, like the keyboard, but we can also control the outputs. How? With the text itself. Actually, an output does not contain only the text but it includes control functions. It's like HTML: Around a text, you can have an element, specifying that the text is a link. It's exactly the same for terminals! To specify that a text must be in red, we must add a control function around it. Hopefully, these control functions have been standardized in the ECMA-48</a> document: Control Functions for Coded Character Set. However, not all terminals implement all this standard, and for historical reasons, some terminals use slightly different control functions. Moreover, some information do not belong to this standard (because this is out of its scope), like: How many colours does the terminal support? or does the terminal support the meta key? Consequently, each terminal has a list of capabilities. This list is splitted in 3 categories: boolean capabilities,</li> number capabilities,</li> string capabilities.</li> </ul> For instance: the “does the terminal support the meta key” is a boolean capability called meta_key</code> where its value is true</code> or false</code>,</li> the “number of colours supported by the terminal” is a… number capability called max_colors</code> where its value can be 2</code>, 8</code>, 256</code> or more,</li> the “clear screen control function” is a string capability called clear_screen</code> where its value might be \e[H\e[2J</code>,</li> the “move the cursor one column to the right” is also a string capability called cursor_right</code> where its value might be \e[C</code> .</li> </ul> All the capabilities can be found in the documentation of terminfo(5)</code></a> or in the documentation of xcurses</a>. I encourage you to follow these links and see how rich the terminal capabilities are! Terminal information#</a> </h2> Terminal capabilities are stored as information in databases. Where are these databases located? In files with a binary format. Favorite locations are: /usr/share/terminfo</code>,</li> /usr/share/lib/terminfo</code>,</li> /lib/terminfo</code>,</li> /usr/lib/terminfo</code>,</li> /usr/local/share/terminfo</code>,</li> /usr/local/share/lib/terminfo</code>,</li> etc.</li> or the TERMINFO</code> or TERMINFO_DIRS</code> environment variables.</li> </ul> Inside these directories, we have a tree of the form: _xx_/_name_</code>, where _xx_</code> is the ASCII value in hexadecimal of the first letter of the terminal name _name_</code>, or _n_/_name_</code> where _n_</code> is the first letter of the terminal name. The terminal name is stored in the TERM</code> environment variable. For instance, on my computer: $ echo $TERM xterm-256color $ file /usr/share/terminfo/78/xterm-256color /usr/share/terminfo/78/xterm-256color: Compiled terminfo entry</code></pre> We can use the Hoa\Console\Tput</code> class</a> to retrieve these information. The getTerminfo</code> static method allows to get the path of the terminal information file. The getTerm</code> static method allows to get the terminal name. Finally, the whole class allows to parse a terminal information database (it will use the file returned by getTerminfo</code> by default). For instance: <?php $tput = new Hoa\Console\Tput(); var_dump($tput->count('max_colors')); /** * Will output: * int(256) */</code></pre> On my computer, with xterm-256color</code>, I have 256 colours, as expected. If we parse the information of xterm</code> and not xterm-256color</code>, we will have: $tput = new Hoa\Console\Tput(Hoa\Console\Tput::getTerminfo('xterm')); var_dump($tput->count('max_colors')); /** * Will output: * int(8) */</code></pre>The power in your hand: Control the cursor#</a> </h2> Let's summarize. We are able to parse and know all the terminal capabilities of a specific terminal (including the one of the current user). If we would like a powerful terminal API, we need to control the basis, like the cursor. Remember. We said that the terminal is a canvas of columns and lines. The cursor is like a pen. We can move it and write something. We are going to (partly) see how the Hoa\Console\Cursor</code> class</a> works. I like to move it!#</a> </h3> The moveTo</code> static method allows to move the cursor to an absolute position. For example: <?php Hoa\Console\Cursor::moveTo($x, $y);</code></pre> The control function we use is cursor_address</code>. So all we need to do is to use the Hoa\Console\Tput</code> class and call the get</code> method on it to get the value of this string capability. This is a parameterized one: On xterm-256color</code>, its value is e[%i%p1%d;%p2%dH</code>. We replace the parameters by $x</code> and $y</code> and we output the result. That's all! We are able to move the cursor on an absolute position on all terminals! This is the right way to do. We use the same strategy for the move</code> static method that moves the cursor relatively to its current position. For example: <?php Hoa\Console\Cursor::move('right up');</code></pre> We split the steps and for each step we read the appropriated string capability using the Hoa\Console\Tput</code> class. For right</code>, we read the parm_right_cursor</code> string capability, for up</code>, we read parm_up_cursor</code> etc. Note that parm_right_cursor</code> is different of cursor_right</code>: The first one is used to move the cursor a certain number of times while the second one is used to move the cursor only one time. With performances in mind, we should use the first one if we have to move the cursor several times. The getPosition</code> static method returns the position of the cursor. This way to interact is a little bit different. We must write a control function on the output, and then, the terminal replies on the input. See the implementation by yourself</a>. <?php print_r(Hoa\Console\Cursor::getPosition()); /** * Will output: * Array * ( * [x] => 7 * [y] => 42 * ) */</code></pre> In the same way, we have the save</code> and restore</code> static methods that save the current position of the cursor and restore it. This is very useful. We use the save_cursor</code> and restore_cursor</code> string capabilities. Also, the clear</code> static method splits some parts to clear. For each part (direction or way), we read from Hoa\Console\Tput</code> the appropriated string capabilities: clear_screen</code> to clear all the screen, clr_eol</code> to clear everything on the right of the cursor, clr_eos</code> to clear everything bellow the cursor etc. <?php Hoa\Console\Cursor::clear('left');</code></pre> See what we learnt in action: <?php echo 'Foobar', "\n", 'Foobar', "\n", 'Foobar', "\n", 'Foobar', "\n", 'Foobar', "\n"; Hoa\Console\Cursor::save(); sleep(1); Hoa\Console\Cursor::move('LEFT'); sleep(1); Hoa\Console\Cursor::move('↑'); sleep(1); Hoa\Console\Cursor::move('↑'); sleep(1); Hoa\Console\Cursor::move('↑'); sleep(1); Hoa\Console\Cursor::clear('↔'); sleep(1); echo 'Hahaha!'; sleep(1); Hoa\Console\Cursor::restore(); echo "\n", 'Bye!', "\n";</code></pre> The result is presented in the following figure. Saving, moving, clearing and restoring the cursor with Hoa\Console</code>. </figcaption> </figure> The resulting API is portable, clean, simple to read and very easy to maintain! This is the right way to do. To get more information, please read the documentation</a>. Colours and decorations#</a> </h3> Now: Colours. This is mainly the reason why I decided to write this article. We see the same and the same libraries, again and again, doing only colours in the terminal, but unfortunately not in the right way 😞. A terminal has a palette of colours. Each colour is indexed by an integer, from 0 to potentially +∞ . The size of the palette is described by the max_colors</code> number capability. Usually, a palette contains 1, 2, 8, 256 or 16 million colours. xterm-256color</code> palette" loading="lazy" decoding="async" /> The xterm-256color</code> palette (source</a>). </figcaption> </figure> So first thing to do is to check whether we have more than 1 colour. If not, we must not colorize the given text. Next, if we have less than 256 colours, we have to convert the style into a palette containing 8 colours. Same with less than 16 million colours, we have to convert into 256 colours. Moreover, we can define the style of the foreground or of the background with respectively the set_a_foreground</code> and set_a_background</code> string capabilities. Finally, in addition to colours, we can define other decorations like bold, underline, blink or even inverse the foreground and the background colours. One thing to remember is: With this capability, we only define the style at a given “pixel” and it will apply on the following text. In this case, it is not exactly like HTML where we have a beginning and an end. Here we only have a beginning. Let's try! <?php Hoa\Console\Cursor::colorize('underlined foreground(yellow) background(#932e2e)'); echo 'foo'; Hoa\Console\Cursor::colorize('!underlined background(normal)'); echo 'bar', "\n";</code></pre> The API is pretty simple: We start to underline the text, we set the foreground to yellow and we set the background to #932e2e</code> . Then we output something. We continue with cancelling the underline decoration in addition to resetting the background. Finally we output something else. Here is the result: Fun with Hoa\Console\Cursor::colorize</code>. </figcaption> </figure> What do we observe? My terminal does not support more than 256 colours. Thus, #932e2e</code> is automatically converted into the closest colour in my actual palette! This is the right way to do. For fun, you can change the colours in the palette with the Hoa\Console\Cursor::changeColor</code> static method. You can also change the style of the cursor, like ▋</code>, _</code> or |</code>. To get more information, please read the documentation</a>. The power in your hand: Readline#</a> </h2> A more complete usage of Hoa\Console\Cursor</code> and even Hoa\Console\Window</code> is the Hoa\Console\Readline</code> class</a> that is a powerful readline. More than autocompleters, history, key bindings etc., it has an advanced use of cursors. See this in action: An autocompletion menu, made with Hoa\Console\Cursor</code> and Hoa\Console\Window</code>. </figcaption> </figure> We use Hoa\Console\Cursor</code> to move the cursor or change the colours and Hoa\Console\Window</code> to get the dimensions of the window, scroll some text in it etc. I encourage you to read the implementation. To get more information, please read the documentation</a>. The power in your hand: Sound 🎵#</a> </h2> Yes, even sound is defined by terminal capabilities. The famous bip is given by the bell</code> string capability. You would like to make a bip? Easy: <?php $tput = new Hoa\Console\Tput(); echo $tput->get('bell');</code></pre> That's it! Bonus: Window#</a> </h2> As a bonus, a quick demo of Hoa\Console\Window</code> because it's fun. The video shows the execution of the following code: <?php Hoa\Console\Window::setSize(80, 35); var_dump(Hoa\Console\Window::getPosition()); foreach( [ [100, 100], [150, 150], [200, 100], [200, 80], [200, 60], [200, 100] ] as list($x, $y) ) { sleep(1); Hoa\Console\Window::moveTo($x, $y); } sleep(2); Hoa\Console\Window::minimize(); sleep(2); Hoa\Console\Window::restore(); sleep(2); Hoa\Console\Window::lower(); sleep(2); Hoa\Console\Window::raise();</code></pre> We resize the window, we get its position, we move the window on the screen, we minimize and restore it, and finally we put it behind all other windows just before raising it. </iframe> To get more information, please read the documentation</a>. Conclusion#</a> </h2> In this article, we saw how to control the terminal by: Firstly, detecting the type of pipes, and secondly, reading and using the terminal capabilities. We know where these capabilities are stored and we saw few of them in action. This approach ensures your code will be portable, easy to maintain and easy to use. The portability is very important because, like browsers and user devices, we have a lot of terminal emulators released in the wild. We have to care about them. I encourage you to take a look at the Hoa\Console</code> library</a> and to contribute to make it even more awesome 😄. atoum has two release managers 2014-11-28T00:00:00+00:00 What is atoum?#</a> </h2> Short introduction: atoum is a simple, modern and intuitive unit testing framework for PHP. Originally created by Frédéric Hardy</a>, a good friend, it has grown thanks to many contributors</a>. atoum's logo. </figcaption> </figure> No one can say that atoum is not simple or intuitive. The framework offers several awesome features and is more like a meta unit testing framework. Indeed, the “user-land” of atoum, I mean all the assertions API (“this is an integer and it is equal to…”) is based on a very flexible mechanism, handled or embedded in runners, reporters etc. Thus, the framework is very extensible. You can find more informations in the README.md</code> of the project: Why atoum?</a>. Several important projects or companies use atoum. For instance, Pickle</a>, the PHP Extension installer, created by Pierre Joye</a>, another friend (the world is very small 😉) use atoum for its unit tests. Another example with M6Web</a>, the geeks working at M6</a>, the most profitable private national French TV channel, also use atoum. Another example, Mozilla</a> is using atoum to test some of their applications. Where is the cap'tain?#</a> </h2> Since the beginning, Frédéric has been a great leader for the project. He has inspired many people, users and contributors. In real life, on stage, on IRC… its personality and charisma were helpful in all aspects. However, leading such a project is a challenging and nerve-wracking daily work. I know what I am talking about with Hoa</a>. Hopefully for Frédéric, some contributors were here to help. Where to go cap'tain?#</a> </h2> However, having contributors do not create a community. A community is a group of people that share something together. A project needs a community with strong connections. They do no need to all look at the same direction, but they have to share something. In the case of atoum, I would say the project has been victim of its own success. We have seen the number of users increasing very quickly and the project was not yet ready for such a massive use. The documentation was not ready, a lot of features were not finalized, there were few contributors and the absence of a real community did not help. Put all these things together, blend them together and you obtain a bomb 😄. The project leaders were under a terrible pressure. In these conditions, this is not easy to work. Especially when users ask for new features. The needs to have a roadmap and people taking decisions were very strong. When the community acts#</a> </h2> After a couple of months under the sea, we have decided that we need to create a structure around the project. An organization. Frédéric is not able to do everything by himself. That's why 2 release managers have been elected: Mikaël Randy and I. Thank you to Julien Bianchi</a>, another friend 😉, for having organized these elections and being one of the most active contributor of atoum! Our goal is to define the roadmap of atoum: what will be included in the next version and what will not,</li> what features need work,</li> what bugs or issues need to be solved,</li> etc.</li> </ul> Well, a release manager is a pretty common job. Why 2? To avoid the bus effect and delegate. We all have a family, friends, jobs and side projects. With 2 release managers, we have 2 times more time to organize this project, and it deserves such an amount of time. The goal is also to organize the community if it is possible. New great features are coming and they will allow more people to contribute and build their “own atoum”. See below. Features to port!#</a> </h2> Everything is not defined at 100% but here is an overview of what is coming. First of all, you will find the latest issues and bugs</a> we have to close before the first release. Second, you will notice the version number… 1.0.0. Yes! atoum will have tags! After several discussions (#261</a>, #300</a>, #342</a>, #349</a>…), even if atoum is rolling-released, it will have tags. And with the semver format</a>. More informations on the blog of Julien Bianchi: atoum embraces semver</a>. Finally, a big feature is the Extension API</a>, that allows to write extension, such as: atoum/visibility-extension</code></a>, allows to override methods visibility; example:</li> </ul> <?php class Foo { protected function bar($arg) { return $arg; } } // and… class Foo extends atoum\test { public function testBaz() { $this ->if($sut = new \Foo()) ->and($arg = 'bar') ->then ->variable($this->invoke($sut)->bar($arg)) ->isEqualTo($arg); } }</code></pre> Now you will be able to test your protected and private methods! atoum/bdd-extension</code></a>, allows to write tests with the behavior-driven development style and vocabulary; example:</li> </ul> <?php class Formatter extends atoum\spec { public function should_format_underscore_separated_method_name() { $this ->given($formatter = new testedClass()) ->then ->invoking->format(__FUNCTION__)->on($formatter) ->shouldReturn('should format underscore separated method name'); } }</code></pre> Even the output looks familiar: Possible output with the `atoum/bdd-extension`. </figcaption> </figure> atoum/json-schema-extension</code></a>, allows to validate JSON payloads against a schema; example:</li> </ul> <?php class Foo extends atoum\test { public function testIsJson() { $this ->given($string = '{"foo": "bar"}') ->then ->json($string); } public function testValidatesSchema() { $this ->given($string = '["foo", "bar"]') ->then ->json($string)->validates('{"title": "test", "type": "array"}') ->json($string)->validates('/path/to/json.schema'); } }</code></pre> atoum/praspel-extension</code></a>, allows to use Praspel</a> inside atoum: automatically generate and validate advanced test data and unit tests; example:</li> </ul> <?php class Foo extends atoum\test { public function testFoo() { $this ->if($regex = $this->realdom->regex('/[\w\-_]+(\.[\w\-\_]+)*@\w\.(net|org)/')) ->and($email = $this->sample($regex)) ->then … } }</code></pre> Here, we have generated a string based on its regular expression. Reminder, you might have seen this on this blog: Generate strings based on regular expressions</a> . Fun fact: the atoum/json-schema-extension</code> is tested with atoum obviously and… atoum/praspel-extension</code>! Conclusion#</a> </h2> atoum has a bright future with exciting features! We sincerely hope this new direction will gather existing and new contributors 😄. ❤️ open-source! Hello fruux! 2014-11-24T00:00:00+00:00 Leaving the research world#</a> </h2> I have really enjoyed my time at INRIA and Femto-ST, 2 research institutes in France. But after 8 years at the university and a hard PhD thesis (but with great results by the way!), I would like to see other things. My time as an intern at Mozilla and my work in the open-source world have been very seductive. Open-source contrasts a lot with the research world, where privacy and secrecy are first-citizens of every project. All the work I have made and all the algorithms I have developed during my PhD thesis have been implemented under an open-source license, and I ran into some issues because of such decision (patents are sometimes better, sometimes not… long story). So, I like research but I also like to hack and share everything. And right now, I have to get a change of air! So I asked on Twitter: I (Ivan Enderlin, a fresh PhD, creator of Hoa) am looking for a job. Here is my CV: http://t.co/dAdLm35RUu</a>. Please, contact me! #hoajob</a> — Hoa project (@hoaproject) July 24th, 2014</a> </blockquote> And what a surprise! A lot of companies answered to my tweet (most of them in private of course), but the most interesting one at my eyes was… fruux 😉. fruux#</a> </h2> fruux defines itself as: “A unified contacts/calendaring system that works across platforms and devices</a>. We are behind sabre/dav</code></a>, which is the most popular open-source implementation of the CardDAV</a> and CalDAV</a> standards. Besides us, developers and companies around the globe use our sabre/dav</code> technology to deliver sync functionality to millions of users”. fruux's logo. </figcaption> </figure> Several things attract me at fruux: low-layers are open-source,</li> viable ecosystem based on open-source,</li> accepts remote working,</li> close timezone to mine,</li> touching millions of people,</li> standards in minds.</li> </ol> The first point is the most important for me. I don't want to make a company richer without any benefits for the rest of the world. I want my work to be beneficial to the rest of the world, to share my work, I want my work to be reused, hacked, criticized, updated and shared again. This is the spirit of the open-source and the hackability paradigms. And fortunately for me, fruux's low-layers are 100% open-source, namely sabre/dav</code> & co. However, being able to eat at the end of the month with open-source is not that simple. Fortunately for me, fruux has a stable economic model, based on open-source. Obviously, I have to work on closed projects, obviously, I have to work for some specific customers, but I can go back to open-source goodnesses all the time 😉. In addition, I am currently living in Switzerland and fruux is located in Germany. Fortunately for me, fruux's team is kind of dispatched all around Europe and the world. Naturally, they accept me to work remotely. Whilst it can be inconvenient for some people, I enjoy to have my own hours, to organize myself as I would like etc. Periodical meetings and phone-calls help to stay focused. And I like to think that people are more productive this way. After 4 years at home because of my Master thesis and PhD thesis, I know how to organize myself and exchange with a decentralized team. This is a big advantage. Moreover, Germany is in the same timezone as Switzerland! Compared to companies located at, for instance, California, this is simpler for my family. Finally, working on an open-source project that is used by millions of users is very motivating. You know that your contributions will touch a lot of people and it gives meaning to my work on a daily basis. Also, the last thing I love at fruux is this desire to respect standards, RFC, recommandations etc. They are involved in these processes, consortiums and groups (for instance CalConnect</a>). I love standards and specifications, and this methodology reminds me the scientific approach I had with my PhD thesis. I consider that a standard without an implementation has no meaning, and a well-designed standard is a piece of a delicious cake, especially when everyone respects this standard 😄. (… but the cake is a lie!) sabre/*</code>#</a> </h2> fruux has mostly hired me because of my experience on Hoa</a>. One of my main public job is to work on all the sabre/*</code> libraries, which include: sabre/dav</code></a>,</li> sabre/davclient</code></a>,</li> sabre/event</code></a>,</li> sabre/http</code></a>,</li> sabre/proxy</code></a>,</li> sabre/tzserver</code></a>,</li> sabre/vobject</code></a>,</li> sabre/xml</code></a>.</li> </ul> You will find the documentations and the news on sabre.io</a>. All these libraries serve the first one: sabre/dav</code>, which is an implementation of the WebDAV technology, including extensions for CalDAV, and CardDAV, respectively for calendars, tasks and address books. For the one who does not know what is WebDAV, in few words: The Web is mostly a read-only media, but WebDAV extends HTTP in order to be able to write and collaborate on documents. The way WebDAV is defined is fascinating, and even more, the way it can be extended. Most of the work is already done by Evert</a> and many contributors, but we can go deeper! More extensions, more standards, better code, better algorithms etc.! If you are interested in the work I am doing on sabre/*</code>, you can check this search result on Github</a>. Future of Hoa#</a> </h2> Certain people have asked me about the future of Hoa: Whether I am going to stop it or not since I have a job now. Firstly, a PhD thesis is exhausting, and believe me, it requires more energy than a regular job, even if you are passionate about your job and you did not count working hours. With a PhD thesis, you have no weekend, no holidays, you are always out of time, you always have a ton (sic) of articles and documents to read… there is no break, no end. In these conditions, I was able to maintain Hoa and to grow the project though, thanks to a very helpful and present community! Secondly, fruux is planning to use Hoa. I don't know how or when, but if at a certain moment, using Hoa makes sense, they will. What does it imply for Hoa and me? It means that I will be paid to work on Hoa at a little percentage. I don't know how much, it will depend of the moments, but this is a big step forward for the project. Moreover, a project like fruux using Hoa is a big chance! I hope to see the fruux's logo very soon on the homepage of the Hoa's website. Thus, to conclude, I will have more time (on evenings, weekends, holidays and sometimes during the day) to work on Hoa. Do not be afraid, the future is bright 😄. Conclusion#</a> </h2> Bref, I am working at fruux! Generate strings based on regular expressions 2014-09-30T00:00:00+00:00 During my PhD thesis, I have partly worked on the problem of the automatic accurate test data generation. In order to be complete and self-contained, I have addressed all kinds of data types, including strings. This article aims at showing how to generate accurate and relevant strings under several constraints. What is a regular expression?#</a> </h2> We are talking about formal language theory here. In the known world, there are four kinds of languages. More formally, in 1956, the Chomsky hierarchy</a> has been formulated, classifying grammars (which define languages) in four levels: unrestricted grammars, matching langages known as Turing languages, no restriction,</li> context-sensitive grammars, matching contextual languages,</li> context-free grammars, matching algebraic languages, based on stacked automata,</li> regular grammars, matching regular languages.</li> </ol> Each level includes the next level. The last level is the “weaker”, which must not sound negative here. Regular expressions</a> are used often because of their simplicity and also because they solve most problems we encounter daily. A regular expression is a small language with very few operators and, most of the time, a simple semantics. For instance ab(c|d)</code> means: a word (a data) starting by ab</code> and followed by c</code> or d</code>. We also have quantification operators (also known as repetition operators), such as ?</code>, *</code> and +</code>. We also have {_x_,_y_}</code> to define a repetition between _x_</code> and _y_</code>. Thus, ? </code> is equivalent to {0,1}</code>, *</code> to {0,}</code> and +</code> to {1,}</code>. When _y_</code> is missing, it means +∞, so unbounded (or more exactly, bounded by the limits of the machine). So, for instance ab(c|d){2,4}e?</code> means: a word starting by ab</code>, followed 2, 3 or 4 times by c</code> or d</code> (so cc</code>, cd</code>, dc</code>, ccc</code>, ccd</code>, cdc</code> and so on) and potentially followed by e</code>. The goal here is not to teach you regular expressions but this is kind of a tiny reminder. There are plenty of regular languages. You might know POSIX regular expression</a> or Perl Compatible Regular Expressions (PCRE)</a>. Forget the first one, please. The syntax and the semantics are too much limited. PCRE is the regular language I recommend all the time. Behind every formal language there is a graph. A regular expression is compiled into a Finite State Machine (FSM)</a>. I am not going to draw and explain them, but it is interesting to know that behind a regular expression there is a basic automaton. No magic. Why focussing regular expressions?#</a> </h3> This article focuses on regular languages instead of other kind of languages because we use them very often (even daily). I am going to address context-free languages in another article, be patient young padawan. The needs and constraints with other kind of languages are not the same and more complex algorithms must be involved. So we are going easy for the first step. Understanding PCRE: lex and parse them#</a> </h2> The Hoa\Compiler</code> library</a> provides both LL(1) LL(k) compiler-compilers. The documentation</a> describes how to use it. We discover that the LL(k) compiler comes with a grammar description language called PP. What does it mean? It means for instance that the grammar of the PCRE can be written with the PP language and that Hoa\Compiler\Llk</code> will transform this grammar into a compiler. That's why we call them “compiler of compilers”. Fortunately, the Hoa\Regex</code> library</a> provides the grammar of the PCRE language in the hoa://Library/Regex/Grammar.pp</code></a> file. Consequently, we are able to analyze regular expressions written in the PCRE language! Let's try in a shell at first with the hoa compiler:pp</code> tool: $ echo 'ab(c|d){2,4}e?' | hoa compiler:pp hoa://Library/Regex/Grammar.pp 0 --visitor dump > #expression > > #concatenation > > > token(literal, a) > > > token(literal, b) > > > #quantification > > > > #alternation > > > > > token(literal, c) > > > > > token(literal, d) > > > > token(n_to_m, {2,4}) > > > #quantification > > > > token(literal, e) > > > > token(zero_or_one, ?)</code></pre> We read that the whole expression is composed of a single concatenation of two tokens: a</code> and b</code>, followed by a quantification, followed by another quantification. The first quantification is an alternation of (a choice betwen) two tokens: c</code> and d</code>, between 2 to 4 times. The second quantification is the e</code> token that can appear zero or one time. Pretty simple. The final output of the Hoa\Compiler\Llk\Parser</code> class is an Abstract Syntax Tree (AST)</a>. The documentation of Hoa\Compiler</code> explains all that stuff, you should read it. The LL(k) compiler is cut out into very distinct layers in order to improve hackability. Again, the documentation teach us we have four levels in the compilation process</a>: lexical analyzer, syntactic analyzer, trace and AST. The lexical analyzer (also known as lexer) transforms the textual data being analyzed into a sequence of tokens (formally known as lexemes). It checks whether the data is composed of the good pieces. Then, the syntactic analyzer (also known as parser) checks that the order of tokens in this sequence is correct (formally we say that it derives the sequence, see the Matching words section</a> to learn more). Still in the shell, we can get the result of the lexical analyzer by using the --token-sequence</code> option; thus: $ echo 'ab(c|d){2,4}e?' | hoa compiler:pp hoa://Library/Regex/Grammar.pp 0 --token-sequence # … token name token value offset ----------------------------------------- 0 … literal a 0 1 … literal b 1 2 … capturing_ ( 2 3 … literal c 3 4 … alternation | 4 5 … literal d 5 6 … _capturing ) 6 7 … n_to_m {2,4} 7 8 … literal e 12 9 … zero_or_one ? 13 10 … EOF 15</code></pre> This is the sequence of tokens produced by the lexical analyzer. The tree is not yet built because this is the first step of the compilation process. However this is always interesting to understand these different steps and see how it works. Now we are able to analyze any regular expressions in the PCRE format! The result of this analysis is a tree. You know what is fun with trees? Visiting them</a>. Visiting the AST#</a> </h2> Unsurprisingly, each node of the AST can be visited thanks to the Hoa\Visitor</code> library</a>. Here is an example with the “dump” visitor: <?php use Hoa\Compiler; use Hoa\File; // 1. Load grammar. $compiler = Compiler\Llk\Llk::load( new File\Read('hoa://Library/Regex/Grammar.pp') ); // 2. Parse a data. $ast = $compiler->parse('ab(c|d){2,4}e?'); // 3. Dump the AST. $dump = new Compiler\Visitor\Dump(); echo $dump->visit($ast);</code></pre> This program will print the same AST dump we have previously seen in the shell. How to write our own visitor? A visitor is a class with a single visit</code> method. Let's try a visitor that pretty print a regular expression, i.e. transform: ab(c|d){2,4}e?</code></pre> into: a b ( c | d ){2,4} e?</code></pre> Why a pretty printer? First, it shows how to visit a tree. Second, it shows the structure of the visitor: we filter by node ID (#expression</code>, #quantification</code>, token</code> etc.) and we apply respective computations. A pretty printer is often a good way for being familiarized with the structure of an AST. Here is the class. It catches only useful constructions for the given example: <?php use Hoa\Visitor; class PrettyPrinter implements Visitor\Visit { public function visit( Visitor\Element $element, &$handle = null, $eldnah = null ) { static $_indent = 0; $out = null; $nodeId = $element->getId(); switch($nodeId) { // Reset indentation and… case '#expression': $_indent = 0; // … visit all the children. case '#quantification': foreach($element->getChildren() as $child) $out .= $child->accept($this, $handle, $eldnah); break; // One new line between each children of the concatenation. case '#concatenation': foreach($element->getChildren() as $child) $out .= $child->accept($this, $handle, $eldnah) . "\n"; break; // Add parenthesis and increase indentation. case '#alternation': $oout = []; $pIndent = str_repeat(' ', $_indent); ++$_indent; $cIndent = str_repeat(' ', $_indent); foreach($element->getChildren() as $child) $oout[] = $cIndent . $child->accept($this, $handle, $eldnah); --$_indent; $out .= $pIndent . '(' . "\n" . implode("\n" . $cIndent . '|' . "\n", $oout) . "\n" . $pIndent . ')'; break; // Print token value verbatim. case 'token': $tokenId = $element->getValueToken(); $tokenValue = $element->getValueValue(); switch($tokenId) { case 'literal': case 'n_to_m': case 'zero_or_one': $out .= $tokenValue; break; default: throw new RuntimeException( 'Token ID ' . $tokenId . ' is not well-handled.' ); } break; default: throw new RuntimeException( 'Node ID ' . $nodeId . ' is not well-handled.' ); } return $out; } }</code></pre> And finally, we apply the pretty printer on the AST like previously seen: <?php $compiler = Compiler\Llk\Llk::load( new File\Read('hoa://Library/Regex/Grammar.pp') ); $ast = $compiler->parse('ab(c|d){2,4}e?'); $prettyprint = new PrettyPrinter(); echo $prettyprint->visit($ast);</code></pre> Et voilà ! Now, put all that stuff together! Isotropic generation#</a> </h2> We can use Hoa\Regex</code> and Hoa\Compiler</code> to get the AST of any regular expressions written in the PCRE format. We can use Hoa\Visitor</code> to traverse the AST and apply computations according to the type of nodes. Our goal is to generate strings based on regular expressions. What kind of generation are we going to use? There are plenty of them: uniform random, smallest, coverage based… The simplest is isotropic generation, also known as random generation. But random says nothing: what is the repartition, or do we have any uniformity? Isotropic means each choice will be solved randomly and uniformly. Uniformity has to be defined: does it include the whole set of nodes or just the immediate children of the node? Isotropic means we consider only immediate children. For instance, a node #alternation</code> has c immediate children, the probability C to choose one child is: $P(C) = \frac{1}{c}</annotation> </semantics> </math> Yes, simple as that! We can use the Hoa\Math</code> library</a> that provides the Hoa\Math\Sampler\Random</code> class to sample uniform random integers and floats. Ready? Structure of the visitor#</a> </h3> The structure of the visitor is the following: <?php use Hoa\Visitor; use Hoa\Math; class IsotropicSampler implements Visitor\Visit { protected $_sampler = null; public function __construct(Math\Sampler $sampler) { $this->_sampler = $sampler; return; } public function visit( Visitor\Element $element, &$handle = null, $eldnah = null ) { switch($element->getId()) { // … } } }</code></pre> We set a sampler and we start visiting and filtering nodes by their node ID. The following code will generate a string based on the regular expression contained in the $expression</code> variable: <?php $expression = '…'; $ast = $compiler->parse($expression); $generator = new IsotropicSampler(new Math\Sampler\Random()); echo $generator->visit($ast);</code></pre> We are going to change the value of $expression</code> step by step until having ab(c|d){2,4}e?</code>. Case of #expression</code>#</a> </h3> A node of type #expression</code> has only one child. Thus, we simply return the computation of this node: <?php case '#expression': return $element->getChild(0)->accept($this, $handle, $eldnah); break;</code></pre>Case of token</code>#</a> </h3> We consider only one type of token for now: literal</code>. A literal can contain an escaped character, can be a single character or can be .</code> (which means everything). We consider only a single character for this example (spoil: the whole visitor already exists). Thus: <?php case 'token': return $element->getValueValue(); break;</code></pre> Here, with $expression = 'a';</code> we get the string a</code>. Case of #concatenation</code>#</a> </h3> A concatenation is just the computation of all children joined in a single piece of string. Thus: <?php case '#concatenation': $out = null; foreach($element->getChildren() as $child) $out .= $child->accept($this, $handle, $eldnah); return $out; break;</code></pre> At this step, with $expression = 'ab';</code> we get the string ab</code>. Totally crazy. Case of #alternation</code>#</a> </h3> An alternation is a choice between several children. All we have to do is to select a child based on the probability given above. The number of children for the current node can be known thanks to the getChildrenNumber</code> method. We are also going to use the sampler of integers. Thus: <?php case '#alternation': $childIndex = $this->_sampler->getInteger( 0, $element->getChildrenNumber() - 1 ); return $element->getChild($childIndex) ->accept($this, $handle, $eldnah); break;</code></pre> Now, with $expression = 'ab(c|d)';</code> we get the strings abc</code> or abd</code> at random. Try several times to see by yourself. Case of #quantification</code>#</a> </h3> A quantification is an alternation of concatenations. Indeed, e{2,4}</code> is strictly equivalent to ee|eee|eeee</code>. We have only two quantifications in our example: ?</code> and {_x_,_y_}</code>. We are going to find the value for _x_</code> and _y_</code> and then choose at random between these bounds. Let's go: <?php case '#quantification': $out = null; $x = 0; $y = 0; // Filter the type of quantification. switch($element->getChild(1)->getValueToken()) { // ? case 'zero_or_one': $y = 1; break; // {x,y} case 'n_to_m': $xy = explode( ',', trim($element->getChild(1)->getValueValue(), '{}') ); $x = (int) trim($xy[0]); $y = (int) trim($xy[1]); break; } // Choose the number of repetitions. $max = $this->_sampler->getInteger($x, $y); // Concatenate. for($i = 0; $i < $max; ++$i) { $out .= $element->getChild(0)->accept($this, $handle, $eldnah); } return $out; break;</code></pre> Finally, with $expression = 'ab(c|d){2,4}e?';</code> we can have the following strings: abdcce</code>, abdc</code>, abddcd</code>, abcde</code> etc. Nice isn't it? Want more? <?php for($i = 0; $i < 42; ++$i) { echo $generator->visit($ast), "\n"; } /** * Could output: * abdce * abdcc * abcdde * abcdcd * abcde * abcc * abddcde * abddcce * abcde * abcc * abdcce * abcde * abdce * abdd * abcdce * abccd * abdcdd * abcdcce * abcce * abddc */</code></pre>Performance#</a> </h2> This is difficult to give numbers because it depends of a lot of parameters: your machine configuration, the PHP VM, if other programs run etc. But I have generated 1 million strings in less than 25 seconds on my machine (an old MacBook Pro), which is pretty reasonable. Conclusion and surprise#</a> </h2> So, yes, now we know how to generate strings based on regular expressions! Supporting all the PCRE format is difficult. That's why the Hoa\Regex</code> library</a> provides the Hoa\Regex\Visitor\Isotropic</code> class that is a more advanced visitor. This latter supports classes, negative classes, ranges, all quantifications, all kinds of literals (characters, escaped characters, types of characters —\w</code>, \d</code>, \h</code>…—) etc. Consequently, all you have to do is: <?php use Hoa\Regex; // … $generator = new Regex\Visitor\Isotropic(new Math\Sampler\Random()); echo $generator->visit($ast);</code></pre> This algorithm is used in Praspel</a>, a specification language I have designed during my PhD thesis. More specifically, this algorithm is used inside realistic domains. I am not going to explain it today but it allows me to introduce the “surprise”. Generate strings based on regular expressions in atoum#</a> </h3> atoum</a> is an awesome unit test framework. You can use the Atoum\PraspelExtension</code> extension</a> to use Praspel and therefore realistic domains inside atoum. You can use realistic domains to validate and to generate data, they are designed for that. Obviously, we can use the Regex</code> realistic domain. This extension provides several features including sample</code>, sampleMany</code> and predicate</code> to respectively generate one datum, generate many data and validate a datum based on a realistic domain. To declare a regular expression, we must write: <?php $regex = $this->realdom->regex('/ab(c|d){2,4}e?/');</code></pre> And to generate a datum, all we have to do is: <?php $datum = $this->sample($regex);</code></pre> For instance, imagine you are writing a test called test_mail</code> and you need an email address: <?php public function test_mail() { $this ->given( $regex = $this->realdom->regex('/[\w\-_]+(\.[\w\-\_]+)*@\w\.(net|org)/'), $address = $this->sample($regex), $mailer = new \Mock\Mailer(…), ) ->when($mailer->sendTo($address)) ->then ->… }</code></pre> Easy to read, fast to execute and help to focus on the logic of the test instead of test data (also known as fixtures). Note that most of the time the regular expressions are already in the code (maybe as constants). It is therefore easier to write and to maintain the tests. I hope you enjoyed this first part of the series :-)! This work has been published in the International Conference on Software Testing, Verification and Validation: Grammar-Based Testing using Realistic Domains in PHP</a>. Rüsh Release 2014-09-15T00:00:00+00:00 Since 2 years, at Hoa</a>, we are looking for the perfect release process. Today, we have finalized the last thing related to this new process: we have found a name. It is called Rüsh Release, standing for Rolling Ünd ScHeduled Release. The following explanations are useful from the user point of view, not from the developer point of view. It means that we do not explain all the branches and the workflow between all of them. We will settle for the user final impact. Rolling Release#</a> </h2> On one hand, Hoa is not and will never be finished. We will never reach the “Holy 1.0 Grail”. So, one might reckon that Hoa is rolling-released? Let's dive into this direction. There are plenty rolling release types</a> out there, such as: partially rolling,</li> fully rolling,</li> truly rolling,</li> pseudo-rolling,</li> optionally rolling,</li> cyclically rolling,</li> and synonyms…</li> </ul> I am not going to explain all of them. All you need to know is that Hoa is partly and truly rolling released, or part- and true- rolling released for short. Why? Firstly, “Part-rolling project has a subset of software packages that are not rolling”. If we look at Hoa only, it is fully rolling but Hoa depends on PHP virtual machines to be executed, which are not rolling released (for the most popular ones at least). Thus, Hoa is partly rolling released. Secondly, “True-rolling [project] are developed solely using a rolling release software development model”, which is the case of Hoa. Consequently and finally, the master</code> branch is the final public branch, it means that it always contains the latest version, and users constantly fetch updates from it.$

Operation</th>	Time</th>	“Human scale”</th></tr></thead>
Fetch from L1 cache</td>	1ns</td>	1mn</td></tr>
Branch misprediction</td>	3ns</td>	3mn</td></tr>
Fetch from L2 cache</td>	4ns</td>	4mn</td></tr>
Mutex lock/unlock</td>	17ns</td>	17mn</td></tr>
Fetch from the main memory</td>	100ns</td>	1h40mn</td></tr>
SSD random read</td>	16'000ns</td>	11.11 days</td></tr> </tbody></table> Latency numbers for the year 2020 for various operations (source: Latency Numbers Every Programmer Shoud Know</cite> from Colin Scott (UC Berkeley)</a>).</p> The time in the second column is given in nanoseconds, i.e. $1</mn>1'000'000'000</mn></mfrac></math> second. The time in the third column is “humanized” to give us a better sense of the scale here: we imagine 1ns maps to 1min.</p> </figcaption> </figure>Do you see the difference between the L1/L2 caches and the main memory? 1ns to 100ns is the same difference as 1mn to 1h40. So, yes, it takes time to read from memory. That's why we try to avoid allocations as much as possible.</p>#memory-race text { font-size: 4pt } #memory-race circle { fill: oklch(69.50% .140 76.18); animation: 4s linear 0s infinite alternate slide; } #memory-race .l1 { animation-duration: .5s } #memory-race .l2 { animation-duration: 2s } #memory-race .ram { animation-duration: 50s } @keyframes slide { from { transform: translateX(15%); } to { transform: translateX(85%); } } </style> CPU</text> CPU</text> CPU</text> L1</text> L2</text> RAM</text> </svg> Not comfortable with numbers? Let's try to visualise it with 1ns = 1s! On the left: the CPU. On the right, the L1 cache, the L2 cache, and the RAM. The “balls” represent the time it takes to move information between the CPU and the L1/L2 caches or the RAM.</p> </figcaption> </figure> Sadly, in our case, it appears we are allocating 322'042 times to sort the initial rooms of the Room List, for a total of 743'151'616 bits allocated, with 287 bytes per allocation. Of course, if we are doing quick napkin maths 5</a></sup>, it should take around 200ms. We are far from The Frozen Room List TM</abbr></sup>, but there is more going on. 6</a></sup></p> Do you remember the memory allocator? Its role is to also avoid fragmentation</em> as much as possible. The number of memory “blocks” isn't infinite: when memory blocks are freed, and new ones are allocated later, maybe the previous blocks are no longer available and cannot be reused. The allocator has to find a good place, while keeping fragmentation under control. Maybe the blocks must be moved to create enough space to insert the new blocks (it's often preferable to have contiguous blocks).</p> That's what I call memory pressure</strong>. We are asking too much, too fast, and the memory allocator we use in the Matrix Rust SDK is not designed to handle this use case.</p> What are our solutions then?</p> Le Factotum</span> </picture> </div> May I suggest an approach? What about finding where we are allocating and deallocating memory? Then we might be able to reduce either the number of allocations, or the size of the value being allocated (and deallocated), with the hope of making the memory allocator happier. Possible solutions:</p> If the allocated value is too large to fit in the stack, we could return a pointer to it if possible,</li> Maybe we don't need the full value: we could return just a pointer to a fragment of it?</li> </ul> </div> </div> Excellent ideas. Let's track which sorter creates the problem. We start with the sorter that was recently modified: latest_event</code>. In short, this sorter compares the LatestEventValue</code> of two rooms: the idea is that rooms with a LatestEventValue</code> representing a local event</em>, i.e. an event that is not sent yet, or is sending, must be at the top of the Room List. Alright, let's look at its core part</a>:</p> pub fn</span> new_sorter</span> ()</span> -></span> impl</span> Sorter</span> {</span></span> let</span> latest_events</span> =</span></span> \|</span> left</span> : &</span> Room</span>,</span> right</span> : &</span> Room</span> \|</span> (</span> left</span> .</span> latest_event</span> (),</span> right</span> .</span> latest_event</span> ());</span></span> </span> Alright. For each sorting iteration, the Room::latest_event</code> method is called twice. This method is as follows</a>:</p> pub fn</span> latest_event</span> (</span> &</span> self</span>)</span> -></span> LatestEventValue</span> {</span></span> self</span> .</span> info</span> .</span> read</span> ()</span> .</span> latest_event</span> .</span> clone</span> ()</span></span>}</span></span></code></pre> Oh, there it is. We are acquiring a read lock over the info</code> value, then we are reading the latest_event</code> field, and we are cloning the value. Cloning is important here as we don't want to hold the read lock for too long. This is our culprit. The size of the LatestEventValue</code></a> type is 144 bytes (it doesn't count the size of the event itself, because this size is dynamic).</p> Before going further, let's check whether another sorter has a similar problem, shall we? Look at the other sorters</i>, oh!, turns out the recency</code> sorter</a> also uses the latest_event</code> method! Damn, this is becoming really annoying.</p> Question: do we need the entire LatestEventValue</code>? Probably not!</p> For the latest_event</code> sorter, we actually only need to know when this LatestEventValue</code> is local</em>, that's it.</li> For the recency</code> sorter, we only need to know the timestamp of the LatestEventValue</code>.</li> </ul> So instead of copying the whole value in memory twice per sorter iteration, for two sorters, let's try to write more specific methods:</p> pub fn</span> latest_event_is_local</span> (</span> &</span> self</span>)</span> -></span> bool</span> {</span></span> self</span> .</span> info</span> .</span> read</span> ()</span> .</span> latest_event</span> .</span> is_local</span> ()</span></span> Just like that, the throughput has been improved by 18%</strong> according to the room_list</code> benchmark. You can see the patch in “action”</a>. Can we declare victory over memory pressure?</p> Le Comte</span> </picture> </div> I beg your pardon, but I don't believe it's a victory. We have reduced the size of allocations, but not the number of allocations itself.</p> Well, actually, latest_event_is_local</code> returns a bool</code>: it can fit in the stack. And latest_event_timestamp</code> returns an Option<MilliSecondsSinceUnixEpoch></code>, where MilliSecondsSinceUnixEpoch</code> is a Uint</code></a>, which itself is a f64</code></a>: it can also fit in the stack.</p> So, yes, we may have reduced the number of allocations greatly, that's agreed, it explains the 18% throughput improvement. However, issue reporters were mentioning a lag of 5 minutes or so, do you remember? How do you explain the remaining 4 minutes 6 seconds then? This is still unacceptable, right?</p> </div> </div> Definitely yes! Everything above 200ms (from our napkin maths) is unacceptable here. Memory pressure was an important problem, and it's now solved, but it wasn't the only problem.</p> Lock Contention #</a> </h2> The assiduous reader may have noticed that we are still dealing with a lock here.</p> self</span> .</span> info</span> .</span> read</span> ()</span> .</span> latest_event</span> .</span> …</span></span> // ^^^^^^</span></span> Le Comte</span> </picture> </div> … the lock is acquired 322'042 times!</p> …</p> … no?</p> </div> </div> … yes… and please, stop interrupting me, I was trying to build up the suspense for a climax.</p> Anyway. Avoiding a lock isn't an easy task. However, this lock around info</code> is particularly annoying because it's called by almost all sorters! They need information about a Room</code>; all the information is in this info</code> field, which is a read-write lock. Hmmm.</p> Let's change our strategy. We need to take a step back:</p> The sorters need this data.</li> Running the sorters won't change this data.</li> When the data does change the sorters will be re-run.</li> </ol> Maybe we could fetch, ahead of time, all the necessary data for all sorters in a single type: it will be refreshed when the data changes, which is right before the sorters run again.</p> Le Procureur</span> </picture> </div> The idea here is to organise the data around a specific layout. The focus on the data layout aims at being CPU cache friendly as much as possible. This kind of approach is called$

From 19k to 4.2M events/sec: story of a SQLite query optimisation

Sliding Sync at the Matrix Conference

Building a new site!

Observability

I've loved Wasmer, I still love Wasmer

Bye bye WhatsApp, hello ?

Announcing the first Java library to run WebAssembly: Wasmer JNI

Announcing the first Postgres extension to run WebAssembly

Announcing the fastest WebAssembly runtime for Go: wasmer

🐘+🦀+🕸 php-ext-wasm: Migrating from wasmi to Wasmer

Bye bye Automattic, hello Wasmer

The PHP galaxy

The C galaxy

The ASM.js galaxy

The WebAssembly galaxy

Prelude

`Bindings#</a> </h2> </p> </figure> From now, our goal is to expose the root</code> function and the Node</code> enum in different platforms or environments. Ready?</p> 3… 2… 1… lift-off!</p>`

How Automattic (WordPress.com & co.) partly moved away from PHPUnit to atoum?

One conference per day, for one year (2017)

Random thoughts about `::class` in PHP

Automattic, Grand Meetup 2017

atoum supports TeamCity

Export functions in PHP à la Javascript

Finite-State Machine as a Type System illustrated with a store product

Tagua VM, a safe PHP virtual machine

Faster find algorithms in nom

Welcome to Chaos

Bye bye Liip, hello Automattic

DuckDuckGo in a Shell

sabre/katana

RFCs should provide executable test suites

`Conclusion#</a> </h2> sabre/vobject</code> supports xCal and xCard.</p>`

Control the terminal, the right way

atoum has two release managers

`Conclusion#</a> </h2> atoum has a bright future with exciting features! We sincerely hope this new direction will gather existing and new contributors 😄.</p> ❤️ open-source!</p>`

Hello fruux!

Conclusion #</a> </h2>
Bref</em>, I am working at fruux!</p>

Generate strings based on regular expressions

Rüsh Release

About memory pressure, lock contention, and Data-oriented Design

About memory pressure, lock contention, and Data-oriented Design

From 19k to 4.2M events/sec: story of a SQLite query optimisation

Sliding Sync at the Matrix Conference

Building a new site!

Site's features#</a> </h2> The site contains articles and series. A series is composed of several episodes. That's it. The URL patterns are the followings:</p> /articles/<article-id>/</code> to view an article,</li> /series/<series-id>/</code> to view all episodes of a series,</li>

Homepage#</a> </h3> The homepage provides:</p> the latest series, and</li>

Articles#</a> </h3> An article has some metadata like:</p> the publishing time,</li> the reading time,</li> keywords,</li>

Observability

I've loved Wasmer, I still love Wasmer

Bye bye WhatsApp, hello ?

Announcing the first Java library to run WebAssembly: Wasmer JNI

Announcing the first Postgres extension to run WebAssembly

Announcing the fastest WebAssembly runtime for Go: wasmer

🐘+🦀+🕸 php-ext-wasm: Migrating from wasmi to Wasmer

Bye bye Automattic, hello Wasmer

The PHP galaxy

The C galaxy

Rust 🚀 C#</a> </h2> </p> </figure> In order to use Rust from C, one may need 2 elements:</p>

C 🚀 executable#</a> </h2> </p> </figure> Now the Rust part is ready, the C part must be written to call it.</p>

The ASM.js galaxy

The WebAssembly galaxy

Prelude

Why Rust?#</a> </h2> That's an excellent question! Thanks for asking. I can summarize my choice with a bullet list:</p> It is fast, and we need speed,</li> It is memory safe, and also memory efficient,</li> No garbage collector, which simplifies memory management across environments,</li>

The parser#</a> </h2> The parser is written in Rust. It relies on the fabulous nom library</a>.</p> </p> nom will happily take a byte out of your files</em> 🙂</p> </figcaption> </figure>

How Automattic (WordPress.com & co.) partly moved away from PHPUnit to atoum?

One conference per day, for one year (2017)

Random thoughts about `::class` in PHP

Automattic, Grand Meetup 2017

atoum supports TeamCity

Glance#</a> </h2> The default CLI report looks like this:</p> </p>

Export functions in PHP à la Javascript

Finite-State Machine as a Type System illustrated with a store product

The Product FSM#</a> </h2> A product in a store might have the following states:</p> Active: Can be purchased,</li> Inactive: Has been cancelled or discontinued (a discontinued product can no longer be purchased),</li> Purchased and renewable,</li> Purchased and not renewable,</li>