Using MoonBit with Golem Cloud

2025-01-03T00:00:00+00:00

Introduction</h2>
MoonBit</a>, a new programming language has been open sourced a few weeks ago - see this blog post</a>. MoonBit is an exciting modern programming language that natively supports WebAssembly, including the component model - this makes it a perfect fit for writing applications for Golem Cloud</a>.
In this post I'm exploring the current state of MoonBit and whether it is ready for writing Golem components, by implementing an example application more complex than a simple "hello world" example.
The application to be implemented is a simple collaborative list editor - on the launch event of Golem 1.0</a> I have live-coded the same example using three different programming languages (TypeScript, Rust and Go) for the three main modules it requires. In this post I am implementing all three using MoonBit, including the e-mail sending feature that was omitted from the live demo due to time constraints.
The application can handle an arbitrary number of simultaneously open lists. Each list consists of a list of string items. These items can be appended, inserted and deleted simultaneously by multiple users; the current list state can be queried any time, as well as the active connections (users who can perform editing operations on the list). Modification is only allowed for connected editors, and there is a poll</code> function exposed for them which returns the new changes since the last poll call. Lists can be archived, in which case they are no longer editable and their contents are saved in a separate list archive. Then the list itself can be deleted, its last state remains stored forever in the archive. An additional feature is that if a list is not archived and there were no changes for a certain period of time, all the connected editors are notified by sending an email to them.

`Golem Architecture</h2> In Golem a good architecture to run this is to have three different golem components:`
the list</li>
the archive</li>
the email notifier</li> </ul>
These are compiled WebAssembly components, each exporting a distinct set of functions. Golem provides APIs to invoke these functions from the external world (for example mapping them to a HTTP API) and also allows workers (instances of these components) to invoke each other. A component can have an arbitrary number of instances, each such worker being identified by a unique name.
We can use this feature to have a very simple and straightforward implementation of the list editor - each document (editable list) will be mapped to its own worker, identified by the list's identifier. This way our list component only has to deal with a single list; scaling it up to handle multiple (possibly even millions) of lists is done automatically by Golem.
For archiving lists, we want to store each archived list in a single place - so we are going to have only a single instance of our archive component, where each archived list information is sent to. This singleton worker can store the archived lists in some database if needed - but because Golem's durable execution guarantees, it is enough to just store them in memory (one important exception is if we want to store a really large amount of archived lists not fitting in a single worker's memory). Golem guarantees that the worker's state is restored in any case of failure or rescaling event so the archive component can really remain very simple.
Finally, because Golem workers are single threaded and does not support async calls overlapping with its invocations at the moment, we need a third component to implement the delayed email sending functionality. There will be an email sending worker corresponding to each list worker and this worker will be suspended for an extended period of time (the amount we want to wait before sending out the email). Again, because of Golem's durable execution feature we can just "sleep" for an arbitrary long time in this component and we don't need to care about what can happen to our execution environment during that long period.
Initial MoonBit implementation</h2>
Before going into details of how to develop Golem components with MoonBit, let's try to implement the above described components in this new language, without any Golem or WebAssembly specifics.
First we create a new `lib</code> project using moon new</code>. This creates a new project with a single package. To match our architecture let's start by creating multiple packages, one for each component to develop (list</code>, archive</code>, email</code>)`
`We create a folder for each package, with a moon.pkg.json</code> in each:`
`{ "import": [ ] } </code></pre>`
List model</h3>
Let's start by modelling our list. The edited "document" itself is just an array of strings:
`struct Document { mut items: Array[String] } </code></pre> We can implement methods on Document</code> corresponding to the document editing operations we want to support. On this level we don't care about collaborative editing or connected users, just model our document as a pure data structure:`
///| Creates an empty document pub fn Document::new() -> Document { { items: [] } } ///| Adds a new item to the document pub fn add(self : Document, item : String) -> Unit { if self.items.search(item).is_empty() { self.items.push(item) } } ///| Deletes an item from the document pub fn delete(self : Document, item : String) -> Unit { self.items = self.items.filter(fn(i) { item != i }) } ///| Inserts an item to the document after an existing item. If `after` is not in the document, the new item is inserted at the end. pub fn insert(self : Document, after~ : String, value~ : String) -> Unit { let index = self.items.search(after) match index { Some(index) => self.items.insert(index + 1, value) None => self.add(value) } } ///| Gets a view of the document's items pub fn get(self : Document) -> ArrayView[String] { self.items[:] } ///| Iterates the items in the document pub fn iter(self : Document) -> Iter[String] { self.items.iter() } </code></pre> We can also use MoonBit's built-in test feature to write unit tests for this. The following test contains an assertion that the initial document is empty:
test "new document is empty" { let empty = Document::new() assert_eq!(empty.items, []) } </code></pre> With the inspect</code> function tests can use snapshot values to compare values with. The moon</code> CLI tool and the IDE integration provides a way to automatically update the snapshot values (the content=</code> part) in these test functions when needed: test "basic document operations" { let doc = Document::new() ..add("x") ..add("y") ..add("z") ..insert(after="y", value="w") ..insert(after="a", value="b") ..delete("z") ..delete("f") inspect!( doc.get(), content= #|["x", "y", "w", "b"] , ) } </code></pre> List editor state</h3> The next step is to implement the editor state management on top of this Document</code> type. As a reminder, we decided that every instance (Golem worker) of the list component will be only responsible for editing a single list. So we don't need to care about storing and indexing the lists, or routing connections to the corresponding node where the list state is - this is all going to be managed by Golem. What we need to do, however, is write stateful code to handle connecting and disconnecting users ("editors"), adding some validation on top of the document editing API so only connected editors can make changes, and collect change events for the polling API. We can start by defining a new datatype holding our document editing state: ///| Document state struct State { document : Document connected : Map[ConnectionId, EditorState] mut last_connection_id : ConnectionId mut archived : Bool mut email_deadline : @datetime.DateTime mut email_recipients : Array[EmailAddress] } </code></pre> Beside the actual document we are going to store: A map of connected editors, with some per-editor state associated with them</li> The last used connection ID so we can always generate a new unique one</li> Whether the document has been archived or not</li> When should we send out the email notification, and to what recipients</li> </ul> So far we have only defined the Document</code> type so let's continue by specifying all these other types used in State</code>s fields. ConnectionId</code> is going to be a newtype wrapping an integer: ///| Identifier of a connected editor type ConnectionId Int derive(Eq, Hash) ///| Generates a next unique connection ID fn next(self : ConnectionId) -> ConnectionId { ConnectionId(self._ + 1) } </code></pre> We want to use this type as a key of a Map</code> so we need instances of the Eq</code> and Hash</code> type classes. MoonBit can derive it for us automatically for newtypes. In addition to that, we also define a method called next</code> that generates a new connection ID with an incremented value. The EditorState</code> structure holds information for each connected editor. To keep things simple, we only store the editor's email address and a buffer of change events since the last call to poll</code>. An email address is a newtype of a String</code>: ///| Email address of a connected editor type EmailAddress String </code></pre> The Change</code> enum describes the possible changes made to the document: ///| An observable change of the edited document enum Change { Added(String) Deleted(String) Inserted(after~ : String, value~ : String) } derive(Show) </code></pre> Deriving Show</code> (or implementing it by hand) makes it possible to use the inspect</code> test function to compare string snapshots of array of changes with the results of our poll</code> function. Finally, let's define EditorState</code> using these two new types: ///| State per connected editor struct EditorState { email : EmailAddress mut events : Array[Change] } </code></pre> The email</code> field never changes of a connected editor - but the events</code> array is, as every call to poll</code> will reset this so the next poll returns only the new changes. To be able to do this, we have to mark it as mut</code>-able. The last new type we need to introduce for State</code> is something representing a point in time. MoonBit's core</code> standard library does not have currently anything for this, but there is already a package database, mooncakes</a>, with published MoonBit packages. Here we can find a package called datetime</code></a>. Adding it to our project can be done with the moon</code> CLI: moon add suiyunonghen/datetime </code></pre> and then importing it into the list</code> package by modifying its moon.pkg.json</code>: { "import": [ "suiyunonghen/datetime" ] } </code></pre> With this we can refer to the DateTime</code> type in this package using @datetime.DateTime</code>. Before starting to implement methods for State</code>, we have to think about error handling too - some of the operations on State</code> may fail, for example if a wrong connection ID is used, or a document editing operation comes in for an already archived list. MoonBit has built-in support for error handling, and it starts by defining our own error type in the following way: ///| Error type for editor state operations type! EditorError { ///| Error returned when an invalid connection ID is used InvalidConnection(ConnectionId) ///| Error when trying to modify an already archived document AlreadyArchived } </code></pre> With this we are ready to implement the collaborative list editor! I'm not going to list all the methods of State</code> in this post, but the full source code is available on GitHub</a>. The connect</code> method associates a new connection ID with the connected user, and also returns the current document state. This is important to be able to use the results of poll</code> - the returned list of changes have to be applied to exactly this document state on the client side. ///| Connects a new editor pub fn connect( self : State, email : EmailAddress ) -> (ConnectionId, ArrayView[String]) { let connection_id = self.last_connection_id.next() self.last_connection_id = connection_id self.connected.set(connection_id, EditorState::new(email)) (connection_id, self.document.get()) } </code></pre> The editing operations are more interesting. They build on top of the editing operations we already defined for Document</code>, but in addition to that, they all perform the following tasks: Validating the connection ID</li> Validating that the document is not archived yet</li> Adding a Change</code> event to each connected editor's state</li> Updating the email_deadline</code> and email_recipients</code> fields, as each editing operation resets the timeout for sending out the emails</li> </ul> Let's go through these steps one by one. For validations, we define two helper methods as we want to reuse them in all editing methods: ///| Fails if the document is archived fn ensure_not_archived(self : State) -> Unit!EditorError { guard not(self.archived) else { raise AlreadyArchived } } ///| Fails if the given `connection_id` is not in the connection map fn ensure_is_connected( self : State, connection_id : ConnectionId ) -> Unit!EditorError { guard self.connected.contains(connection_id) else { raise InvalidConnection(connection_id) } } </code></pre> The Unit!EditorError</code> result type indicates that these methods can fail with EditorError</code>. We can also define a helper method for adding a change event to each connected editor's state: ///| Adds a change event to each connected editor's state fn add_event(self : State, change : Change) -> Unit { for editor_state in self.connected.values() { editor_state.events.push(change) } } </code></pre> And finally one for resetting the email-sending deadline and list of recipients: ///| Updates the `email_deadline` and `email_recipients` fields after an update. fn update_email_properties(self : State) -> Unit { let now = @datetime.DateTime::from_unix_mseconds(0) // TODO let send_at = now.inc_hour(12) let email_list = self.connected_editors() self.email_deadline = send_at self.email_recipients = email_list } </code></pre> Note that the datetime</code> library we imported has no concept of getting the current date and time which we need for this function to work properly. We are going to address this problem once we start targeting WebAssembly (and Golem) as getting the current system time is something depending on the target platform. With these helper functions, implementing the editor functions, for example add</code>, is straightforward: ///| Adds a new element to the document as a connected editor pub fn add( self : State, connection_id : ConnectionId, value : String ) -> Unit!EditorError { self.ensure_not_archived!() self.ensure_is_connected!(connection_id) self.document.add(value) self.add_event(Change::Added(value)) self.update_email_properties() } </code></pre> Implementing poll</code> is also easy, as we already maintain the list of changes per connection, we just need to reset it after each call: ///| Returns the list of changes occurred since the last call to poll pub fn poll( self : State, connection_id : ConnectionId ) -> Array[Change]!EditorError { match self.connected.get(connection_id) { Some(editor_state) => { let events = editor_state.events editor_state.events = [] events } None => raise InvalidConnection(connection_id) } } </code></pre> List archiving</h3> As mentioned in the introduction, we are going to have a singleton Golem worker to store archived lists. At this point we are still not having anything Golem or WebAssembly specific, like RPC calls, so let's just implement the list archive store in the simplest possible way. As I wrote earlier, we can simply store the archived lists in memory, and Golem will take care of persisting it. We don't want to reuse the same Document</code> type as it represents a live, editable document. Instead we define a few new types in the archive</code> package: ///| Unique name of a document type DocumentName String derive(Eq, Hash) ///| Show instance for DocumentName impl Show for DocumentName with output(self, logger) { self._.output(logger) } ///| A single archived immutable document, encapsulating the document's name and its items struct ArchivedDocument { name : DocumentName items : Array[String] } derive(Show) ///| Archive is a list of archived documents struct Archive { documents : Map[DocumentName, ArchivedDocument] } </code></pre> All we need is an insert</code> method and a way to iterate all the archived documents: ///| Archives a named document pub fn insert( self : Archive, name : DocumentName, items : Array[String] ) -> Unit { self.documents.set(name, { name, items }) } ///| Iterates all the archived documents pub fn iter(self : Archive) -> Iter[ArchivedDocument] { self.documents.values() } </code></pre> With this done, we first implement the list archiving in the list</code> package using simple method calls. Later we are going to replace it with Golem's own Worker to Worker communication. As there will be a singleton archive worker, we can simulate this for now by having a top-level Archive</code> instance in the archive</code> package: pub let archive: Archive = Archive::new() </code></pre> And calling this in our State::archive</code> method: pub fn archive(self : State) -> Unit { self.archived = true let name = @archive.DocumentName("TODO") @archive.archive.insert(name, self.document.iter().to_array()) } </code></pre> Note that so far we have no way to know the document's name in State</code> - we did not store it anywhere. This is intentional, as we discussed earlier the worker name will be used as the document's unique identifier. Getting the worker's name will be done in a Golem specific way once we get there. Sending an email</h3> We already prepared some part of the email sending logic in the State</code> type: it has a deadline and a list of recipients. The idea is that we start an email sending worker when a new list is created, and this runs in parallel to our editing session, in a loop. In this loop it first queries the deadline and list of recipients from our list editing state, and then just sleeps until that given deadline. When it wakes up (after 12 hours), it queries the list again, and if it is past the deadline, it means there were no further editing operations in the meantime. Then it sends the notification emails to the list of recipients. There is no library on mooncakes</a> yet for sending emails or even for making HTTP requests, so this is something we will have to do ourselves. Also, spawning the worker to run it in parallel is something Golem specific, so at this point we are not going to implement anything for the email</code> package. We will get back to it once the rest of the application is already compiled as Golem components. Compiling as Golem Components</h2> It is time to try to compile our code as Golem components - these are WebAssembly components (using the component model</a>) exporting an API described with the Wasm Interface Type (WIT) language. Bindings</h3> In the current world of the WASM component model, components are defined in a spec-first way - first we write the WIT files describing types and exported interfaces, and then use a binding generator to generate language-specific glue code from them. Fortunately the wit-bindgen</code> tool</a> already has MoonBit support, so we can start by installing the latest version: cargo install wit-bindgen-cli </code></pre> Note that Golem's documentation recommends an older, specific version of wit-bindgen</code> - but that version did not support MoonBit yet. The new version should work well but the example codes for Golem were not tested with it. We will reuse the WIT definitions that were created for the Golem 1.0 launch demo. For the list</code> component, it is the following: package demo:lst; interface api { record connection { id: u64 } record insert-params { after: string, value: string } variant change { added(string), deleted(string), inserted(insert-params) } add: func(c: connection, value: string) -> result<_, string>; delete: func(c: connection, value: string) -> result<_, string>; insert: func(c: connection, after: string, value: string) -> result<_, string>; get: func() -> list<string>; poll: func(c: connection) -> result<list<change>, string>; connect: func(email: string) -> tuple<connection, list<string>>; disconnect: func(c: connection) -> result<_, string>; connected-editors: func() -> list<string>; archive: func(); is-archived: func() -> bool; } interface email-query { deadline: func() -> option<u64>; recipients: func() -> list<string>; } world lst { // .. imports to be explained later .. export api; export email-query; } </code></pre> This interface definition exports two APIs - one is the public API of our list editors, very similar to the methods we already implemented for the State</code> type. The other is an internal API for the email</code> component to query the deadline and recipients as it was explained earlier. For simplicity, we are using string</code> as an error type on the public API. For the archive</code> component, we define a much simpler interface: package demo:archive; interface api { record archived-list { name: string, items: list<string> } store: func(name: string, items: list<string>); get-all: func() -> list<archived-list>; } world archive { // .. imports to be explained later .. export api; } </code></pre> And finally, for the email</code> component: package demo:email; interface api { use golem:rpc/types@0.1.0.{uri}; send-email: func(list-uri: uri); } world email { // .. imports to be explained later .. export api; } </code></pre> Here we are using a Golem specific type: uri</code>. This is needed because the email</code> workers need to call the specific list</code> worker it was spawned from. The details of this will be explained later. These WIT definitions need to be put in wit</code> directories of each package, and dependencies in subdirectories of wit/deps</code>. Check the repository</a> for reference. We started with defining a single MoonBit module (identified by moon.mod.json</code> in the root) and just created list</code>, email</code> and archive</code> as internal packages. At this point we have to change this because we need to have a separate module for each chunk of code we want to compile to a separate Golem component. By running wit-bindgen</code> in each of the three subdirectories (shown below), it actually generates module definitions for us. We reorganize the directory structure a bit, moving src/archive</code> to archive</code> etc, and moving the previously written source code to archive/src</code>. This way the generated bindings and our hand-written implementation will be put next to each other. We can also delete the top-level module definition JSON. Now in all the three directories we can generate the bindings: wit-bindgen moonbit wit </code></pre> Note that once we start modifying the generated stub.wit</code> files, running this command again will overwrite our changes. To avoid that, it can be run in the following way: wit-bindgen moonbit wit --ignore-stub </code></pre> With this done, moon build --target wasm </code></pre> will compile a WASM module for us in ./target/wasm/release/build/gen/gen.wasm</code>. This is not yet a WASM component - so it's not ready to be used directly in Golem. To do so, we will have to use another command line tool, wasm-tools</code></a> to convert this module into a component that self-describes its higher level exported interface. WIT dependencies</h3> We are going to need to depend on some WIT packages, some from WASI (WebAssembly System Interface) to access things like environment variables and the current date/time, and some Golem specific ones to implement worker-to-worker communication. The simplest way to get the appropriate version of all the dependencies Golem provides is to use Golem's "all" packaged interfaces with the wit-deps</code></a> tool. So first we install wit-deps</code>: cargo install wit-deps-cli </code></pre> And create a deps.toml</code> file in each wit</code> directory we have created with the following contents: all = "https://github.com/golemcloud/golem-wit/archive/main.tar.gz" </code></pre> And finally we run the following command to fill the wit/deps</code> directory: wit-deps update </code></pre> Implementing the exports</h3> Before setting up this compilation chain let's see how we can connect the generated bindings with our existing code. Let's start with the archive</code> component, as it is the simplest one. The binding generator creates a stub.mbt</code> file at archive/gen/interface/demo/archive/api/stub.mbt</code> with the two exported functions to be implemented. Here we face the usual question when working with code generators: we have a definition of archived-list</code> in WIT and the binding generator generated the following MoonBit definition from it: // Generated by `wit-bindgen` 0.36.0. DO NOT EDIT! pub struct ArchivedList { name : String; items : Array[String] } derive() </code></pre> But we already defined a very similar structure called ArchivedDocument</code>! The only differences are the use of the DocumentName</code> newtype and that our version was deriving a Show</code> instance. We could decide to give up using the newtype, and use the generated type in our business logic, or we could keep the generated types separated from our actual code. (This is not really specific to MoonBit or the WASM tooling, we face the same issue with any code generator based approach). In this post I will keep the generated code separate from our already written business logic, and just show how to implement the necessary conversions to implement the stub.mbt</code> file(s). The first exported function to implement is called store</code>. We can implement it by just calling insert</code> on our singleton top level Archive</code> as we did before when we directly wired the archive</code> package to the list</code> package: pub fn store(name : String, items : Array[String]) -> Unit { @src.archive.insert(@src.DocumentName(name), items) } </code></pre> Note that we need to import our main archive</code> source in the stub's package JSON: { "import": [ { "path" : "demo/archive/ffi", "alias" : "ffi" }, { "path" : "demo/archive/src", "alias" : "src" } ] } </code></pre> The second function to be implemented needs to convert between the two representations of an archived document: pub fn get_all() -> Array[ArchivedList] { @src.archive .iter() .map(fn(archived) { { name: archived.name._, items: archived.items } }) .to_array() } </code></pre> Note that for this to work, we also have to make the previously defined struct ArchivedDocument</code> a pub struct</code> otherwise we cannot access it's name</code> and items</code> fields from the stub package. (Note: at the time of writing https://github.com/bytecodealliance/wit-bindgen/pull/1100 was not merged yet, and it is needed for the binding generator to produce working code with Golem wasm-rpc; Until it is merged, it is possible to compile the fork and use it directly) The same way we can implement the two generated stubs in the list</code> module (in list/gen/interface/demo/lst/api/stub.mbt</code> and list/gen/interface/demo/lst/emailQuery/stub.mbt</code>) using our existing implementation of State</code>. One interesting details is how we can map the EditorError</code> failures into the string errors used in the WIT definition. First we define a to_string</code> method for EditorError</code>: pub fn to_string(self : EditorError) -> String { match self { InvalidConnection(id) => "Invalid connection ID: \{id._}" AlreadyArchived => "Document is already archived" } } </code></pre> Then use ?</code> and map_err</code> in the stubs: pub fn add(c : Connection, value : String) -> Result[Unit, String] { @src.state .add?(to_connection_id(c), value) .map_err(fn(err) { err.to_string() }) } </code></pre> Using host functions</h3> When we implemented the update_email_properties</code> function earlier, we could not properly query the current time to calculate the proper deadline. Now that we are targeting Golem, we can use the WebAssembly system interface (WASI) to access things like the system time. One way would be to use the published wasi-bindings</code> package</a> but as we are already generating bindings from WIT anyway, we can just use our own generated bindings to imported host functions. First, we need to import the WASI wall-clock interface into our WIT world: world lst { export api; export email-query; import wasi:clocks/wall-clock@0.2.0; } </code></pre> Then we regenerate the bindings (make sure to use --ignore-stub</code> to avoid rewriting our stub implementation!) and import it into our main (src</code>) package: { "import": [ "suiyunonghen/datetime", { "path" : "demo/lst/interface/wasi/clocks/wallClock", "alias" : "wallClock" } ] } </code></pre> With that we can call the WASI now</code> function to query the current system time, and convert it to the datetime</code> module's DateTime</code> type which we were using before: ///| Queries the WASI wall clock and returns it as a @datetime.DateTime /// /// Note that DateTime has only millisecond precision fn now() -> @datetime.DateTime { let wasi_now = @wallClock.now() let base_ms = wasi_now.seconds.reinterpret_as_int64() * 1000; let nano_ms = (wasi_now.nanoseconds.reinterpret_as_int() / 1000000).to_int64(); @datetime.DateTime::from_unix_mseconds(base_ms + nano_ms) } </code></pre> Golem app manifest</h2> In the next step of our implementation we will have to connect our two existing components: list</code> and archive</code> in a way that list</code> can do remote procedure calls to archive</code>. With the same technique we will be able to implement the third component, email</code> which needs to be both called from list</code> (when started) and called back (when getting the deadline and recipients). Golem has tooling supporting this - but before trying to use it, let's convert our project into a golem application described by app manifests. This will enable us to use golem-cli</code> to generate the necessary files for worker-to-worker communication, and will also make it easier to deploy the compiled components into Golem. The build steps</h3> To build a single MoonBit module into a Golem component, without any worker-to-worker communication involved, we have to perform the following steps: (Optionally) regenerate the WIT bindings with wit-bindgen ... --ignore-stub</code></li> Compile the MoonBit source code into a WASM module with moon build --target wasm</code></li> Embed the WIT specification into a custom WASM section using wasm-tools component embed</code></li> Convert the WASM module into a WASM component using wasm-tools component new</code></li> </ul> When we will start to use worker-to-worker communication it will require even more steps, as we are going to generate stub WIT interfaces, and compile and link multiple WASM components. An earlier version of this was described in the Worker to Worker communication in Golem</a> blog post last year. The Golem app manifest and the corresponding CLI tool, introduced with Golem 1.1, automates all these steps for us. Manifest template</h3> We start by creating a root app manifest, golem.yaml</code>, in the root of our project. We start by setting up a temporary directory and a shared directory for the WIT dependencies we previously fetched with wit-deps</code>: # Schema for IDEA: # $schema: https://schema.golem.cloud/app/golem/1.1.0/golem.schema.json # Schema for vscode-yaml # yaml-language-server: $schema=https://schema.golem.cloud/app/golem/1.1.0/golem.schema.json tempDir: target/golem-temp witDeps: - common-wit/deps </code></pre> By moving our previous deps.toml</code> into common-wit</code> and doing a wit-deps update</code> in the root, we can fill up this deps</code> directory with all the WASI and Golem APIs we need. Then we define a template for building MoonBit components with Golem CLI. In the template, we are going to define two profiles - one for doing a release build and one for debug. In the post I'm only going to show the release build. It starts by specifying some directory names and where the final WASM files will be placed: templates: moonbit: profiles: release: sourceWit: wit generatedWit: wit-generated componentWasm: ../target/release/{{ componentName }}.wasm linkedWasm: ../target/release/{{ componentName }}-linked.wasm </code></pre> These directories are relative to the components subdirectories (for example archive</code>) so what we say here is that once all the components are built, they al will be put in the root target/release</code> directory. Then we specify the build steps, described in the previous section: build: - command: wit-bindgen moonbit wit-generated --ignore-stub --derive-error --derive-show sources: - wit-generated targets: - ffi - interface - world - command: moon build --target wasm - command: wasm-tools component embed wit-generated target/wasm/release/build/gen/gen.wasm -o ../target/release/{{ componentName }}.module.wasm --encoding utf16 mkdirs: - ../target/release - command: wasm-tools component new ../target/release/{{ componentName }}.module.wasm -o ../target/release/{{ componentName }}.wasm </code></pre> Finally, we can define additional directories to be cleaned by the golem app clean</code> command, and we can even define custom commands to be executed with golem app xxx</code>: clean: - target - wit-generated customCommands: update-deps: - command: wit-deps update dir: .. regenerate-stubs: - command: wit-bindgen moonbit wit-generated </code></pre> With this set, we can add a new MoonBit module* to this Golem project by creating a golem.yaml</code> in its directory - so archive/golem.yaml</code> and list/golem.yaml</code> for now. In these sub-manifests we can use the above defined template to tell Golem that this is a MoonBit module. It is possible to mix Golem components written in different languages in a single application. For example the archive</code> component's manifest will look like this: # Schema for IDEA: # $schema: https://schema.golem.cloud/app/golem/1.1.0/golem.schema.json # Schema for vscode-yaml # yaml-language-server: $schema=https://schema.golem.cloud/app/golem/1.1.0/golem.schema.json components: archive: template: moonbit </code></pre> Building the components</h3> With this set, the whole application (with its two already written components) can be compiled by simply saying golem app build </code></pre> There are a few organizational things to do first, as golem app build</code> does some transformations on the WIT definitions. This means that our previously written stubs are a wrong place. The easiest way to fix this is to delete all the wit-bindgen generated directories (but first backup the hand-written stubs!) and then copy back the stubs into the new directories created. We are not going to discuss this in more details here. The blog post incrementally discovers how to build Golem applications with MoonBit and introduces the app manifest in a late stage, but the recommended way is to start immediately with an app manifest and then there is no need to do these fixes. First try</h3> Running the build command results in two WASM files that are ready to be used with Golem! Although they are not able to communicate with each other yet (so the archiving functionality does not work), it is already possible to try them out with Golem. To do so, we can start Golem locally by downloading the latest release of single-executable Golem</a> or using our hosted Golem Cloud. With the golem</code> binary, we just use the following command to start up the services locally: $ golem start -vv </code></pre> Then, from the root of our project, we can upload the two compiled components using the same command: $ golem component add --component-name archive Added new component archive Component URN: urn:component:bde2da89-75a8-4adf-953f-33b360c978d0 Component name: archive Component version: 0 Component size: 9.35 KiB Created at: 2025-01-03 15:09:05.166785 UTC Exports: demo:archive-interface/api.{get-all}() -> list<record { name: string, items: list<string> }> demo:archive-interface/api.{store}(name: string, items: list<string>) </code></pre> and $ golem component add --component-name list Added new component list Component URN: urn:component:b6420554-62b5-4902-8994-89c692a937f7 Component name: list Component version: 0 Component size: 28.46 KiB Created at: 2025-01-03 15:09:09.743733 UTC Exports: demo:lst-interface/api.{add}(c: record { id: u64 }, value: string) -> result<_, string> demo:lst-interface/api.{archive}() demo:lst-interface/api.{connect}(email: string) -> tuple<record { id: u64 }, list<string>> demo:lst-interface/api.{connected-editors}() -> list<string> demo:lst-interface/api.{delete}(c: record { id: u64 }, value: string) -> result<_, string> demo:lst-interface/api.{disconnect}(c: record { id: u64 }) -> result<_, string> demo:lst-interface/api.{get}() -> list<string> demo:lst-interface/api.{insert}(c: record { id: u64 }, after: string, value: string) -> result<_, string> demo:lst-interface/api.{is-archived}() -> bool demo:lst-interface/api.{poll}(c: record { id: u64 }) -> result<list<variant { added(string), deleted(string), inserted(record { after: string, value: string }) }>, string> demo:lst-interface/email-query.{deadline}() -> option<u64> demo:lst-interface/email-query.{recipients}() -> list<string> </code></pre> We can try out the archive</code> component by first invoking the store</code> function, and then the get-all</code> function, using the CLI's worker invoke-and-await</code> command: $ golem worker invoke-and-await --worker urn:worker:bde2da89-75a8-4adf-953f-33b360c978d0/archive --function 'demo:archive-interface/api.{store}' --arg '"list1"' --arg '["x", "y", "z"]' Empty result. $ golem worker invoke-and-await --worker urn:worker:bde2da89-75a8-4adf-953f-33b360c978d0/archive --function 'demo:archive-interface/api.{get-all}' Invocation results in WAVE format: - '[{name: "list1", items: ["x", "y", "z"]}]' </code></pre> Similarly we can try out the list</code> component, keeping in mind that the worker name is the list name: </code></pre> When we try out list, we get an error (and if we used the debug</code> profile - using --build-profile debug</code> then we also get a nice call stack): Failed to create worker b6420554-62b5-4902-8994-89c692a937f7/list6: Failed to instantiate worker -1/b6420554-62b5-4902-8994-89c692a937f7/list6: error while executing at wasm backtrace: 0: 0x19526 - wit-component:shim!indirect-wasi:clocks/[email protected] 1: 0x414b - <unknown>!demo/lst/interface/wasi/clocks/wallClock.wasmImportNow 2: 0x4165 - <unknown>!demo/lst/interface/wasi/clocks/wallClock.now 3: 0x42c1 - <unknown>!demo/lst/src.now 4: 0x433d - <unknown>!@demo/lst/src.State::update_email_properties 5: 0x440e - <unknown>!@demo/lst/src.State::new 6: 0x5d81 - <unknown>!init/38 </code></pre> The reason is we are creating a global variable of State</code> and in its constructor we are tryting to call a WASI function (to get the current date-time). This is too early for that; so let's modify the State::new</code> method to not call any host functions: ///| Creates a new empty document editing state pub fn State::new() -> State { let state = { document: Document::new(), connected: Map::new(), last_connection_id: ConnectionId(0), archived: false, email_deadline: @datetime.DateTime::from_unix_mseconds(0), // Note: can't use now() here because it will run in initialization-time (due to the global `state` variable) email_recipients: [], } state } </code></pre> This fixes the issue! Now we can create and play with our collaboratively editable lists: $ golem worker start --component urn:component:b6420554-62b5-4902-8994-89c692a937f7 --worker-name list7 Added worker list7 Worker URN: urn:worker:b6420554-62b5-4902-8994-89c692a937f7/list7 Component URN: urn:component:b6420554-62b5-4902-8994-89c692a937f7 Worker name: list7 $ golem worker invoke-and-await --component urn:component:b6420554-62b5-4902-8994-89c692a937f7 --worker-name list7 --function 'demo:lst-interface/api.{connect}' --arg '"[email protected]"' Invocation results in WAVE format: - '({id: 1}, [])' $ golem worker invoke-and-await --component urn:component:b6420554-62b5-4902-8994-89c692a937f7 --worker-name list7 --function 'demo:lst-interface/api.{add}' --arg '{ id: 1}' --arg '"a"' Invocation results in WAVE format: - ok $ golem worker invoke-and-await --component urn:component:b6420554-62b5-4902-8994-89c692a937f7 --worker-name list7 --function 'demo:lst-interface/api.{add}' --arg '{ id: 1}' --arg '"b"' Invocation results in WAVE format: - ok $ golem worker invoke-and-await --component urn:component:b6420554-62b5-4902-8994-89c692a937f7 --worker-name list7 --function 'demo:lst-interface/api.{connect}' --arg '"[email protected]"' Invocation results in WAVE format: - '({id: 2}, ["a", "b"])' </code></pre> Worker to Worker communication</h2> List calling archive</h3> The first worker-to-worker communication we want to set up is the list</code> component calling the archive</code> component - basically, when we call archive()</code> on the list, it needs to call store</code> in a singleton archive worker, sending its data to it. The first step is to simply state this dependency in the app manifest of list</code>: components: list: template: moonbit dependencies: list: - type: wasm-rpc target: archive </code></pre> Running golem app build</code> after this will run a lot of new build steps - including generating and compiling some Rust source code, which is something that will no longer be needed in the next release of Golem. We are not going into details of what is generated for worker to worker communication in this post - what is important is that after this change, and running build once, we can import a generated stub of our archive</code> component in our list</code> component's moonbit package: { "import": [ "suiyunonghen/datetime", { "path" : "demo/lst/interface/wasi/clocks/wallClock", "alias" : "wallClock" }, { "path" : "demo/lst/interface/demo/archive_stub/stubArchive", "alias": "stubArchive" }, { "path" : "demo/lst/interface/golem/rpc/types", "alias": "rpcTypes" } ] } </code></pre> Then we can add the following code into our archive</code> function to call the remote worker: let archive_component_id = "bde2da89-75a8-4adf-953f-33b360c978d0"; // TODO let archive = @stubArchive.Api::api({ value: "urn:worker:\{archive_component_id}/archive"}); let name = "TODO"; // TODO archive.blocking_store(name, self.document.iter().to_array()) </code></pre> In line 2 we construct the remote interface by pointing to a specific worker, by using the component ID and the worker's name. (In the next Golem release this is going to be simplified by being able to use the component's name instead). In line 5 we call the remote store</code> function. What is missing are two things: We should not hard-code the archive component's ID as it is automatically generated when the component is first uploaded to Golem</li> We need to know our own worker name to be used as the list's name</li> </ul> The solution to both is to use environment variables - Golem automatically sets the GOLEM_WORKER_NAME</code> environment variable to the worker's name, and we can manually provide values to workers through custom environment variables. This allows us to inject the component ID from the outside (until a more sophisticated configuration feature is added in Golem 1.2). We have already seen how we can use WASI to query the current date/time; we can use another WASI interface to get environment variables. So once again, we add an import to our WIT file: import wasi:cli/environment@0.2.0; </code></pre> Then run golem app build</code> to regenerate the bindings, and import it in the list/src</code> MoonBit package: { "path" : "demo/lst/interface/wasi/cli/environment", "alias": "environment" } </code></pre> and implement a helper function to get a specific key from the environment variables: ///| Gets an environment variable using WASI fn get_env(key : String) -> String? { @environment.get_environment() .iter() .find_first(fn(pair) { pair.0 == key }) .map(fn(pair) { pair.1 }) } </code></pre> We can use this to get the worker's name and the archive component ID: let archive_component_id = get_env("ARCHIVE_COMPONENT_ID").or("unknown"); // ... let name = get_env("GOLEM_WORKER_NAME").or("unknown"); </code></pre> When starting the list workers, we have to explicitly specify ARCHIVE_COMPONENT_ID</code>: $ golem worker start --component urn:component:b6420554-62b5-4902-8994-89c692a937f7 --worker-name list10 --env "ARCHIVE_COMPONENT_ID=bde2da89-75a8-4adf-953f-33b360c978d0" </code></pre> With that we can try connecting to the list, adding some items and then calling archive</code> on it, and finally calling get-all</code> on the archive worker - we can see that the remote procedure call works! List and email</h3> We haven't implemented the third component of the application yet - the one responsible for sending an email after some deadline. Setting up the component and the worker-to-worker communication works exactly the same as it was demonstrated above. The app manifest supports circular dependencies, so we can add say that list</code> depends on email</code> via wasm-rpc</code>, and also email</code> depends on list</code> via wasm-rpc</code>. We need to communicate in both directions. We will have to use the WASI monotonic-clock</code> interface's subscribe-instant</code> function to sleep until the given deadline. Without showing all the details, here is the MoonBit code implementing the single send-email</code> function we defined in the email.wit</code> file: ///| Structure holding an email sender's configuration pub(all) struct Email { list_worker_urn : String } ///| Run the email sending loop pub fn run(self : Email) -> Unit { while true { match self.get_deadline() { Some(epoch_ms) => { let now = @wallClock.now() let now_ms = now.seconds * 1000 + (now.nanoseconds.reinterpret_as_int() / 1000000).to_uint64() let duration_ms = epoch_ms.reinterpret_as_int64() - now_ms.reinterpret_as_int64() if duration_ms > 0 { sleep(duration_ms.reinterpret_as_uint64()) } else { send_emails(self.get_recipients()) } continue } None => break } } } </code></pre> We use the wall-clock</code> interface again to query the current time and calculate the duration to sleep for based on the deadline got from the corresponding list worker. The get_deadline</code> and get_recipients</code> methods are just using Golem's Worker to Worker communication as shown before: ///| Get the current deadline from the associated list worker fn get_deadline(self : Email) -> UInt64? { let api = @stubLst.EmailQuery::email_query({ value: self.list_worker_urn }) api.blocking_deadline() } ///| Get the current list of recipients from the associated list worker fn get_recipients(self : Email) -> Array[String] { let api = @stubLst.EmailQuery::email_query({ value: self.list_worker_urn }) api.blocking_recipients() } </code></pre> The two remaining interesting part are sleeping and sending emails. We can sleep by calling the subscribe-duration</code> function in the WASI monotonic-clock</code> package to get a pollable, and then poll for it. As we only pass a single pollable to the list, it won't return until the deadline we want to wait for expires: ///| Sleep for the given amount of milliseconds fn sleep(ms : UInt64) -> Unit { let ns = ms * 1000000 let pollable = @monotonicClock.subscribe_duration(ns) let _ = @poll.poll([pollable]) } </code></pre> On the list</code> side, we don't want to block until this email sending loop runs - as it would block our list from receiving new requests. The generated RPC stubs support this, we simply use the non-blocking version on the generated Api</code> type: if not(self.email_worker_started) { let email_component_id = get_env("EMAIL_COMPONENT_ID").or("unknown"); let name = get_env("GOLEM_WORKER_NAME").or("unknown") let self_component_id = get_env("GOLEM_COMPONENT_ID").or("unknown") let api = @stubEmail.Api::api({ value: "urn:worker:\{email_component_id}:\{name}"}) api.send_email({ value: "urn:worker:\{self_component_id}:\{name}"}) self.email_worker_started = true; } </code></pre> Sending emails</h2> Sending actual emails is a bit more difficult, as there are no HTTP client libraries in the MoonBit ecosystem at the moment. But Golem implements the WASI HTTP interface, so we can use the already demonstrated techniques to import WASI HTTP through WIT, generate bindings for it, and then use it from MoonBit code to send emails through a third party provider. In the example we are going to use Sendgrid</a> as a provider. This means we have to send a HTTP POST request to https://api.sendgrid.com/v3/mail/send</code> with a pre-configured authorization header, and a JSON body describing our email sending request. First we are going to define a few helper constants and functions to assemble the parts of the requests: const AUTHORITY : String = "api.sendgrid.com" const PATH : String = "/v3/mail/send" type! HttpClientError String </code></pre> The payload is a JSON, which can be constructed using MoonBit's built-in JSON literal feature. However in the WASI HTTP interface we have to write it out as a byte array. MoonBit strings are UTF-16 but SendGrid requires the payload to be in UTF-8. Unfortunately there isn't any string encoding library available for MoonBit yet, so we write a simple function that fails if any of the characters is not ASCII: ///| Converts a string to ASCII byte array if all characters are ASCII characters, otherwise fails fn string_to_ascii( what : String, value : String ) -> FixedArray[Byte]!HttpClientError { let result = FixedArray::makei(value.length(), fn(_) { b' ' }) for i, ch in value { if ch.to_int() < 256 { result[i] = ch.to_int().to_byte() } else { raise HttpClientError("The \{what} contains non-ASCII characters") } } result } </code></pre> With this we can construct the payload and we can also read the sendgrid API key from an environment variable: ///| Constructs a SendGrid send message payload as an ASCII byte array fn payload(recipients : Array[String]) -> FixedArray[Byte]!HttpClientError { let email_addresses = recipients .iter() .map(fn(email) { { "email": email, "name": email } }) .to_array() .to_json() let from : Json = { "email": "[email protected]", "name": "Daniel Vigovszky" } let json : Json = { "personalizations": [{ "to": email_addresses, "cc": [], "bcc": [] }], "from": from, "subject": "Collaborative list editor warning", "content": [ { "type": "text/html", "value": "The list opened for editing has not been changed in the last 12 hours", }, ], } let json_str = json.to_string() string_to_ascii!("constructed JSON body", json_str) } ///| Gets the SENDGRID_API_KEY environment variable as an ASCII byte array fn authorization_header() -> FixedArray[Byte]!HttpClientError { let key_str = @environment.get_environment() .iter() .find_first(fn(pair) { pair.0 == "SENDGRID_API_KEY" }) .map(fn(pair) { pair.1 }) .unwrap() string_to_ascii!( "provided authorization header via SENDGRID_API_KEY", key_str, ) } </code></pre> The next step is to create the data structures for sending out the HTTP request. In WASI HTTP, outgoing requests are modeled as WIT resources, which means we have to construct them with a constructor and call various methods to set properties of the request. All these methods have a Result</code> result type so our code is going to be quite verbose: let headers = @httpTypes.Fields::fields() headers .append("Authorization", authorization_header!()) .map_err(fn(error) { HttpClientError("Failed to set Authorization header: \{error}") }) .unwrap_or_error!() let request = @httpTypes.OutgoingRequest::outgoing_request(headers) request .set_authority(Some(AUTHORITY)) .map_err(fn(_) { HttpClientError("Failed to set request authority") }) .unwrap_or_error!() request .set_method(@httpTypes.Method::Post) .map_err(fn(_) { HttpClientError("Failed to set request method") }) .unwrap_or_error!() request .set_path_with_query(Some(PATH)) .map_err(fn(_) { HttpClientError("Failed to set request path") }) .unwrap_or_error!() request .set_scheme(Some(@httpTypes.Scheme::Https)) .map_err(fn(_) { HttpClientError("Failed to set request scheme") }) .unwrap_or_error!() let outgoing_body = request .body() .map_err(fn(_) { HttpClientError("Failed to get the outgoing body") }) .unwrap_or_error!() let stream = outgoing_body .write() .map_err(fn(_) { HttpClientError("Failed to open the outgoing body stream") }) .unwrap_or_error!() let _ = stream .blocking_write_and_flush(payload!(recipients)) .map_err(fn(error) { HttpClientError("Failed to write request body: \{error}") }) .unwrap_or_error!() let _ = outgoing_body .finish(None) .map_err(fn(_) { HttpClientError("Failed to close the outgoing body") }) .unwrap_or_error!() </code></pre> At this point we have our request</code> variable initialized with everything we need, so we can call the handle</code> function to initiate the HTTP request: let future_incoming_response = @outgoingHandler.handle(request, None) .map_err(fn(error) { HttpClientError("Failed to send request: \{error}") }) .unwrap_or_error!() </code></pre> Sending a request is an async operation and what we have a result here is just a handle for a future value we have to await somehow. As we don't want to do anything else in parallel in this example, we just write a loop that awaits for the result and checks for errors: while true { match future_incoming_response.get() { Some(Ok(Ok(response))) => { let status = response.status() if status >= 200 && status < 300 { break } else { raise HttpClientError("Http request returned with status \{status}") } } Some(Ok(Err(code))) => raise HttpClientError("Http request failed with \{code}") Some(Err(_)) => raise HttpClientError("Http request failed") None => { let pollable = future_incoming_response.subscribe() let _ = @poll.poll([pollable]) } } } </code></pre> We are ignoring the response body in this example - but in other applications, response</code> could be used to open an incoming body stream and read chunks from it. With this we implemented the simplest possible way to call the SendGrid API for sending an e-mail using WASI HTTP provided by Golem. Debugging</h2> When compiled to debug (using golem app build --build-profile debug</code>), Golem shows a nice stack trace when something goes wrong in a MoonBit component. Another useful way to observe a worker is to write a log in it, which can be realtime watched (or queried later) using tools like golem worker connect</code> or the Golem Console. The best way to write logs from MoonBit is to use the WASI Logging interface. We can import it as usual in our WITs: import wasi:logging/logging; </code></pre> and then to our MoonBit packages: "demo/archive/interface/wasi/logging/logging" </code></pre> and then write out log messages of various levels from our application logic: let recipients = self.get_recipients(); @logging.log(@logging.Level::INFO, "", "Sending emails to recipients: \{recipients}") match send_emails?(recipients) { Ok(_) => @logging.log(@logging.Level::INFO, "", "Sending emails succeeded") Err(error) => @logging.log(@logging.Level::ERROR, "", "Failed to send emails: \{error}") } </code></pre> Conclusion</h2> MoonBit is a nice new language that is quite powerful and expressive, and seems to be a very good fit for developing applications for Golem. The resulting WASM binaries are very small - a few tens of kilobytes for this application (only increased by the generated Rust stubs - but those are going away soon). A few things in the language felt a little bit inconvenient - but maybe it is just a matter of personal taste - mostly the JSON files describing MoonBit packages, the anonymous function syntax and the way the built-in formatter organizes things. I'm sure some of these, especially the tooling, will greatly improve in the future. The support for WASM and the Component Model are still in an early stage - but working. It requires many manual steps, but fortunately Golem's app manifest feature can automate most of this for us. Still the generated directory structure of wit-bindgen moonbit</code> felt a little overwhelming first. I hope the MoonBit ecosystem will get some useful libraries in the near future, convenient wappers for WASI and WASI HTTP, (and Golem specific ones!), string encoding utilities, etc. As there are not many libraries yet, it is very easy to find something useful to work on. I'm looking forward to have official support for MoonBit in Golem, such as templates for the golem new ...</code> command and extensive documentation on our website.

desert part 1 - features

2024-02-19T00:00:00+00:00

Introduction</h2>
This is the first part of a series of blog posts about my serialization library, desert</a>. I also gave an overview of this library on Functional Scala 2022 - you can check the talk on YouTube if interested</a>.
In this post I'm going to give an overview of the features this serialization library provides, and then going to dive into the details of how it supports evolving data types.

Where is it coming from?</h2>
The idea of creating desert</code> came after some serious disappointment in our previously chosen serialization library. It was used for serialization of both persistent Akka actors and for the distributed actor messages, and it turned out that just by updating the Scala version from 2.12 to 2.13 completely broke our serialization format.
None of the alternatives looked good enough to me - I wanted something that is code first and fits well to our functional Scala style. Support for multiple platforms or programming languages were not a requirement.
So I started thinking about what would a perfect serialization library look like, at least for our use cases? It was something that has first-class support for ADTs, for Scala's collection libraries (I don't want to see Scala lists serialized via Java reflection ever again!), with a focus of supporting evolution of the serialized data types. We knew that our persisted data and actor messages will change over time, and we had to be able to survive these changes without any downtime.
Features</h2> Let's just go through all the features provided by the library before we talk about how exactly it supports these kind of changes in the serialized data structures. desert</code> is a Scala library. As probably expected, it captures the core concept of binary serialization though a simple trait</code> called BinaryCodec[T]</code>:trait BinarySerializer[T] { def serialize(value: T)(implicit context: SerializationContext): Unit def contramap[U](f: U => T): BinarySerializer[U] = // ... def contramapOrFail[U](f: U => Either[DesertFailure, T]): BinarySerializer[U] = // ... } trait BinaryDeserializer[T] { def deserialize()(implicit ctx: DeserializationContext): T def map[U](f: T => U): BinaryDeserializer[U] = // ... def mapOrFail[U](f: T => Either[DesertFailure, U]): BinaryDeserializer[U] = // ... } trait BinaryCodec[T] extends BinarySerializer[T] with BinaryDeserializer[T] </code></pre> These BinaryCodec</code> instances should be made implicitly available for each type we need to serialize. There are multiple ways to create an instance of a binary codec:
There are many built-in codecs for primitive types, standard collections, date-time classes, etc.</li> The map</code> and contramap</code> operators can be used to construct new codecs from existing ones</li> There is a codec derivation macro for ADTs (case classes and sealed traits / enums)</li> Custom implementation can directly read/write the binary data and access some of the built-in features like the type registry, references, string deduplication and compression</li>
It is also possible to define these custom implementations in a more functional way on top of ZPure</code></li> </ul> Under the hood there is a simple BinaryInput</code> / BinaryOutput</code> abstraction which is extensible, by default implemented for Java InputStream</code> and OutputStream</code>. On the lowest level, in addition to having an interface for serializing primitive types we also have support for variable length integer encoding and for gzip compression. Custom codecs can also use the built-in string deduplication feature, and encode cyclic graphs using support for storing references. Sometimes you want to serialize only a part of your data structure - a real-world example we had was having a set of typed actor messages where only a subset of the cases were designed to be used between different nodes. Some cases were only used locally, and in those we would store things that are not serializable at all - for example open websocket connection handles. This is supported by desert</code> by having the concept of both transient fields and transient constructors. What if a field is not an ADT but contains a reference to an arbitrary type with a given interface? Or if we don't know the root type of a message, only a set of possible types which are otherwise unrelated? The library provides a type registry for this purpose. Every type registered into this will have an associated identifier, and in places where we don't know the exact type, we can use these to get the codec by it's unique ID from the type registry. On the top level desert</code> also comes with a set of integration modules. The following modules are available at the time of writing:
desert-akka</code> provides helper functions to serialize from/to ByteString</code>, provides codecs for both typed and untyped ActorRef</code>s, and provides an implementation of Akka's Serializer</code> interface.</li>desert-cats</code> adds codecs for Validation</code>, NonEmptyList</code>, NonEmptySet</code> and NonEmptyMap</code> from the cats library</a>.</li>
desert-cats-effect</code> gives a cats-effect</a> IO</code> version of the top level serialization and deserialization functions</li>
desert-zio</code> provides ZIO</code> version of the top level serialization and deserialization functions and adds codec and helper functions to work with Chunk</code>s,</li>
desert-zio-prelude</code> provides a more functional interface for defining custom codecs, as well as having built-in codecs for</li>
desert-shardcake</code> provides easy integration within the Shardcake</a> library</li> </ul> There are two more modules which implement the same core functionality, codec derivation, with different tradeoffs: desert-shapeless</code> is a shapeless</a> based codec deriver, the original implementation of desert</code>'s derivation logic. It only works for Scala 2 but it has no additional requirements.</li> desert-zio-scheme</code> is an alternative implementation of the same codec derivation, built on the Deriver</code> feature of zio-schema</a>. This works both with Scala 2 and Scala 3, and supposed to provide better compile-time error messages, but requires to derive an implicit Schema</code> for each serialized type beside the binary codec.</li> </ul> I wrote a detailed post about typeclass derivation</a> a few months ago. Data evolution</h2> Let's see in details what it means that desert</code> supports evolving data structures. Primitives vs newtype wrappers</h3> Let's start with a simple example: we are serializing a single Int</code>. The default codec just uses the fixed width 32-bit representation of the integer: val x: Int = 100 </code></pre> results in: 0</td> 0</td> 0</td> 100</td> </tr> </table> Imagine that later we decide that Int</code> is just too generic, and what we have here is in fact a Coordinate</code>. We can define a a newtype wrapper like the following: final case class Coordinate(value: Int) extends AnyVal </code></pre> and then define the binary codec either by using map</code> and contramap</code> on the integer codec, or by using the deriveForWrapper</code> macro: object Coordinate { implicit val codec: BinaryCodec[Coordinate] = DeriveBinaryCodec.deriveForWrapper } </code></pre> The binary representation of a Coordinate</code> will be exactly the same as for an Int</code>, so we are still fully backward and forward compatible regarding our serialization format: val x: Coordinate = Coordinate(100) </code></pre> results in: 0</td> 0</td> 0</td> 100</td> </tr> </table> Collections</h3> First let's see what happens if we try to serialize a pair of coordinates: val xy = (Coordinate(1), Coordinate(2)) </code></pre> results in: 0</td> 0</td> 0</td> 0</td> 1</td> 0</td> 0</td> 0</td> 2</td> </tr> </table> the binary representation starts with a 0</code>, which is an ADT header. We will talk about it later. The rest of the data is just a flat representation of the two coordinates, taking in total 9 bytes. Now we start storing arrays of these coordinates: val coordinates: Array[(Coordinate, Coordinate)] = Array( (Coordinate(1), Coordinate(2)), (Coordinate(3), Coordinate(4)), (Coordinate(5), Coordinate(6)) ) </code></pre> Arrays are serialized simply by writing the length of the array as a variable-length integer and then serializing all elements. 6</td> 0</td> 0</td> 0</td> 0</td> 1</td> 0</td> 0</td> 0</td> 2</td> 0</td> 0</td> 0</td> 0</td> 3</td> 0</td> 0</td> 0</td> 4</td> 0</td> 0</td> 0</td> 0</td> 5</td> 0</td> 0</td> 0</td> 6</td> </tr> </table> The variable-length integer encoding of 3</code> is 6</code>, and that is simply followed by the three 9-byte long serialized representation of the coordinate pairs. What if we decide we don't want to use Array</code> but ZIO's Chunk</code> instead? Or if we realize our data model is more precise if we talk about a set of coordinate pairs? Nothing! Desert uses the same encoding for all collection types, allowing us to always choose the best data type without being worried about breaking the serialization format. In some collections, such as linked lists, there is no way to know the number of elements without iterating through the whole data set. Desert supports these collection types by writing -1</code> as the number of elements, and then prefixing each element with a single byte where 1</code> represents we have a next element and 0</code> that we don't. This is actually exactly the same binary format as a series of Option[T]</code> values where the first and only None</code> represents the end of the sequence. Records</h3> Maybe using tuples of coordinates was a good idea in the beginning but as our data model evolves we want to introduced a named record type instead: final case class Point(x: Coordinate, y: Coordinate) </code></pre> We can use desert</code>'s codec derivation feature to get a binary codec for this type: object Point { implicit val schema: Schema[Point] = DeriveSchema.gen implicit val codec: BinaryCodec[Point] = DerivedBinaryCodec.derive } </code></pre> When using desert-zio-scheme</code> we also need to derive a Schema</code> instance - this is not required when using the desert-shapeless</code> version of the codec derivation. Let's see how desert</code> serializes an instance of this Point</code> type: val pt = Point(Coordinate(1), Coordinate(2)) </code></pre> results in: 0</td> 0</td> 0</td> 0</td> 1</td> 0</td> 0</td> 0</td> 2</td> </tr> </table> This is exactly the same as the tuple's binary representation was, which probably isn't a big surprise as they are structurally equivalent. Still this is an important property as it allows us to replace any tuple with an equivalent record type and keeping the binary format exactly the same! If we have to change a record's type, we can only change any of its fields if that field's new type has a compatible binary representation with the old one. All the cases described in this post are valid data evolution steps. Beside those there are a few special type of changes desert</code> supports for records. Let's see!s Adding a field</h3> As a next step let's imagine our data type requires a new field. Let's add a z</code> coordinate to our point: final case class Point(x: Coordinate, y: Coordinate, z: Coordinate) object Point { implicit val codec: BinaryCodec[Point] = DerivedBinaryCodec.derive } val pt = Point(Coordinate(1), Coordinate(2), Coordinate(3)) </code></pre> Serializing this pt</code> value results in: 0</td> 0</td> 0</td> 0</td> 1</td> 0</td> 0</td> 0</td> 2</td> 0</td> 0</td> 0</td> 3</td> </tr> </table> If we try to read this value with the deserializer of our original Point</code> type, it will read Point(Coordinate(1), Coordinate(2))</code>, but the next deserialized value will be corrupt as the input stream will point to the beginning of the 0, 0, 0, 3</code> value. Similarly, if we would try to read a binary serialized with the old Point</code> serializer, it would read the next four bytes from the data stream which, if even exists, belongs to some other serialized element. The solution for this in desert</code> is to explicitly document data evolution. This is done by listing each modification in an attribute called evolutionSteps</code>: @evolutionSteps(FieldAdded[Coordinate]("z", Coordinate(0))) final case class Point(x: Coordinate, y: Coordinate, z: Coordinate) object Point { implicit val codec: BinaryCodec[Point] = DerivedBinaryCodec.derive } val pt = Point(Coordinate(1), Coordinate(2), Coordinate(3)) </code></pre> With this annotation, we mark z</code> as a newly added field, and provide a default value for it which will be used in cases when reading an old version of the serialized data which did not have this field yet. Every time we change the data type we record the change as a new element in this attribute. There are other supported evolution step types as we will see soon. But first let's see what changes in the binary representation of Point</code> now that we added this attribute! 1</td> 16</td> 8</td> 0</td> 0</td> 0</td> 1</td> 0</td> 0</td> 0</td> 2</td> 0</td> 0</td> 0</td> 3</td> </tr> </table> Now that we have an evolution step the first byte, which was always 0</code> before, becomes 1</code>. Every evolution step increases this value, which is interpreted as the type's version. For each ADT which has a version other than 0, this first version byte is followed by a list of the binary encoding of the evolution steps. Here the 16</code> is the variable-length encoding of the value 8</code>, which is the length of the "version 0" part of the data type. This is followed by 8</code> which is just the variable-length encoding of the value 4</code>, and it represents the field added evolution step, encoding the newly added field's size. With this format when the old deserializer reads the point, it knows it needs to skip additional 4 bytes after reading the x</code> and y</code> coordinates. Also when the new deserializer encounters an old point, that binary data will begin with 0</code>, so the deserializer is aware that it's an older version and can set the deserialized value's z</code> coordinate to the provided default. By documenting the data type change we get full forward and backward compatibility in this case. The cost is that instead of 13</code> bytes, now each Point</code> takes 15</code> bytes. Making a field optional</h3> Another special data type change is making an existing field optional. Staying with the previous example we could change our Point</code> type like this: @evolutionSteps( FieldAdded[Coordinate]("z", Coordinate(0)), FieldMadeOptional("z") ) final case class Point(x: Coordinate, y: Coordinate, z: Option[Coordinate]) object Point { implicit val codec: BinaryCodec[Point] = DerivedBinaryCodec.derive } val pt = Point(Coordinate(1), Coordinate(2), None) </code></pre> This of course can no longer guarantee full forward and backward compatibility - but it can be useful as an intermediate step in getting rid of some unused parts of the data model, while still being able to access it when it's available from older serialized data. This evolution step is represented by a variable-length integer -1</code> in the ADT header. All positive values are representing the field added case, with the actual value containing the size of the added field. -1 is a special marker for field removed, and it is followed by another variable-length integer encoding the field position which has been made optional. Then serializing the Option</code> field, the integer gets prefixed by a 1</code> if the value was Some</code>, or the whole option is serialized as a 0</code> if it was None</code>. The total serialized record of the above example would look like this: 2</td> 16</td> 2</td> 1</td> 1</td> 0</td> 0</td> 0</td> 1</td> 0</td> 0</td> 0</td> 2</td> 0</td> </tr> </table> The first byte is now 2</code> as we have two evolution steps. The next one still defines that the original part of the data is 8 bytes long, the third byte shows that this time the new z field is taking only 1 byte (as it was set to None</code>). The header is now containing two more bytes, as described above: the first 1</code> means a field has been made optional, and the second points to the field. This can be still loaded by the very first point serializer (or even as the coordinate pair tuple), as everything after the first two coordinates would be skipped. It can also be loaded as a Point</code> with non-optional z coordinate, but only if the serialized data is a Some</code>. So in the above example it would lead to a deserialization error. The change is fully backward compatible so our latest deserializer can still load all the variants we have seen before. Removing a field</h3> The final special data evolution step supported by the library is removing a field completely. This is more limited than the previous ones though - backward compatibility is easy, newer versions of the deserializer just have to skip the removed fields which they can easily do. But forward compatibility is only possible if the removed field was an option field - that's the only type desert</code> can automatically provide a default value, None</code> for. The binary header for removing a field needs to store the actual field name because it cannot otherwise identify the field which is not actually in the rest of the data set. To make this more space-efficient, desert</code> uses string deduplication and only needs to serialize the actual field name once. Sum types</h3> Scala 2 sealed trait hierarchies and Scala 3 enums are simply serialized with the same techniques mentioned above, but with a constructor ID serialized as a prefix to the binary. Constructor identifiers are associated in order - as the constructors appear in the source code. This means that adding new constructors is backward and forward compatible, as long as they are added as the last constructor. Otherwise the identifiers will be rearranged and binary compatibility breaks. Transients</h3> It is possible to make a previously non-transient field transient and maintain binary compatibility. The rules are the same as for removing a field. Type registry</h3> As mentioned earlier, a type registry can be used to associate identifiers to types, and then serialize arbitrary values using these identifiers. Maintaining the stability of this mapping is also very important when evolving data types. What if we want to delete a type which was added to the type registry because we never want to use it again, and we already migrated our serialized data and we are sure we will never encounter that ID again during deserialization? We still cannot just simply remove the entry from the type registry, because it will break all the following identifiers as they get assigned sequentially. The library has a solution for this - it is possible to registry empty placeholders where we previously had an actual type - it will maintain the identifier order, but will lead to a runtime error when that identifier is encountered during deserialization. Summary</h2> In this post I summarized the key features of the desert</code> serialization library, and explained in detail how it supports changes into the data model while trying to keep maximal backward and forward compatibility. In the next post I will show how the same library can be implemented for Rust, how the Scala solution maps into different concepts in the other language and what difficulties I've encountered during the migration process. [Video] Beyond OpenAPI @ Functional Scala 2023 2024-01-21T00:00:00+00:00 My talk at Functional Scala 2023</a> about my experience with generating client libraries from OpenAPI specifications, and an alternative code-first approach using ZIO Http</a>. </iframe> Type class derivation with ZIO Schema 2023-12-02T00:00:00+00:00 Introduction</h2> Making the compiler to automatically derive implementations of a type class for your custom algebraic data types is a common technique in programming languages. Haskell, for example, has built-in syntax for it: data Literal = StringLit String | BoolLit Bool deriving (Show) </code></pre> and Rust is using macros instantiated by annotations to do the same: #[deriving(Debug)] enum Literal { StringLit(String), BoolLit(bool) } </code></pre> Scala 3 has its own syntax for deriving type classes: enum Literal deriving Show: case StringLit(value: String) case BoolLit(value: Boolean) </code></pre> but the more traditional way that works with Scala 2 as well is to define an implicit in the type's companion object by an explicit macro invocation: sealed trait Literal object Literal { final case class StringLit(value: String) extends Literal final case class BoolLit(value: String) extends Literal implicit val show: Show[Literal] = DeriveShow[Literal] } </code></pre> All these examples from different languages are common in a way that in order to automatically generate an implementation for an arbitrary type we need to be able to gather information about these types as (compilation-) runtime values, and to generate new code fragments (or actual abstract syntax tree) which then takes part of to the compilation, producing the same result as writing the implementation by hand. This means using some kind of macro, depending on which programming language we use. But writing these macros is never easy, and in some cases can be very different from the usual way of writing code - so in each programming language people are writing libraries helping type class derivation in one way or the other. In this post I will show a library like that for Scala, the Deriver</code> feature of ZIO Schema</a> that I added at the end of last year (2022). But before that let's see a real world example and what alternatives we had. Example</h2> Desert</a> is a Scala serialization library I wrote in 2020. Not surprisingly in the core of Desert is a trait that describes serialization and deserailization of a type T</code>: trait BinaryCodec[T] extends BinarySerializer[T] with BinaryDeserializer[T] trait BinarySerializer[T] { def serialize(value: T)(implicit context: SerializationContext): Unit // ... } trait BinaryDeserializer[T] { def deserialize()(implicit ctx: DeserializationContext): T // ... } </code></pre> Although we can implement these traits manually, in order to take advantage of Desert's type evolution capabilities, for complex types like case classes or enums we want the user to be able to write something like this: final case class Point(x: Int, y: Int, z: Int) object Point { implicit val codec: BinaryCodec[Point] = DerivedBinaryCodec.derive } </code></pre> Alternatives</h2> Scala 3 mirrors</h3> First of all, Scala 3 has some built-in support for implementing derivation macros using its Mirror</code> type, explained in the official documentation</a>. We can see a simple example of this technique in the ZIO codebase</a> where I have implemented a deriving mechanism for the Gen[R, A]</code> trait which is Scala 3 specific. (The Scala 2 version is using the Magnolia library, introduced below, which did not have a Scala 3 version back then). The Mirror</code> values are summoned by the compiler and they provide the type information: inline def gen[T](using m: Mirror.Of[T]): DeriveGen[T] = new DeriveGen[T] { def derive: Gen[Any, T] = { val elemInstances = summonAll[m.MirroredElemTypes] inline m match { case s: Mirror.SumOf[T] => genSum(s, elemInstances) case p: Mirror.ProductOf[T] => genProduct(p, elemInstances) } } } </code></pre> As this function is an inline function</a>, it gets evaluated compile time, using this summoned Mirror</code> value to produce an implementation of Gen[Any, T]</code>. This is a little low level and requires knowledge of inline functions and things like summonAll</code> etc., but otherwise a relatively easy way to solve the type class derivation problem. But it is Scala 3 only. Back in 2020 when I wrote the first version of Desert, there was no Scala 3 at all, and the three main way to do this were writing a (Scala 2) macro by hand</li> using Shapeless</a></li> using Magnolia</a></li> </ul> Scala 2 macros</h3> Writing a custom derivation logic with Scala 2 macros is not easy, but it is completely possible. It starts by defining a whitebox macro</a>: object Derive { def derive[A]: BinaryCodec[A] = macro deriveImpl[A] def deriveImpl[A: c.WeakTypeTag]( c: whitebox.Context ): c.Tree = { import c.universe._ // ... } } </code></pre> The job of deriveImpl</code> is to examine the type of A</code> and generate a Tree</code> that represents the implementation of the BinaryCodec</code> trait for A</code>. We can start by getting a Type</code> value for A</code>: val tpe: Type = weakTypeOf[A] </code></pre> and then use that to get all kind of information about this type. For example to check if it is a case class, we could write def isCaseClass(tpe: Type): Boolean = tpe.typeSymbol.asClass.isCaseClass </code></pre> and then try to collect all the fields of that case class: val fields = tpe.decls.sorted.collect { case p: TermSymbol if p.isCaseAccessor && !p.isMethod => p } </code></pre> As we can see this is a very direct and low level way to work with the types, much harder then the Mirror</code> type we used for Scala 3. Once we gathered all the necessary information for generating the derived type class, we can use quotes to construct fragments of Scala AST: val fieldSerializationStatements = // ... val codec = q"new BinaryCodec[$tpe] { def serialize(value: T)(implicit context: SerializationContext): Unit = { ..$fieldSerializationStatements } } </code></pre> In the end, this quoted codec</code> value is a Tree</code> which we can return from the macro. Shapeless</h3> Shapeless</a> is a library for type level programming in Scala 2 (and there is a new version</a> for Scala 3 too). It provides things like type-level heterogeneous lists and all of operations on them, and it also defines macros that can convert an arbitrary case class into a generic representation, which is essentially a type level list containing all the fields. Similarly it can convert an arbitrary sum type (sealed trait in Scala 2) to a generic representation of coproducts. For example the Point</code> case class we used in an earlier example would be represented like this: final case class Point(x: Int, y: Int, z: Int) val point: Point = Point(1, 2, 3) val genericPoint: Int :: Int :: Int :: HNil = // type 1 :: 2 :: 3 :: HNil // value val labelledGenericPoint = // type too complex to show here ("x" ->> 1) :: ("y" ->> 2) :: ("z" ->> 3) :: HNil // value </code></pre> In connection with type class derivation the idea is that by using Shapeless we no longer have to write macros to extract type information for our types - we can work with these generic representations instead using advanced type level programming techniques. So the complexity of writing macros is replaced with the complexity of doing type level computation. Let's see how it would look like. First we start by creating a derive</code> method that gets the type we are deriving the codec for as a type parameter: def derive[T] = // ... </code></pre> This T</code> is an arbitrary type, for example our Point</code> structure. In order to get its generic representation provided by Shapeless we have to start using type level techniques, by introducing new type parameters for the things we want to calculate (as types) and implicits to drive these computations. The following version, when compiles, will "calculate" the generic representation of T</code> as the type parameter H</code>: def derive[T, H](implicit gen: LabelledGeneric.Aux[T, H]) = { new BinaryCodec[T] { def serialize(value: T)(implicit context: SerializationContext): Unit = { val h: H = gen.to(value) // generic representation of (value: T) // ... } // ... } } </code></pre> This is not that hard yet but we need to recursively summon implicit codecs for our fields, so we can't just use this H</code> value to go through all the fields in a traditional way - we need to traverse it on the type level. To do that we need to write our own type level computations implemented as implicit instances for HNil</code> and ::</code> etc. The serialization part of the codec would look something like this: implicit val hnilSerializer: BinarySerializer[HNil] = new BinarySerializer[HNil] { def serialize(value: HNil)(implicit context: SerializationContext) => { // no (more) fields } } implicit def hlistSerializer[K <: Symbol, H, T <: HList](implicit witness: Witness.Aux[K] // type level extraction of the field's name headSerializer: BinarySerializer[H] // type class summoning for the field tailSerializer: BinarySerializer[T] // hlist recursion ): BinarySerializer[FieldType[K, H] :: T] = // ... </code></pre> Similar methods have to be implemented for coproducts too, and also in the codec example we would have to simultaneously derive the serializer and the deserializer. A real implementation would also require access to the annotations of various fields to drive the serialization logic, which requires more and more type level calculations and complicates these type signatures. I did chose to use Shapeless in the first version of Desert, and the real derive</code> method has the following signature: def derive[T, H, Ks <: HList, Trs <: HList, Trcs <: HList, KsTrs <: HList, TH](implicit gen: LabelledGeneric.Aux[T, H], keys: Lazy[Symbols.Aux[H, Ks]], transientAnnotations: Annotations.Aux[transientField, T, Trs], transientConstructorAnnotations: Annotations.Aux[transientConstructor, T, Trcs], taggedTransients: TagTransients.Aux[H, Trs, Trcs, TH], zip: Zip.Aux[Ks :: Trs :: HNil, KsTrs], toList: ToTraversable.Aux[KsTrs, List, (Symbol, Option[transientField])], serializationPlan: Lazy[SerializationPlan[TH]], deserializationPlan: Lazy[DeserializationPlan[TH]], toConstructorMap: Lazy[ToConstructorMap[TH]], classTag: ClassTag[T] ): BinaryCodec[T] </code></pre> Although this works, there are many problems with this approach. All these type and implicit resolutions can make the compilation quite slow, the code is very complex and hard to understand or modify, and most importantly error messages will be a nightmare. A user trying to derive a type class for our serialization library should not get an error that complains about not being able to find an implicit value of Zip.Aux</code> for a weird type that does not even fit on one screen! Magnolia</h3> The Magnolia</a> library provides a much more friendly solution for deriving type classes for algebraic data types - it moves the whole problem into the value space by hiding the necessary macros. The derivation implementation for a given type class then only requires defining two functions (one for working with products, one for working with coproducts) that are regular Scala functions getting a "context" value and producing an instance of the derived type class. The context value contains type information - for example the name and type of all the fields of a case class - and also contains an instance of the derived type class for each of these inner elements. To write a Magnolia based deriver you have to create an object</code> with a join</code> and a split</code> method and a Typeclass</code> type: object BinaryCodecDerivation { type Typeclass[T] = BinaryCodec[T] def join[T](ctx: CaseClass[BinaryCodec, T]): BinaryCodec[T] = new BinaryCodec[T] { def serialize(value: T)(implicit context: SerializationContext) => { for (parameter <- ctx.parameters) { // recursively serialize the fields parameter.typeclass.serialize(parameter.dereference(value)) } // ... } } def split[T](ctx: SealedTrait[BinaryCodec, T]): BinaryCodec[T] = // ... def gen[T]: BinaryCodec[T] = macro Magnolia.gen[T] } </code></pre> There is a Magnolia version for Scala 3 too, which is although quite similar, it is not source compatible with the Scala 2 version, leading to the need to define these derivations twice in cross-compiled projects. Why not Magnolia?</h2> Magnolia already existed when I wrote the first version of Desert, but I could not use it because of two reasons. In that early version of the library the derivation had to take a user defined list of evolution steps, so the actual codec definitions looked something like this: object Point { implicit val codec: BinaryCodec[Point] = BinaryCodec.derive(FieldAdded[Int]("z", 1)) } </code></pre> It was not clear how could I pass these parameters to Magnolia context - with Shapeless it was not a problem because it is possible to simply pass them as a parameter to the derive</code> function that "starts" the type level computation. This requirement no longer exists though, as in recent versions the evolution steps are defined by attributes, which are fully supported by Magnolia as well: @evolutionSteps(FieldAdded[Int]("z", 1)) final case class Point(x: Int, y: Int, z: Int) </code></pre> The second reason was a much more important limitation in Magnolia that still exists - it is not possible to shortcut the derivation tree. Desert has transient field and transient constructor support. For those fields and constructors which are marked as transient we don't want to, and cannot define codec instances. They can be things like open files, streams, actor references, sockets etc. Even though Magnolia only instantiates the type class instances when they are accessed, the derivation fails if there are types in the tree that does not have an instance. This issue is tracked here</a>. There was one more decision I did not like regarding Magnolia - the decision to have an incompatible Scala 3 version. I believe it was a big missed opportunity to seamlessly support cross-compiled type class derivation code. ZIO Schema based derivation</h2> All these issues lead to writing a new derivation library - as part of the ZIO Schema</a> project. It was first released in version v0.3.0</a> in November of 2022. From the previously demonstrated type class derivation techniques the closest to ZIO Schema's deriver is Magnolia. On the other hand it does supports the transient field use case, and it is fully cross-compilation compatible between Scala 2 and Scala 3. To implement type class derivation based on ZIO Schema you need to implement a trait called Deriver</code>: trait Deriver[F[_]] { def deriveRecord[A]( record: Schema.Record[A], fields: => Chunk[WrappedF[F, _]], summoned: => Option[F[A]] ): F[A] // more deriveXXX methods to impelment } </code></pre> This looks similar to Magnolia's join</code> method but has some significant differences. The first thing to notice is that we get a Schema.Record</code> value describing our case class. This is one of the cases of the core data type Schema[T]</code> which describes Scala data types and provides a lot of features to work with them. So having a Schema[A]</code> is a requirement to derive an F[A]</code> with Deriver</code> - but luckily ZIO schema has derivation support for Schema itself. The second thing to notice is that Schema[A]</code> itself does not know anything about type class derivation and especially about the actual F</code> type class that is being derived, so the second parameter of deriveRecord</code> is a collection of potentially derived instances of our derived type class for each field. WrappedF</code> is just making this lazy so if we decide we don't need instances for (some of) the fields they won't be traversed (they still need to have a Schema</code> though - but it can even be a Schema.fail</code> for things not representable by ZIO Schema - it will be fine if we never touch them by unwrapping the WrappedF</code> value). The third parameter is also interesting as it provides full control to the developer to choose between the summoned implicit and the derivation logic. If your deriveRecord</code> is called for a record type A</code> and there is already an implicit F[A]</code> that the compiler can find (for example defined in A</code>'s companion object), it will be passed in the summoned</code> parameter to deriveRecord</code>. The usual logic is to choose the summoned value when it is available and only derive an instance when there isn't any. By calling .autoAcceptSummoned</code> on our Deriver</code> class we can automatically enable this behavior - in this case deriveRecord</code> will only be called for the cases where summoned</code> was None</code>. Another method we have on Deriver</code> is .cached</code> which stores the generated type class instances in a concurrent hash map shared between the macro invocations. Our ZIO Schema based Desert codec derivation is defined using these modifiers: object DerivedBinaryCodec { lazy val deriver = BinaryCodecDeriver().cached.autoAcceptSummoned private final case class BinaryCodecDeriver() extends Deriver[BinaryCodec] { // ... } } </code></pre> As ZIO Schema is not only describing records and enums but also primitive types, tuples, and special cases like Option</code> and Either</code> and collection types, the deriver has to support all these. The minimum set of methods to implement is deriveRecord</code>, deriveEnum</code>, derivePrimitive</code>, deriveOption</code>, deriveSequence</code>, deriveMap</code> and deriveTransformedRecord</code>. In addition to that we can also override deriveEither</code>, deriveSet</code> and deriveTupleN</code> (1-22) to handle these cases specially. In case of Desert the deriveRecord</code> and deriveEnum</code> are calling to the implementation of the same data-evolution aware binary format that was previously implemented using Shapeless, but this time it is automatically supporting Scala 2 and Scala 3 the same time. The derivePrimitive</code> is just choosing from predefined BinaryCodec</code> instances based on the primitive's type: override def derivePrimitive[A]( st: StandardType[A], summoned: => Option[BinaryCodec[A]] ): BinaryCodec[A] = st match { case StandardType.UnitType => unitCodec case StandardType.StringType => stringCodec case StandardType.BoolType => booleanCodec case StandardType.ByteType => byteCodec // ... } </code></pre> Same applies for option, either, sequence etc - it is just a mapping to the library's own definition of these binary codecs. Under the hood Deriver</code> is a macro (implemented separately both for Scala 2 and Scala 3) that traverses the types simultaneously with the provided Schema</code> (so it does not need to regenerate those) and maps these informations into calls through the Deriver</code> interface. The whole process is initiated by calling the derive</code> method on our Deriver</code>, which is the entry point of these macros, so it has a different looking (but source-code compatible) definition for Scala 2 and Scala 3: // Scala 3 inline def derive[A](implicit schema: Schema[A]): F[A] // Scala 2 def derive[F[_], A](deriver: Deriver[F])( implicit schema: Schema[A] ): F[A] = macro deriveImpl[F, A] </code></pre> These are compatible if you are directly calling them: so you can write val binaryCodecDeriver: Deriver[BinaryCodec] = // ... val pointCodec: BinaryCodec[Point] = binaryCodecDeriver.derive[Point] </code></pre> Or even: object BinaryCodecDeriver extends Deriver[BinaryCodec] { // ... } val pointCodec: BinaryCodec[Point] = BinaryCodecDeriver.derive[Point] </code></pre> But if you want to wrap this derive call you have to be aware that they are macro calls, and they have to be wrapped by (version-specific) macros. This is what Desert is doing - as shown before, it uses the cached</code> and autoAcceptSummoned</code> modifiers to create a deriver, but still exposes a simple derive</code> method through an object</code>. To do so it needs to wrap the inner deriver macro with its own macro like this: // Scala 2 trait DerivedBinaryCodecVersionSpecific { def deriver: Deriver[BinaryCodec] def derive[T](implicit schema: Schema[T]): BinaryCodec[T] = macro DerivedBinaryCodecVersionSpecific.deriveImpl[T] } object DerivedBinaryCodecVersionSpecific { def deriveImpl[T: c.WeakTypeTag]( c: whitebox.Context)( schema: c.Expr[Schema[T]] ): c.Tree = { import c.universe._ val tpe = weakTypeOf[T] q"_root_.zio.schema.Derive.derive[BinaryCodec, $tpe] (_root_.io.github.vigoo.desert.zioschema.DerivedBinaryCodec.deriver)($schema)" } } // Scala 3 trait DerivedBinaryCodecVersionSpecific { lazy val deriver: Deriver[BinaryCodec] inline def derive[T](implicit schema: Schema[T]): BinaryCodec[T] = Derive.derive[BinaryCodec, T](DerivedBinaryCodec.deriver) } </code></pre> Conclusion</h2> We have a new alternative for deriving type class instances from type information, based on ZIO Schema. You may want to use it if you want to have a single deriver source code for both Scala 2 and Scala 3, if you need more flexibility than what Magnolia provides, or if you are already using ZIO Schema in your project. Generating a Rust client library for ZIO Http endpoints 2023-09-07T00:00:00+00:00 We at Golem Cloud</a> built our first developer preview on top of the ZIO ecosystem, including ZIO Http</a> for defining and implementing our server's REST API. By using ZIO Http we immediately had the ability to call our endpoints using endpoint clients, which allowed us to develop the first version of Golem's CLI tool very rapidly. Although very convenient for development, using a CLI tool built with Scala for the JVM is not a pleasant experience for the users due to the slow startup time. One possible solution is to compile to native using GraalVM Native Image</a> but it is very hard to set up and even when it works, it is extremely fragile - further changes to the code or updated dependencies can break it causing unexpected extra maintenance cost. After some initial experiments we dropped this idea - and instead chose to reimplement the CLI using Rust - a language being a much better fit for command line tools, and also already an important technology in our Golem stack. ZIO Http</h2> If we rewrite golem-cli</code> to Rust, we lose the convenience of using endpoint definitions (written in Scala with ZIO Http, the ones we have for implementing the server) for calling our API, and we would also lose all the types used in these APIs as they are all defined as Scala case classes and enums. Just to have more context, let's take a look at one of the endpoints! A ZIO Http endpoint is just a definition of a single endpoint of a HTTP API, describing the routing as well the inputs and outputs of it: val getWorkerMetadata = Endpoint(GET / "v1" / "templates" / rawTemplateId / "workers" / workerName) .header(Auth.tokenSecret) .outErrorCodec(errorCodec) .out[WorkerMetadata] ?? Doc.p("Get the current worker status and metadata") </code></pre> Let's see what we have here: the endpoint is reached by sending a GET request</li> the request path consists of some static segments as well as the template id and the worker name</li> it also requires an authorization header</li> we define the kind of errors it can return</li> and finally it defines that the response's body will contain a JSON representation (default in ZIO Http) of a type called WorkerMetadata</code></li> </ul> What are rawTemplateId</code> and workerName</code>? These are so called path codecs, defined in a common place so they can be reused in multiple endpoints. They allow us to have dynamic parts of the request path mapped to specific types - so when we implement the endpoint (or call it in a client) we don't have to pass strings and we can directly work with the business domain types, in this case RawTemplateId</code> and WorkerName</code>. The simplest way to define path codecs is to transform an existing one: val workerName: PathCodec[WorkerName] = string("worker-name").transformOrFailLeft(WorkerName.make(_).toErrorEither, _.value) </code></pre> Here the make</code> function is a ZIO Prelude Validation</code></a> which we have to convert to an Either</code> for the transform function. Validations can contain more than one failures, as opposed to Either</code>s, which allows us to compose them in a way that we can keep multiple errors instead of immediately returning with the first failure. The tokenSecret</code> is similar, but it is a HeaderCodec</code> describing what type of header it is and how the value of the given header should be mapped to a specific type (a token, in this case). What is WorkerMetadata</code> and how does ZIO Http know how to produce a JSON from it? It's just a simple case class: final case class WorkerMetadata( workerId: ComponentInstanceId, accountId: AccountId, args: Chunk[String], env: Map[String, String], status: InstanceStatus, templateVersion: Int, retryCount: Int ) </code></pre> But with an implicit derived ZIO Schema: object WorkerMetadata { implicit val schema: Schema[WorkerMetadata] = DeriveSchema.gen[WorkerMetadata] } </code></pre> We will talk more about ZIO Schema below - for now all we need to know is it describes the structure of Scala types, and this information can be used to serialize data into various formats, including JSON. Once we have our endpoints defined like this, we can do several things with them - they are just data describing what an endpoint looks like! Implementing an endpoint</h3> When developing a server, the most important thing to do with an endpoint is to implement it. Implementing an endpoint looks like the following: val getWorkerMetadataImpl = getWorkerMetadata.implement { Handler.fromFunctionZIO { (rawTemplateId, workerName, authTokenId) => // ... ZIO program returning a WorkerMetadata } } </code></pre> The type of getWorkerMetadataImpl</code> is Route</code> - it is no longer just a description of what an endpoint looks like, it defines a specific HTTP route and its associated request handler, implemented by a ZIO effect (remember that ZIO effects are also values - we describe what we need to do when a request comes in, but executing it will be the responsibility of the server implementation). The nice thing about ZIO Http endpoints is that they are completely type safe. I've hidden the type signature in the previous code snippets but actually getWorkerMetadata</code> has the type: Endpoint[ (RawTemplateId, WorkerName), (RawTemplateId, WorkerName, TokenSecret), WorkerEndpointError, WorkerMetadata, None ] </code></pre> Here the second type parameter defines the input of the request handler and the forth type parameter defines the output the server constructs the response from. With these types, we really just have to implement a (ZIO) function from the input to the output: (RawTemplateId, WorkerName, TokenSecret) => ZIO[Any, WorkerEndpointError, WorkerMetadata] </code></pre> and this is exactly what we pass to Handler.fromFunctionZIO</code> in the above example. Calling an endpoint</h3> The same endpoint values can also be used to make requests to our API from clients such as golem-cli</code>. Taking advantage of the same type safe representation we can just call apply</code> on the endpoint definition passing its input as a parameter to get an invocation: val invocation = getInstanceMetadata(rawTemplateId, workerName, token) </code></pre> this invocation can be executed to perform the actual request using an EndpointExecutor</code> which can be easily constructed from a ZIO Http Client</code> and some other parameters like the URL of the remote server: executor(invocation).flatMap { workerMetadata => // ... } </code></pre> The task</h2> So can we do anything to keep this convenient way of calling our endpoints when migrating the CLI to Rust? At the time of writing we already had more than 60 endpoints, with many complex types used in them - defining them by hand in Rust, and keeping the Scala and Rust code in sync sounds like a nightmare. The ideal case would be to have something like this in Rust: #[async_trait] pub trait Worker { // ... async fn get_worker_metadata(&self, template_id: &TemplateId, worker_name: &WorkerName, authorization: &Token) -> Result<WorkerMetadata, WorkerError>; } </code></pre> with an implementation that just requires the same amount of configuration as the Scala endpoint executor (server URL, etc), and all the referenced types like WorkerMetadata</code> would be an exact clone of the Scala types just in Rust. Fortunately we can have (almost) this by taking advantage of the declarative nature of ZIO Http and ZIO Schema! In the rest of this post we will see how we can generate Rust code using a combination of ZIO libraries to automatically have all our type definitions and client implementation ready to use from the Rust version of golem-cli</code>. The building blocks</h2> We want to generate from an arbitrary set of ZIO Http Endpoint</code> definitions a Rust crate ready to be compiled, published and used. We will take advantage of the following libraries: ZIO Http</a> as the source of endpoint definitions</li> ZIO Schema</a> for observing the type definitions</li> ZIO Parser</a> because it has a composable printer concept</li> ZIO NIO</a> for working with the filesystem</li> ZIO Prelude</a> for implementing the stateful endpoint/type discovery in a purely functional way</li> </ul> Generating Rust code</h2> Let's start with the actual source code generation. This is something that can be done in many different ways - one extreme could be to just concatenate strings (or use a StringBuilder</code>) while the other is to build a full real Rust AST and pretty print that. I had a talk on Function Scala 2021 about the topic</a>. For this task I chose a technique which is somewhere in the middle and provides some extent of composability while also allowing use to do just the amount of abstraction we want to. The idea is that we define a Rust code generator model which does not have to strictly follow the actual generated language's concepts, and then define a pretty printer for this model. This way we only have to model the subset of the language we need for the code generator, and we can keep simplifications or even complete string fragments in it if that makes our life easier. Let's see how this works with some examples! We will have to generate type definitions so we can define a Scala enum describing what kind of type definitions we want to generate: enum RustDef: case TypeAlias(name: Name, typ: RustType, derives: Chunk[RustType]) case Newtype(name: Name, typ: RustType, derives: Chunk[RustType]) case Struct(name: Name, fields: Chunk[RustDef.Field], derives: Chunk[RustType], isPublic: Boolean) case Enum(name: Name, cases: Chunk[RustDef], derives: Chunk[RustType]) case Impl(tpe: RustType, functions: Chunk[RustDef]) case ImplTrait(implemented: RustType, forType: RustType, functions: Chunk[RustDef]) case Function(name: Name, parameters: Chunk[RustDef.Parameter], returnType: RustType, body: String, isPublic: Boolean) </code></pre> We can make this as convenient to use as we want, for example adding constructors like: def struct(name: Name, fields: Field*): RustDef </code></pre> The Name</code> is an opaque string type with extension methods to convert between various cases like pascal case, snake case, etc. RustType</code> is a similar enum to RustDef</code>, containing all the different type descriptions we will have to use. But it is definitely not how a proper Rust parser would define what a type is - for example we can have a RustType.Option</code> as a shortcut for wrapping a Rust type in Rust's own option type, just because it makes our code generator simpler to write. So once we have this model (which in practice evolves together with the code generator, usually starting with a few simple case classes) we can use ZIO Parser's printer feature to define composable elements constructing Rust source code. We start by defining a module and a type alias for our printer: object Rust: type Rust[-A] = Printer[String, Char, A] </code></pre> and then just define building blocks - what these building blocks are depends completely on us, and the only thing it affects is how well you can compose them. Having very small building blocks may reduce the readability of the code generator, but using too large chunks reduces their composability and makes it harder to change or refactor. We can define some short aliases for often used characters or string fragments: def gt: Rust[Any] = Printer.print('>') def lt: Rust[Any] = Printer.print('<') def bracketed[A](inner: Rust[A]): Rust[A] = lt ~ inner ~ gt </code></pre> and we have to define Rust</code> printers for each of our model types. For example for the RustType</code> enum it could be something like this: def typename: Rust[RustType] = Printer.byValue: case RustType.Primitive(name) => str(name) case RustType.Option(inner) => typename(RustType.Primitive("Option")) ~ bracketed(typename(inner)) case RustType.Vec(inner) => typename(RustType.Primitive("Vec")) ~ bracketed(typename(inner)) case RustType.SelectFromModule(path, typ) => Printer.anyString.repeatWithSep(dcolon)(path) ~ dcolon ~ typename(typ) case RustType.Parametric(name, params) => str(name) ~ bracketed(typename.repeatWithSep(comma)(params)) // ... </code></pre> We can see that typename</code> uses itself to recursively generate inner type names, for example when generating type parameters of tuple members. It also demonstrates that we can extract patterns such as bracketed</code> to simplify our printer definitions and eliminate repetition. Another nice feature we get by using a general purpose printer library like ZIO Parser is that we can use the built-in combinators to get printers for new types. One example is the sequential composition of printers. For example the following fragment: val p = str("pub ") ~ name ~ str(": ") ~ typename </code></pre> would have the type Rust[(Name, RustType)]</code> and we can even make that a printer of a case class like: final case class PublicField(name: Name, typ: RustType) val p2 = p.from[PublicField] </code></pre> where p2</code> will have the type Rust[PublicField</code>]. Another very useful combinator is repetition. For example if we have a printer for an enum's case: def enumCase: Rust[RustDef] = // ... </code></pre> we can simply use one of the repetition combinators to make a printer for a list of enum cases: def enumCases: Rust[Chunk[RustDef]] = enumCase.* </code></pre> or as in the typename</code> example above: typename.repeatWithSep(comma) </code></pre> to have a Rust[Chunk[RustType]]</code> that inserts a comma between each element when printed. Inspecting the Scala types</h2> As we have seen the endpoint DSL uses ZIO Schema to capture information about the types being used in the endpoints (usually as request or response bodies, serialized into JSON). We can use the same information to generate Rust types from our Scala types! The core data type defined by the ZIO Schema library is called Schema</code>: sealed trait Schema[A] { // ... } </code></pre> Schema describes the structure of a Scala type A</code> in a way we can inspect it from regular Scala code. Let's imagine we have Schema[WorkerMetadata]</code> coming from our endpoint definition and we have to generate an equivalent Rust struct</code> with the same field names and field types. The first thing to notice is that type definitions are recursive. Unless WorkerMetadata</code> only contains fields of primitive types such as integer or string, our job does not end with generating a single Rust struct - we need to recursively generate all the other types WorkerMetadata</code> is depending on! To capture this fact let's introduce a type that represents everything we have to extract from a single (or a set of) schemas in order to generate Rust types from them: final case class RustModel( typeRefs: Map[Schema[?], RustType], definitions: Chunk[RustDef], requiredCrates: Set[Crate] ) </code></pre> We have typeRefs</code> which associates a RustType</code> with a schema so we can use it in future steps of our code generator to refer to a generated type in our Rust codebase. We have a list of RustDef</code> values which are the generated type definitions, ready to be printed with out Rust</code> pretty printer. And finally we can also gather a set of required extra rust crates, because some of the types considered primitive types by ZIO Schema are not having proper representations in the Rust standard library, only in external crates. Examples are UUIDs and various date/time types. So our job now is to write a function of def fromSchemas(schemas: Seq[Schema[?]]): Either[String, RustModel] </code></pre> The Either</code> result type is used to indicate failures. Even if we write a transformation that can produce from any Schema</code> a proper RustModel</code>, we always have to have an error result when working with ZIO Schema because it has an explicit failure case called Schema.Fail</code>. If we process a schema and end up with a Fail</code> node, we can't do anything else than fail our code generator. There are many important details to consider when implementing this function, but let's just see first what the actual Schema</code> type looks like. When we have a value of Schema[?]</code> we can pattern match on it and implement the following cases: Schema.Primitive</code> describes a primitive type - there are a lot of primitive types defined by ZIO Schema's StandardType</code> enum</li> Schema.Enum</code> describes a type with multiple cases (a sum type) such as a sealed trait</code> or enum</code></li> Schema.Record</code> describes a type with multiple fields (a product type) such as a case class</code></li> Schema.Map</code> represents a map with a key and value type</li> Schema.Sequence</code> represents a sequence of items of a given element type</li> Schema.Set</code> is a set of items of a given element type</li> Schema.Optional</code> represents an optional type (like an Option[T]</code>)</li> Schema.Either</code> is a special case of sum types representing either one or the other type (like an Either[A, B]</code>)</li> Schema.Lazy</code> is used to safely encode recursive types, it contains a function that evaluates into an inner Schema</code></li> Schema.Dynamic</code> represents a type that is dynamic - like a JSON</code> value</li> Schema.Transform</code> assigns a transformation function that converts a value of a type represented by the schema to a value of some other type. As we have no way to inspect these functions (they are compiled Scala functions) in our code generator, this is not very interesting for us now.</li> Schema.Fail</code> as already mentioned represents a failure in describing the data type</li> </ul> When traversing a Schema</code> recursively (for any reason), it is important to keep in mind that it can encode recursive types! A simple example is a binary tree: final case class Tree[A](label: A, left: Option[Tree], right: Option[Tree]) </code></pre> We can construct a Schema[Tree[A]]</code> if we have a Schema[A]</code>. This will be something like (pseudo-code): lazy val tree: Schema[Tree] = Schema.Record( Field("label", Schema[A]), Field("left", Schema.Optional(Schema.Lazy(() => tree))), Field("right", Schema.Optional(Schema.Lazy(() => tree))) ) </code></pre> If we are not prepared for recursive types we can easily get into an endless loop (or stack overflow) when processing these schemas. This is just one example of things to keep track of while converting a schema into a set of Rust definitions. If fields refer to the self type we want to use Box</code> so to put them on the heap. We also need to keep track of if everything within a generated type derives Ord</code> and Hash</code> - and if yes, we should derive an instance for the same type classes for our generated type as well. My preferred way to implement such recursive stateful transformation functions is to use ZIO Prelude's ZPure</code> type. It's type definition looks a little scary: sealed trait ZPure[+W, -S1, +S2, -R, +E, +A] </code></pre> ZPure</code> describes a purely functional computation which can: Emit log entries of type W</code></li> Works with an inital state of type S1</code></li> Results in a final state of type S2</code></li> Has access to some context of type R</code></li> Can fail with a value of E</code></li> Or succeed with a value of A</code></li> </ul> In this case we need the state, failure and result types only, but we could also take advantage of W</code> to log debug information within our schema transformation function. To make it easier to work with ZPure</code> we can introduce a type alias: type Fx[+A] = ZPure[Nothing, State, State, Any, String, A] </code></pre> where State</code> is our own case class containing everything we need: final case class State( typeRefs: Map[Schema[?], RustType], definitions: Chunk[RustDef], requiredCrates: Set[Crate], processed: Set[Schema[?]], stack: Chunk[Schema[?]], nameTypeIdMap: Map[Name, Set[TypeId]], schemaCaps: Map[Schema[?], Capabilities] ) </code></pre> We won't get into the details of the state type here, but I'm showing some fragments to get a feeling of working with ZPure</code> values. Some helper functions to manipulate the state can make our code much easier to read: private def getState: Fx[State] = ZPure.get[State] private def updateState(f: State => State): Fx[Unit] = ZPure.update[State, State](f) </code></pre> For example we can use updateState</code> to manipulate the stack</code> field of the state around another computation - before running it, we add a schema to the stack, after that we remove it: private def stacked[A, R](schema: Schema[A])(f: => Fx[R]): Fx[R] = updateState(s => s.copy(stack = s.stack :+ schema)) .zipRight(f) .zipLeft(updateState(s => s.copy(stack = s.stack.dropRight(1)))) </code></pre> This allows us to decide whether we have to wrap a generated field's type in Box</code> in the rust code: private def boxIfNeeded[A](schema: Schema[A]): Fx[RustType] = for state <- getState backRef = state.stack.contains(schema) rustType <- getRustType(schema) yield if backRef then RustType.box(rustType) else rustType </code></pre> By looking into state.stack</code> we can decide if we are dealing with a recursive type or not, and make our decision regarding boxing the field. Another example is to guard against infinite recursion when traversing the schema definition, as I explained before. We can define a helper function that just keeps track of all the visited schemas and shortcuts the computation if something has already been seen: private def ifNotProcessed[A](value: Schema[A])(f: => Fx[Unit]): Fx[Unit] = getState.flatMap: state => if state.processed.contains(value) then ZPure.unit else updateState(_.copy(processed = state.processed + value)).zipRight(f) </code></pre> Putting all these smaller combinators together we have an easy-to-read core recursive transformation function for converting the schema: private def process[A](schema: Schema[A]): Fx[Unit] = ifNotProcessed(schema): getRustType(schema).flatMap: typeRef => stacked(schema): schema match // ... </code></pre> In the end to run a Fx[A]</code> all we need to do is to provide an initial state: processSchema.provideState(State.empty).runEither </code></pre> Inspecting the endpoints</h2> We generated Rust code for all our types but we still need to generate HTTP clients. The basic idea is the same as what we have seen so far: Traversing the Endpoint</code> data structure for each endpoint we have</li> Generate some intermediate model</li> Pretty print this model to Rust code</li> </ul> The conversion once again is recursive, can fail, and requires keeping track of various things, so we can use ZPure</code> to implement it. Not repeating the same details, in this section we will talk about what exactly the endpoint descriptions look like and what we have be aware of when trying to process them. The first problem to solve is that currently ZIO Http does not have a concept of multiple endpoints. We are not composing Endpoint</code> values into an API, instead we first implement them to get Route</code> values and compose those. We can no longer inspect the endpoint definitions from the composed routes, so unfortunately we have to repeat ourselves and somehow compose our set of endpoints for our code generator. First we can define a RustEndpoint</code> class, similar to the RustModel</code> earlier, containing all the necessary information to generate Rust code for a single endpoint. We can construct it with a function: // ... object RustEndpoint: def fromEndpoint[PathInput, Input, Err, Output, Middleware <: EndpointMiddleware]( name: String, endpoint: Endpoint[PathInput, Input, Err, Output, Middleware], ): Either[String, RustEndpoint] = // ... </code></pre> The second thing to notice: endpoints do not have a name! If we look back to our initial example of getWorkerMetadata</code>, it did not have a unique name except the Scala value it was assigned to. But we can't observe that in our code generator (without writing a macro) so here we have chosen to just get a name as a string next to the definition. Then we can define a collection of RustEndpoint</code>s: final case class RustEndpoints(name: Name, originalEndpoints: Chunk[RustEndpoint]) </code></pre> and define a ++</code> operator between RustEndpoint</code> and RustEndpoints</code>. In the end we can use these to define APIs like this: for getDefaultProject <- fromEndpoint("getDefaultProject", ProjectEndpoints.getDefaultProject) getProjects <- fromEndpoint("getProjects", ProjectEndpoints.getProjects) postProject <- fromEndpoint("postProject", ProjectEndpoints.postProject) getProject <- fromEndpoint("getProject", ProjectEndpoints.getProject) deleteProject <- fromEndpoint("deleteProject", ProjectEndpoints.deleteProject) yield (getDefaultProject ++ getProjects ++ postProject ++ getProject ++ deleteProject).named("Project") </code></pre> The collection of endpoints also have a name ("Project"</code>). In the code generator we can use these to have a separate client (trait and implementation) for each of these groups of endpoints. When processing a single endpoint, we need to process the following parts of data: Inputs (endpoint.input</code>)</li> Outputs (endpoint.output</code>)</li> Errors (endpoint.error</code>)</li> </ul> Everything we need is encoded in one of these three fields of an endpoint, and all three are built on the same abstraction called HttpCodec</code>. Still there is a significant difference in what we want to do with inputs versus what we want to do with outputs and errors, so we can write two different traversals for gathering all the necessary information from them. Inputs</h3> When gathering information from the inputs, we are going to run into the following cases: HttpCodec.Combine</code> means we have two different inputs; we need both, so we have to process both inner codecs sequentially, both extending our conversion function's state.</li> HttpCodec.Content</code> describes a request body. Here we have a Schema</code> of our request body type and we can use the previously generated schema-to-rust type mapping to know how to refer to the generated rust type in our client code. It is important that in case there are multiple content codecs, that means the endpoint receives a multipart/form-data</code> body, while if there is only one codec, it accepts an application/json</code> representation of that.</li> HttpCodec.ContentStream</code> represents a body containing a stream of a given element type. We can model this as just a Vec<A></code> in the Rust side, but there is one special case here - if the element is a Byte</code>, ZIO Http expects a simple byte stream of type application/octet-stream</code> instead of a JSON-encoded array of bytes.</li> HttpCodec.Fallback</code> this represents the case when we should either use the first codec, or the second. A special case is when the right</code> value of Fallback</code> is HttpCodec.Empty</code>. This is how ZIO Http represents optional inputs! We have to handle this specially in our code generator to mark some of the input parameters of the generated API as optional parameters. We don't support currently the other cases (when right</code> is not empty) as it is not frequently used and was not required for the Golem API.</li> HttpCodec.Header</code> means we need to send a header in the request, which can be a static (value described by the endpoint) or dynamic one (where we need to add an extra parameter to the generated function to get a value of the header). There are a couple of different primitive types supported for the value, such as string, numbers, UUIDs.</li> HttpCodec.Method</code> defines the method to be used for calling the endpoint</li> HttpCodec.Path</code> describes the request path, which consists of a sequence of static and dynamic segments - for the dynamic segments the generated API need to have exposed function parameters of the appropriate type</li> HttpCodec.Query</code> similar to the header codec defines query parameters to be sent</li> HttpCodec.TransformOrFail</code> transforms a value with a Scala function - the same case as with Schema.Transform</code>. We cannot use the Scala function in our code generator so we just need to ignore this and go to the inner codec.</li> HttpCodec.Annotated</code> attaches additional information to the codecs that we are currently not using, but it could be used to get documentation strings and include them in the generated code as comments, for example.</li> </ul> Outputs</h3> For outputs we are dealing with the same HttpCodec</code> type but there are some significant differences: We can ignore Path</code>, Method</code>, Query</code> as they have no meaning for outputs</li> We could look for output headers but currently we ignore them</li> Fallback</code> on the other hand needs to be properly handled for outputs (errors, especially) because this is how the different error responses are encoded.</li> Status</code> is combined with Content</code> in these Fallback</code> nodes to describe cases. This complicates the code generator because we need to record "possible outputs" which are only added as real output once we are sure we will not get any other piece of information for them.</li> </ul> To understand the error fallback handling better, let's take a look at how it is defined in one of Golem's endpoint groups: val errorCodec: HttpCodec[HttpCodecType.Status & HttpCodecType.Content, LimitsEndpointError] = HttpCodec.enumeration[LimitsEndpointError]( HttpCodec.error[LimitsEndpointError.Unauthorized](Status.Unauthorized), HttpCodec.error[LimitsEndpointError.ArgValidationError](Status.BadRequest), HttpCodec.error[LimitsEndpointError.LimitExceeded](Status.Forbidden), HttpCodec.error[LimitsEndpointError.InternalError](Status.InternalServerError) ) </code></pre> This leads to a series of nested HttpCodec.Fallback</code>, HttpCodec.Combine</code>, HttpCodec.Status</code> and HttpCodec.Content</code> nodes. When processing them we first add values of possible outputs: final case class PossibleOutput(tpe: RustType, status: Option[Status], isError: Boolean, schema: Schema[?]) </code></pre> and once we have fully processed one branch of a Fallback</code>, we finalize these possible outputs and make them real outputs. The way these different error cases are mapped into different case classes of a a single error type (LimitsEndpointError</code>) also complicates things. When we reach a HttpCodec.Content</code> referencing Schema[LimitsEndpointError.LimitExceeded</code>] for example, all we see is a Schema.Record</code> - and not the parent enum! For this reason in the code generator we are explicitly defining the error ADT type: val fromEndpoint = RustEndpoint.withKnownErrorAdt[LimitsEndpointError].zio </code></pre> and we detect if all cases are subtypes of this error ADT and generate the client code according to that. The Rust client</h3> It is time to take a look at what the output of all this looks like. In this section we will examine some parts of the generated Rust code. Let's take a look at the Projects API. We have generated a trait</code> for all the endpoints belonging to it: #[async_trait::async_trait] pub trait Project { async fn get_default_project(&self, authorization: &str) -> Result<crate::model::Project, ProjectError>; async fn get_projects(&self, project_name: Option<&str>, authorization: &str) -> Result<Vec<crate::model::Project>, ProjectError>; async fn post_project(&self, field0: crate::model::ProjectDataRequest, authorization: &str) -> Result<crate::model::Project, ProjectError>; async fn get_project(&self, project_id: &str, authorization: &str) -> Result<crate::model::Project, ProjectError>; async fn delete_project(&self, project_id: &str, authorization: &str) -> Result<(), ProjectError>; } </code></pre> This is quite close to our original goal! One significant difference is that some type information is lost: project_id</code> was ProjectId</code> in Scala, and authorization</code> was TokenSecret</code> etc. Unfortunately with the current version of ZIO Schema these newtypes (or Scala 3 opaque types) are represented as primitive types transformed by a function. As explained earlier, we can't inspect the transformation function so all we can do is to use the underlying primitive type's schema here. This can be solved by introducing the concept of newtypes into ZIO Schema. The ProjectError</code> is a client specific generated enum</code> which can represent a mix of internal errors (such as not being able to call the endpoint) as well as the endpoint-specific domain errors: pub enum ProjectError { RequestFailure(reqwest::Error), InvalidHeaderValue(reqwest::header::InvalidHeaderValue), UnexpectedStatus(reqwest::StatusCode), Status404 { message: String, }, Status403 { error: String, }, Status400 { errors: Vec<String>, }, Status500 { error: String, }, Status401 { message: String, }, } </code></pre> So why are these per-status-code error types inlined here instead of generating the error ADT as a Rust enum</code> and using that? The reason is a difference between Scala and Rust: we have a single error ADT in Scala and we can still use its cases directly in the endpoint definition: sealed trait ProjectEndpointError object ProjectEndpointError { final case class ArgValidation(errors: Chunk[String]) extends ProjectEndpointError // ... } // ... HttpCodec.error[ProjectEndpointError.ArgValidation](Status.BadRequest), </code></pre> We do generate the corresponding ProjectEndpointError</code> enum in Rust: #[derive(Debug, Clone, PartialEq, Eq, Hash, Ord, PartialOrd, serde::Serialize, serde::Deserialize)] pub enum ProjectEndpointError { ArgValidation { errors: Vec<String>, }, // ... } </code></pre> but we cannot use ProjectEndpointError::ArgValidation</code> as a type in the above ProjectError</code> enum. And we cannot safely do something like Either[ClientError, ProjectEndpointError]</code> because in the endpoint DSL we just have a sequence of status code - error case pairs. There is no guarantee that one enum case is only used once in that mapping, or that every case is used at least once. For this reason the mapping from ProjectError</code> to ProjectEndpointError</code> is generated as a transformation function: impl ProjectError { pub fn to_project_endpoint_error(&self) -> Option<crate::model::ProjectEndpointError> { match self { ProjectError::Status400 { errors } => Some(crate::model::ProjectEndpointError::ArgValidation { errors: errors.clone() }), // ... } } } </code></pre> For each client trait we also generate a live implementation, represented by a struct containing configuration for the client: #[derive(Clone, Debug)] pub struct ProjectLive { pub base_url: reqwest::Url, pub allow_insecure: bool, } </code></pre> And the implementation of the client trait for these live structs are just using reqwest</code></a> (a HTTP client library for Rust) to construct the request from the input parameters exactly the way the endpoint definition described: async fn get_project(&self, project_id: &str, authorization: &str) -> Result<Project, ProjectError> { let mut url = self.base_url.clone(); url.set_path(&format!("v1/projects/{project_id}")); let mut headers = reqwest::header::HeaderMap::new(); // ... let mut builder = reqwest::Client::builder(); // ... let client = builder.build()?; let result = client .get(url) .headers(headers) .send() .await?; match result.status().as_u16() { 200 => { let body = result.json::<crate::model::Project>().await?; Ok(body) } 404 => { let body = result.json::<ProjectEndpointErrorNotFoundPayload>().await?; Err(ProjectError::Status404 { message: body.message }) } // ... } } </code></pre> Putting it all together</h2> At this point we have seen how ZIO Http describes endpoints, how ZIO Schema encodes Scala types, how we can use ZIO Parser to have composable printers and how ZIO Prelude can help with working with state in a purely functional code. The only thing remaining is to wire everything together and define an easy to use function that, when executed, creates all the required Rust files ready to be compiled. We can create a class for this: final case class ClientCrateGenerator(name: String, version: String, description: String, homepage: String, endpoints: Chunk[RustEndpoints]): </code></pre> Here endpoints</code> is a collection of a group of endpoints, as it was shown earlier. So first you can use RustEndpoint.fromEither</code> and ++</code> to create a RustEndpoints</code> value for each API you have, and then generate a client for all of those in one run with this class. The first thing to do is collect all the referenced Schema</code> from all the endpoints: private val allSchemas = endpoints.map(_.endpoints.toSet.flatMap(_.referredSchemas)).reduce(_ union _) </code></pre> Then we define a ZIO function (it is an effectful function, manipulating the filesystem!) to generate the files: def generate(targetDirectory: Path): ZIO[Any, Throwable, Unit] = for clientModel <- ZIO.fromEither(RustModel.fromSchemas(allSchemas.toSeq)) .mapError(err => new RuntimeException(s"Failed to generate client model: $err")) cargoFile = targetDirectory / "Cargo.toml" srcDir = targetDirectory / "src" libFile = srcDir / "lib.rs" modelFile = srcDir / "model.rs" requiredCrates = clientModel.requiredCrates union endpoints.map(_.requiredCrates).reduce(_ union _) _ <- Files.createDirectories(targetDirectory) _ <- Files.createDirectories(srcDir) _ <- writeCargo(cargoFile, requiredCrates) _ <- writeLib(libFile) _ <- writeModel(modelFile, clientModel.definitions) _ <- ZIO.foreachDiscard(endpoints): endpoints => val clientFile = srcDir / s"${endpoints.name.toSnakeCase}.rs" writeClient(clientFile, endpoints) yield () </code></pre> The steps are straightforward: Create a RustModel</code> using all the collected Schema[?]</code> values</li> Create all the required directories</li> Write a cargo file - having all the dependencies and other metadata required to compile the Rust project</li> Write a lib file - this is just a series of pub mod xyz;</code> lines, defining the generated modules which are put in different fiels</li> Write all the generated Rust types into a model.rs</code></li> For each endpoint group create a xyz.rs</code> module containing the client trait and implementation</li> </ul> For working with the file system - creating directories, writing data into files, we can use the [ZIO NIO</a>] library providing ZIO wrapprers for all these functionalities. Links</h3> Finally, some links: The code generator is open source and available at https://github.com/vigoo/zio-http-rust - the code and the repository itself is not documented at the moment, except by this blog post.</li> The generated Golem client for Rust is published as a crate to https://crates.io/crates/golem-client</li> The new Golem CLI, using the generated client, is also open sourced and can be found at https://github.com/golemcloud/golem-cli</li> Finally you can learn more about Golem itself at https://www.golem.cloud</li> </ul> [Video] Introducing ZIO Flow @ ZIO World 2023 2023-09-06T00:00:00+00:00 My short talk at ZIO World 2023</a> about the zio-flow library</a>. </iframe> [Video] Binary Serialization Of Evolving Data Types @ Functional Scala 2022 2022-12-01T00:00:00+00:00 My talk at Functional Scala 2022</a> about the binary serialization library desert</a>. </iframe> ZIO Kafka with transactions - a debugging story 2022-06-15T00:00:00+00:00 Introduction</h2> With one of our clients, we were working on a chain of services responsible for processing some logs coming from a Kafka topic, partition them by some properties like user and date, infer and aggregate the log schema and eventually store the partitioned data in a different format. The details of this use case are not important for understanding this post, in which I'm going to explain the recent changes to ZIO Kafka</a>, how was it implemented and how did we know it's not perfect, and the long story of investigation that finally resulted in a fix making this new feature usable in production. We only have to know about the first component of this data pipeline, which is a zio-kafka service: Consumes it's source topic. Each record in this topic consists one or more log entries for a given user. The kafka topic's partitions are not aligned with our target partition (of user/date), all kafka partitions may contain data from all users.</li> The service partitions the source data per user/date/hour and writes the log entries into Avro files in the local file system</li> It also computes and aggregates a log schema in memory for each of these files</li> It is using Kafka transactions to achieve exactly-once delivery</a>. This means that the processed records are not committed when they are written to the Avro files - there is a periodic event triggered every 30 seconds and at each rebalance that uploads the Avro files to S3, and then it emits Kafka messages to downstream containing references to the uploaded files and their aggregated schema, and it commits all the offsets of all the input Kafka topic's transactionally.</li> </ul> Stream restarting mode in zio-kafka</h2> When we first implemented this using zio-kafka and started to test it we have seen a lot of errors like Transiting to abortable error state due to org.apache.kafka.clients.consumer.CommitFailedException: Transaction offset Commit failed due to consumer group metadata mismatch: Specified group generation id is not valid."}</code> Group generation ID is a counter that gets incremented at each rebalance. The problem was that zio-kafka by default provides a continuous stream for partitions that survives rebalances. So we have a single stream per Kafka partition and after a rebalance we end up with some of them revoked and their streams stopped, some new streams created, but the ones that remained assigned are not going to be recreated. This works fine without using transactions, but it means your stream can contain messages from multiple generations. I first tried to solve this by detecting generation switches downstream but quickly realized this cannot work. It's too late to commit the previous generation when there are already records from the new generation; we have to do it before the rebalance finishes. To solve this I introduced a new mode in zio-kafka back in February 2022, with this pull request</a>. This adds a new mode to zio-kafka's core run loop which guarantees that every rebalance stops all the partition streams and create new ones every time. With this approach the library user can build the following logic on top of the "stream of partition streams" API of zio-kafka: Get the next set of partition streams</li> Merge and drain them all</li> Perform a flush - upload and commit everything before start working on the new set of streams</li> Repeat</li> </ul> This alone is still not enough - we have to block the rebalancing until we are done with the committing otherwise we would still get the invalid generation ID error. The onRevoke</code> and onAssigned</code> callbacks from the underlying Java Kafka library are working in a way that they block the rebalance process so that's the place where we can finish every processing for the revoked partitions. This extension point is provided by zio-kafka too but it's completely detached from the streaming API so I have introduced a rebalance event queue with with some promises and timeouts to coordinate this: In onRevoke</code> we publish a rebalance event and wait until it gets processed.</li> Because the new run loop mode is guaranteed to terminate all streams on rebalance (which is already happening, as we are in onRevoke</code>) we can be sure that eventually the main consumer stream's current stage - that drains the previous generation's partition streams will finish soon</li> and then it performs the rotation and fulfills the promise in the rebalance event.</li> </ul> With these changes our service started to work - but we had to know if it works correctly. QoS tests</h2> We implemented a QoS test running on Spark which periodically checks that we are not loosing any data with our new pipeline. Our log entries have associated unique identifiers coming from upstream - so what we can do in this test is to consume an hour amount of log records from the same Kafka topic our service is consuming from, and read all the Avro files produced in that period (with some padding of course to have some tolerance for lag) and then see if there are any missing records in our output. Another source of truth for the investigation was an older system doing something similar, resulting in the same input being available as archived CSV files in some cases. Comparing the archived CSV files with the archived Avro files I could verify that the QoS test itself works correctly, by checking that both methods report the same set of missing records. What we learned from these tests was that: there is data loss</li> the data loss is related to rebalances</li> </ul> To understand it's related to rebalances I was comparing failing QoS reports from several hours, figured out the ingestion time for some of the missing log records within these hours, and checked our service and infrastructure logs around that time. Every time there was a rebalance near the reported errors. Additional tests</h2> During the investigation I added some additional debug features and logs to the system. One of them is an extra verification step, enabled only temporarily in our development cluster, that aggregates all the log identifiers at the earliest point - as soon as they got in the zio-kafka partition stream</li> after uploading the Avro files and committing the records, it re-downloads all the files from S3 and checks if they got all the log identifiers that they should.</li> </ul> This never reported any error so based on that I considered the flow after zio-kafka correct. We also have a lot of debug logs coming from the Java Kafka library, from zio-kafka and from our service to help understanding the issue: After each rebalance, the Java library logs the offset it's starting to read from</li> When committing I'm logging the minimum and maximum offset contained by the committed and uploaded Avro files per kafka partition</li> All streams creation and termination are logged</li> If records within a partition stream are skipping an offset (this was never logged actually)</li> </ul> I wrote a test app that reads our service's logs from a given period, logged from all the Kubernetes pods it's running on, and runs a state machine that verifies that all the logged offsets from the different pods are in sync. It fails in two cases: When a pod resets its offsets to something that was previously seen in the logs and there is a gap</li> When a pod rotates a kafka without it got assigned to that pod first (so if multiple pods would somehow consume the same partition which Kafka prevents)</li> </ul> I tried for long to write integration tests using embedded Kafka (similar to how it's done in zio-kafka's test suite) that reproduces the data loss issue, without any luck. In all my simulated cases everything works perfectly. Theories and fixes</h2> From logs from the time ranges where the data loss is reported from, these additional checks were not showing any discrepancies. This could only mean two things: All the kafka/zio-kafka level is correct but we are still loosing data in our service-specific logic, somewhere in writing to Avro-s and uploading to S3.</li> On Kafka level everything is fine but somehow zio-kafka does not pass all the records to our service's logic</li> </ul> I trusted the validation mode I described earlier (the one that re-downloads the data) so I ruled out the first option. zio-kafka internals</h2> Before discussing the fixes I tried to make in zio-kafka, first let's talk about how the library works. The zio-kafka library wraps the Java library for Kafka and provides a ZIO Stream interface for consuming the records. As I mentioned earlier, it creates a separate stream for each kafka partition assigned to the consumer. The primary operation on the Java interface is called poll</code>. This method is responsible for fetching data for all the subscribed partitions for a given timeout. Another important property is that in case of rebalancing, the poll</code> is blocked until the rebalancing completes, and it calls the already mentioned revoked/assigned callbacks in this blocked state. Another thing it has to support is back pressure. We don't want this poll</code> to fetch more and more data for partitions where we did process the previous records yet. In other words, upstream demand in our ZIO Streams must control what partitions we poll</code>. In the Java level this is controlled by pausing and resuming individual partitions. So let's see a summary of how the consumer streams work: Each partition stream is a repeated ZIO effect that enqueues a Request</code> in a queue and then waits for the promise contained in this request to be fulfilled. The promise will contain a chunk of records fetched from Kafka if everything went well.</li> There is a single (per consumer) run loop which periodically calls poll</code>. Before calling it, it pauses/resumes partitions based on which partitions has at least one Request</code> since the last poll</code>.</li> This, as ZIO streams are pull based, implements the back pressure semantics mentioned earlier.</li> </ul> There is a similar mechanism for gathering commit requests and then performing them as part of the run loop but in our use case that is not used - the transactional producer is independent of this mechanism. There is one more concept which is very important for to understand the problem: buffered records. Imagine that we are consuming five partitions, 1 .. 5</code> and only have a request (downstream pull) for partition 1</code>. This means we are pausing 2 .. 5</code> and do a poll</code> but what if the resulting record set contains records from other partitions? There could be multiple reason for this (and some of them may not be possible in practice), for example there could be some data already buffered within the Java library for the paused partitions, or maybe a rebalance assigns some new partitions which are not paused yet (as we don't know we are going to get them) resulting in immediately fetching some data for them. The library handles these cases in a simple way: it buffers these records which were not requested in a per-partition map, and when a partition is pulled next time, it will not only give the records returned by poll</code> to the request's promise, but also all the buffered ones, prepended to the new set of records. Another important detail for this investigation is that we don't care about graceful shutdown, or if records got lost during shutdown. This is also very interesting in general, but our service is not trying to finish writing and uploading all data during shutdown, it simply ignores the partial data and quits without committing them so they get reprocessed as soon as possible in another consumer. What happens during rebalancing? Let's forget the default mode of zio-kafka for this discussion and focus on the new mode which restarts all the partition streams every time. We don't know in advance that a rebalance will happen, it happens during the call to poll</code>. The method in the run loop that contains this logic is called handlePoll</code> and does roughly the following (in our case): store the current state (containing the current streams, requests, buffered records etc) in a ref</li> pause/resume partitions based on the current requests, as described earlier</li> call poll</code> during poll</code> in the revoked callback we end all partition streams. This means they get an interrupt signal and they stop. As I mentioned earlier, in this mode the consumer merges the partition streams and drain them; this is the other side of it, interrupting all the streams so we know that eventually this merged stream will also stop.</li> dropping all the buffered records, but first adding them to a drain queue (this is a fix that was not part of the original implementation). It is now guaranteed that the partition streams will get the remaining buffered elements before they stop.</li> storing the fact of the rebalancing, so the rest of handlePoll</code> knows about it when poll</code> returns.</li> </ul> </li> once poll</code> returned, buffer all records for all unrequested partitions. this is another place where a fix was made, currently we treat all records unrequested in case of a rebalancing, because all the streams were restarted, so the original requests were made by the previous set of streams; fulfilling them would loose data because the new streams are not waiting for the same promises.</li> the next step would be to fulfill all the requests that we can by using the combination of buffered records and the poll</code> result. But we had a rebalance and dropped all the requests! So this step must not do anything.</li> finally we start new streams for each assigned partition</li> </ul> So based on all this, and the theory that the commits/offsets are all correct but somehow data is lost between the Java library and the service logic, the primary suspect was the buffered records. Let's see what fixes and changes I made, in time order: Fix attempt 1</h2> The first time I suspected buffered records are behind the issue I realized that when we end all partition streams during rebalancing, we loose the buffered records. This is not a problem if those partitions are really revoked - it means there was no demand for those partitions, so it's just that some records were read ahead and now they get dropped and will be reprocessed on another consumer. But if the same partition is "reassigned" to the same consumer, this could be a data loss! The reason is that there is an internal state in Kafka which is a per-consumer, per-partition position. In this case this position would point to after the buffered records, so the next poll</code> will get the next records and the previously buffered ones will not be prepended as usual because the revocation clears the buffer. Note that this whole problem would not exist if the reassigned partitions get reseted to the last committed offset after rebalancing. I don't think this is the case, only when a new partition is assigned to a consumer with no previous position. My first fix was passing the buffered records to the user-defined revoke handler so it could write the remaining records to the Avro files before uploading them. This was just a quick test, as it does not really fit into the API of zio-kafka. Fix attempt 2</h2> After playing with the first fix for a while I thought it solved the issue but it was just not reproducing - it is not completely clear why, probably I missed some test results. But I wrote a second version of the same fix, this time by adding the remaining buffered elements to the end of the partition streams before they stop, instead of explicitly passing them to the revoke handler. This should work exactly the same but handles the problem transparently. Fix attempt 3</h2> After some more testing it was clear that the QoS tests were still showing data loss. The investigation continued and the next problem I have found was that in handlePoll</code> after a rebalance we were not storing the buffered records anymore in this "restarting streams" mode. I did not catch this in the first fix attempts I was focusing on dealing with the buffered records at the end of the revoked streams. What does it mean it was not storing the buffered records? In handlePoll</code> there is a series of state manipulation functions and the buffered records map is part of this state. The logic here is quite complicated and it very much depends on whether we are running the consumer in normal or stream restarting mode. The problem was that for some reason after a rebalance (in the new mode only) this buffered records field was cleared instead of preserving records from before the rebalance. Fix attempt 4</h2> Very soon turned out that my previous fix was not doing anything, because there was one more problem in the state handling in handlePoll</code>. As I wrote, it bufferes only those records which were not requested. For those partitions which have a request, it fulfills these requests with the new records instead. When the reassigned partitions are not restarted during rebalancing (as in the normal mode) this is OK but for us, as we are creating new streams, the old requests must be dropped and not taken into account when deciding which records to buffer. In other words, in restarting streams mode we have to buffer all records after a rebalance. Fix attempt 5</h2> I was very confident about the previous fix but something was still not OK, the test continued to report data loss. After several code reviews and discussions, I realized that it is not guaranteed that the onRevoked</code> and onAssigned</code> callbacks are called within a single poll</code>! My code was not prepared for this (the original zio-kafka code was, actually, but I did not realize this for a long time). First of all I had to change the way how the rebalance callbacks are passing information to the poll handler. The previously added rebalance event (which was a simple case class) was changed to be either Revoked</code>, Assigned</code> or RevokedAndAssigned</code> and I made sure that for each case all the run loop state variables are modified correctly. Immediately after deploying this, I saw evidence in the logs that indeed the revoked and assigned callbacks are called separately, so the fix was definitely needed. The only problem was that I did not really understand how could this cause data loss, and by doing some rebalancing tests it turned out that the problem still exists. Fix attempt 6</h2> One more thing I added in the previous attempt was a log in a place that was suspicious to me and I did not care about it earlier. When adding requests to the run loop - these are added to the run loop's command queue when a partition stream tries to pull, completely asynchronous to the run loop itself - it was checking if currently the run loop is in the middle of a rebalancing. So in case the rebalancing takes multiple poll</code>s, as we have seen, it is possible that between the onRevoked</code> and onAssigned</code> events we get some new requests from the streams. In the restart-streams mode all partition streams are interrupted on the revoke event, and no new streams are created until the assigned event. This means that these requests can only come from the previous streams so they should be ignored. But what zio-kafka was doing was to add these requests to the run loop's pending requests. This is correct behavior in its normal mode, because on rebalance some of the streams survive it and their requests can be still fulfilled. But in our case it is incorrect, because after the assignment is done and some records are fetched by poll</code>, these pending requests get fulfilled with them, "stealing" the records from the new partition streams! At this point I really felt like this was the last missing piece of the puzzle. Conclusion</h2> And it was! The final set of fixes are published in this pull request</a>. The service and its tests are running perfectly since more than 10 days, proving that it is correct. [Video] ZIO Parser @ ZIO World 2022 2022-03-11T00:00:00+00:00 My talk at ZIO World 2022</a> introducing ZIO Parser</a> </iframe> [Video] Generating Libraries @ Functional Scala 2021 2021-12-03T00:00:00+00:00 My talk at Functional Scala 2021</a> about generating libraries in Scala: </iframe> Writing kubectl plugins with ZIO K8s 2021-03-07T00:00:00+00:00 Originally posted at the Ziverge blog</a>. Andrea Peruffo recently published a blog post on the Lightbend blog</a> about how they migrated a kubectl</code> plugin from Golang to Scala using the Fabric8</a> Kubernetes client and a few Scala libraries. This is a perfect use case for the zio-k8s library</a> announced two weeks ago</a>, so we decided to write this post demonstrating how to implement the same example using the ZIO ecosystem. We are going to implement the same example, originally described in the Write a kubectl plugin in Java with JBang and fabric8</a> article, using the following libraries: ZIO</a></li> ZIO K8s</a></li> ZIO Logging</a></li> clipp</a></li> sttp</a></li> circe</a></li> </ul> The source code of the example can be found here</a>. The linked blog post does a great job in explaining the benefits and difficulties of compiling to native image with GraalVM so we are not going to repeat it here. Instead, we will focus on how the implementation looks in the functional Scala world. The example has to implement two kubectl commands: version</code> to print its own version and list</code> to list information about all Pods of the Kubernetes cluster in either ASCII table, JSON or YAML format. CLI parameters</h3> Let's start with defining these command line options with the clipp</a> library! First, we define the data structures that describe our parameters: sealed trait Format object Format { case object Default extends Format case object Json extends Format case object Yaml extends Format } sealed trait Command object Command { final case class ListPods(format: Format) extends Command case object Version extends Command } final case class Parameters(verbose: Boolean, command: Command) </code></pre> When parsing the arguments (passed as an array of strings), we need to either produce a Parameters</code> value or fail and print some usage information. With clipp</code>, this is done by defining a parameter parser using its parser DSL in a for comprehension: val spec = for { _ <- metadata("kubectl lp") verbose <- flag("Verbose logging", 'v', "verbose") commandName <- command("version", "list") command <- commandName match { case "version" => pure(Command.Version) case "list" => for { specifiedFormat <- optional { namedParameter[Format]( "Output format", "default|json|yaml", 'o', "output" ) } format = specifiedFormat.getOrElse(Format.Default) } yield Command.ListPods(format) } } yield Parameters(verbose, command) </code></pre> As we can see, it is possible to make decisions in the parser based on the previously parsed values, so each command can have a different set of arguments. In order to parse the possible output formats, we also implement the ParameterParser</code> type class for Format</code>: implicit val parameterParser: ParameterParser[Format] = new ParameterParser[Format] { override def parse(value: String): Either[String, Format] = value.toLowerCase match { case "default" => Right(Format.Default) case "json" => Right(Format.Json) case "yaml" => Right(Format.Yaml) case _ => Left(s"Invalid output format '$value', use 'default', 'json' or 'yaml'") } override def example: Format = Format.Default } </code></pre> This is all we need to bootstrap our command line application. The following main function parses the arguments and provides the parsed Parameters</code> value to the ZIO</code> program: def run(args: List[String]): URIO[zio.ZEnv, ExitCode] = { val clippConfig = config.fromArgsWithUsageInfo(args, Parameters.spec) runWithParameters() .provideCustomLayer(clippConfig) .catchAll { _: ParserFailure => ZIO.succeed(ExitCode.failure) } } def runWithParameters(): ZIO[ZEnv with ClippConfig[Parameters], Nothing, ExitCode] = // ... </code></pre> Working with Kubernetes</h3> In runWithParameters</code>, we have everything needed to initialize the logging and Kubernetes modules and perform the actual command. Before talking about the initialization though, let's take a look at how we can list the pods! We define a data type holding all the information we want to report about each pod: case class PodInfo(name: String, namespace: String, status: String, message: String) </code></pre> The task now is to fetch all pods from Kubernetes and construct PodInfo</code> values. In zio-k8s</code> getting a list of pods is defined as a ZIO Stream, which under the hood sends multiple HTTP requests to Kubernetes taking advantage of its pagination capability. In this stream each element will be a Pod</code> and we can start processing them one by one as soon they arrive over the wire. This way the implementation of the list</code> command can be something like this: def run(format: Format) = for { _ <- log.debug("Executing the list command") _ <- pods .getAll(namespace = None) .mapM(toModel) .run(reports.sink(format)) .catchAll { k8sFailure => console.putStrLnErr(s"Failed to get the list of pods: $k8sFailure") } } yield () </code></pre> Let's take a look at each line! First, log.debug</code> uses the ZIO logging library. We are going to initialize logging in a way that these messages only appear if the --verbose</code> option was enabled. Then pods.getAll</code> is the ZIO Stream provided by the ZIO K8s library. Not providing a specific namespace means that we are getting pods from all namespaces. With mapM(toModel)</code> we transform each Pod</code> in the stream to our PodInfo</code> data structure. Finally we run</code> the stream into a sink that is responsible for displaying the PodInfo</code> structures with the specific output format. The Pod</code> objects returned in the stream are simple case classes containing all the information available for the given resource. Most of the fields of these case classes are optional though, even though we can be sure that in our case each pod would have a name, a namespace and a status. To make working with these data structures easier within a set of expectations, they feature getter methods that are ZIO functions either returning the field's value, or failing if they are not specified. With these we can implement toModel</code>: def toModel(pod: Pod): IO[K8sFailure, PodInfo] = for { metadata <- pod.getMetadata name <- metadata.getName namespace <- metadata.getNamespace status <- pod.getStatus phase <- status.getPhase message = status.message.getOrElse("") } yield PodInfo(name, namespace, phase, message) </code></pre> An alternative would be to just store the optional values in PodInfo</code> and handle their absence in the report sink. Let's talk about the type of the above defined run</code> function: ZIO[Pods with Console with Logging, Nothing, Unit] </code></pre> The ZIO environment precisely specifies the modules used by our run</code> function: Module</th> Description</th></tr></thead> Pods</code></td> for accessing K8s pods</td></tr> Console</code></td> for printing errors on the standard error channel with putStrLnErr</code></td></tr> Logging</code></td> for emitting some debug logs</td></tr> </tbody></table> The error type is Nothing</code> because it can never fail - all errors are catched and displayed for the user within the run function. Initialization</h3> Now we can see that in order to run the list</code> command in runWithParameters</code>, we must provide Pods</code> and Logging</code> modules to our implementation (Console</code> is part of the default environment and does not need to be provided). These modules are described by ZIO Layers which can be composed together to provide the environment for running our ZIO program. In this case we need to define a logging layer and a kubernetes pods client layer and then compose the two for our list</code> implementation. Let's start with logging: def configuredLogging(verbose: Boolean): ZLayer[Console with Clock, Nothing, Logging] = { val logLevel = if (verbose) LogLevel.Trace else LogLevel.Info Logging.consoleErr(logLevel) >>> initializeSlf4jBridge } </code></pre> We create a simple ZIO console logger that will print lines to the standard error channel; the enabled log level is determined by the verbose</code> command line argument. As this logger writes to the console and also prints timestamps, our logging layer requires Console with Clock</code> to be able to build a Logging</code> module. Enabling the SLF4j bridge guarantees that logs coming from third party libraries will also get logged through ZIO logging. In our example this means that when we enable verbose logging, our kubectl</code> plugin will log the HTTP requests made by the Kubernetes library! The second layer we must define constructs a Pods</code> module: val pods = k8sDefault >>> Pods.live) </code></pre> By using k8sDefault</code> we ask zio-k8s</code> to use the default configuration chain, which first tries to load the kubeconfig</code> and use the active context stored in it. This is exactly what kubectl</code> does, so it is the perfect choice when writing a kubectl</code> plugin. Other variants provide more flexibility such as loading custom configuration with the ZIO Config</a> library. Once we have a k8s configuration we just feed it to the set of resource modules we need. In this example we only need to access pods. In more complex applications this would be something like k8sDefault >>> (Pods.live ++ Deployments.live ++ ...)</code>. With both layers defined, we can now provide them to our command implementation: runCommand(parameters.command) .provideCustomLayer(logging ++ pods) </code></pre> Output</h3> The last thing missing is the report sink that we are running the stream of pods into. We are going to define three different sinks for the three output types. Let's start with JSON! def sink[T: Encoder]: ZSink[Console, Nothing, T, T, Unit] = ZSink.foreach { (item: T) => console.putStrLn(item.asJson.printWith(Printer.spaces2SortKeys)) } </code></pre> The JSON sink requires Console</code> and then for each element T</code> it converts it to JSON and pretty prints it to console. Note that this is going to be a JSON document per each line. We could easily define a different sink that collects each element and produces a single valid JSON array of them: def arraySink[T: Encoder]: ZSink[Console, Nothing, T, T, Unit] = ZSink.collectAll.flatMap { (items: Chunk[T]) => ZSink.fromEffect { console.putStrLn(Json.arr(items.map(_.asJson): _*).printWith(Printer.spaces2SortKeys)) } } </code></pre> The T</code> type paramter in our example will always be PodInfo</code>. By requiring it to have an implementation of circe's Encoder</code> type class we can call .asJson</code> on instances of T</code>, encoding it into a JSON object. We can derive these encoders automatically: implicit val encoder: Encoder[PodInfo] = deriveEncoder </code></pre> Producing YAML output is exactly the same except of first converting the JSON model to YAML with asJson.asYaml</code>. The third output format option is to generate ASCII tables. We implement that with the same Java library as the original post, called asciitable</code></a>. In order to separate the specification of how to convert a PodInfo</code> to a table from the sink implementation, we can define our own type class similar to the JSON Encoder</code>: trait Tabular[T] { /** Initializes a table by setting properties and adding header rows */ def createTableRenderer(): ZManaged[Any, Nothing, AsciiTable] /** Adds a single item of type T to the table created with [[createTableRenderer()]] */ def addRow(table: AsciiTable)(item: T): UIO[Unit] /** Adds the table's footer and renders it to a string */ def renderTable(table: AsciiTable): UIO[String] } </code></pre> We can implement this for PodInfo</code> and then use a generic sink for printing the result table, similar to the previous examples: def sink[T](implicit tabular: Tabular[T]): ZSink[Console, Nothing, T, T, Unit] = ZSink.managed[Console, Nothing, T, AsciiTable, T, Unit](tabular.createTableRenderer()) { table => // initialize the table ZSink.foreach(tabular.addRow(table)) <* // add each row printResultTable[T](table) // print the result } def printResultTable[T]( table: AsciiTable )(implicit tabular: Tabular[T]): ZSink[Console, Nothing, T, T, Unit] = ZSink.fromEffect { tabular .renderTable(table) .flatMap(str => console.putStrLn(str)) } </code></pre> Trying it out</h3> With the report sinks implemenented we have everything ready to try out our new kubectl</code> plugin! We can compile the example to native image and copy the resulting image to a location on the PATH</code>: sbt nativeImage cp target/native-image/kubectl-lp ~/bin </code></pre> Then use kubectl lp</code> to access our custom functions: The Coralogix Operator: A Tale of ZIO and Kubernetes 2021-02-16T00:00:00+00:00 My blog post published at the Coralogix blog</a> about using zio-k8s</a> for writing operators. ZIO-AWS with ZIO Query 2020-11-01T00:00:00+00:00 A few years ago I wrote a post</a> about how I refactored one of our internal tools at Prezi</a>. This command line tool was able to discover a set of AWS resources and present them in a nice human readable way. The primary motivation at that time was to introduce circuit breaking to survive AWS API rate limits. I have recently published a set of libraries, zio-aws</a>, and thought it would be interesting to rewrite this tool on top of it, and use this opportunity to try out ZIO Query</a> on a real-world example. In this post I'm going to show step by step how to build an efficient and easily extensible query tool with the help of ZIO libraries. The full source can be found on GitHub</a>. The task</h2> The CLI tool we build will get an arbitrary string as an input, and search for it in various AWS resources. Once it has a match, it has to traverse a graph of these resources and finally pretty-print all the gathered information to the console. The provided input could mean any of the following: An EC2 instance ID</li> An ELB (load balancer)'s name</li> An ElasticBeanstalk environment name or ID</li> An ElasticBeanstalk application name</li> An ASG (auto-scaling group) ID</li> </ul> For the level of detail to be reported I copied the original tool. This means finding all the related resources in the above sets (plus among launch configurations) but only include a single EC2 instance in the output if it was explicitly queried. So for example if the search term matches an ELB that belongs to an ElasticBeanstalk environment, the report will contain the EB app and all its other environments as well, but won't show individual instances. This choice does not affect the design and could be easily changed or extended with additional resource types. AWS client</h2> For querying the above mentioned resources, we have to call four different AWS services. The zio-aws</code> project adds a streaming ZIO wrapper for all the libraries in AWS Java SDK v2</a>, each published as separate artifact: libraryDependencies ++= Seq( "io.github.vigoo" %% "zio-aws-autoscaling" % zioAwsVersion, "io.github.vigoo" %% "zio-aws-ec2" % zioAwsVersion, "io.github.vigoo" %% "zio-aws-elasticloadbalancing" % zioAwsVersion, "io.github.vigoo" %% "zio-aws-elasticbeanstalk" % zioAwsVersion "io.github.vigoo" %% "zio-aws-netty" % zioAwsVersion, ) </code></pre> In addition to loading the necessary client libraries, we also need one of the http implementations, in this case I chose the default Netty. Other possibilities are akka-http and http4s. If your application already uses one of these for other HTTP communications you may want to use them to share their configuration and pools. The client libraries have a ZStream</code> API for all the operations that either support streaming (like for example S3 download/upload) or pagination, and ZIO</code> wrapper for non-streaming simple operations. Instead of using the Java SDK's builders, the requests are described by case classes, and the result types have convenience accessors to handle the nullable results. Let's see some examples! We can get information about EB applications with the ElasticBeanstalk API's DescribeApplications</code> operation</a>. This is defined like the following in zio-aws-elasticbeanstalk</code>: def describeApplications(request: DescribeApplicationsRequest): ZIO[ElasticBeanstalk, AwsError, DescribeApplicationsResponse.ReadOnly] type ApplicationName = String case class DescribeApplicationsRequest(applicationNames: Option[Iterable[ApplicationName]]) case class DescribeApplicationsResponse(applications : Option[Iterable[ApplicationDescription]]) object DescribeApplicationsResponse { trait ReadOnly { def editable: DescribeApplicationsResponse def applicationsValue: Option[List[ApplicationDescription.ReadOnly]] def applications: ZIO[Any, AwsError, List[ApplicationDescription.ReadOnly]] } } </code></pre> A few things to notice here: The client function requires the ElasticBeanstalk</code> module. We will see how to set up the dependencies in the Putting all together section.</li> The primitive types defined by the AWS schema are currently simple type aliases. In the future they will be probably replaced by zio-prelude</a>'s newtypes.</li> Each wrapper type has a ReadOnly</code> trait and a case class. The case classes are used as input, and the read-only interfaces as outputs. This way the result provided by the Java SDK can be accessed directly and it only has to be rewrapped in the case class if it is passed to another call as input.</li> In many cases the AWS SDK describes fields as optional even if in normal circumstances it would never be None</code>. To make it more convenient to work with these, the ReadOnly</code> interface contains accessor functions which fail with FieldIsNone</code> in case the field did not have any value. The pure optional values can be accessed with the xxxValue</code> variants. See applications</code> and applicationsValue</code> in the above example.</li> </ul> For operations support pagination, the wrapper functions return a stream. The actual first AWS call happens when the stream is first pulled. An example for this that we have to use in this application is the EC2 API's DescribeInstances</code> operation</a>. def describeInstances(request: DescribeInstancesRequest): ZStream[Ec2, AwsError, Reservation.ReadOnly] </code></pre> The pagination can be controlled by setting the MaxResults</code> property in DescribeInstancesRequest</code>. For the user of the describeInstances</code> function this is completely transparent, the returned stream will gather all the results, possibly by performing multiple AWS requests. Queries</h2> We could implement the resource discovery directly using the low level AWS wrappers described above, using ZIO's tools to achieve concurrency. There are several things to consider though: We don't know what resource we are looking for, so we should start multiple queries in parallel to find a match as soon as possible</li> Some queries return additional data that could be reused later. For example it is not possible to search for an ELB by a instance ID contained by it; for that we have to query all load balancers and check the members on client side.</li> There are AWS operations that support querying multiple entities, for example by providing a list of IDs to look for</li> We should minimize the number of calls to AWS, both for performance reasons, and to avoid getting rate limited</li> </ul> We can achieve all this by expressing our AWS queries with a higher level abstraction, delegating the execution to a library called ZIO Query</a>. This library let us define composable queries to arbitrary data sources, and it automatically provides pipelining, batching and caching. A perfect match for the problem we have to solve here. To be able to cache results that became available as a side effect of a query, we need a recent improvement</a> that is not published yet, so aws-query</code> currently uses a snapshot release of zio-query</code>: libraryDependencies += "dev.zio" %% "zio-query" % "0.2.5+12-c41557f7-SNAPSHOT" </code></pre> The first step is to define custom data sources. Data sources must implement a function runAll</code> with the following signature: def runAll(requests: Chunk[Chunk[A]]): ZIO[R, Nothing, CompletedRequestMap] </code></pre> Here A</code> is the request type specific to a given data source (extending Request[E, A]</code>, and the returned CompletedRequestMap</code> will store an Either[E, A]</code> result for each request. The two nested chunks model sequential and parallel execution: the requests in the inner chunks can be executed in parallel, while these batches contained by the outer chunk must be performed sequentially. In practice we won't implement this method but use DataSource.Batched</code> that is a simplified version that can perform requests in parallel but does not make further optimizations on the requests to be performed sequentially. What should belong to one data source? It could be a single data source for all the AWS queries, or one per service, or one per resource type. The best choice in this case is to have one for each resource type, for the following reasons: There are no opportunities to do any cross-resource-type caching. For example when we are querying EC2 instances, we won't fetch auto scaling groups as a side effect.</li> If all requests are about the same data type, implementing the data source is much simpler</li> </ul> Let's see a simple example. EC2 instances can be queried by instance ID with the DescribeInstances</code></a> operation, and it supports querying for multiple IDs in a single request. We first define a request type: case class GetEc2Instance(id: InstanceId) extends Request[AwsError, Instance.ReadOnly] </code></pre> Then the data source: val ec2InstancesDataSource: DataSource[Logging with Ec2, GetEc2Instance] = DataSource.Batched.make("ec2") { (requests: Chunk[GetEc2Instance]) => import AwsDataSource._ for { result <- ec2.describeInstances(DescribeInstancesRequest(instanceIds = Some(requests.map(_.id)))) .mapM(_.instances) .flatMap(instances => ZStream.fromIterable(instances)) .foldM(CompletedRequestMap.empty) { (resultMap, item) => for { instanceId <- item.instanceId } yield resultMap.insert(GetEc2Instance(instanceId))(Right(item)) } .recordFailures("DescribeInstances", requests) } yield result } </code></pre> Here requests</code> holds a set of GetEc2Instance</code> requests to be performed in parallel. We can simply do this by taking all the instance IDs from these requests and performing a single describeInstances</code> AWS call. The result, as I explained before, is a ZStream</code> of instances. We have to construct a CompletedRequestMap</code> holding one entry for each request in requests</code>. To do this we foldM</code> the stream, using the instanceId</code> accessor function to reconstruct the request value for each item in the result stream. The .recordFailures</code> function is a helper extension method defined in AwsDataSource</code>. It catches all errors and produces a CompletedRequestMap</code> where all requested items are recorded as failures: def recordFailures[A](description: String, requests: Iterable[Request[AwsError, A]]): ZIO[R, Nothing, CompletedRequestMap] = f.catchAll { error => log.error(s"$description failed with $error") *> ZIO.succeed { requests.foldLeft(CompletedRequestMap.empty) { case (resultMap, req) => resultMap.insert(req)(Left(error)) } } } </code></pre> This is necessary because the data source requires a function of type Chunk[A] => ZIO[R, Nothing, CompletedRequestMap]</code> that cannot fail. With the data source defined, we can define primitive queries on it: def getEc2Instance(id: InstanceId): ZQuery[Logging with Ec2, AwsError, Instance.ReadOnly] = ZQuery.fromRequest(GetEc2Instance(id))(ec2InstancesDataSource) </code></pre> A more complex example is ebEnvDataSource</code>, the data source of ElasticBeanstalk environments. For this resource, we have different request types: sealed trait EbEnvRequest[+A] extends Request[AwsError, A] case class GetEnvironmentByName(name: EnvironmentName) extends EbEnvRequest[Option[EnvironmentDescription.ReadOnly]] case class GetEnvironmentById(id: EnvironmentId) extends EbEnvRequest[Option[EnvironmentDescription.ReadOnly]] case class GetEnvironmentByApplicationName(name: ApplicationName) extends EbEnvRequest[List[EnvironmentDescription.ReadOnly]] </code></pre> In the data source implementation we get a Chunk</code> of EbEnvRequest</code> to be performed in parallel. We start it by separating it per request type: val byName = requests.collect { case GetEnvironmentByName(name) => name } val byId = requests.collect { case GetEnvironmentById(id) => id } val byAppName = requests.collect { case GetEnvironmentByApplicationName(name) => name } </code></pre> Then for each of these collections, if not empty, we can perform a describeEnvironments</code> AWS call and then fold the result stream to create partial CompletedRequestMap</code> values. What is interesting here is that if we already queried an environment by either name or id or it's application name, we already know both its identifier and name, so we can store additional items in CompletedRequestMap</code> that will be cached and reused in future queries. For example this is how the query by-id gets processed: resultMap <- elasticbeanstalk .describeEnvironments(DescribeEnvironmentsRequest(environmentIds = Some(byId))) .foldM(initialResultMap) { (resultMap, item) => for { name <- item.environmentName id <- item.environmentId } yield resultMap .insert(GetEnvironmentById(id))(Right(Some(item))) .insert(GetEnvironmentByName(name))(Right(Some(item))) } .recordFailures("DescribeEnvironmentRequest(id)", byId.map(GetEnvironmentById)) </code></pre> For all three request types we describe the computation to create a partial CompletedRequestMap</code> for them. Then we can implement the data source by executing these (maximum) three queries in parallel and combining the results: byNameResultMap .zipWithPar(byIdResultMap)(_ ++ _) .zipWithPar(byAppNameResultMap)(_ ++ _) </code></pre> There are some cases where being able to query all instances of a given resource is also a requirement. An example is load balancers, where the only way to find if an ELB contains a given EC2 instance is to query all ELBs and check their members. There are a few more cases that require a very similar implementation, so it makes sense extracting it to a common place. We define an AllOrPerItem</code> trait that defines the specifics per use case: trait AllOrPerItem[R, Req, Item] { val name: String def isGetAll(request: Req): Boolean def isPerItem(request: Req): Boolean val allReq: Req def itemToReq(item: Item): ZIO[R, AwsError, Req] def getAll(): ZStream[R, AwsError, Item] def getSome(reqs: Set[Req]): ZStream[R, AwsError, Item] def processAdditionalRequests(requests: Chunk[Req], partialResult: CompletedRequestMap): ZIO[R, Nothing, CompletedRequestMap] = ZIO.succeed(partialResult) } </code></pre> By implementing these one-liners the actual data source implementation can be a shared code defined in AllOrPerItem.make</code>. It's very similar to the examples already seen. If any of the requests is the get all request, that's the only thing to be performed, and all the result items will be cached. Otherwise a single batched request is made. These primitive ZQuery</code>s then can be composed to more complex queries. For example the following code: for { instance <- ec2query.getEc2Instance(instanceId) imageId <- ZQuery.fromEffect(instance.imageId) imgElb <- (ec2query.getImage(imageId) <&> elbquery.loadBalancerOf(instanceId)) (image, elb) = imgElb elbReport <- optionally(elb)(getElbReport) result <- // ... } yield result </code></pre> This is part of the definition of a query of type ZQuery[QueryEnv, AwsError, LinkedReport[Ec2InstanceKey, Ec2InstanceReport]]</code>. We will talk about QueryEnv</code> and LinkedReport</code> later, for now it's enough to understand that this is a more complex query that provides an EC2 instance report; the data type that will be used to render the human-readable output. The query first gets an EC2 instance by instance ID. Then with ZQuery.fromEffect</code> we lift a ZIO</code> effect to the query. In this case this is a zio-aws</code> accessor function that fails if imageId</code> is None</code>. By this we express that we expect that imageId</code> is always specified, and if not, we fail the whole query. Then we use <&></code> (it's alias is zipPar</code>) to perform two queries in parallel: getting an EC2 image and finding the load balancer containing the instance. Once both queries are finished, we optionally generate a load balancer report (if we have found an ELB link) and then we construct the result. Here optionally</code> is a simple helper function that makes our query more readable. It could have been written as elb.fold(ZQuery.none)(getElbReport)</code>. Another useful combinator on ZQuery</code> is collectAllPar</code> that runs a subquery on each item of a collection in parallel: for { elbNames <- ZQuery.fromEffect(asg.loadBalancerNames) result <- ZQuery.collectAllPar(elbNames.map(name => elbquery.getLoadBalancer(name) >>= getElbReport)) } yield result </code></pre> As I mentioned earlier, we have no way to know what resource we are looking for (in fact we could for example detect EC2 instance IDs by a pattern but let's ignore that for now). So on top level we simply start _all the possible queries at once and let print all the non-failing ones: for { renderers <- ZQuery.collectAllPar(possibleQueries).run _ <- ZIO.foreach_(renderers.flatten)(identity) } yield () </code></pre> Where possibleQueries</code> is a where we list all the queries we want to support, tied to the renderer to show it on the console. Report cache</h2> ZIO Query solves caching and optimizes the requests on the AWS resource level, but we still have a problem. The queries form a cyclic graph. For example an EC2 instance holds a link to its load balancer, that holds a link to the EB environment it is defined in. The environment refers back to the ELB, and it also links to the EB app and the application has again links to all the environments it contains. We want to collect all these resources exactly once, and there is a chance that parallel queries reach to the same resource. To solve this we can add an extra caching layer on top of ZIO Query. Let's define this caching layer as a ZIO module: object ReportCache { trait Service { def storeIfNew[A <: Report](reportKey: ReportKey, query: ZQuery[Any, AwsError, A]): ZQuery[Any, AwsError, Boolean] def retrieve[A <: Report](key: ReportKey): ZIO[Any, AwsError, Option[A]] } } </code></pre> The storeIfNew</code> function is a query, to be used in high level queries to shortcut cycles in case a given report is already stored in the cache. We can define a helper function cached</code> like the following: protected def cached[R <: ReportCache with Logging, A, B <: Report, K <: ReportKey] (input: A) (keyFn: A => ZIO[Any, AwsError, K]) (query: K => ZQuery[R, AwsError, B]): ZQuery[R, AwsError, LinkedReport[K, B]] = for { key <- ZQuery.fromEffect(keyFn(input)) env <- ZQuery.environment[R] _ <- storeIfNew( key, query(key).provide(env ? "provided environment") ) } yield LinkedReport[K, B](key) </code></pre> Then we can use it in queries like this: def getEbAppReport(name: ApplicationName): ZQuery[QueryEnv, AwsError, LinkedReport[EbAppKey, EbAppReport]] = cached(name)(name => ZIO.succeed(EbAppKey(name))) { (key: EbAppKey) => // ... } </code></pre> Let's see in detail how this works! First of all, we define the following types: final case class LinkedReport[+K <: ReportKey, +R <: Report](key: K) sealed trait ReportKey final case class Ec2InstanceKey(instanceId: InstanceId) extends ReportKey // ... sealed trait Report final case class Ec2InstanceReport(instanceId: ec2.model.primitives.InstanceId, // ... elb: Option[LinkedReport[ElbKey, ElbReport]] ) extends Report </code></pre> In cached</code>, we provide a keyFn</code> that is an effectful function to extract the ReportKey</code> from the arbitrary input that can be the key itself, or an already fetched resource. Then we call the ReportCache</code> module's storeIfNew</code> query and return a LinkedReport</code>. A linked report is just a wrapper around a report key, it is the type to be used in Report</code> types to refer to each other. We store the cyclic resource graph by using these report keys and the cache's retrieve</code> function to resolve the references on demand. One thing to notice is the .provide</code> in the code of cached</code>. The report cache does not know about the environments needed for the queries it caches the results of; the query</code> parameter of storeIfNew</code> has the type ZQuery[Any, AwsError, A]</code>. For this reason cached</code> eliminates the environment of its inner query by getting it and calling .provide(env)</code> before passing it to the cache. The report cache itself can be implemented with STM</a>. First we create a TMap</code></a>: cache <- TMap.empty[ReportKey, Promise[AwsError, Report]].commit </code></pre> We want to store the fact that a query has been started for a given report key. This can be modelled with a Promise</code> that eventually gets a Report</code> value. With this TMap</code> structure, the storeIfNew</code> function can be defined as: override def storeIfNew[A <: Report](reportKey: ReportKey, query: ZQuery[Any, AwsError, A]): ZQuery[Any, AwsError, Boolean] = ZQuery.fromEffect { for { promise <- Promise.make[AwsError, Report] finalQuery <- cache.get(reportKey).flatMap { case Some(report) => // replacing the query with the cached value ZSTM.succeed(ZQuery.succeed(false)) case None => // replacing the query with the cached value cache.put(reportKey, promise).map { _ => query.foldM( failure => ZQuery.fromEffect(promise.fail(failure)) *> ZQuery.fail(failure), success => ZQuery.fromEffect(promise.succeed(success)) ) } }.commit } yield finalQuery }.flatMap(identity) </code></pre> This may seem simple but actually we are combining three different layers of abstraction here! The whole thing is a query. But we first run a ZIO effect that produces a query, and then execute that result query (in .flatMap(identity)</code>)</li> In the effect we create a promise that might be used or not, depending on the outcome of the transaction. Then we do cache.get</code> which is an STM transaction.</li> In the transaction we produce a ZQuery</code> value that is either returning a simple false</code> value if the report was already cached, or we store the already created promise in the map and return the query that constructs the report as the result of the transaction.</li> As it is an STM transaction it may be retried multiple times but eventually it returns with a query that is either a NOP or calculates the report and sets the promise in the end.</li> </ul> The other function of ReportCache</code>, retrieve</code> will be used when traversing the gathered reports to follow the LinkedReport</code> links. It is simply a combination of getting an item from the TMap</code> and then waiting for the stored promise. Throttling</h2> The original implementation of this tool did not control the amount and rate of AWS requests in any way, and a few years ago API rate limits made it somewhat unusable. As I explained in a previous post</a>, I solved it by centralizing the calls to AWS then adding circuit breaking and retry to handle the throttling errors. In this new implementation ZIO Query 's batching feature already reduces the load but AWS has a global rate limit that can be reached any time, regardless of the actual request rate provided by this application. So how could we handle this with zio-aws</code> and ZIO Query? There is useful ZIO library called rezilience</a> that defines utilities to express circuit breaking, retries, rate limiting and other similar policies. With this library we can create a policy that detects AwsError</code>s representing throttling failures: private def throttlingPolicy: ZManaged[Random with Clock with Logging, Nothing, Policy[AwsError]] = for { cb <- CircuitBreaker.make[AwsError]( trippingStrategy = TrippingStrategy.failureCount(1), resetPolicy = Retry.Schedules.exponentialBackoff(min = 1.second, max = 1.minute), isFailure = { case GenericAwsError(error: AwsServiceException) if error.isThrottlingException => true } ) retry <- Retry.make(min = 1.second, max = 1.minute) retryComposable = retry.widen[PolicyError[AwsError]] { case Policy.WrappedError(e) => e } } yield cb.toPolicy compose retryComposable.toPolicy </code></pre> This will open a circuit breaker in case of throttling errors, and retry the operation with exponential back-off. These policies can be applied to ZIO</code> effects. What we really need is to apply a policy like this to all AWS call. It should be the actual call to the underlying AWS Java SDK, not on the zio-aws</code> wrapper level, because for example a streaming API function may produce multiple AWS requests. The zio-aws</code> library supports applying AwsCallAspect</code>s on the AWS service client layers to modify the underlying SDK calls. This is exactly what we need to apply the throttling policy to all calls! What's even better, by creating a single throttlingPolicy</code> and applying it to all the service layers (ec2</code>, elasticloadbalancing</code>, elasticbeanstalk</code> and autoscaling</code>) they will share a common circuit breaker that matches the situation perfectly as the AWS API rate limiting is applied to globally to all services. An AWS call aspect has the following form: val throttling = new AwsCallAspect[Any] { override def apply[R1, A](f: ZIO[R1, AwsError, Described[A]]): ZIO[R1, AwsError, aspects.Described[A]] = policy(f).mapError { case Policy.WrappedError(e) => e case Policy.BulkheadRejection => AwsError.fromThrowable(new RuntimeException(s"Bulkhead rejection")) case Policy.CircuitBreakerOpen => AwsError.fromThrowable(new RuntimeException(s"AWS rate limit exceeded")) } } </code></pre> Another simple example could be logging all AWS requests: val callLogging: AwsCallAspect[Logging] = new AwsCallAspect[Logging] { override final def apply[R1 <: Logging, A](f: ZIO[R1, AwsError, Described[A]]): ZIO[R1, AwsError, Described[A]] = f.flatMap { case r@Described(_, description) => log.info(s"[${description.service}/${description.operation}]").as(r) } } </code></pre> These aspects can be applied to a zio-aws</code> ZLayer</code> directly, such as: ec2.live @@ (throttling >>> callLogging) </code></pre> Rendering</h2> With the queries and report cache ready the last missing building block is rendering the gathered reports. We implement it in its own ZIO module with the following interface: object Rendering { trait Service { def renderEc2Instance(report: LinkedReport[Ec2InstanceKey, Ec2InstanceReport]): UIO[Unit] def renderElb(report: LinkedReport[ElbKey, ElbReport], context: Option[String]): UIO[Unit] def renderAsg(report: LinkedReport[AsgKey, AsgReport]): UIO[Unit] def renderEbEnv(report: LinkedReport[EbEnvKey, EbEnvReport]): UIO[Unit] def renderEbApp(report: LinkedReport[EbAppKey, EbAppReport]): UIO[Unit] } } </code></pre> The live implementation of course needs access to ReportCache</code> and writes the report out to Console</code>: val live: ZLayer[Console with ReportCache, Nothing, Rendering] = // ... </code></pre> We need two main things to implement report rendering: A way to pretty-print reports to the console</li> We have to track which report was already rendered to be able to traverse the cyclic result graph</li> </ul> To track the already printed reports we can simply create a Ref</code> holding a set of visited ReportKey</code>s: private case class State(alreadyVisited: Set[ReportKey]) // ... alreadyVisited <- Ref.make(State(Set.empty)) </code></pre> For pretty printing the reports there are several possibilities. Eventually we want to call console.putStr</code> to write to the console. The original implementation of this tool used a string templating engine to define the output. Instead of doing that we can write a pretty-printing DSL to define our output in Scala. Take a look at the following example: ifNotVisitedYet(report) { env => sectionHeader("Beanstalk/Env") <-> highlighted(env.name) <-> details(env.id) <-> normal(s"is a Beanstalk environment of the application ${env.appName}") \\ indented { keyword("AWS Console") <:> link(s"https://console.aws.amazon.com/elasticbeanstalk/home?region=${env.region}#/environment/dashboard?applicationName=${env.appName}&environmentId=${env.id}") \\ keyword("Health") <:> highlighted(env.health.toString) \\ keyword("Currently running version") <:> normal(env.version) \\ normal(s"${env.asgs.size} ASGs, ${env.instanceCount} instances, ${env.elbs.size} ELBs") \\ env.elbs.foreach_(elb(_, None)) \\ env.asgs.foreach_(asg) \\ ebApp(env.app) } } </code></pre> We can see here a couple of functions and operators, all created to the specific task of printing AWS resource reports: ifNotYetVisitedYet</code> must somehow interact with the Ref</code> we defined above</li> <-></code> concatenates two texts with a space</li> <:></code> concatenates two texts with a colon and a space</li> \\</code> concatenates two texts with a newline</li> keyword</code>, link</code>, normal</code>, highlighted</code> etc. add styling to the given text</li> foreach_</code> is coming from zio-prelude</code>-s Traversable</code>. We will see why is it used soon.</li> </ul> We could define these styling functions as ZIO</code> effects and the helper operators as general extension methods on ZIO</code>. Then we could store required state (for example for indentation) in a Ref</code> for example. This works but we can do better. By defining our own monadic data type Print[A]</code> we get the following advantages: It is more type safe. The pretty printing operators will be only applicable to pretty printing functions, not to arbitrary ZIO effects</li> Pretty printing state gets completely hidden from the pretty printing definitions</li> We can easily do some optimizations such as collapsing multiple newlines into one, which makes rendering optional lines more convenient</li> </ul> So let's define a data type to represent pretty printing: sealed trait Print[+A] final case class PrintPure[A](a: A) extends Print[A] final case class PrintS(s: String) extends Print[Unit] final case class PrintModified(s: String, modifiers: String) extends Print[Unit] final case object PrintNL extends Print[Unit] final case class PrintIndented[A](p: Print[A]) extends Print[A] final case class PrintFlatMap[A, B](a: Print[A], f: A => Print[B]) extends Print[B] final case class PrintEffect[A](f: UIO[A]) extends Print[A] </code></pre> PrintPure</code> and PrintFlatMap</code> can be used to implement zio-prelude</code>s type classes: implicit val print = new Covariant[Print] with IdentityFlatten[Print] with IdentityBoth[Print] { override def map[A, B](f: A => B): Print[A] => Print[B] = fa => PrintFlatMap(fa, (a: A) => PrintPure(f(a))) override def any: Print[Any] = PrintPure(()) override def flatten[A](ffa: Print[Print[A]]): Print[A] = PrintFlatMap(ffa, (fa: Print[A]) => fa) override def both[A, B](fa: => Print[A], fb: => Print[B]): Print[(A, B)] = PrintFlatMap(fa, (a: A) => map((b: B) => (a, b))(fb)) } </code></pre> What are these type classes providing to us? Covariant</code> basically gives us map</code></li> IdentityFlatten</code> means that the data type can be "flattened" associatively and has an identity element. This gives us flatten</code> and flatMap</code>.</li> IdentityBoth</code> means we have an associative binary operator to combine two values. This enables syntax like <*></code>.</li> </ul> Having this we can define primitive pretty printing operators like: def normal(text: String): Print[Unit] = PrintS(text) val space: Print[Unit] = PrintS(" ") implicit class PrintOps[A](self: Print[A]) { def <->[B](next: => Print[B]): Print[B] = self *> space *> next // ... } </code></pre> Then we can use the syntax provided by zio-prelude</code> to compose these pretty printer values. The only thing remaining is to provide a transformation of Print[A]</code> to UIO[A]</code>. This is where we can hide the pretty printer state and can handle special rules like collapsing newlines: private trait PrettyConsole { protected val console: Console.Service private case class PrettyState(indentation: String, afterNL: Boolean) private def printFlatMap[A, B](a: Print[A], f: A => Print[B], state: PrettyState): UIO[(B, PrettyState)] = for { r1 <- runImpl(a, state) r2 <- runImpl(f(r1._1), r1._2) } yield r2 private def runImpl[A](p: Print[A], state: PrettyState): UIO[(A, PrettyState)] = p match { case PrintPure(a) => ZIO.succeed((a, state)) case PrintS(s) => ZIO.when(state.afterNL)(console.putStr(state.indentation)) *> console.putStr(s).as(((), state.copy(afterNL = false))) case PrintModified(s, modifiers) => ZIO.when(state.afterNL)(console.putStr(state.indentation)) *> console.putStr(s"${modifiers}$s$RESET").as(((), state.copy(afterNL = false))) case PrintNL => if (state.afterNL) ZIO.succeed(((), state)) else console.putStrLn("").as(((), state.copy(afterNL = true))) case PrintIndented(f) => runImpl(f, state.copy(indentation = state.indentation + " ")).map { case (a, s) => (a, s.copy(indentation = state.indentation)) } case PrintFlatMap(a, f) => printFlatMap(a, f, state) case PrintEffect(f) => f.map((_, state)) } def run[A](p: Print[A]): UIO[A] = runImpl(p, PrettyState("", afterNL = false)).map(_._1) } </code></pre> A couple of things to notice here: PrettyState</code> holds the indentation and a flag that is true when the last print was a new line</li> runImpl</code> gets the state as input and has the capability to modify it, by returning the modified state together with the computation's result</li> there is a PrintEffect</code> constructor that allows lifting arbitrary ZIO</code> effects to the pretty printer. This is needed for interacting with the Ref</code> that holds the record of already printed reports.</li> </ul> Putting all together</h2> Putting all this together means getting command line arguments, setting up the AWS client libraries, the report cache and the rendering modules and running the top level queries. To parse the command line arguments we can use my clipp library</a>: final case class Parameters(verbose: Boolean, searchInput: String, region: String) // ... val paramSpec = for { _ <- metadata("aws-query", "search for AWS infrastructure resources") verbose <- flag("Verbose logging", 'v', "verbose") searchInput <- parameter[String]("Search input", "NAME_OR_ID") region <- optional { namedParameter[String]("AWS region", "REGION", "region") } } yield Parameters(verbose, searchInput, region.getOrElse("us-east-1")) val params = clipp.zioapi.config.fromArgsWithUsageInfo(args, paramSpec) </code></pre> The verbose</code> flag is used to set up logging. We use zio-logging</a> with SLF4j support (to be able to see logs from the underlying AWS Java SDK) with lo4j2 backend. In order to control the log level by the command line verbose</code> flag, instead of the usual XML-based configuration for log4j2 we define a ZIO layer that's only purpose is to perform the configuration programmatically: private def log4j2Configuration: ZLayer[Has[ClippConfig.Service[Parameters]], Throwable, Has[Log4jConfiguration]] = { ZLayer.fromServiceM[ClippConfig.Service[Parameters], Any, Throwable, Log4jConfiguration] { params => ZIO.effect { val builder = ConfigurationBuilderFactory.newConfigurationBuilder() // ... Configurator.initialize(builder.build()) Log4jConfiguration() } } </code></pre> This way the root logger's level can depend on the Parameters</code> parsed by clipp</code>. Composing this layer with zio-logger</code>s Slf4jLogger</code> gives us a working Logging</code> layer: val logging = log4j2Configuration >+> Slf4jLogger.make { (_, message) => message } </code></pre> By bootstrapping the parameters and the logging we can run our main application like this: for { result <- awsQuery() .provideCustomLayer(params >+> logging) .catchAll { _ => ZIO.succeed(ExitCode.failure) } _ <- ZIO.effect(LogManager.shutdown()).orDie } yield result </code></pre> The clipp</code> parser will print detailed usage info in case it fails, and other runtime errors are logged, so we can simply catch all errors and exit with a failure on top level. In awsQuery</code> we create all the other layers necessary for running the queries. First we need to create the throttling policy that is used by all the AWS service clients as I explained above: private def awsQuery(): ZIO[Random with Clock with Console with Logging with ClippConfig[Parameters], Nothing, ExitCode] = throttlingPolicy.use { policy => </code></pre> The zio-aws</code> library uses ZIO Config</a> for configuration. This means we need a ZConfig[CommonAwsConfig]</code> to construct the AwsConfig</code> layer: val commonConfig = ZLayer.succeed(CommonAwsConfig( region = Some(Region.of(params.region)), credentialsProvider = DefaultCredentialsProvider.create(), endpointOverride = None, commonClientConfig = None )) val awsCore = (netty.default ++ commonConfig) >>> core.config.configured() </code></pre> The AwsConfig</code> layer combines the configuration with a selected HTTP backend. In our case this is the Netty backend, using its default configuration. Then we define the per-service client layers, applying the throttling and call logging aspects as I described before: val awsClients = ec2.live @@ (throttling >>> callLogging) ++ elasticloadbalancing.live @@ (throttling >>> callLogging) ++ elasticbeanstalk.live @@ (throttling >>> callLogging) ++ autoscaling.live @@ (throttling >>> callLogging) </code></pre> To produce the final layer, we feed the logging and the AwsConfig</code> layers to the client layers, and add the ReportCache</code> and Render</code> implementations: val finalLayer = ((ZLayer.service[Logger[String]] ++ awsCore) >>> awsClients) ++ ((Console.any ++ cache.live) >+> render.live) </code></pre> This has the environment ClippConfig[Parameters] with Console with Logging with ReportCache with Rendering with AllServices</code> where type AllServices = Ec2 with ElasticLoadBalancing with ElasticBeanstalk with AutoScaling </code></pre> Conclusion</h2> We reimplemented the tool to query AWS resources using functional programming techniques, built on top of ZIO libraries. By separating the execution from the problem specification we get an easily readable and maintainable code that can be easily extended with new queries or reports without having to thing about how caching and concurrency is implemented under the hood. We can rate limit AWS requests without touching the actual queries, and take advantage of batching AWS operations while keeping the query logic simple and unaware of this optimization. Code generation in ZIO-AWS 2020-09-23T00:00:00+00:00 I have recently published a set of libraries, zio-aws</a>, aiming to provide a better interface for working with AWS services from ZIO</a> applications. For more information about how the ZIO interface works and how to get started with these libraries, read the repository's README. In this post, I will focus on how these libraries are generated from the schema provided by the AWS Java SDK v2</a>. Generating code</h2> I wanted to cover all AWS services at once. This means client libraries for more than 200 services, so the only possible approach was to generate these libraries on top of a small hand-written core. Schema</h3> The first thing we need for generating code is a source schema. This is the model that we use to create the source code from. It is usually constructed by some kind of DSL or more directly described by a JSON or YAML or similar data model. In the case of zio-aws this was already defined in the AWS Java SDK v2</a> project. The way it works is: There is a codegen</code> project, published in the software.amazon.awssdk</code> group among the client libraries, that contains the Java classes used for generating the Java SDK itself. This contains the data model classes for parsing the actual schema as well.</li> In the AWS Java SDK v2 repository, the schema is located in the subdirectory called services</code></a>. There is a directory for each AWS service and it contains among other things some relevant JSON schema files: service-2.json</code> is the main schema of the service, describing the data structures and operations</li> paginators-1.json</code> describes the operations that the Java SDK creates a paginator interface for</li> customization.config</code> contains extra information, including changes to be applied on top of the service model</li> </ul> </li> Fortunately, these are also embedded in the generated AWS Java SDK libraries as resources, so getting all client libraries on the classpath gives us an easy way to get the corresponding schemas as well</li> </ul> I decided to use the low-level data classes from the AWS codegen</code> library to parse these files and using that build a higher-level model that can be then used as an input for the code generator. This is encapsulated in a ZIO layer called Loader</code>, which has two functions: def findModels(): ZIO[Blocking, Throwable, Set[ModelId]] def loadCodegenModel(id: ModelId): ZIO[Blocking, Throwable, C2jModels] </code></pre> The first one, findModels</code> uses the ClassLoader</code> to enumerate all codegen-resources</code> folders on the classpath and just returns a set of ModelId</code>s. ModelId</code> is a pair of a model name (such as s3</code>) and an optional submodule name (for example dynamodb:dynamodbstreams</code>). Then for each detected model we can load it with the loadCodegenModel</code> function, C2jModels</code> is a class from the AWS codegen</code> library. Figuring out how to interpret these data structures, and how to map them to the generated Java API was the hardest part, but it's out of scope for this post. Our next topic here is how we generate code from our model. Scalameta</h3> There are several possibilities to generate source code and I tried many of them during the past years. Let's see some examples: Using a general-purpose text template engine. An example we used at Prezi</a> is the Java implementation of the Liquid templating engine</a>. Another example is the OpenAPI generator project</a> that uses Mustache</a> templates to generate server and client code from OpenAPI specifications.</li> Generating from code with some general-purpose pretty-printing library. With this approach, we are using the pretty-printer library's composability features to create source code building blocks, and map the code generator model to these constructs. It is easier to express complex logic in this case, as we don't have to encode it in a limited dynamic template model. On the other hand, reading the code generator's source and imagining the output is not easy, and nothing enforces that the pretty-printer building blocks are actually creating valid source code.</li> If the target language has an AST with a pretty-printing feature, we can map the model to the AST directly and just pretty print at the end. With this, we get a much more efficient development cycle, as the generated code is at least guaranteed to be syntactically correct. But the AST can be far from how the target language's textual representation looks like, which makes it difficult to read and write this code.</li> With a library that supports building ASTs with quasiquotes, we can build the AST fragments with a syntax that is very close to the generated target language. For Scala, a library that supports this and is used in a lot of tooling projects is Scalameta</a></li> </ul> I wanted to try using Scalameta ever since I met Devon Stewart and he mentioned how he uses it in guardrail</a>. Finally, this was a perfect use case to do so! To get an understanding of what kind of Scala language constructs can be built with quasiquotes with Scalameta, check the list of them in the official documentation</a>. We get a good mix of both worlds with this. It is possible to express complex template logic in real code, creating higher-level constructs, taking advantage of the full power of Scala. On the other hand, the actual quasiquoted fragments are still close to the code generator's target language (which is in this case also Scala). Let's see a short example of this: private def generateMap(m: Model): ZIO[GeneratorContext, GeneratorFailure, ModelWrapper] = { for { keyModel <- get(m.shape.getMapKeyType.getShape) valueModel <- get(m.shape.getMapValueType.getShape) keyT <- TypeMapping.toWrappedType(keyModel) valueT <- TypeMapping.toWrappedType(valueModel) } yield ModelWrapper( code = List(q"""type ${m.asType} = Map[$keyT, $valueT]""") ) } </code></pre> For each AWS service-specific model type we generate some kind of wrapper code into the ZIO service client library. This is done by processing the schema model to an intermediate format where for each such wrapper, we have a ModelWrapper</code> value that already has the Scalameta AST for that particular wrapper. The above code fragment creates this for map types, which is a simple type alias for a Scala Map</code>. It's a ZIO</code> function, taking advantage of passing around the context in the environment and safely handling generator failures, while the actual generated code part in the q"""..."""</code> remained quite readable. Then the whole model package can be expressed like this: for { // ... primitiveModels <- ZIO.foreach(primitiveModels.toList.sortBy(_.name))(generateModel) models <- ZIO.foreach(complexModels.toList.sortBy(_.name))(generateModel) } yield q"""package $fullPkgName { import scala.jdk.CollectionConverters._ import java.time.Instant import zio.{Chunk, ZIO} import software.amazon.awssdk.core.SdkBytes ..$parentModuleImport package object model { object primitives { ..${primitiveModels.flatMap(_.code)} } ..${models.flatMap(_.code)} }}""" </code></pre> This can be then pretty printed simply with.toString</code> and saved to a .scala</code> file. Building the libraries</h2> We have a way to collect the service models and generate source code from that, but we still have to use that generated code somehow. In zio-aws</code> the goal was to generate a separate client library for each AWS service. At the time of writing, there were 235 such services. The generated libraries have to be built and published to Sonatype. First version</h3> In the first version I simply wired together the above described loader</code> and generator</code> module into a ZIO</code> command line app, using clipp</a> for command line parsing. It's main</code> was really just something like the following: val app = for { svcs <- config.parameters[Parameters].map(_.serviceList) ids <- svcs match { case Some(ids) => ZIO.succeed(ids.toSet) case None => loader.findModels().mapError(ReflectionError) } _ <- ZIO.foreachPar(ids) { id => for { model <- loader.loadCodegenModel(id).mapError(ReflectionError) _ <- generator.generateServiceCode(id, model).mapError(GeneratorError) } yield () } _ <- generator.generateBuildSbt(ids).mapError(GeneratorError) _ <- generator.copyCoreProject().mapError(GeneratorError) } yield ExitCode.success val cfg = config.fromArgsWithUsageInfo(args, Parameters.spec).mapError(ParserError) val modules = loader.live ++ (cfg >+> generator.live) app.provideCustomLayer(modules) </code></pre> Then created a multi-module sbt</code> project with the following modules: zio-aws-codegen</code> the CLI code generator we were talking about so far</li> zio-aws-core</code> holding the common part of all AWS service wrapper libraries. This contains things like how to translate AWS pagination into ZStream</code> etc.</li> zio-aws-akka-http</code>, zio-aws-http4s</code> and zio-aws-netty</code> are the supported HTTP layers, all depend on zio-aws-core</code></li> </ul> I also created a first example project in a separate sbt</code> project, that demonstrated the use of some of the generated AWS client libraries. With this primitive setup, building everything from scratch and running the example took the following steps: sbt compile</code> the root project</li> manually running zio-aws-codegen</code> to generate all client libs at once to a separate directory, with a corresponding build.sbt</code> including all these projects in a single sbt</code> project</li> sbt publishLocal</code> in the generated sbt</code> project</li> sbt run</code> in the examples project</li> </ol> For the second, manual step I created some custom sbt tasks called generateAll</code>, buildAll</code>, and publishLocalAll</code>, that downloaded an sbt-launch-*.jar</code> and used it to run the code generator and fork an sbt</code> to build the generated project. The generateAll</code> task was quite simple: generateAll := Def.taskDyn { val root = baseDirectory.value.getAbsolutePath Def.task { (codegen / Compile / run).toTask(s" --target-root ${root}/generated --source-root ${root} --version $zioAwsVersion --zio-version $zioVersion --zio-rs-version $zioReactiveStreamsInteropVersion").value } }.value </code></pre> Launching a second sbt</code> took more effort: buildAll := Def.taskDyn { val _ = generateAll.value val generatedRoot = baseDirectory.value / "generated" val launcherVersion = sbtVersion.value val launcher = s"sbt-launch-$launcherVersion.jar" val launcherFile = generatedRoot / launcher Def.task[Unit] { if (!launcherFile.exists) { val u = url(s"https://oss.sonatype.org/content/repositories/public/org/scala-sbt/sbt-launch/$launcherVersion/sbt-launch-$launcherVersion.jar") sbt.io.Using.urlInputStream(u) { inputStream => IO.transfer(inputStream, launcherFile) } } val fork = new ForkRun(ForkOptions() .withWorkingDirectory(generatedRoot)) fork.run( "xsbt.boot.Boot", classpath = launcherFile :: Nil, options = "compile" :: Nil, log = streams.value.log ) } }.value </code></pre> With these extra tasks, I released the first version of the library manually, but there was a lot of annoying difficulties: Having to switch between various sbt</code> projects</li> The need to publishLocal</code> the generated artifacts in order to build the examples, or any kind of integration tests that I planned to add</li> The only way to build only those client libraries that are needed for the examples/tests was to build and publish them manually, as this dependency was not tracked at all between the unrelated sbt</code> projects</li> Because the generated sbt</code> project could not refer to the outer zio-aws-core</code> project, it has to be copied into the generated project in the code generator step</li> Building and publishing all the 235 projects at once required about 16Gb memory and hours of compilation time. It was too big to run on any of the (freely available) CI systems.</li> </ul> Proper solution</h3> When I mentioned this, Itamar Ravid recommended trying to make it an sbt code generator. sbt</code> has built-in support for generating source code, as described on it's documentation page</a>. This alone though would not be enough to cover our use case, as in zio-aws</code> even the set of projects is dynamic and comes from the enumeration of schema models. Fortunately, there is support for that in too, through the extraProjects</code> property of sbt</code> plugins. With these two tools, the new project layout became the following: zio-aws-codegen</code> is an sbt plugin, having it's own sbt</code> project in a subdirectory</li> the zio-aws-core</code> and the HTTP libraries are all in the top-level project as before</li> examples and integration tests are also part of the top-level project</li> the zio-aws-codegen</code> plugin is referenced using a ProjectRef</code> from the outer project</li> the plugin adds all the AWS service client wrapper libraries to the top-level project</li> these projects generate their source on-demand</li> </ul> In this setup, it is possible to build any subset of the generated libraries without the need to process and compile all of them, so it needs much less memory. It is also much simpler to run tests or build examples on top of them, as the test and example projects can directly depend on the generated libraries as sbt</code> submodules. And even developing the code generator itself is convenient - although for editing it, it has to be opened as in a separate IDE session, but otherwise, sbt reload</code> on the top level project automatically recompiles the plugin when needed. Let's see piece by piece how we can achieve this! Project as a source dependency</h4> The first thing I wanted to do is having the zio-aws-codegen</code> project converted to an sbt</code> plugin, but still having it in the same repository and be able to use it without having to install to a local repository. Although the whole code generator code could have been added to the top level sbt</code> project's project</code> source, I wanted to keep it as a separate module to be able to publish it as a library or a CLI tool in the future if needed. This can be achieved by putting it in a subdirectory of the top level project, with a separate build.sbt</code> that contains the sbtPlugin := true </code></pre> (beside the usual ones). Then it can be referenced in the top level project's project/plugins.sbt</code> in the following way: lazy val codegen = project .in(file(".")) .dependsOn(ProjectRef(file("../zio-aws-codegen"), "zio-aws-codegen")) </code></pre> and enabled in the build.sbt</code> as enablePlugins(ZioAwsCodegenPlugin) </code></pre> Dynamically generating projects</h4> To generate the subprojects dynamically, we need the Set[ModelId]</code> coming from the loader</code> module. It is a ZIO</code> module, so from the sbt</code> plugin we have to use Runtime.default.unsafeRun</code> to execute it. As the code generator project is now an sbt</code> plugin, all the sbt</code> data structures are directly available, so we can just write a function that maps the ModelId</code>s to Project</code>s: protected def generateSbtSubprojects(ids: Set[ModelId]): Seq[Project] = ??? </code></pre> One interesting part here is that some of the subprojects are depending on each other. This happens with AWS service submodules, indicated by the second parameter of ModelId</code>. An example is dynamodbstreams</code> that depends on dynamodb</code>. When creating the Project</code> values, we have to be able to dependOn</code> on some other already generated projects, and they have to be generated in the correct order to do so. We could do a full topological sort, but it is not necessary, here we know that the maximum depth of dependencies is 1, so it is enough to put the submodules at the end of the sequence: val map = ids .toSeq .sortWith { case (a, b) => val aIsDependent = a.subModuleName match { case Some(value) if value != a.name => true case _ => false } val bIsDependent = b.subModuleName match { case Some(value) if value != b.name => true case _ => false } bIsDependent || (!aIsDependent && a.toString < b.toString) } </code></pre> Then in order to be able get the dependencies, we do a fold on the ordered ModelId</code>s: .foldLeft(Map.empty[ModelId, Project]) { (mapping, id) => // ... val deps = id.subModule match { case Some(value) if value != id.name => Seq(ClasspathDependency(LocalProject("zio-aws-core"), None), ClasspathDependency(mapping(ModelId(id.name, Some(id.name))), None)) case _ => Seq(ClasspathDependency(LocalProject("zio-aws-core"), None)) } val project = Project(fullName, file("generated") / name) .settings( libraryDependencies += "software.amazon.awssdk" % id.name % awsLibraryVersion.value, // ... .dependsOn(deps: _*) mapping.updated(id, project) } </code></pre> To make it easier to work with the generated projects, we also create a project named all</code> that aggregates all the ones generated above. Applying settings to the generated projects</h4> The code generator only sets the basic settings for the generated projects: name, path and dependencies. We need a lot more, setting organization and version, all the publishing options, controlling the Scala version, etc. I decided to keep these settings outside of the code generator plugin, in the top-level sbt</code> project. By creating an AutoPlugin</code> end enabling it for all projects, we can inject all the common settings for both the hand-written and the generated projects: object Common extends AutoPlugin { object autoImport { val scala212Version = "2.12.12" val scala213Version = "2.13.3" // ... } import autoImport._ override val trigger = allRequirements override val requires = Sonatype override lazy val projectSettings = Seq( scalaVersion := scala213Version, crossScalaVersions := List(scala212Version, scala213Version), // ... ) } </code></pre> Source generator task</h4> At this point, we could also add the already existing source code generation to the initialization of the plugin, and just generate all the subproject's all source files every time the sbt</code> project is loaded. With this number of generated projects though, it would have been a very big startup overhead and would not allow us to split the build (at least not the code generation part) on CI, to solve the memory and build time issues. As sbt</code> has built-in support for defining source generator tasks, we can do much better! Instead of generating the source codes in one step, we define a generateSources</code> task and add it to each generated subproject as a source generator: Compile / sourceGenerators += generateSources.taskValue, awsLibraryId := id.toString </code></pre> The awsLibraryId</code> is a custom property that we the generateSources</code> task can use to determine which schema to use for the code generation. The first part of this task is to gather the information from the project it got applied on, including the custom awsLibraryId</code> property: lazy val generateSources = Def.task { val log = streams.value.log val idStr = awsLibraryId.value val id = ModelId.parse(idStr) match { case Left(failure) => sys.error(failure) case Right(value) => value } val targetRoot = (sourceManaged in Compile).value val travisSrc = travisSource.value val travisDst = travisTarget.value val parallelJobs = travisParallelJobs.value </code></pre> From these, we create a Parameters</code> data structure to pass to the generator</code> module. This is what we used to construct with clipp</code> from CLI arguments: val params = Parameters( targetRoot = Path.fromJava(targetRoot.toPath), travisSource = Path.fromJava(travisSrc.toPath), travisTarget = Path.fromJava(travisDst.toPath), parallelTravisJobs = parallelJobs ) </code></pre> And finally, construct the ZIO</code> environment, load a single schema model, and generate the library's source code: zio.Runtime.default.unsafeRun { val cfg = ZLayer.succeed(params) val env = loader.live ++ (cfg >+> generator.live) val task = for { _ <- ZIO.effect(log.info(s"Generating sources for $id")) model <- loader.loadCodegenModel(id) files <- generator.generateServiceCode(id, model) } yield files.toSeq task.provideCustomLayer(env).catchAll { generatorError => ZIO.effect(log.error(s"Code generator failure: ${generatorError}")).as(Seq.empty) } } } </code></pre> The generateServiceCode</code> function returns a Set[File]</code> value containing all the generated source files. This is the result of the source generator task, and sbt</code> uses this information to add the generated files to the compilation. Referencing the generated projects</h4> When defining downstream projects in the build.sbt</code>, such as integration tests and other examples, we have to refer to the generated projects somehow. There is no value of type Project</code> in scope to do so, but we can do it easily by name using LocalProject</code>. The following example shows how the example1</code> subproject does this: lazy val example1 = Project("example1", file("examples") / "example1") .dependsOn( core, http4s, netty, LocalProject("zio-aws-elasticbeanstalk"), LocalProject("zio-aws-ec2") ) </code></pre> Parallel build on Travis CI</h4> The last thing that I wanted to solve is building the full zio-aws</code> suite on a CI. I am using Travis CI</a> for my private projects, so that's what I built it for. The idea is to split the set of service client libraries to chunks and create build matrix</a> to run those in parallel. The tricky part is that the set of generated service libraries is dynamic, collected by the code generator. To solve this, I started to generate the .travis.yml</code> build descriptor as well. The hand-written part has been moved to .travis.base.yml</code>: language: scala services: - docker scala: - 2.12.12 - 2.13.3 cache: directories: - $HOME/.cache/coursier - $HOME/.ivy2/cache - $HOME/.sbt env: - COMMANDS="clean zio-aws-core/test zio-aws-akka-http/test zio-aws-http4s/test zio-aws-netty/test" - COMMANDS="clean examples/compile" - COMMANDS="clean integtests/test" before_install: - if [ "$COMMANDS" = "clean integtests/test" ]; then docker pull localstack/localstack; fi - if [ "$COMMANDS" = "clean integtests/test" ]; then docker run -d -p 4566:4566 --env SERVICES=s3,dynamodb --env START_WEB=0 localstack/localstack; fi script: - sbt ++$TRAVIS_SCALA_VERSION -jvm-opts travis/jvmopts $COMMANDS </code></pre> I use the COMMANDS</code> environment variable to define the parallel sets of sbt</code> commands here. There are three predefined sets: building zio-aws-core</code> and the HTTP implementations, building the example projects and running the integration test. The last two involve generating actual service client code and building them - but only the few that are necessary, so it is not an issue to do that redundantly. The real .travis.yml</code> file is then generated by running a task manually, sbt generateTravisYaml</code>. It is implemented in the zio-aws-codegen</code> plugin and it loads the .travis.base.yml</code> file and extends the env</code> section with a set of COMMANDS</code> variants, each compiling a subset of the generated subprojects. Conclusion</h2> Travis CI can now build zio-aws</code> and run its integration tests. A build runs for hours, but it is stable, and consists of 22 parallel jobs to build all the libraries for both Scala 2.12 and 2.13. At the same time, developing the code generator and the other subprojects and tests became really convenient. prox part 4 - simplified redesign 2020-08-03T00:00:00+00:00 Blog post series</h2> Part 1 - type level programming</a></li> Part 2 - akka streams with cats effect</a></li> Part 3 - effect abstraction and ZIO</a></li> Part 4 - simplified redesign</a></li> </ul> Intro</h2> In Part 1</a> I described how the advanced type level programming techniques can be used to describe the execution of system processes. It was both a good playground to experiment with these and the result has been proven useful as we started to use it in more and more production systems and test environments at Prezi</a>. On the other hand as I mentioned at the end of the first post, there is a tradeoff. These techniques made the original version of prox very hard to maintain and improve, and the error messages library users got by small mistakes were really hard to understand. Last December (in 2019) I redesigned the library to be simpler and easier to use by making some compromises. Let's discover how! A single process</h2> We start completely from scratch and try to design the library with the same functionality but with simplicity in mind. The code snippets shown here are not necessarily the final, current state of the traits and objects of the library, but some intermediate steps so we see the thought process. First let's focus on defining a single process: trait Process { val command: String val arguments: List[String] val workingDirectory: Option[Path] val environmentVariables: Map[String, String] val removedEnvironmentVariables: Set[String] } </code></pre> Without deciding already how it will be implemented, we know we need these information to be able to launch the process alone. And how to execute it? Let's separate it completely: trait ProcessResult { val exitCode: ExitCode } trait ProcessRunner { def start(process: Process): Resource[IO, Fiber[IO, ProcessResult]] } </code></pre> I decided that better integration with the IO library (cats-effect</a> in this case) is also a goal of the redesign, so for starter modelled the running process as a cancellable fiber resulting in ProcessResult</code>, where cancellation means terminating the process. At this stage of the redesign I worked directly with IO</code> instead of the IO typeclasses and later replaced it like I described in the previous post</a>. Let's see how a simple runner implementation would look like: import java.lang.{Process => JvmProcess} class JVMProcessRunner(implicit contextShift: ContextShift[IO]) extends ProcessRunner { import JVMProcessRunner._ override def start(process: Process): Resource[IO, Fiber[IO, ProcessResult]] = { val builder = withEnvironmentVariables(process, withWorkingDirectory(process, new ProcessBuilder((process.command :: process.arguments).asJava))) val start = IO.delay(new JVMRunningProcess(builder.start())).bracketCase { runningProcess => runningProcess.waitForExit() } { case (_, Completed) => IO.unit case (_, Error(reason)) => IO.raiseError(reason) case (runningProcess, Canceled) => runningProcess.terminate() >> IO.unit }.start Resource.make(start)(_.cancel) } } </code></pre> Here withEnvironmentVariables</code> and withWorkingDirectories</code> are just helper functions around the JVM process builder. The more important part is the cancelation and that we expose it as a resource. First we wrap the started JVM process in a JVMRunningProcess</code> class which really just wraps some of it's operations in IO operations: case class SimpleProcessResult(override val exitCode: ExitCode) extends ProcessResult class JVMRunningProcess(val nativeProcess: JvmProcess) extends RunningProcess { override def isAlive: IO[Boolean] = IO.delay(nativeProcess.isAlive) override def kill(): IO[ProcessResult] = IO.delay(nativeProcess.destroyForcibly()) >> waitForExit() override def terminate(): IO[ProcessResult] = IO.delay(nativeProcess.destroy()) >> waitForExit() override def waitForExit(): IO[ProcessResult] = for { exitCode <- IO.delay(nativeProcess.waitFor()) } yield SimpleProcessResult(ExitCode(exitCode)) } </code></pre> Then we wrap the starting of the process with bracketCase</code>, specifying the two cases: On normal execution, we waitForExit</code> for the process to stop and create the ProcessResult</code> as the result of the bracketed IO operation.</li> In the release case, if JVM thrown an exception it is raised to the IO level</li> And if it got canceled, we terminate</code> the process</li> </ul> This way the IO cancelation interface gets a simple way to wait for or terminate an executed process. By calling .start</code> on this bracketed IO operation we move it to a concurrent fiber. Finally we wrap it in a Resource</code>, so if the user code starting the process got canceled, it releases the resource too that ends up terminating the process, leaving no process leaks. This is something that was missing from the earlier versions of the library. To make starting processes more convenient we can create an extension method on the Process</code> trait: implicit class ProcessOps(private val process: Process) extends AnyVal { def start(implicit runner: ProcessRunner): Resource[IO, Fiber[IO, ProcessResult]] = runner.start(process) } </code></pre> Redirection</h2> The next step was to implement input/output/error redirection. In the original prox library we had two important features, both implemented with type level techniques: Allow redirection only once per channel</li> The redirection source or target was a type class with dependent result types</li> </ul> To keep the type signatures simpler I decided to work around these by sacrificing some genericity and terseness. Let's start by defining an interface for redirecting process output: trait RedirectableOutput[+P[_] <: Process[_]] { def connectOutput[R <: OutputRedirection, O](target: R)(implicit outputRedirectionType: OutputRedirectionType.Aux[R, O]): P[O] // ... } </code></pre> This is not very much different than the output redirection operator in the previous prox versions: def >[F[_], To, NewOut, NewOutResult, Result <: ProcessNode[_, _, _, Redirected, _]] (to: To) (implicit contextOf: ContextOf.Aux[PN, F], target: CanBeProcessOutputTarget.Aux[F, To, NewOut, NewOutResult], redirectOutput: RedirectOutput.Aux[F, PN, To, NewOut, NewOutResult, Result]) </code></pre> One of the primary differences is that we don't allow arbitrary targets just by requiring a CanBeProcessOutput</code> type class. Instead we can only connect the output to a value of OutputRedirection</code> which is an ADT: sealed trait OutputRedirection case object StdOut extends OutputRedirection case class OutputFile(path: Path, append: Boolean) extends OutputRedirection case class OutputStream[O, +OR](pipe: Pipe[IO, Byte, O], runner: Stream[IO, O] => IO[OR], chunkSize: Int = 8192) extends OutputRedirection </code></pre> We still need a type level calculation to extract the result type of the OutputStream</code> case (which is the OR</code> type parameter). This extracted by the following trait with the help of the Aux</code> pattern: trait OutputRedirectionType[R] { type Out def runner(of: R)(nativeProcess: JvmProcess, blocker: Blocker, contextShift: ContextShift[IO]): IO[Out] } </code></pre> The important difference from earlier versions of the library is that this remains completely an implementation detail. OutputRedirectionType</code> is implemented for all three cases of the OutputRedirection</code> type and connectOutput</code> is not even used in the default use cases, only when implementing redirection for something custom. Instead the RedirectableOutput</code> trait itself defines a set of operators and named function versions for redirecting to different targets. With this we loose a general-purpose, type class managed way to redirect to anything but improve a lot on the usability of the library. All these functions are easily discoverable from the IDE and there would not be any weird implicit resolution errors. Let's see some examples of these functions: trait RedirectableOutput[+P[_] <: Process[_]] { // ... def >(sink: Pipe[IO, Byte, Unit]): P[Unit] = toSink(sink) def toSink(sink: Pipe[F, Byte, Unit]): P[Unit] = connectOutput(OutputStream(sink, (s: Stream[F, Unit]) => s.compile.drain)) def >#[O: Monoid](pipe: Pipe[F, Byte, O]): P[O] = toFoldMonoid(pipe) def toFoldMonoid[O: Monoid](pipe: Pipe[F, Byte, O]): P[O] = connectOutput(OutputStream(pipe, (s: Stream[F, O]) => s.compile.foldMonoid)) def >>(path: Path): P[Unit] = appendToFile(path) def appendToFile(path: Path): P[Unit] = connectOutput(OutputFile[F](path, append = true)) // ... } </code></pre> All of them are just using the connectOutput</code> function so implementations of the RedirectableOutput</code> trait need to define that single function to get this capability. Note that connectOutput</code> has a return type of P[O]</code> instead of being just Process</code>. This is important for multiple reasons. First, in order to actually execute the output streams, we need to store it somehow in the Process</code> data type itself. For this reason we add a type parameter to the Process</code> trait representing the output type and store the output stream runner function itself in it: trait Process[O] { // ... val outputRedirection: OutputRedirection val runOutputStream: (JvmProcess, Blocker, ContextShift[IO]) => IO[O] } </code></pre> Note that runOutputStream</code> is actually the OutputRedirectiontype.runner</code> function, got from the "hidden" type level operation and stored in the process data structure. With this, the process runner can be extended to pass the started JVM process to this function that sets up the redirection, and then store the result of type O</code> in ProcessResult[O]</code>: override def start[O](process: Process[O], blocker: Blocker): Resource[IO, Fiber[IO, ProcessResult[O]]] = { // ... process builder val outputRedirect = process.outputRedirection match { case StdOut => ProcessBuilder.Redirect.INHERIT case OutputFile(path) => ProcessBuilder.Redirect.to(path.toFile) case OutputStream(_, _, _) => ProcessBuilder.Redirect.PIPE } builder.redirectOutput(outputRedirect) val startProcess = for { nativeProcess <- IO.delay(builder.start()) runningOutput <- process.runOutputStream(nativeProcess, blocker, contextShift).start } yield new JVMRunningProcess(nativeProcess, runningOutput) // ... bracketCase, start, Resource.make } </code></pre> It is also important that this RedirectableOutput</code> trait is not something all process has: it is a capability, and only processes with unbound output should implement it. This is the new encoding of fixing the three channels of a process. Instead of having three type parameters with phantom types, now we have a combination of capability traits mixed with the Process</code> trait, constraining what kind of redirections we can do. As this is not something unbounded and have relatively small number of cases, I chose to implement the combinations by hand, designing it in a way to minimize the redundancy in these implementation classes. This means, in total 8 classes representing the combinations of bound input, output and error. I will demonstrate this with a single example. The Process</code> constructor now returns a type with everything unbound, represented by having all the redirection capability traits: object Process { def apply(command: String, arguments: List[String] = List.empty): ProcessImpl = ProcessImpl( command, arguments, workingDirectory = None, environmentVariables = Map.empty, removedEnvironmentVariables = Set.empty, outputRedirection = StdOut, runOutputStream = (_, _, _) => IO.unit, errorRedirection = StdOut, runErrorStream = (_, _, _) => IO.unit, inputRedirection = StdIn ) case class ProcessImpl(override val command: String, override val arguments: List[String], override val workingDirectory: Option[Path], override val environmentVariables: Map[String, String], override val removedEnvironmentVariables: Set[String], override val outputRedirection: OutputRedirection[F], override val runOutputStream: (java.io.InputStream, Blocker, ContextShift[F]) => F[Unit], override val errorRedirection: OutputRedirection[F], override val runErrorStream: (java.io.InputStream, Blocker, ContextShift[F]) => F[Unit], override val inputRedirection: InputRedirection[F]) extends Process[Unit, Unit] with RedirectableOutput[ProcessImplO[*]] with RedirectableError[ProcessImplE[*]] with RedirectableInput[ProcessImplI]] { // ... def connectOutput[R <: OutputRedirection, RO](target: R)(implicit outputRedirectionType: OutputRedirectionType.Aux[R, RO]): ProcessImplO[RO] = ProcessImplO( // ... target, outputRedirectionType.runner(target), // ... ) } case class ProcessImplO[O](// ... override val runOutputStream: (java.io.InputStream, Blocker, ContextShift[F]) => F[O], // ... ) extends Process[O, Unit] with RedirectableError[ProcessImplOE[O, *]] with RedirectableInput[ProcessImplIO[O]] { // ... } } </code></pre> Each implementation class only has the necessary subset of type parameters O</code> and E</code> (E</code> is the error output type), and the I</code> O</code> and E</code> postfixes in the class names represent which channels are bound. Each redirection leads to a different implementation class with less and less redirection capabilities. ProcessImplIOE</code> is the fully bound process. This makes all the redirection operators completely type inferable and very pleasant to use for building up concrete process definitions. And we don't loose the ability to create generic function either. We can do it by requiring redirection capabilities: def withInput[O, E, P <: Process[O, E]](s: String)(process: Process[O, E] with RedirectableInput[P]): P = { val input = Stream("This is a test string").through(text.utf8Encode) process < input } </code></pre> Here we know we want to have a Process</code> with the RedirectableInput</code> capability. We also know that by binding the input we get a something without that trait, so we know the result is a process P</code> but know nothing else about its further capabilities. This is where this solution gets a bit inconvenient, if we want to chain these wrapper functions. To help with it, the library contains type aliases for the whole redirection capability chain that can be used in these functions. For example: /** Process with unbound input, output and error streams */ type UnboundProcess = Process[Unit, Unit] with RedirectableInput[UnboundOEProcess] with RedirectableOutput[UnboundIEProcess[*]] with RedirectableError[UnboundIOProcess[*]] </code></pre> Process piping</h2> The other major feature beside redirection that prox had is piping processes together, meaning the first process' output gets redirected to the second process' input. Now that we have redesigned processes and redirection capabilities, we can try to implement this on top of them. The idea is that when we construct a process group from a list of Process</code> instances with the necessary redirection capabilities, this construction could set up the redirection and store the modified processes instead, then running them together. And it can reuse the RedirectableOutput</code> and RedirectableInput</code> capabilities to bind the first/last process! Let's again start by defining what we need for the process group: trait ProcessGroup[O, E] extends ProcessLike { val firstProcess: Process[Stream[IO, Byte], E] val innerProcesses: List[Process.UnboundIProcess[Stream[IO, Byte], E]] val lastProcess: Process.UnboundIProcess[O, E] val originalProcesses: List[Process[Unit, Unit]] } </code></pre> ProcessLike</code> is a common base trait for Process</code> and ProcessGroup</code>. By introducing it, we can change the RedirectableOutput</code> trait's self type bounds so it works for both processes and process groups. A valid process group always have at least 2 processes and they get pre-configured during the construction of the group so when they get started, their channels can be joined. This means the group members can be split into three groups: The first process has it's output redirected to a stream, but running the stream just returns the stream itself; this way it can be connected to the next process's input</li> The inner processes are all having their output redirected in the same way, and it is also a requirement that these must have their input channel unbound. This is needed for the operation described above, when we plug the previous process' output into the input</li> The last process can have its output freely redirected by the user, but it's input must be unbound so the previous process can be plugged in</li> </ul> We also store the original process values for reasons explained later. So as we can see the piping has two stages: First we prepare the processes by setting up their output to return an un-executed stream</li> And we need a process group specific start function into the ProcessRunner</code> that plugs everything together</li> </ol> The first step is performed by the pipe operator (|</code>), which is defined on Process</code> via an extension method to construct group of two processes, and on ProcessGroupImpl</code> to add more. For simplicity the piping operator is currently not defined on the bound process group types. So it has to be first constructed, and then the redirection set up. Let's see the one that adds one more process to a group: def pipeInto(other: Process.UnboundProcess, channel: Pipe[IO, Byte, Byte]): ProcessGroupImpl = { val pl1 = lastProcess.connectOutput(OutputStream(channel, (stream: Stream[IO, Byte]) => IO.pure(stream))) copy( innerProcesses = pl1 :: innerProcesses, lastProcess = other, originalProcesses = other :: originalProcesses ) } def |(other: Process.UnboundProcess): ProcessGroupImpl = pipeInto(other, identity) </code></pre> Other than moving processes around in the innerProcesses</code> and lastProcess</code>, we also set up the previous last process's output in the way I described: It gets redirected to a pipe which is by default identity</code></li> And it's runner instead of actually running the stream, just returns the stream definition</li> </ul> This way we can write a process group specific start function into the process runner: override def startProcessGroup[O, E](processGroup: ProcessGroup[O, E], blocker: Blocker): IO[RunningProcessGroup[O, E]] = for { first <- startProcess(processGroup.firstProcess, blocker) firstOutput <- first.runningOutput.join innerResult <- if (processGroup.innerProcesses.isEmpty) { IO.pure((List.empty, firstOutput)) } else { val inner = processGroup.innerProcesses.reverse connectAndStartProcesses(inner.head, firstOutput, inner.tail, blocker, List.empty) } (inner, lastInput) = innerResult last <- startProcess(processGroup.lastProcess.connectInput(InputStream(lastInput, flushChunks = false)), blocker) runningProcesses = processGroup.originalProcesses.reverse.zip((first :: inner) :+ last).toMap } yield new JVMRunningProcessGroup[O, E](runningProcesses, last.runningOutput) </code></pre> where connectAndStartProcesses</code> is a recursive function that does the same as we do with the first process: start it with the startProcess</code> function (this is the same function we discussed in the first section, that starts Process</code> values)</li> then "join" the output fiber; this completes immediately as it is not really running the output stream just returning it</li> we connect the input of the next process to the previous process' output</li> </ul> One thing we did not talk about yet is getting the results of a process group. This is where the old implementation again used some type level techniques and returned a RunningProcess</code> value with specific per-process output and error types for each member of the group, as a HList</code> (or converted to a tuple). By making the library a bit more dynamic we can drop this part too. What is that we really want to do with a running process group? Terminating the whole group together. Terminating just one part is something we does not support currently although it would not be hard to add.</li> Waiting for all processes to stop</li> Examining the exit code for each member of the group</li> Redirecting the error channel of each process to something and getting them in the result</li> Redirecting the input of the group's first process</li> Redirecting the output of the group's last process, and getting it in the result</li> </ul> The most difficult and primary reason for the HList</code> in the old version is the error redirection, as it can be done per process. With some restrictions we can make a reasonable implementation though. First, we require that the processes participating in forming a process group does not have their error channel bound yet. Then we create a RedirectableErrors</code> capability that is very similar to the existing RedirectableError</code> trait, but provides an advanced interface through it's customizedPerProcess</code> field: trait RedirectableErrors[+P[_] <: ProcessGroup[_, _]] { lazy val customizedPerProcess: RedirectableErrors.CustomizedPerProcess[P] = // ... } </code></pre> where the CustomizedPerProcess</code> interface contains the same redirection functions but accept a function of a Process</code> as parameter. For example: def errorsToSink(sink: Pipe[IO, Byte, Unit]): P[Unit] // vs def errorsToSink(sinkFn: Process[_, _] => Pipe[IO, Byte, Unit]): P[Unit] = </code></pre> The limitation is that for all process we need to have the same error result type but it still gets a lot of freedom via the advanced interface: we can tag the output with the process and split their processing further in the stream. With this choice, we can finally define the result type of the process group too: trait ProcessGroupResult[+O, +E] { val exitCodes: Map[Process[Unit, Unit], ExitCode] val output: O val errors: Map[Process[Unit, Unit], E] } </code></pre> The error results and the exit codes are in a map indexed by the original process. This is the value passed to the piping operator, the one that the user constructing the group has. That's why in the ProcessGroup</code> trait we also had to store the original process values. As the output of all the inner processes are piped to the next process, we only have to care about the last process' output. Conclusion</h2> With a full redesign and making some compromises, we get a library that has a much more readable and easier to maintain code, and an API that is discoverable by the IDE and does not produce any weird error messages on misuse. Note that in all the code snippets above I removed the effect abstraction and just used IO</code> to make them simpler. The real code of course can be used with any IO library such as ZIO, just like the previous versions. prox part 3 - effect abstraction and ZIO 2019-08-13T00:00:00+00:00 Blog post series</h2> Part 1 - type level programming</a></li> Part 2 - akka streams with cats effect</a></li> Part 3 - effect abstraction and ZIO</a></li> Part 4 - simplified redesign</a></li> </ul> Intro</h2> The first post</a> introduced the prox library and demonstrated the advanced type level programming techniques it uses. Then in the second part</a> of this series we experimented with replacing the streaming library from fs2</a> to Akka Streams</a>. In both cases the library used cats-effect</a> for describing side effects. But it did not really take advantage of cats-effect's effect abstraction: it explicitly defined everything to be a computation in IO</code></a>, cats-effect's implementation of describing effectful computations. But we can do better! By not relying on IO</code> but the various type classes the cats-effect library provides we can make prox work with any kind of effect library out of the box. One such example is ZIO</a>. Effect abstraction</h2> Let's see an example of how IO</code> used to be used in the library! The following function is in the Start</code> type class, and it starts a process or piped process group: def apply(process: PN, dontStartOutput: Boolean = false, blocker: Blocker) (implicit contextShift: ContextShift[IO]): IO[RunningProcesses] </code></pre> We can observe two things here: The function returns an effectful computation in IO</code></li> An implicit context shifter is needed by the implementations which are calling some streaming functions needing it.</li> </ul> To make it independent of the effect library implementation we have to get rid of IO</code> and use a generic type instead, let's call it F</code>: def apply(process: PN, dontStartOutput: Boolean = false, blocker: Blocker) (implicit concurrent: Concurrent[F], contextShift: ContextShift[F]): F[RunningProcesses] </code></pre> Beside using F</code> instead of IO</code> everywhere we also have a new requirement, our context type (F</code>) have to have an implementation of the Concurrent</code></a> type class. Cats-effect defines a hierarchy of type classes to deal with effectful computations. At the time of writing it looks like this: Read the official documentation</a> for more information. Prox is based on the ProcessNode</code> type which has two implementations, a single Process</code> or a set of processes piped together to a PipedProcess</code>. Because these types store their I/O redirection within themselves, they also have to be enriched with a context type parameter. For example Process</code> will look like this: class Process[F[_], Out, Err, OutResult, ErrResult, IRS <: RedirectionState, ORS <: RedirectionState, ERS <: RedirectionState] (val command: String, val arguments: List[String], val workingDirectory: Option[Path], val inputSource: ProcessInputSource[F], val outputTarget: ProcessOutputTarget[F, Out, OutResult], val errorTarget: ProcessErrorTarget[F, Err, ErrResult], val environmentVariables: Map[String, String], val removedEnvironmentVariables: Set[String]) extends ProcessNode[Out, Err, IRS, ORS, ERS] { // ... } </code></pre> The context parameter (F</code>) is needed because the input source and output target are all representing effectful code such as writing to the standard output, reading from a file, or passing data through concurrent streams. Let's see some examples of how the abstract types of cats-effect can be used to describe the computation, when we cannot rely on IO</code> itself! The most basic operation is to delay the execution of some code that does not use the effect abstractions. This is how we wrap the Java process API, for example. While with the original implementation of prox it was done by using the IO</code> constructor: IO { systemProcess.isAlive } </code></pre> with an arbitrary F</code> we only need to require that it has an implementation of the Sync</code> type class: private class WrappedProcess[F[_] : Sync, // ... </code></pre> and then use the delay</code> function: Sync[F].delay { systemProcess.isAlive } </code></pre> Similarily the Concurrent</code> type class can be used to start a concurrent computation on a fiber: Concurrent[F].start(stream.compile.toVector) </code></pre> Type level</h2> This would be it - except that we need one more thing because of the type level techniques described in the first post</a>. To understand the problem, let's see how the output redirection operator works. It is implemented as an extension method on the ProcessNode</code> type: implicit class ProcessNodeOutputRedirect[PN <: ProcessNode[_, _, _, NotRedirected, _]](processNode: PN) { def >[F[_], To, NewOut, NewOutResult, Result <: ProcessNode[_, _, _, Redirected, _]] (to: To) (implicit target: CanBeProcessOutputTarget.Aux[F, To, NewOut, NewOutResult], redirectOutput: RedirectOutput.Aux[F, PN, To, NewOut, NewOutResult, Result]): Result = { redirectOutput(processNode, to) } } </code></pre> This extension method basically just finds the appropriate type class implementations and then call it to alter the process node to register the output redirection: we are redirecting the output of processNode</code> (of type PN</code>) to to</code> (of type To</code>)</li> target</code> is the CanBeProcessOutputTarget</code> implementation, containing the actual code to set up the redirection</li> redirectOutput</code> is the process node type specific implementation of the RedirectOutput</code> interface, knowing how to set up the redirection of a Process</code> or a PipedProcess</code></li> </ul> This code would compile, but we won't be able to use it. For example for the following code: running <- (Process[IO]("echo", List("Hello world!")) > tempFile.toPath).start(blocker) </code></pre> It fails with not being able to resolve the implicits correctly. The exact error of course depends much on the context but one example for the above line could be: [error] prox/src/test/scala/io/github/vigoo/prox/ProcessSpecs.scala:95:63: diverging implicit expansion for type cats.effect.Concurrent[F] [error] starting with method catsIorTConcurrent in object Concurrent [error] running <- (Process[IO]("echo", List("Hello world!")) > tempFile.toPath).start(blocker) </code></pre> This does not really help understanding the real problem though. As we have seen earlier, in this library the Process</code> types have to be parameterized with the context as well, because they store their redirection logic within themselves. That's why we specify it explicitly in the example to be IO</code>: Process[IO](...)</code>. What we would expect is that by tying F[_]</code> to IO</code> at the beginning, all the subsequent operations such as the ></code> redirection would respect this and the context gets inferred to be IO</code> everywhere in the expression. The compiler cannot do this. If we check the definition of ></code> again, you can see that there is no connection expressed between the type PN</code> (the actual process node type) and F</code> which is used as a type parameter for the implicit parameters. The fix is to link the two, and we have a technique exactly for this that I described earlier: the aux pattern. First let's write some code that, in compile time, can "extract" the context type from a process node type: trait ContextOf[PN] { type Context[_] } object ContextOf { type Aux[PN, F[_]] = ContextOf[PN] { type Context[_] = F[_] } def apply[PN <: ProcessNode[_, _, _, _, _], F[_]](implicit contextOf: ContextOf.Aux[PN, F]): Aux[PN, F] = contextOf implicit def contextOfProcess[F[_], Out, Err, OutResult, ErrResult, IRS <: RedirectionState, ORS <: RedirectionState, ERS <: RedirectionState]: Aux[Process[F, Out, Err, OutResult, ErrResult, IRS, ORS, ERS], F] = new ContextOf[Process[F, Out, Err, OutResult, ErrResult, IRS, ORS, ERS]] { override type Context[_] = F[_] } implicit def contextOfPipedProcess[ F[_], Out, Err, PN1 <: ProcessNode[_, _, _, _, _], PN2 <: ProcessNode[_, _, _, _, _], IRS <: RedirectionState, ORS <: RedirectionState, ERS <: RedirectionState]: Aux[PipedProcess[F, Out, Err, Byte, PN1, PN2, IRS, ORS, ERS], F] = new ContextOf[PipedProcess[F, Out, Err, Byte, PN1, PN2, IRS, ORS, ERS]] { override type Context[_] = F[_] } } </code></pre> Both Process</code> and PipedProcess</code> have the context as their first type parameter. By creating the ContextOf</code> type class and the corresponding Aux</code> type we can extend the ></code> operator to require such a connection (a way to get a F[_]</code> context out of a type PN</code>) in compile time, and with the aux pattern it unifies the type parameters and the context type gets chained through all the subsequent calls as we desired: def >[F[_], To, NewOut, NewOutResult, Result <: ProcessNode[_, _, _, Redirected, _]] (to: To) (implicit contextOf: ContextOf.Aux[PN, F], target: CanBeProcessOutputTarget.Aux[F, To, NewOut, NewOutResult], redirectOutput: RedirectOutput.Aux[F, PN, To, NewOut, NewOutResult, Result]): Result = { redirectOutput(processNode, to) } </code></pre> ZIO</h2> Now that everything is in place, we can try out whether prox is really working with other effect libraries such as ZIO</a>. ZIO has a compatibility layer for cats-effect. It's the implementation of the type classes cats-effect provides. It is in an extra library called zio-interop-cats</a>. For running processes with prox we can use the following variants of the ZIO</code> type: RIO[-R, +A]</code> which is an alias for ZIO[R, scala.Throwable, A]</code></li> or Task[A]</code> which is an alias for ZIO[scala.Any, scala.Throwable, A]</code> if we don't take advantage of the environment parameter R</code>.</li> </ul> This in fact assuming the correct context only means switching IO</code> to RIO</code> or Task</code> in the type parameter for Process</code>: import zio.interop.catz._ Blocker[RIO[Console, ?]].use { blocker => for { // ... _ <- console.putStrLn("Starting external process...") _ <- (Process[Task]("echo", List("Hello world!")) > tempFile.toPath).start(blocker) // ... } yield () } </code></pre> A nice way to have everything set up for this is to use the interop library's CatsApp</code></a> trait as an entrypoint for the application. This brings all the necessary implicits in scope and requires you to implement the following function as the entry point of the application: def run(args: List[String]): ZIO[Environment, Nothing, Int] </code></pre> prox part 2 - akka streams with cats effect 2019-03-07T00:00:00+00:00 Blog post series</h2> Part 1 - type level programming</a></li> Part 2 - akka streams with cats effect</a></li> Part 3 - effect abstraction and ZIO</a></li> Part 4 - simplified redesign</a></li> </ul> Intro</h2> In the previous post we have seen how prox</a> applies advanced type level programming techniques to express executing external system processes. The input and output of these processes can be connected to streams. The current version of prox</a> uses the fs2</a> library to describe these streams, and cats-effect</a> as an IO abstraction, allowing it to separate the specification of a process pipeline from its actual execution. In this post we will keep cats-effect</a> but replace fs2</a> with the stream library of the Akka toolkit, Akka Streams</a>. This will be a hybrid solution, as Akka Streams is not using any kind of IO abstraction, unlike fs2</a> which is implemented on top of cats-effect</a>. We will experiment with implementing prox</a> purely with the Akka libraries in a future post. Replacing fs2 with Akka Streams</h2> We start by removing the fs2</a> dependency and adding Akka Streams: - "co.fs2" %% "fs2-core" % "1.0.3", - "co.fs2" %% "fs2-io" % "1.0.3", + "com.typesafe.akka" %% "akka-stream" % "2.5.20", </code></pre> Then we have to change all the fs2 types used in the codebase to the matching Akka Streams types. The following table describe these pairs: fs2</th> Akka Streams</th></tr></thead> Stream[IO, O]</code></td> Source[O, Any]</code></td></tr> Pipe[IO, I, O]</code></td> Flow[I, O, Any]</code></td></tr> Sink[IO, O]</code></td> Sink[O, Future[Done]</code></td></tr> </tbody></table> Another small difference that requires changing a lot of our functions is the implicit context these streaming solutions require. With the original implementation it used to be: an implicit ContextShift[IO]</code> instance</li> and an explicitly passed blocking execution context of type ExecutionContext</code></li> </ul> We can treat the blocking execution context as part of the implicit context for prox too, and could refactor the library to pass both of them wrapped together within a context object. Let's see what we need for the Akka Streams based implementation! an implicit ContextShift[IO]</code> is still needed because we are still using cats-effect</code> as our IO abstraction</li> The blocking execution context however was only used for passing it to fs2, so we can remove that</li> And for Akka Streams we will need an execution context of type ExecutionContext</code> and also a Materializer</code>. The materializer is used by Akka Streams to execute blueprints of streams. The usual implementation is ActorMaterializer</code> which does that by spawning actors implementing the stream graph.</li> </ul> So for example the start</code> extension method, is modified like this: - def start[RP](blockingExecutionContext: ExecutionContext) (implicit start: Start.Aux[PN, RP, _], contextShift: ContextShift[IO]): IO[RP] + def start[RP]() (implicit start: Start.Aux[PN, RP, _], contextShift: ContextShift[IO], materializer: Materializer, executionContext: ExecutionContext): IO[RP] </code></pre> It turns out that there is one more minor difference that needs changes in the internal type signatures. In Akka Streams byte streams are represented by not streams of element type Byte</code>. like in fs2, but streams of chunks called ByteString</code>s. So everywhere we used Byte</code> as element type, such as on the process boundaries, we now simply have to use ByteStrings</code>, for example: - def apply(from: PN1, to: PN2, via: Pipe[IO, Byte, Byte]): ResultProcess + def apply(from: PN1, to: PN2, via: Flow[ByteString, ByteString, Any]): ResultProcess </code></pre> Another thing to notice is that fs2 had a type parameter for passing the IO</code> monad to run on. As I wrote earlier, Akka Streams does not depend on such abstractions, so this parameter is missing. On the other hand, it has a third type parameter which is set in the above example to Any</code>. This parameter is called Mat</code> and represents the type of the value the flow will materialize to. At this point we don't care about it so we set it to Any</code>. Let's take a look of the connect</code> function of the ProcessIO</code> trait. With fs2 the InputStreamingSource</code> is implemented like this: class InputStreamingSource(source: Source[ByteString, Any]) extends ProcessInputSource { override def toRedirect: Redirect = Redirect.PIPE override def connect(systemProcess: lang.Process, blockingExecutionContext: ExecutionContext) (implicit contextShift: ContextShift[IO]): Stream[IO, Byte] = { source.observe( io.writeOutputStream[IO]( IO { systemProcess.getOutputStream }, closeAfterUse = true, blockingExecutionContext = blockingExecutionContext)) } override def run(stream: Stream[IO, Byte])(implicit contextShift: ContextShift[IO]): IO[Fiber[IO, Unit]] = Concurrent[IO].start(stream.compile.drain) } </code></pre> We have a source</code> stream and during the setup of the process graph, when the system process has been already created, we have to set up the redirection of this source stream to this process. This is separated to a connect</code> and a run</code> step: The connect</code> step creates an fs2 stream that observers the source stream and sends each byte to the system process's standard input. This just defines this stream, and returns it as a pure functional value.</li> The run</code> step on the other hand has the result type IO[Fiber[IO, Unit]]</code>. It defines the effect of starting a new thread and running the stream on it.</li> </ul> In the case of fs2 we can be sure that the source.observe</code> function is pure just by checking it's type signature: def observe(p: Pipe[F, O, Unit])(implicit F: Concurrent[F]): Stream[F, O] </code></pre> All side-effecting functions in fs2 are defined as IO</code> functions, so we simply know that this one is not among them, and that's why the connect</code> was a pure, non-IO</code> function in the original implementation. With Akka Streams we don't have any information about this encoded in the type system. We use the source.alsoTo</code> function: def alsoTo(that: Graph[SinkShape[Out], _]): Repr[Out] </code></pre> which is actually also pure (only creating a blueprint of the graph to be executed), so we can safely replace the implementation to this in the Akka Streams version: class InputStreamingSource(source: Source[ByteString, Any]) extends ProcessInputSource { override def toRedirect: Redirect = Redirect.PIPE override def connect(systemProcess: lang.Process)(implicit contextShift: ContextShift[IO]): Source[ByteString, Any] = source.alsoTo(fromOutputStream(() => systemProcess.getOutputStream, autoFlush = true)) override def run(stream: Source[ByteString, Any]) (implicit contextShift: ContextShift[IO], materializer: Materializer, executionContext: ExecutionContext): IO[Fiber[IO, Unit]] = { Concurrent[IO].start(IO.async { finish => stream.runWith(Sink.ignore).onComplete { case Success(Done) => finish(Right(())) case Failure(reason) => finish(Left(reason)) } }) } } </code></pre> The implementation of run</code> above is a nice example of how we can integrate asynchronous operations not implemented with cats-effect</code> to an IO</code> based program. With IO.async</code> we define how to start the asynchronous operation (in this case running the Akka stream) and we get a callback function, finish</code> to be called when the asynchronous operation ends. The stream here materializes to a Future[T]</code> value, so we can use it's onComplete</code> function to notify the IO system about the finished stream. The IO</code> value returned by IO.async</code> represents the whole asynchronous operation, it returns it's final result when the callback is called, and "blocks" the program flow until it is done. This does not mean actually blocking a thread; but the next IO function will be executed only when it finished running (as it's type is IO[A]</code>). That is not what we need here, so we use Concurrent[IO].start</code> to put this IO</code> action on a separate fiber. This way all streams involved in the process graph will be executing in parallel. Calculating the result</h3> prox</a> supports multiple ways to calculate a result of running a process graph: If the target is a Sink</code>, the result type is Unit</code></li> If the pipe's output is Out</code> and there is a Monoid</code> instance for Out</code>, the stream is folded into an Out</code> value</li> Otherwise if the pipe's output is Out</code>, the result type will be Vector[Out]</code></li> </ul> These cases can be enforced by the Drain</code>, ToVector</code> and Fold</code> wrapper classes. Let's see how we can implement them with Akka Streams compared to fs2. Drain sink</h4> The sink version was implemented like this with fs2: Concurrent[IO].start(stream.compile.drain) </code></pre> .compile</code> gets an interface that can be used to convert the stream to a IO[A]</code> value in multiple ways.</li> .drain</code> is one of them. It runs the stream but ignores its elements, having a result type of IO[Unit]</code>.</li> We want to run this concurrently with the other streams so we move it to a fiber</li> </ul> With Akka Streams there is one big difference. In fs2 the sink is represented as a Pipe[F, E, Unit]</code>, so we could treat it in the same way as other stream segments. In this case the Sink</code> is not a Flow</code>, so we do a trick to keep the interface as close to the original one as possible: create((sink: Sink[ByteString, Future[R]]) => new OutputStreamingTarget(Flow.fromFunction(identity)) with ProcessOutputTarget[ByteString, R] { override def run(stream: Source[ByteString, Any]) (implicit contextShift: ContextShift[IO], materializer: Materializer, executionContext: ExecutionContext): IO[Fiber[IO, R]] = Concurrent[IO].start(IO.async { complete => stream.runWith(sink).onComplete { case Success(value) => complete(Right(value)) case Failure(reason) => complete(Left(reason)) } }) } </code></pre> The trick is that we create the OutputStreamingTarget</code> with an identity flow, and only use the Sink</code> when we actually run the stream, passing it to the runWith</code> function. This materializes the stream into a Future[Done]</code> value, that we can tie back to our IO</code> system with IO.async</code> as I already described it. Combine with Monoid</h4> When the element type is a monoid we can fold it into a single value. Fs2 directly supports this: Concurrent[IO].start(stream.compile.foldMonoid) </code></pre> Akka Streams does not use cats type classes, but it also has a way to fold the stream, so we can easily implement it using the monoid instance: Concurrent[IO].start(IO.async { complete => stream.runFold(monoid.empty)(monoid.combine).onComplete { case Success(value) => complete(Right(value)) case Failure(reason) => complete(Left(reason)) } }) </code></pre> Vector of elements</h4> Finally let's see the version that keeps all the stream elements in a vector as a result: Concurrent[IO].start(stream.compile.toVector) </code></pre> With Akka Streams we can do it by running the stream into a sink created for this, Sink.seq</code>. It materializes into a Future[Seq[T]]</code> value that holds all the elements of the executed stream: Concurrent[IO].start(IO.async { complete => stream.runWith(Sink.seq).onComplete { case Success(value) => complete(Right(value.toVector)) case Failure(reason) => complete(Left(reason)) } }) </code></pre> Testing</h3> At this point the only remaining thing is to modify the tests too. One of the more complex examples is the customProcessPiping</code> test case. With fs2 it takes advantage of some text processing pipe elements coming with the library: val customPipe: Pipe[IO, Byte, Byte] = (s: Stream[IO, Byte]) => s .through(text.utf8Decode) .through(text.lines) .map(_.split(' ').toVector) .map(v => v.map(_ + " !!!").mkString(" ")) .intersperse("\n") .through(text.utf8Encode) val proc = Process("echo", List("This is a test string")) .via(customPipe) .to(Process("wc", List("-w")) > text.utf8Decode[IO]) </code></pre> There are similar tools in Akka Streams to express this in the Framing</code> module: val customPipe = Framing.delimiter( delimiter = ByteString("\n"), maximumFrameLength = 10000, allowTruncation = true ).map(_.utf8String) .map(_.split(' ').toVector) .map(v => v.map(_ + " !!!").mkString(" ")) .intersperse("\n") .map(ByteString.apply) val proc = Process("echo", List("This is a test string")) .via(customPipe) .to(Process("wc", List("-w")) > utf8Decode) </code></pre> where utf8Decode</code> is a helper sink defined as: val utf8Decode: Sink[ByteString, Future[String]] = Flow[ByteString] .reduce(_ ++ _) .map(_.utf8String) .toMat(Sink.head)(Keep.right) </code></pre> First it concatenates the ByteString</code> chunks, then simply calls .utf8String</code> on the result. Final thoughts</h2> We have seen that it is relatively easy to replace the stream library in prox</a> without changing it's interface much, if we keep cats-effect</a> for expressing the effectful computations. The complete working example is available on the akka-streams</code> branch</a>. prox part 1 - type level programming 2019-02-10T00:00:00+00:00 Blog post series</h2> Part 1 - type level programming</a></li> Part 2 - akka streams with cats effect</a></li> Part 3 - effect abstraction and ZIO</a></li> Part 4 - simplified redesign</a></li> </ul> Intro</h2> I started writing prox</a> at the end of 2017 for two reasons. First, I never liked any of the existing solutions for running external processes and capture their input/output streams. And I just returned from the scala.io conference</a> full of inspiration; I wanted to try out some techniques and libraries and this seemed to be a nice small project to do so. Since then, prox</a> has been proved to be useful, we are using it at Prezi</a> in all our Scala projects where we have to deal with external processes. The last stable version was created last October, after cats-effect 1.0</a> and fs2 1.0</a> was released. This is the first part of a series of blog posts dedicated to this library. In the first one I'm going to talk about shapeless</a> and type level programming techniques are used to create a strongly typed interface for starting system processes. In future posts I will explore replacing its dependencies such as using akka-streams</a> instead of fs2</a> or ZIO</a> instead of cats-effect</a>. These different versions will be a good opportunity to do some performance comparison, and to close the series with creating a new version of the library which is easier to use in the alternative environments. Limiting redirection</h2> When I started writing the library I wanted to explore how I can express some strict constraints on the type level: A process can have its input, output and error streams redirected, but only once</li> Processes without redirected output can be piped to processes without a redirected input</li> </ul> In prox 0.2.1 a single system process is described by the following type: class Process[Out, Err, OutResult, ErrResult, IRS <: RedirectionState, ORS <: RedirectionState, ERS <: RedirectionState]( val command: String, val arguments: List[String], val workingDirectory: Option[Path], val inputSource: ProcessInputSource, val outputTarget: ProcessOutputTarget[Out, OutResult], val errorTarget: ProcessErrorTarget[Err, ErrResult], val environmentVariables: Map[String, String]) extends ProcessNode[Out, Err, IRS, ORS, ERS] { // ... } </code></pre> but let's focus first on the requirement to be able to redirect one of the streams maximum once. This is encoded by the IRS</code>, ORS</code> and ERS</code> type parameters, which are all have to be subtypes of RedirectionState</code>. RedirectionState</code> is a phantom type; there are no values ever created of this type, it is only used in type signatures to encode whether one of the three streams are already redirected or not: /** Phantom type representing the redirection state of a process */ sealed trait RedirectionState /** Indicates that the given channel is not redirected yet */ trait NotRedirected extends RedirectionState /** Indicates that the given channel has already been redirected */ trait Redirected extends RedirectionState </code></pre> So for example with a simplified model of a process, Process[IRS <: RedirectionState, ORS <: RedirectionState, ERS <: RedirectionState]</code>, using the output redirection operator ></code> would change the types in the following way: val p1: Process[NotRedirected, NotRedirected, NotRedirected] = ??? val p2: Process[NotRedirected, Redirected, NotRedirected] = p1 > (home / "tmp" / "out.txt") val p3 = p2 > (home / "tmp" / "another.txt") // THIS MUST NOT COMPILE </code></pre> How can we restrict the redirect function to only work on Process[_, NotRedirected, _]</code>? We can define it as an extension method with an implicit class (once again this is a simplified version focusing only on the redirection state handling): implicit class ProcessNodeOutputRedirect[ IRS <: RedirectionState, ERS <: RedirectionState, PN <: Process[IRS, NotRedirected, ERS]](process: PN) { def >[To](to: To)(implicit target: CanBeProcessOutputTarget[To]): Process[IRS, Redirected, ERS] = ??? } </code></pre> By forcing the ORS</code> type parameter to be NotRedirected</code> and setting it to Redirected</code> in the result type we can guarantee that this function can only be called on a process that does not have their output redirected yet. The target of the redirection is extensible through the CanBeProcessOutputTarget</code> type class, as we will see later. Dependent types</h2> Reality is much more complicated, because of process piping and because the process types encode the redirection result types too. Let's get back to our ></code> function and see how we could modify it so it works with piped processes too. Anyway, how is process piping encoded in this library? Two processes connected through a pipe are represented by the PipedProcess</code> class. Both Procses</code> and PipedProcess</code> implements the following trait: sealed trait ProcessNode[Out, Err, IRS <: RedirectionState, ORS <: RedirectionState, ERS <: RedirectionState] </code></pre> We've already seen Process</code>. PipedProcess</code> is a bit more complicated: class PipedProcess[Out, Err, PN1Out, PN1 <: ProcessNode[_, _, _, _, _], PN2 <: ProcessNode[_, _, _, _, _], IRS <: RedirectionState, ORS <: RedirectionState, ERS <: RedirectionState] (val from: PN1, val createTo: PipeConstruction[PN1Out] => PN2) extends ProcessNode[Out, Err, IRS, ORS, ERS] { // ... } </code></pre> To make ></code> work on both, we can start by modifying its definition to work on any ProcessNode</code> not just Process</code> (omitting the output type params for now): implicit class ProcessNodeOutputRedirect[ IRS <: RedirectionState, ERS <: RedirectionState, PN <: ProcessNode[IRS, NotRedirected, ERS]](process: PN) { def >[To](to: To)(implicit target: CanBeProcessOutputTarget[To]): ProcessNode[IRS, Redirected, ERS] = ??? } </code></pre> This has a serious problem though. The output type is ProcessNode</code> and not the "real" process type, which means that we lose type information and all the other dependent typed operations will not work. We have to make the result type depend on the input! We may try to use the RedirectionOutput</code> type class like this: implicit class ProcessNodeOutputRedirect[ IRS <: RedirectionState, ERS <: RedirectionState, PN <: ProcessNode[IRS, NotRedirected, ERS]](process: PN) { def >[To](to: To) (implicit target: CanBeProcessOutputTarget[To], redirectOutput: RedirectOutput[PN, To]): redirectOutput.Result = redirectOutput(to) } </code></pre> Here the result (redirectOutput.Result</code>) is a path dependent type. This may work in some simple cases but have two serious issues: It is not possible to use redirectOutput.Result</code> in the parameter block of the function, so if another type class needed it as a type parameter we could not pass it.</li> Further implicit resolutions and type level operations will quickly break as the compiler will not be able to unify the various path dependent types</li> </ul> The Aux pattern, used heavily in the shapeless</a> library provides a nice pattern for fixing both problems. We start by defining a type class for describing the operation, in this case redirecting the output channel of a process: trait RedirectOutput[PN <: ProcessNode[_, NotRedirected, _], To] { type Result <: ProcessNode[_, Redirected, _] def apply(process: PN, to: To)(implicit target: CanBeProcessOutputTarget[To]): Result } object RedirectOutput { type Aux[PN <: ProcessNode[_, NotRedirected, _], To, Result0] = RedirectOutput[PN, To] { type Result = Result0 } // ... type class instances } </code></pre> The type class itself is straightforward. We have to implement it for both Process</code> and PipedProcess</code> and set the Result</code> type accordingly, then implement apply</code> that sets up the actual redirection. But what the Aux</code> type is for? It solves the problems with the path dependent version if we use it like this: implicit class ProcessNodeOutputRedirect[ IRS <: RedirectionState, ERS <: RedirectionState, PN <: ProcessNode[IRS, NotRedirected, ERS]](process: PN) { def >[To, Result <: ProcessNode[_, Redirected, _]](to: To) (implicit target: CanBeProcessOutputTarget[To], redirectOutput: RedirectOutput.Aux[PN, To, Result]): Result = redirectOutput(to) } </code></pre> By lifting the Result</code> from the type class instance to a type parameter the compiler can now "extract" the calculated type from redirectOutput.Result</code> to the ></code> function's Result</code> type parameter and use it directly, both for other further type requirements or as we do here, in the result type. This is the basic pattern used for all the operations in prox. You can check Luigi's short introduction to the Aux</code> pattern</a> for a more detailed explanation. Starting the processes</h2> So far we just combined purely functional data structures in a complicated way. The result value may encode the launching of several system processes that are connected via pipes to each other and possibly other streams as we will see. When we eventually decide to start these processes, we need a way to observe their status, wait for them to stop, get their exit code, and to access the data sent to the output streams if they were redirected. And we need this per process, while launching the whole process graph in a single step. First let's model a single running process: trait RunningProcess[Out, OutResult, ErrResult] { def isAlive: IO[Boolean] def waitForExit(): IO[ProcessResult[OutResult, ErrResult]] def terminate(): IO[ProcessResult[OutResult, ErrResult]] } </code></pre> and ProcessResult</code> that represents an already terminated process: case class ProcessResult[OutResult, ErrResult]( exitCode: Int, fullOutput: OutResult, fullError: ErrResult ) </code></pre> Now we need to define a start</code> extension method on ProcessNode</code> that returns somehow one well typed RunningProcess</code> for each system process that it starts. Let's forget for a second about having multiple processes piped together and just consider the single process case. For that, we would need somehing like this (the Out</code> parameter is needed only for piping so I omitted it): def start: IO[RunningProcess[OutResult, ErrResult]] </code></pre> Now we can see why Process</code> has those additional type paramters. It is not enough to encode whether the output and error channels were redirected or not, we also have to encode the expected result type of redirecting these. By storing these types in type parameters of Process</code> we can easily imagine that by using the pattern described in the previous section, the result type can depend on what we redirected the process to. Let's see some examples of what this means! Target</th> Result type</th></tr></thead> A file system path</td> The result type is Unit</code>, the redirection happens on OS level</td></tr> Sink</td> The result type is Unit</code>, only the sink's side effect matters</td></tr> Pipe with monoid elem type</td> The stream is folded by the monoid, the result type is T</code></td></tr> Pipe with non-monoid elem type</td> The stream captures the elements in a vector, the result type is Vector[T]</code></td></tr> Custom fold function</td> The result type is the function's result type</td></tr> </tbody></table> The CanBeProcessOutputTarget</code> type class we've seen earlier defines both the stream element type and the result type: trait CanBeProcessOutputTarget[To] { /** Output stream element type */ type Out /** Result type of running the output stream */ type OutResult def apply(to: To): ProcessOutputTarget[Out, OutResult] } </code></pre> ProcessOutputTarget</code> contains the actual IO code to build the redirection of the streams, I won't get into details in this post. Note that there are similar type classes for error and input redirection too. For two processes piped together we have to provide two RunningProcess</code> instances with the proper result type parameters. So we can see that it is not enough that the redirection stores the result type in the process type, the start method must be dependent typed too. One way to encode this in the type system would be something like this (simplified): val p1 = Process() val p2 = Process() val p3 = Process() val rp1: IO[RunningProcess] = p1.start val rp2: IO[(RunningProcess, RunningProcess)] = (p1 | p2).start val rp3: IO[(RunningProcess, RunningProcess, RunningProcess)] = (p1 | p2 | p3).start </code></pre> We encode piped processes with tuples of RunningProcess</code> and single process with a single RunningProcess</code>. To implement this we can make use of the shapeless</a> library's HList</code> implementation. HLists are heterogeneous lists; basically similar to a tuple, but with all the "usual" list-like functions implemented as dependent typed functions. It's type describes the types of all its elements, and you can split it to head/tail, append two, etc. And we can do it both on the type level (computing the result type of appending two HList</code>'s, for example) and on the value leve (appending the two values creating a third HList</code> value). We can implement the start</code> method more easily by building a HList</code>, while still keep the desired interface as shapeless</a> implements a conversion from HList</code> to tuples. We can define two separate start functions, one producing HList</code> and another the tuples (IO releated parameters omitted): def start[RP](implicit start: Start.Aux[PN, RP, _]]): IO[RP] = ??? def startHL[RPL <: HList](implicit start: Start.Aux[PN, _, RP[IO]): IO[RPL] = ??? </code></pre> The Start</code> type class calculates both the tupled and the HList</code> version's result type. The implementation's responsibility is to start the actual system processes and wire the streams together. The interesting part is how we use type level calculations from shapeless</a> to calculte the tuple and HList</code> types for piped processes. This is all done using the technique I described earlier, but may look a bit shocking first. Let's take a look! implicit def startPipedProcess[ Out, Err, PN1 <: ProcessNode[_, _, _, _, _], PN2 <: ProcessNode[_, _, _, _, _], IRS <: RedirectionState, ORS <: RedirectionState, ERS <: RedirectionState, RP1, RPL1 <: HList, RP1Last <: RunningProcess[_, _, _], RP2, RPL2 <: HList, RP2Head <: RunningProcess[_, _, _], RP2Tail <: HList, RPT, RPL <: HList] (implicit start1: Start.Aux[PN1, RP1, RPL1], start2: Start.Aux[PN2, RP2, RPL2], last1: Last.Aux[RPL1, RP1Last], rp1LastType: RP1Last <:< RunningProcess[Byte, _, _], hcons2: IsHCons.Aux[RPL2, RP2Head, RP2Tail], prepend: Prepend.Aux[RPL1, RPL2, RPL], tupler: Tupler.Aux[RPL, RPT]): Aux[PipedProcess[Out, Err, Byte, PN1, PN2, IRS, ORS, ERS], RPT, RPL] = new Start[PipedProcess[Out, Err, Byte, PN1, PN2, IRS, ORS, ERS]] { override type RunningProcesses = RPT override type RunningProcessList = RPL // ... } </code></pre> The way to parse this is to follow the type level computations performed through the Aux types in the implicit parameter list: PN1</code> and PN2</code> are the types of the two processes piped together</li> The first two implicit definition calculates the running process tuple and the running process HList types of these inidividual process nodes and "stores" the results in RP1</code>, RPL1</code>, RP2</code> and RPL2</code> type parameters. For example if the two processes pipe together are single Process</code> instances, then RP1</code> and RP2</code> would be some kind of RunningProcess</code>, and the HLists would be one element long, like RunningProcess :: HNil</code>.</li> The last1</code> implicit parameter is a type level last functinon on the first process's HList</code>. This is required because PN1</code> itself can also be a sequence of piped processes, and we are connecting PN2</code> to the last of these. The RP1Last</code> type parameter becomes the type of the last running process of the first process node.</li> The next line, rp1LastType</code> is an additional constraint fixing the output stream element type of RP1Last</code> to Byte</code>. The piping implementation is not able to connect streams of arbitrary element types, as the process input is always required to be a byte stream.</li> hcons2</code> is similar to the last1</code> but here we are calculating the type level head type of the HList</code> called RPL2</code>. The head will be in RP2Head</code> and the tail HList</code> in RP2Tail</code>.</li> In the prepend</code> step we concatenate RPL1</code> with RPL2</code> using the Prepend</code> operation, the result HList</code> type is in RPL</code>. This is the HList</code> representation of the piped running process.</li> Finally we use the Tupler</code> operation to calculate the tuple type from the HList</code>, and store it in RPT</code>.</li> </ul> The compiler perform the type level calculations and we can use the result types RPT</code> and RPL</code> to actually implement the start typeclass. This is the most complicated type level calculation in the library. Final thoughts</h2> As we've seen, Scala's type system can bring us quite far in expressing a dependent typed interface. On the other hand writing and reading code in this style is really hard, and if things go wrong, decoding the compiler's error messages is not an easy task either. This is a serious tradeoff that has to be considered and in many cases a more dynamic but much more readable and maintainable approach can be better. With prox</a> I explicitly wanted to explore these features of the Scala language. In the next posts we will ignore the type level parts of the library and focus on different streaming and effect libraries. AWS rate limits vs prezidig 2018-09-21T00:00:00+00:00 At Prezi</a>, we have an internal tool called prezidig for discovering AWS resources. I like it a lot so I was quite annoyed recently that it always fails with a throttling exception because of our increased use of the AWS API. It made it completely unusable, so I decided to try to fix this. Then I decided to write the story in this blog post, as the steps I had to made to achieve the results I aimed for can be useful for writing maintainable, fast and safe Scala code in the future. I will describe the phases as they happened, as I did not really know anything about this codebase so the path to the success was not exactly clear immediately. Wrapping the calls</h2> So my initial thought was to just find the AWS API calls and wrap them in a helper function which catches the throttling error and retries with an increasing delay. I basically wrote this in the base class of all the mirrors (the classes which are responsible for fetching AWS and other resource data for prezidig): protected def byHandlingThrottling[T](awsCall: => T): Future[T] = { def call(remainingTries: Int, wait: FiniteDuration): Future[T] = { Future(Try(awsCall)).flatMap { case Success(result) => Future.successful(result) case Failure(awsException: AmazonServiceException) if awsException.getErrorCode == "Throttling" && remainingTries > 0 => akka.pattern.after(wait, actorSystem.scheduler) { call(remainingTries - 1, wait * 2) } case Failure(reason) => Future.failed(reason) } } call(10, 100.millis) // TODO: make configurable } </code></pre> Then the only thing I had to do was to was wrapping all the existing AWS calls with this. Then I realized that this won’t be this simple, as these calls were not always asynchronous, just sometimes. To see an example, for an ElasticBeanstalk application, it fetches the application metadata with synchronous call, then fetches the related EB environments asynchronously. The whole thing might be wrapped in another future somewhere else, but that’s a different story. While making these discoveries I also found several synchronization points, like the code waiting for some futures to complete in a blocking way. Also that the model is mutable. So… just for trying this out, I still wrapped all the AWS calls with this stuff, by converting the future back to a synchronous call by immediately blocking on it. What did I achieve with this? Well, some throttling errors were fixed, the code became extremely ugly, and I could not even wrap everything so the errors remained, and because of the tons of blocking, timeouts, etc. it was basically impossible to understand whether this would work or deadlock or just be slow. That was the point I decided to do this properly Reflection</h2> Before solving the real problem I found that the mirrors are initialized via reflection, something like this: def buildMirrors[A <: RegionAwareAWSMirror[_, _]](implicit mf: Manifest[A]): Seq[A] = Config.regions.map(region => mf.runtimeClass.getConstructor(classOf[String]).newInstance(region).asInstanceOf[A]) </code></pre> This is something that you should avoid, as it leads to problems that are not detected by the compiler, only at runtime, every time you refactor something around these classes. There are some use cases where this may be required, like dynamically loading plugins or stuff like this, but to just have a factory for something, it is must simple to use… functions! So I could not hold myself back and quickly changed this to: def buildMirrors[A <: RegionAwareAWSMirror[_, _]](factory: (String, ActorSystem) => A) Config.regions.map(region => factory(region, system)) </code></pre> (Since then even this has disappeared, but don’t run that much forward). Async fetching</h2> Ok so the first obvious step was to refactor the whole fetching code in a way that it is just a chain of futures. By making everything async in the process, the AWS calls would be simply replaceable with the throttling function above or anything more sophisticated! But I knew that I cannot safely do this while the model we are building itself is mutable - there is no way I want to debug what happens with it once all the steps are really becoming parallel! Immutable model</h3> I believe the following GitHub diff captures the core change of this step: Of course I had to change all the subtypes of Model, and I went through the code looking for vars</li> mutable collections</li> </ul> and got rid of them. Except for the caching constructs, because I planned to refactor those in the next step, so for now I left them alone. Async mirrors</h3> Once I felt the model is safe enough, I went to the next big change, making everything asynchronous. This took some hours, to be honest. But really, the core idea is only that the result must be a Future[T]</code>, not T</code>. So how do you refactor a code that was previously half synchronous, half asynchronous to achieve this? Let’s see an example! It will be the key-pair mirror as it is the smallest. Originally (with my ugly wrapping in the previous step) it looked like this: override protected def fetch(input: SimpleParsedInput, context: Context): Seq[KeyPair] = try { val futureResult = byHandlingThrottling( buildClient(AmazonEC2ClientBuilder.standard()).describeKeyPairs( new DescribeKeyPairsRequest().withKeyNames(input.id) )) val result = Await.result(futureResult, 10.seconds) result.getKeyPairs.asScala.map(info => KeyPair(info, region)).seq .map(keypair => keypair.withFutureChildren(LaunchConfigurationMirror(region, actorSystem).apply(context.withInput(keypair.description.getKeyName)))) } catch { case _: AmazonEC2Exception => Seq() } </code></pre> So as you can see fetching the key pairs by name was a synchronous request, but then the launch configurations are fetched asynchronously and are being updated back the result model in a mutable way. We want to transform this function so it does not have any side effects, just performs a chain of asynchronous operations and in the end have a fully fetched key pair with the related launch configurations. In every case the only thing needed was a combination of map</code> and flatMap</code> on futures, and of course the for syntax can also be used to make the code more readable: private def fetchKeyPair(client: AmazonEC2, context: Context, info: KeyPairInfo): Future[KeyPair] = { for { launchConfigurations <- LaunchConfigurationMirror(region, actorSystem).apply(context.withInput(info.getKeyName)) } yield KeyPair( description = info, region = region, children = launchConfigurations ) } override protected def fetch(input: SimpleParsedInput, context: Context): Future[List[KeyPair]] = { val client = buildClient(AmazonEC2ClientBuilder.standard()) byHandlingThrottling(client.describeKeyPairs(new DescribeKeyPairsRequest().withKeyNames(input.id))).flatMap { result => Future.sequence( result.getKeyPairs.asScala.toList.map(fetchKeyPair(client, context, _)) ) }.recover { case _: AmazonEC2Exception => List() // TODO: log? } } </code></pre> Note that the Future.sequence</code> function is quite useful in these scenarios, as it makes a Future[List[T]]</code> from List[Future[T]]</code>. Of course the code became more verbose because of all this chaining, this is the price of this transformation. And why I don’t like to express complex logic with a chain of futures, rather with some higher level abstraction such as actors (or for this use case, streams would fit even better). But I wanted to make iterative changes, so I did this transformation on all the mirrors and eventually got a Future[List[Model]]</code> in the main function that I could await for. I also thrown out the global atomic integer that counted the running stuff for completion, as in this model the completion of the composed future should mark the end of the whole computation. So did I succeed at this point? Of course not. Actually this whole thing is a big deadlock :) Caching and circular references</h2> It was not immediately obvious what causes the deadlock. In a system like this it can happen in different ways. For example I knew that there are global singleton caches in the code, protected by locks. This could cause deadlocks if all the executors got blocked and no new threads can be spawned by the active executor. I did not know if this is happening, but would not have been surprised at all, as much more things were happening in parallel because of the previous refactoring step. And circular references in the huge chained future graph can also lead to this. Let’s consider this simplified example: trait Cache { def get(key: String): Future[Work] def put(key: String, compute: () => Future[Work]): Unit } val cache: Cache = ??? val work1: Future[Work] = cache.get("work2").map { w2 => Work(s"Hello $w2")) } val work2: Future[Work] = cache.get("work1").map { w1 => Work(s"Hello $w1")) } cache.put(work1) cache.put(work2) println(Await.result(work1), 1.second) </code></pre> This can never work. If you think about what prezidig does, you will have a feeling that this happens. A lot. But let’s go in order. Non-blocking cache</h3> First I wanted to get rid of the global, lock-protected mutable maps used as caches, and have a non-blocking implementation with more control and better performance and safety. This is the kind of job that an actor can model nicely, so I created a model cache actor that is spawned for each mirror and can store and retrieve lists of AWS models for a given key. I won’t list the whole actor’s code here, let’s see the messages it consumes: sealed trait ModelCacheMessage[M <: Model] final case class Put[M <: Model](key: String, value: List[M]) extends ModelCacheMessage[M] final case class FetchFailed[M <: Model](key: String, failure: Failure[_]) extends ModelCacheMessage[M] final case class GetOrFetch[M <: Model](key: String, fetch: () => Future[List[M]], respondTo: ActorRef[Try[List[M]]]) extends ModelCacheMessage[M] final case class GetRefOrFetch[M <: Model](key: String, fetch: () => Future[List[M]], respondTo: ActorRef[ModelRef[M]]) extends ModelCacheMessage[M] final case class Dump[M <: Model](respondTo: ActorRef[Map[String, List[M]]]) extends ModelCacheMessage[M] </code></pre> This cache itself is responsible for executing the fetch function only if needed, when the value for the given key is not cached yet. It is done by using the pipe pattern: it starts the asynchronous fetch function on a configured worker executor (which can be the actor system, or a fix thread pool, etc.) and registers an onFinish</code> callback for the future which pipes back the future’s result to the actor as actor messages (Put</code> and FetchFailed</code>). I will talk about references and cache dumps in the next section. There was one more big problem with the existing code that prevented introducing these cache actors: that the mirrors were not really singletons but some mirrors created new instances of existing mirrors (without any difference to the ones created in the main function). These shared the singleton mutable lock-protected cache map in the original version, that’s why it worked. But in the new implementation each mirror spawned its own cache actor, so it was no longer allowed to create multiple instances of the same thing. So in this step I collected all the mirrors to a class called Mirrors</code>, which later became the collection of all the resources needed to perform the “dig”, so in the final version it is called DigSite</code>. With this change the caching could be replaced, and with the ask pattern I was able to fit it to the chain of futures created in the previous step. Did it solve the deadlock? No, of course not Circular references</h3> But now it was obvious that there are some circular references. And by simply drawing it, I could see that this is actually the core concept of the whole thing :) Let me show you the drawing: So everything refers back to everything, not a surprise that this chained-together code cannot finish. To be honest, I was not sure how exactly did it work in the original version, whether the boundary of sync and async calls were carefully designed to make this work or just accidentally, whatever. I wanted to have a solution where you don’t have to think about it so nobody will fuck it up next time when it has to be modified. The chosen solution can be summarized in the following way: The models are only storing references to other models encoded by the ModelRef</code> type. A reference is basically selecting a mirror (by its cache) and an item in it by its key</li> When fetching a model, you immediately get back a model reference from the cache so it can be stored in the owner model, even with circular references. The real data is still fetched and cached as before.</li> This works because nobody uses the actual child models until the rendering of the output. So we have the asynchronous, parallel fetching of all the models, and then a completely separate, non-async step where we need the real connections to actually render the output based on the templates. I could change how the rendering works to query the model references from the cache, but I did not want to touch that part. So I introduced a middle step where all the model cache actors dump their state to simple immutable maps, and then the model gets updated by selecting the referenced models from this map and changing a field. Yes, a mutable field. It is a non-threadsafe operation that has a single, well defined place to be called, and this way the whole third part (rendering the output) could remain untouched.</li> Because of decoupling the actual fetching from the result future (it is completed earlier, as it only needs the references!), I had to have something that keeps track of the ongoing tasks ran by the cache actors, so there is also a work monitor actor that notifies the main logic once everything is complete.</li> </ul> Considering all this, the main steps before starting to render the output looks like this: val result = for { models <- runRelevantMirrors(digSite.allMirrors, Context.initial(input)) fetchingDone <- digSite.workMonitor ? WorkMonitor.WaitForReady cacheDumps <- CacheDumps.fromMirrors(digSite.allMirrors) _ = models.foreach(_.resolveChildren(cacheDumps)) // side effect! } yield models </code></pre> Anyone else blocking?</h2> At this point the tool started to work again and produce results. So I went back checking if any other blocking code remained that can be implemented in other ways. The progress tracker was like that, it had mutable state and locks, so I converted that to an actor too. It was quite simple, and on the usage side almost nothing changed compared to the original. And what about the throttling?</h2> Ok so at this point I refactored the whole stuff but still did not solve the throttling issue, right? Right. But now finally I knew how to do it! I already wrapped all AWS calls with that specific function (and at this point it was really all calls, not just almost). So I just had to write it in a better way. I wanted to: Have control on how many AWS requests are we doing in parallel</li> In case of throttling errors delay everything as soon as possible</li> </ul> This can be achieved easily by some standard patterns like treating AWS as an encapsulated resource and putting some circuit breaking logic in it, and explicitly distributing the work among multiple workers. Let’s see the designed solution on a drawing: Note: the classic Akka has built-in support for this routing and circuit breaking, but I prefer Akka-typed because of its type safety, where there are no official reusable higher level components like this yet. The one I implemented here is quite specific, later could be refactored to be built from more reusable typed actor components. So how does this work? There is a single coordinator actor called AWS and multiple (32 by default) worker actors called AWS Worker.</li> The number of worker actors control the maximum number of parallel AWS operations, because each worker actor is guaranteed to run maximum one such operation at the same time. All the other incoming requests are distributed among the workers and gets enqueued.</li> The AWS calls are executed on a different thread pool, not blocking the actors. Their result is sent back by the already mentioned pipe to pattern</li> AWS throttling errors are detected on the worker nodes, and the worker node immediately switches to open circuit state in which it does not start any new AWS command. The length of the open state increases with every throttling error, and gets reseted after a number of successful requests.</li> Opening the circuit breaker on one worker node is immediately followed by opening it on all other worker nodes too, to stop overloading AWS.</li> </ul> This could be further improved with more advanced logic but I believe it is good enough for our current purposes, and now we can use prezidig again! Bari with Visual Studio Code 2016-01-21T00:00:00+00:00 Intro</h2> A few weeks ago I discovered Visual Studio Code</a> and started using it for some of my work. (Note: I'm using multiple editors/IDEs all the time, based on the task; Emacs, Sublime, Atom, IntelliJ, VS, etc.) So far Code is my favourite among the set of similar editors, such as Atom. I was pleasently surprised how well it works with its integrated OmniSharp</a> plugin on bari's</a> codebase, so I decided to try to write a bari plugin for it. Writing an extension for Code was a nice experience. The outcome is the bari build management extension</a>, which I'll demonstrate in the next section. Developing .NET applications with Visual Studio Code and bari</h2> As Code is multiplatform, and bari also works with Mono</a>, I'll demonstrate how you can use these tools to develop a .NET application (actually bari itself) on a Mac. The steps here (except installing Mono) would be the same on Windows or Linux as well. Installing the tools</h3> First, if you are not on Windows, you'll have to install the latest Mono</a> framework. On OSX I recommed to use brew</code></a> to do that: brew install mono mono --version </code></pre> Then get the latest Visual Studio Code</a> version, either by downloading it from its homepage or with brew cask</code></a>: brew cask install visual-studio-code </code></pre> Get the latest bari. On Windows I recommend downloading and extracting the latest official release</a> and adding it to the PATH</code>. On OSX, with mono</code> we already have nuget</code>, so let's use that: cd /opt nuget install bari-mono ln -s bari-mono.1.0.2.2 bari </code></pre> and create a script to execute it somewhere in your PATH</code>: #!/bin/sh mono /opt/bari/tools/bari.exe $@ </code></pre> That's it. Future versions of the bari extension will probably be able to install bari itself. Let's start Code now! Installing the extension</h3> Open the command palette (F1, or ⇧⌘P) and type ext install bari</code> </a> Loading the project</h3> After that restart the editor. Have your bari-built project available somewhere. As we are going to develop bari itself, let's clone its repository: git clone https://github.com/vigoo/bari.git </code></pre> Then open the result bari</code> directory with Code. This should look like the following: </a> The bari plugin automatically detected that the opened folder has a suite.yaml</code> in its root, and loaded it. That's why we can see the two sections on the statusbar's right side: full</code> and debug</code>. The first one is the selected target product</a> and the second one is the selected goal</a>. All the bari commands provided by the extension will be executed with these settings. Changing the target</h3> To change the active product or goal, you can click on the statusbar or use the command palette (F1, or ⇧⌘P) and choose bari: Change goal</code> or bari: Change target product</code>. Let's change the goal to debug-mono</code>, as we are working on a non-Windows environment: </a> Generating the solution</h3> The next step before starting coding is to actually generate the solution and projects files (and fetch the dependencies, etc.) so OmniSharp can load it and provide code completion, analysis, etc. features. To do so, just use the command palette and choose bari: Regenerate solution</code>, which runs the bari vs</code> command</a> with the correct parameters. The command's output is displayed in an output panel called bari</code>. This looks like the following: </a> There's nothing else left than pointing OmniSharp to the generated solution, with the following command: </a> It will automatically find the generated .sln</code> file, just select the correct one: </a> In a few seconds (and with a few warnings for this project), OmniSharp works. To see what it can do, check this page</a>. A simple example is to jump to a given class or interface with ⌘P: </a> Working on the project</h3> You can work on the project and build it from Code or run its tests using the bari: Build</code> and bari: Test</code> commands. The build output will be shown just like in the solution generation step. </a> Whenever the suite definition itself must be modified, you can jump there with the bari: Open suite.yaml</code> command and then just regenerate the solution as it was shown above. Implementation</h2> The implementation was really straightforward. The source code can be found here</a>. It's basically a JSON defining how the plugin is integrated and some implementation code in TypeScript. It's easy to run and debug the plugin from Code itself. For example the following section from the extension definition describes what events triggers the extension: "activationEvents": [ "onCommand:bari.build", "onCommand:bari.test", "onCommand:bari.vs", "onCommand:bari.openSuiteYaml", "onCommand:bari.selfUpdate", "onCommand:bari.goal.changeCurrentGoal", "onCommand:bari.goal.changeCurrentProduct", "workspaceContains:suite.yaml" ], </code></pre> It's either done by invoking one of the defined commands from the command palette, or if the opened workspace contains a suite.yaml</code>. The latter enables the extension to parse the suite definition and initialize the statusbar immediately one the suite has been opened. The package definition also specifies the provided configuration values, such as: "bari.commandLine": { "type": "string", "default": "bari", "description": "Command line to execute bari" }, "bari.verboseOutput": { "type": "boolean", "default": false, "description": "Turns on verbose output for all the executed bari commands" } </code></pre> The implementation itself is really simple, all the user interface elements involved such as the console output window, the command palette, the statusbar panels can be easily managed. For example the panel showing bari</code>'s output is created by the following code snippet: var channel = vscode.window.createOutputChannel('bari'); channel.show(); </code></pre> Or to display the result of an operation: vscode.window.showErrorMessage("No suite.yaml in the current workspace!") </code></pre> or to create the statusbar panel: this.goals = vscode.window.createStatusBarItem(vscode.StatusBarAlignment.Right); this.goals.command = 'bari.goal.changeCurrentGoal'; this.goals.show(); </code></pre> This API is simple and well documented enough so basic integrations like this can be done in an hour. Gradle-Haskell-plugin with experimental Stack support 2015-12-22T00:00:00+00:00 I've released a new version (0.4) of gradle-haskell-plugin</a> today, with experimental stack support. It is not enabled by default, but I used it exclusively for months and it seems to get quite stable. To use it you need stack</a>, have it enabled with -Puse-stack</code> and have to keep some rules in your .cabal</code> file, as explained in the README</a>. How does it work?</h2> The core idea did not change compared to the original, cabal based solution</a>. To support chaining the binary artifacts, I had to add a new option to stack called extra package databases</a>. The databases listed in this section are passed after the global but before the snapshot and the local databases, which means that the snapshot database cannot be used (the packages in the binary artifacts are not "seeing" them). This sounds bad, but gradle-haskell-plugin does a workaround; it generates the stack.yaml</code> automatically, and in a way that: it disables snapshots on stack level (uses a resolver like ghc-7.10.2</code>)</li> lists all the dependencies explicitly in extra-deps</code></li> but it still figures out the versions of the dependencies (to be listed in extra-deps</code>) based on a given stackage snapshot!</li> </ul> With this approach we get the same behavior that was already proven in cabal mode, but with the advantage that the generated stack.yaml</code> completely defines the project for any tool that knows stack. So after gradle extracted the dependencies and generated the stack.yaml</code>, it is no longer needed to succesfully compile/run/test the project, which means that tools like IDE integration will work much better than with the more hacky cabal mode of the plugin. Case Study - Haskell at Prezi 2015-09-21T00:00:00+00:00 I wrote a case study for FPComplete</a> on how we use Haskell at Prezi</a>. It is published here</a>, but I'm just posting it here as well: Prezi</a> is a cloud-based presentation and storytelling tool, based on a zoomable canvas. The company was founded in 2009, and today we have more than 50 million users, with more than 160 million prezis created. The company is using several different platforms and technologies; one of these is Haskell, which we are using server side, for code generation and for testing. PDOM</h2> Prezi's document format is continuously evolving as we add features to the application. It is very important for us that this format is handled correctly on all our supported platforms, and both on client and server side. To achieve this, we created an eDSL in Haskell that defines the schema of a Prezi. From this schema we are able to generate several artifacts. Most importantly we are generating a Prezi Document Object Model (PDOM) library for multiple platforms - Haxe (compiled to JS) code for the web, C++ code for the native platforms, and Haskell code for our tests, tools and the server side. These libraries are responsible for loading, updating, maintaining consistency and saving Prezis. This API also implements collaborative editing functionality by transparently synchronising document changes between multiple clients. This technique is called operational transformation (OT)</a>. We implemented the server side of this in Haskell; it supports clients from any of the supported platforms and it is connected to several other backend services. Benefits</h2> Using Haskell for this project turned out to have huge benefits. We are taking advantage of Haskell's capabilities to create embedded domain specific languages, using it to define the document's schema in our own eDSL which is used not only by Haskell developers but many others too. Haskell's clean and terse code allows us to describe document invariants and rules in a very readable way and the type system guarantees that we handle all the necessary cases, providing a stable base Haskell implementation which we can compare the other language backends to. It was also possible to define a set of merge laws for OT, which are verified whenever we introduce a new element to the document schema, guaranteeing that the collaboration functionality works correctly. We use the QuickCheck testing library on all levels. We can generate arbitrary Prezi documents and test serialization on all the backends. We are even generating arbitrary JavaScript code which uses our generated API to test random collaborative network sessions. These tests turned out to be critical for our success as they caught many interesting problems before we deployed anything to production Haskell plugin for Gradle 2015-04-22T00:00:00+00:00 My team at Prezi</a> uses Haskell for several projects, which usually depend on each other, often with build steps using other languages such as Scala, C++ or Haxe. As Gradle</a> is used heavily in the company, we decided to try to integrate our Haskell projects within Gradle. The result is Gradle Haskell Plugin</a>, which we were using succesfully in the last 2 months in our daily work, and we have open-sourced recently. What makes this solution interesting is that it not just simply wraps cabal within Gradle tasks, but implements a way to define dependencies between Haskell projects and to upload the binary Haskell artifacts to a repository such as artifactory</a>. This makes it easy to modularize our projects, publish them, and also works perfectly with pride</a>, an other open-source Prezi project. This means that we can work on a subset of our Haskell projects while the other dependencies are built on Jenkins, and it also integrates well with our non-Haskell projects. How does it work?</h2> The main idea is that we let cabal manage the Haskell packages, and handle whole Haskell sandboxes on Gradle level. So if you have a single Haskell project, it will be built using cabal and the result sandbox (the built project together with all the dependent cabal packages which are not installed in the global package database) will be packed/published as a Gradle artifact. This is not very interesting so far, but when you introduce dependencies on Gradle level, the plugin does something which (as far as I know) is not really done by anyone else, which I call sandbox chaining. This basically means that to compile the haskell project, the plugin will pass all the dependent sandboxes' package database to cabal and GHC, so for the actual sandbox only the packages which are not in any of the dependent sandboxes will be installed. Example</h2> Let's see an example scenario with 4 gradle-haskell projects. </a> The project called Haskell project depends on two other projects, which taking into accound the transitive dependencies means it depends on three other haskell projects. Each project has its own haskell source and cabal file. Building this suite consists of the following steps: dependency 1 is built using only the global package database, everything not in that database, together with the compiled project goes into its build/sandbox</code> directory, which is a combination of a GHC package database and the project's build output. This is packed as dependency 1's build artifact.</li> For dependency 2, Gradle first downloads the build artifact of dependency 1 and extracts it to build/deps/dependency1</code>.</li> Then it runs SandFix</a> on it</li> And compiles the second project, now passing both the global package database and dependency 1's sandbox to cabal/ghc. The result is that only the packages which are not in any of these two package databases will be installed in the project's own sandbox, which becomes the build artifact of dependency 2.</li> For dependency 3, Gradle extracts both the direct dependency and the transitive dependency's sandbox, to build/deps/dependency2</code> and build/deps/dependency3</code>.</li> Then it runs SandFix</a> on both the dependencies</li> And finally passes three package databases to cabal/ghc to compile the project. Only those cabal dependencies will be installed into this sandbox which are not in global, neither in any of the dependent sandboxes.</li> Finally, for Haskell project it goes the same way, but here we have three sandboxes, all chained together to make sure only the built sandbox only contains what is not in the dependent sandboxes yet.</li> </ul> For more information, check out the documentation</a>. bari 1.0 released 2014-12-08T00:00:00+00:00 I already wrote about bari</a> in May</a>. As a reminder, bari</a> is a build management system primarily for .NET, trying to fix Visual Studio's bad parts while keeping the good ones. After more than two years of development, and being in production at KOTEM</a> for almost half a year, bari reached a state when it can be considered as a stable and usable first version. To indicate this today I released bari 1.0. Try it out and feel free to give any kind of feedback or ask any questions! ScalaFXML 0.2.2 available 2014-10-22T00:00:00+00:00 I've released a new version of ScalaFXML</a>, which now supports both ScalaFX 8</a> with JavaFX 8 on Java 8, and ScalaFX 2.2</a> with JavaFX 2.x on Java 7. The two branches are separated by the sfx2</code> and sfx8</code> postfixes, and both are available for Scala 2.10.x</code> and 2.11.x</code>. To use it with sbt</a> on Java 7: addCompilerPlugin("org.scalamacros" % "paradise" % "2.0.1" cross CrossVersion.full) libraryDependencies += "org.scalafx" %% "scalafx" % "2.2.67-R10" libraryDependencies += "org.scalafx" %% "scalafxml-core-sfx2" % "0.2.2" </code></pre> And on Java 8: addCompilerPlugin("org.scalamacros" % "paradise" % "2.0.1" cross CrossVersion.full) libraryDependencies += "org.scalafx" %% "scalafx" % "8.0.20-R6" libraryDependencies += "org.scalafx" %% "scalafxml-core-sfx8" % "0.2.2" </code></pre> A python/thrift profiling story 2014-09-15T00:00:00+00:00 A few weeks ago I met a problem where a script, running once every night sending out some emails did not run correctly because a remote thrift call timed out in it. As I started investigating it, turned out that it's a search call: staff_users = RemoteUserFactory().search(is_staff=True) </code></pre> The details here are not really important, what this call does is that it asks a service to return a set of users, and the communication is going on thrift</a>. Executing it manually on the server revealed that it should return 5649 users. Checking out the logs I could see that the call took extremely long time, between 8 to 12 seconds. Even when the cron job was moved from 3:00 AM to a less busy time (several other jobs were executing at the same time), it took more than 6 seconds! This was suspicious so I also checked the log of a proxy which runs on the same host as the script itself and provides client side load balancing, circuit breaking, retry logic etc. for thrift connections. This log showed that the service replied in 2.5 seconds, but it took almost 4 seconds to get this response from the proxy to the client on localhost! This seemed to be completely unacceptable, and also the 2.5 second response time from the service seemed to be too big (I ran the query on one of the server nodes and it returned the users from the database almost instantly). I also had similar experience (but without measurements) before. So I decided to find out what's going on. And I found the process interesting enough to write this post about it :) Test environment</h2> I started by adding a test method to the service's thrift API called test_get_users(count, sleep)</code> which returns count</code> fake users after waiting sleep</code> seconds. Then in the following experiments I called it with (5499, 1)</code>. The 1 second sleep was intended to simulate the network latency and database query; there was no advantage from having it at the end, but as it is visible everywhere in the results, I had to mention. For finding out what's going on I used cProfile</a> with gprof2dot</a>, calling the remote test method from a django shell, while everything is running on localhost. First measurement</h3> Without touching anything, returning 5499 dummy users on localhost took 5.272 seconds! The client side of the call looked like this: </a> Here we can see that the call has two major phases: The thrift call itself (65%)</li> Converting the raw results to model objects with _row_to_model</code> (35%)</li> </ul> Let's see first the thrift call (the green branch on the picture). Once again it has two, nearly equivalent branches: send_test_get_users</code> which sends the request and waits for the response. This includes the 1 second sleep as well.</li> recv_test_get_users</code> processes the response</li> </ul> What's interesting here is that recv_test_get_users</code> took ~32% of the overall time which is around ~1.6 seconds for simple data deserialization. Optimizing thrift deserialization</h3> I did not want to believe that the python thrift deserialization is that slow, so I did a search and found that the TBinaryProtocol</code> which we are using is really that slow. But the thrift library contains a class called TBinaryProtocolAccelerated</code> which is about 10x faster (according to a stackoverflow post). First I simply changed the used protocol to this, but nothing happened. Digging deeper I found that this is not a real protocol implementation, but a lower level hack. The documentation of the protocol class says: C-Accelerated version of TBinaryProtocol. This class does not override any of TBinaryProtocol's methods, but the generated code recognizes it directly and will call into our C module to do the encoding, bypassing this object entirely. We inherit from TBinaryProtocol so that the normal TBinaryProtocol encoding can happen if the fastbinary module doesn't work for some reason. (TODO(dreiss): Make this happen sanely in more cases.) In order to take advantage of the C module, just use TBinaryProtocolAccelerated instead of TBinaryProtocol. </code></pre> So why didn't it work? The answer is in TBase.py</a>. The following conditions have to met in order to use the fast deserializer: Protocol must be TBinaryProtocolAccelerated</code> (I changed that)</li> Protocol's transport implementation must implement the TTransport.CReadableTransport</code> interface</li> thrift_spec</code> must be available (this was true in this case)</li> fastbinary</code> must be available (also true)</li> </ul> The problem was that we were replacing the TTransport</code> implementation with a custom class called ThriftifyTransport</code> in order to do thrift logging, HMAC authentication, etc. Fortunately all the default transport implementations implement the CReadableTransport</code> interface, and one of them, TBufferedTransport</code> can be used to wrap another transport to add buffering around it. That's what I did, and it immediately started using the fast deserialization code. The test call now ran in 3.624 seconds. And the new profiling results with this change: </a> The left-hand side of the call graph remained the same, but recv_test_get_users</code> is now only 2.35% of the overall time which is ~0.08 seconds (to be compared with the 1.6 seconds with the original deserializer!) Optimizing thrift serialization</h3> The obvious next step was to apply this change on the server side as well, so our service can use the fast binary protocol for serialization too. For this I simply copied the change and remeasured everything. The test call now ran in 3.328 seconds! Let's see the call graph of this stage: </a> Optimizing result processing</h3> The client side of the test method was written similar to how the original API method is written: def test_get_users_thrift(self, count, sleep): rpc = ThriftRPC(UserDataService, self.name, service_name=self.service_name, client_config=client_config) result = [] for row in rpc.test_get_users(count, sleep).iteritems(): user = self._row_to_model(self.user_factory, row) result.append(user) return result </code></pre> It is clearly visible on the call graph that the 5499 call to _row_to_model</code> takes 53% of the total time, which is ~1.7 seconds. There are two main branches of this call. The left hand side (row_to_model</code>) seemed to be simple data conversion, and its slowest part is date-time deserialization. The other branch however looked like a real problem; why should we resolve HMAC host, or parse configuration for each row? It turned out to be a bug, _row_to_model</code> created a new model factory in each call, which involves a lot of initialization, config parsing, and similar things. So the simple fix was to create a _rows_to_model</code> helper function which does the same for multiple rows with a single factory. Running my test code once again showed that the optimization makes sense. Now it ran in 2.448 seconds, with the following call graph: </a> Further optimizations</h3> I saw two possible ways to further optimize this case: Lazy conversion of raw thrift data to model data (per field). This would make sense because many times only a few fields (the id for example) are used, but it seemed to be a too complex change </li> Checking the server side as well </li> </ol> To profile the server side and only measure the thrift request processing I had to add profiling code to the django view class in the following way: import cProfile cProfile.runctx('self._call_processor(op_data)', globals(), locals(), 'callstats') # self._call_processor(op_data) </code></pre> The server-side call took 1.691 seconds and looked like this: </a> As expected, 60% of this was the 1 second sleep. The rest of the calls are data conversion with no obvious point to improve. Summary</h2> These optimizations are decreasing the response time significantly, especially for calls returning multiple rows. The interesting was that the extremely slow performance was caused by both the slow perfomance of the python thrift serializer and a bug in our code. Conditional blocks in Distributed Documentor 2014-07-13T00:00:00+00:00 I've added a new feature to Distributed Documentor</a> today, conditional blocks. The idea is that parts of the documents can be enabled when a given condition is present. This is very similar to C's ifdef blocks</a>. To use it with the MediaWiki syntax, put [When:X]</code> and [End]</code> commands in separate lines: Unconditional [When:FIRST] First conditional [When:SECOND] First and second conditional [End] [End] [When:SECOND] Second conditional [End] </code></pre> Snippets can also have conditional blocks. There are two possibilities to set which conditionals are enabled: Specifying it with command line arguments, such as java -jar DistributedDocumentor.jar -D FIRST -D SECOND </code></pre> This is useful when exporting a documentation from command line, or to launch the documentation editor with a predefined set of enabled conditions. </li> On the user interface, using View menu's Enabled conditions... menu item: </li> </ol> Introducing bari 2014-05-16T00:00:00+00:00 In the past two years I worked on a project called bari</a> which now reached an usable state. bari is a build management system, trying to fix Visual Studio's bad parts while keeping the good ones. Basically it tries to make .NET development more convenient, when The application may consist of a large number of projects</li> There may be several different subsets of these projects defining valuable target products</li> Custom build steps may be required</li> It is important to be able to reproduce the build environment as easily as possible</li> The developers want to use the full power of their IDE</li> </ul> The main idea is to generate Visual Studio solutions and projects on the fly as needed, from a concise declarative build description. I tried to optimize this build description for human readability. Let's see an example, a short section from bari's own build definition: - name: bari type: executable references: - gac://System - nuget://log4net - nuget://Ninject/3.0.1.10 - nuget://QuickGraph - module://Bari.Core csharp: root-namespace: Bari.Console </code></pre> The main advantage of generating solutions and projects on the fly is that each developer can work on the subset he needs for his current task keeping the IDE fast, but can also open everything in one solution if it is useful for performing a refactoring. To keep build definitions short and readable, bari prefers convention over configuration. For example the directory stucture in which the source code lays defines not only the name of the modules to build, but also the way it is built. For example, in a simple hello world example the C# source code would be put in the src/TestModule/HelloWorld/cs</code> directory, and bari would build target/TestModule/HelloWorld.exe</code>. bari unifies the handling of project references in a way that referencing projects within a suite, from the GAC, using Nuget</a> or from a custom repository works exactly the same. It is also possible to write custom builders in Python. For more information check out the getting started page</a>. ScalaFX with FXML 2014-01-12T00:00:00+00:00 ScalaFX</a> is a nice wrapper around JavaFX for Scala, but currently it lacks support for using FXML</a> instead of Scala code for defining the user interfaces. This can be understood as ScalaFX is in fact a DSL for defining the UI in Scala instead of an XML file. Still I believe that using FXML instead may have its advantages; first of all it has a visual designer (JavaFX Scene Builder</a>). For me, designing an UI without immediate visual feedback is hard, and involves a lot of iterations of tweaking the code, running it and checking the results. I also expect that in the future there will be more tools available which work on FXML data. It is not impossible to use FXML user interfaces from Scala, but the ScalaFX wrappers does not help and the code for the controller classes is not clean enough. See the following example</a> to get a feeling how it looks like. To make it better I wrote a small library called ScalaFXML</a>. In this post I'll go through a small example to explain how it works. The following image shows how our sample application will look like: The From fiels is editable, and the result in the To field is filled as you type using data binding. The Close button's only purpose is to demonstrate event handlers. The conversion logic itself is implemented by small classes</a> sharing the same trait: trait UnitConverter { val description: String def run(input: String): String override def toString = description } object MMtoInches extends UnitConverter { val description: String = "Millimeters to inches" def run(input: String): String = try { (input.toDouble / 25.4).toString } catch { case ex: Throwable => ex.toString } } object InchesToMM extends UnitConverter { val description: String = "Inches to millimeters" def run(input: String): String = try { (input.toDouble * 25.4).toString } catch { case ex: Throwable => ex.toString } } </code></pre> To describe the set of available unit converters, we define one more helper class: class UnitConverters(converters: UnitConverter*) { val available = List(converters : _*) } </code></pre> Now let's start with a pure ScalaFX solution</a>, where the user interface is defined in Scala. I've implemented the view itself in a class called PureScalaFXView</code>, which gets the set of available unit converters as a dependency through its constructor. This makes the main application object very simple: object PureScalaFX extends JFXApp { stage = new PureScalaFXView( new UnitConverters(InchesToMM, MMtoInches)) } </code></pre> The PureScalaFXView</code> class consists of two distinct parts. First we define the user interface using the ScalaFX UI DSL: class PureScalaFXView(converters: UnitConverters) extends JFXApp.PrimaryStage { // UI Definition title = "Unit conversion" private val types = new ComboBox[UnitConverter]() { maxWidth = Double.MaxValue margin = Insets(3) } private val from = new TextField { margin = Insets(3) prefWidth = 200.0 } private val to = new TextField { prefWidth = 200.0 margin = Insets(3) editable = false } scene = new Scene { content = new GridPane { padding = Insets(5) add(new Label("Conversion type:"), 0, 0) add(new Label("From:"), 0, 1) add(new Label("To:"), 0, 2) add(types, 1, 0) add(from, 1, 1) add(to, 1, 2) add(new Button("Close") { // inline event handler binding onAction = (e: ActionEvent) => Platform.exit() }, 1, 3) columnConstraints = List( new ColumnConstraints { halignment = HPos.LEFT hgrow = Priority.SOMETIMES margin = Insets(5) }, new ColumnConstraints { halignment = HPos.RIGHT hgrow = Priority.ALWAYS margin = Insets(5) } ) } } </code></pre> This is not 100% pure UI definition, because it also contains an inline event handler definition for the Close button. The next part fills the combo box and defines the data binding. Filling the combo box is a simple procedural loop: for (converter <- converters.available) { types += converter } types.getSelectionModel.selectFirst() </code></pre> For the data binding we define a low level data binding</a> which depends on the combo box's selected value and the From field's text, and produces the output for the To field: to.text <== new StringBinding { bind(from.text.delegate, types.getSelectionModel.selectedItemProperty) def computeValue() = types.getSelectionModel.getSelectedItem.run(from.text.value) } </code></pre> That's all, the application is fully functional. The next thing is to split this class so the UI definition and the UI logic got separated. This refactored ScalaFX solution</a> is very similar to the previous one, but the initialization of the combo box, the data binding and the event handler are all encapsulated by a new, separate class: class RawUnitConverterPresenter( private val from: TextField, private val to: TextField, private val types: ComboBox[UnitConverter], private val converters: UnitConverters) { // Filling the combo box for (converter <- converters.available) { types += converter } types.getSelectionModel.selectFirst() // Data binding to.text <== new StringBinding { bind(from.text.delegate, types.getSelectionModel.selectedItemProperty) def computeValue() = types.getSelectionModel.getSelectedItem.run(from.text.value) } // Close button event handler def onClose(event: ActionEvent) { Platform.exit() } } </code></pre> What I wanted is to be able to define the controller class exactly like this while building the user interface from FXML. Without ScalaFXML</a> the controller class have some serious limitations: It must implement the Initializable</a> interface</li> It cannot have any constructor arguments</li> The user interface objects must be variable fields of the class</li> And they have to have the type of the JavaFX controls, so to be able to use the ScalaFX wrappers, they have to be explicitly wrapped in the initialize</code> method.</li> </ul> With ScalaFXML</a> the process is really simple. First we create the FXML, for example with the JavaFX Scene Builder</a>: In the FXML we give the from</code>, to</code>, and types</code> identifiers to our controls using the fx:id</code> attribute, for example: <TextField fx:id="from" prefWidth="200.0" GridPane.columnIndex="1" GridPane.margin="$x1" GridPane.rowIndex="1" /> </code></pre> The event handlers can be specified simply by their name: <Button onAction="#onClose" text="Close" mnemonicParsing="false" GridPane.columnIndex="1" GridPane.halignment="RIGHT" GridPane.rowIndex="3" /> </code></pre> and the controller class must be referenced on the root node fx:controller="scalafxml.demo.unitconverter.UnitConverterPresenter" </code></pre> The controller class can be exactly the same as the RawUnitConverterPresenter</code></a>, adding an additional @sfxml</code> annotation for it. Everything else is handled by the library, as we will see. The application object itself looks like this: object ScalaFXML extends JFXApp { val root = FXMLView(getClass.getResource("unitconverter.fxml"), new DependenciesByType(Map( typeOf[UnitConverters] -> new UnitConverters(InchesToMM, MMtoInches)))) stage = new JFXApp.PrimaryStage() { title = "Unit conversion" scene = new Scene(root) } } </code></pre> Beside giving the URI for the FXML file we also has to provide the additional dependencies of the controller class. This is an easily extensible part of the library, and it already has support for SubCut</a> and Guice</a> as well. Here we are using a simple type->value mapping instead. How does this work? What happens behind the scenes? The @sfxml</code> is a macro annotation</a>. In compile-time, the class definition itself is transformed by the sfxmlMacro.impl</code> function</a>. The transformation's result is a class definition with the source class' name, but with a completely different content. The original class is added as an inner class, always called Controller</code>. In our example, the generated class definition would look like something similar: class UnitConverterPresenter(private val dependencyResolver: ControllerDependencyResolver) extends javafx.fxml.Initializable with FxmlProxyGenerator.ProxyDependencyInjection { class Controller( private val from: TextField, private val to: TextField, private val types: ComboBox[UnitConverter], private val converters: UnitConverters) { // … } private var impl: Controller = null // … } </code></pre> The class have four distinct parts: Getting the additional dependencies from the dependency resolver</li> Variable fields for binding the JavaFX controls defined in the FXML</li> Event handler methods</li> The initializable</code> method's implementation</li> </ol> The first one is simple - for each constructor argument of the controller class which is not a ScalaFX control, we query the dependency resolver to get a value for it. These are performed when the outer, generated class is instantiated and stored through the ProxyDependencyInjection</code> trait. The variable fields are simple fields for all the ScalaFX constructor arguments of the controller class, but converted to their JavaFX counterpart. For example the generated field for the controller's from</code> argument will look like this: @javafx.fxml.FXML private var from: javafx.scene.control.TextField = null </code></pre> The event handler's are proxies for all the public methods of the controller, but the ScalaFX event argument types are replaced with JavaFX event argument types and they are wrapped automatically when forwarding the call to the real implementation. For the onClose</code> event handler it would look like the following: @javafx.fxml.FXML def onClose(e: javafx.event.ActionEvent) { impl.onClose(new scalafx.event.ActionEvent(e)) } </code></pre> When JavaFX calls the generated controller's initialize</code> method, the control fields are already set up, and the additional dependencies were already gathered from the dependency resolver so we have all the values required to instantiate the real controller class. For ScalaFX arguments we wrap the JavaFX controls, for the additional dependencies we use the ProxyDependencyInjection</code> trait's getDependency</code> method: def initialize(url: java.net.URL, rb: java.util.ResourceBundle) { impl = new Controller( new scalafx.scene.control.TextField(from), new scalafx.scene.control.TextField(to), new scalafx.scene.control.ComboBox[UnitConverter](types), getDependencies[UnitConverters]("converters")) } </code></pre> That's all. The final interesting bit is the FXMLView</code> object, which overrides JavaFX's default controller factory. This is only necessary to be able to pass the given ControllerDependencyResolver</code> to the generated controller's constructor: def apply(fxml: URL, dependencies: ControllerDependencyResolver): jfxs.Parent = jfxf.FXMLLoader.load( fxml, null, new jfxf.JavaFXBuilderFactory(), new jfxu.Callback[Class[_], Object] { override def call(cls: Class[_]): Object = FxmlProxyGenerator(cls, dependencies) }) </code></pre> FxmlProxyGenerator</code> uses reflection to create a new instance of the generated controller, and pass the dependency resolver as its only constructor argument. Trying out Ceylon - Part 1 2013-11-17T00:00:00+00:00 Ceylon's first production release was announced on 12th of November. I decided to try it out after going through the quick introduction, as it looked quite promising. In a series of posts I'd like to share my first attempts to use this interesting language. This first release came with an eclipse plugin as well - after installing it I was immediately able to start working on my test project. In this few hours the plugin seemed to be stable enough, I did not experience any problems. I have a JVLT</code> file which I created while attending a foreign language course about a year ago. I was using only a limited subset of this application, so basically what I have is a .jvlt file, which is in fact a ZIP archive, in which a dict.xml</code> stores a set of words and for each word one or more translation and the lesson we have learnt it. See the following example: <dictionary language="french" version="1.4"> <entry id="e275"> <orth>à côté de</orth> <sense id="e275-s1"> <trans>mellett</trans> </sense> <sense id="e275-s2"> <trans>mellé</trans> </sense> <lesson>8</lesson> </entry> </dictionary> </code></pre> My idea was to write an application that helps me learning and practicing these words. In this first post I'm going to load the dictionary from the JVLT file. To get started, I created a new Ceylon module with the help of the IDE called jvlt. This immediately created three program units: module.ceylon</code>, package.ceylon</code> and run.ceylon</code>. The module.ceylon</code> contains the module definition, which also describes the module's dependencies. As I was trying to implement the dictionary reader, I ended up with the following module definition: module jvlt "1.0.0" { shared import ceylon.file "1.0.0"; import ceylon.collection "1.0.0"; import ceylon.interop.java "1.0.0"; import javax.xml "7"; import ceylon.test "1.0.0"; } </code></pre> Let's start with the data model we want to build up! The dictionary consists of words: "Represents a foreign word with one or more senses" shared class Word(shared String word, shared Set<string> senses, shared Integer lesson){ } </code></pre> The word, senses and lessons are all shared attributes of this class, accessible from the outside. To make it easy to access the word objects by their foreign word, I'm currently storing them in a map: "Represents a dictionary of words in a given language" shared class Dictionary(shared String language, shared Map<string word=""> words) { } </code></pre> Basically that's the data model, but I wrapped the whole thing in an abstract JVLT class which looks like this: "Represents a JVLT file" abstract shared class JVLT() { "The dictionary stored in this JVLT" formal shared Dictionary dictionary; } </code></pre> The idea is that you get a JVLT instance from one of the helper functions and then use it as a root of the data model. The next thing is to create this data model from the JVLT files. For this, I needed two things: Reading a ZIP archive</li> Parsing XML</li> </ul> It turned out that Ceylon's file module has ZIP support, with the createZipFileSystem</code> function as an entry point. I made two module-level functions beside the JVLT class for creating instances deriving from the abstract JVLT class: loadJVLT</code> which loads a JVLT ZIP archive from the file system</li> loadJVLTFromDictionaryString</code> oads directly a dict.xml-like XML passed as a simple string. I'm using this for unit testing the XML parser.</li> </ul> Let's see the ZIP handling first: "Loads a JVLT file from a `.jvlt` ZIP archive, if possible." shared JVLT? loadJVLT(File file) { value zip = createZipFileSystem(file); value dictPath = zip.parsePath("/dict.xml"); if (is File dictFile = dictPath.resource) { try (reader = dictFile.Reader()) { return loadJVLTFromDictionaryString(readAll(reader)); } } else { return null; } } </code></pre> Well, the error handling is not too sophisticated in this case, it either returns a JVLT or returns null</code> if the given file did not have a dict.xml</code> in it. Other error conditions such as a dict.xml</code> with a wrong format, etc., are not handled currently. As you can see, I'm reusing my other load function here, once the dict.xml</code> is read. There are two interesting things here. First, the if statement where we check if the resource is an instance of File</code> and immediately store it in the value called dictFile</code>. The dictPath.resource</code> attribute has the type Resource</code> which is a Ceylon interface. It is either an ExistingResource</code>: Directory</code>, File</code> or Link</code>, or Nil</code>. In any case if it is not a File</code> instance, we just return null</code>. For simplicity, I'm reading the full dict.xml</code> into a string before parsing it. For this purpose I wrote a small helper function readAll</code>: "Reads all lines from a file reader and returns the concatenated string" String readAll(File.Reader reader) { variable String result = ""; while (exists line = reader.readLine()) { result += line; } return result; } </code></pre> Probably it's not an optimal solution, but works :) Now that we have our data model and have a way to build it up from XML, we can write some unit tests to see how it works. The Ceylon SDK has a test module and the Ceylon IDE supports running the tests. There is a separate page in the documentation</a> describing how. It is really simple, I had to add the test module as a dependency, and I created a separate file to hold my test definitions. The class groups the tests together and optionally supports running extra code before/after each test case, as in other test frameworks: class DictionaryParserTests() { shared test void emptyDictionary() { value dic = loadJVLTFromDictionaryString("<dictionary>"); assert (dic.dictionary.words.empty); assert (dic.dictionary.language == "unknown"); } shared test void languageAttributeRead() { value dic = loadJVLTFromDictionaryString("<dictionary language="testlang">"); assert (dic.dictionary.language == "testlang"); } // ... </code></pre> I won't paste here all the test code, only a few samples to get the feeling how the Ceylon code looks like. To test whether a given word's translations are loaded correctly, I wrote a helper function: void assertSenses(JVLT jvlt, String w, [String+] expectedSenses) { Word? word = jvlt.dictionary.words[w]; if (exists word) { assert (word.senses.equals(HashSet(expectedSenses))); } else { fail("Word does not exists"); } } </code></pre> This helper function can be used to assert that a word has been loaded correctly: shared test void wordWithMultipleSenses() { value dic = loadJVLTFromDictionaryString( "<dictionary> <entry id="e1"> <orth>src1</orth> <sense id="e1-s1"> <trans>dst1</trans> </sense> <sense id="e1-s2"> <trans>dst2</trans> </sense> </entry> </dictionary>"); assertSenses(dic, "src1", ["dst1", "dst2"]); } </code></pre> Now the only problem is that there is no XML parsing support in the Ceylon SDK currently, so it has to be done using Java interop. As I wrote the code to build up the data model from the XML, I wrote several helper functions to make it easier to fit into the language. So let's see first how the dictionary loading is defined, and then I'll show the helper functions. The XML parsing is done by two module level functions which are not shared - only used by the JVLT constructor functions I shown before. The first one creates a map entry for a single word: "Creates a word entry for the dictionary" String->Word loadEntry(Element elem) { value w = Word { word = selectNodeText(elem, "orth") else "???"; lesson = selectNodeInteger(elem, "lesson") else 0; senses = HashSet(selectNodes(elem, "sense/trans") .map((Node n) => n.textContent)); }; return w.word->w; } </code></pre> and the second one loads all the words from the XML document: "Loads a dictionary from JVLT's `dict.xml` format." Dictionary loadDictionaryFromXML(Document doc) { doc.documentElement.normalize(); return Dictionary { language = getAttribute(doc.documentElement, "language") else "unknown"; words = HashMap({ for (node in selectNodes(doc, "dictionary/entry")) if (is Element elem = node) loadEntry(elem) }); }; } </code></pre> The function which returns the JVLT instance uses this function and Java interop to read the dictionary: "Loads a JVLT file by the parsing the dictionary XML directly from a string" shared JVLT loadJVLTFromDictionaryString(String dictXML) { value docBuilderFactory = DocumentBuilderFactory.newInstance(); value builder = docBuilderFactory.newDocumentBuilder(); value doc = builder.parse(ByteArrayInputStream(javaString(dictXML).bytes)); object result extends JVLT() { dictionary = loadDictionaryFromXML(doc); } return result; } </code></pre> There are two things to notice here: we had to convert from Ceylon's string to Java string. This is not done automatically and we need the ceylon.interop.java</code> module to do it. In the last lines we define an anonymous class extending from JVLT and overwriting it's abstract dictionary attribute. Then this anonymous class instance is returned as the loaded JVLT. To make the XML parsing less painful, I defined a few helper functions in a separate compilation unit (XmlHelper.ceylon</code>). I won't show here the full file but there are some interesting parts. First, from Ceylon you cannot call static methods, but you can import them. I'm using the following two import statements: import org.w3c.dom { Node, NodeList, Element } import javax.xml.xpath { XPathFactory { newXPathFactory = newInstance }, XPathConstants { nodeSet = \iNODESET }} </code></pre> The first one is straightforward. It imports three DOM interfaces. The second one first imports the XPathFactory.newInstance</code> static method and also renames it, as newInstance is a too generic name without its class name as a prefix. The third line imports a constant value and gives it a Ceylon-compatible name. Because in Ceylon only the types can start with an uppercase character, we have to use a special and ugly syntax which helps the interoperability - prefixing it with \i</code>. The ceylon.interop.java</code> module has helper classes to make Java Iterable objects iterable in Ceylon, but unfortunately the NodeList</code> interface is not iterable in Java either. So I wrote a simple wrapper that iterates through a node list: class NodeListIterator(NodeList nodes) satisfies Iterable<Node> { shared actual default Iterator<Node> iterator() { object it satisfies Iterator<Node> { variable Integer i = 0; shared actual Node|Finished next() { if (i < nodes.length) { return nodes.item(i++); } else { return finished; } } } return it; } } </code></pre> Using this iterator and the imports I wrote a selectNodes</code> function to run XPath expressions and return the result as a Ceylon iterable: {Node*} selectNodes(Node root, String xpath) { value factory = newXPathFactory(); value xpathCompiler = factory.newXPath(); value expr = xpathCompiler.compile(xpath); value nodeList = expr.evaluate(root, nodeSet); if (is NodeList nodeList) { return NodeListIterator(nodeList); } else { return []; } } </code></pre> Using this function it is very easy to write a variant that selects a single node: Node? selectNode(Node root, String xpath) { return selectNodes(root, xpath).first; } </code></pre> There are some other helper functions returning the node's text, converting it to integer, etc. but I think they are not that interesting. Now that I have my data model which is built from my JVLT file, the next thing is to make a user interface somehow where the vocabulary can be shown an the user's knowledge can be tested/improved. This will be the topic of some future posts, as soon as I have time to experiment more with this new language. Cloning WPF flow document fragments 2013-10-25T00:00:00+00:00 Today I had to write such an ugly hack to fix a bug that I decided to start writing a blog where I can show it to the world :) The software I'm working on has some sort of context sensitive help panel, which is implemented using dynamically generated flow documents</a>. The software loads a large set of flow document sections from a XAML file runtime, and later builds documents from a subset of them. For some reason (which belong to a separate post), it is not possible to reuse these flow document elements in multiple flow documents, not even if there is only one at a time. To work around this, I was cloning these sections before adding them to the document. As WPF elements are not cloneable, I was using the method recommended many places, for example in this StackOverflow post</a>: saving the object tree to an in-memory XAML stream, and loading it back. This worked quite well.. until we discovered a bug, which I still cannot explain. In some cases which were easily reproducible for any developer, but the code running in those cases being exactly the same as in other, working cases, the clone method simply stopped working. Stopped working here means that the following code: var xaml = XamlWriter.Save(block); </code></pre> would write out the correct object hierarchy, but without any properties (no attributes, no content properties, nothing but the element names)! In the same time the objects in the memory were untouched and still had all the relevant properties set. I also tried to write my own XAML serializer based on the code found at this site</a>, but this was only good to find out that the problem lies deep within the MarkupWriter</code> class, which is the same what the XamlWriter</code> uses internally. When the XamlWriter</code> failed, my own code could not find any properties using the returned MarkupObject</a>: MarkupObject markupObj = MarkupWriter.GetMarkupObjectFor(obj); </code></pre> For the same object, in the working scenarios it returned a markup object with a working Properties</code> collection. So here is the final "solution" which I'm not really proud of, but solved the problem. Maybe with some modifications it is useful for someone struggling with the framework: /// <summary> /// Horrible ugly clone hack to issues where XamlWriter/XamlReader based /// clone method did not work. /// </summary> public static class CloneHelper { public static Block Clone<t>(this T block) where T : Block { var result = (T)DeepClone(block); return result; } private static object DeepClone(object obj) { if (obj != null) { // Replacing ResourceDictionary and Style values with null. // In this particular use case it is correct to do if (obj.GetType() == typeof(ResourceDictionary) || obj.GetType() == typeof(Style)) { return null; } else { // Value types and some special cases where we don't want to clone if (obj.GetType().IsValueType || obj.GetType() == typeof (Cursor) || obj.GetType() == typeof (XmlLanguage)) { return obj; } else { // If it is cloneable, use it var cloneable = obj as ICloneable; if (cloneable != null) { return cloneable.Clone(); } else { // Creating the clone with reflection var typ = obj.GetType(); var clone = Activator.CreateInstance(typ); // Property names which are known locally set // dependency properties var usedNames = new HashSet<string>(); // Copying locally set dependency properties from the // source to the target var dobjSource = obj as DependencyObject; var dobjTarget = clone as DependencyObject; if (dobjSource != null && dobjTarget != null) { var locallySetProperties = dobjSource.GetLocalValueEnumerator(); while (locallySetProperties.MoveNext()) { DependencyProperty dp = locallySetProperties.Current.Property; if (!dp.ReadOnly) { dobjTarget.SetValue(dp, dobjSource.GetValue(dp)); usedNames.Add(dp.Name); } } } // Getting all the public, non-static properties of the source foreach (var pi in typ.GetProperties( BindingFlags.Instance | BindingFlags.Public | BindingFlags.FlattenHierarchy)) { // If it is not a dependency property // and not the default property... if (pi.CanRead && !usedNames.Contains(pi.Name) && !IsDependencyProperty(dobjSource, pi) && pi.Name != "Item") { var val = pi.GetValue(obj, null); // ..and it is writeable, then we recursively clone // the value and set the property: if (pi.CanWrite) { pi.SetValue(clone, DeepClone(val), null); } else { // ..otherwise if it is a readonly list property, // go through each item, clone it and add to // the clone's list property if (pi.PropertyType .GetInterfaces() .Contains(typeof (IList))) { var source = val as IList; var target = pi.GetValue(clone, null) as IList; if (source != null && target != null) { foreach (var item in source) target.Add(DeepClone(item)); } } } } } return clone; } } } } else { return null; } } /// <summary> /// Tries to determine if a property is a dependency property, by reflection and /// naming convention /// </summary> /// <param name="dobj">Dependency object /// <param name="pi">Property info /// <returns>Returns <c>true</c> if the given property seems to be a /// CLR access property for a dependency property.</returns> private static bool IsDependencyProperty(DependencyObject dobj, PropertyInfo pi) { if (dobj != null) { var dpProp = dobj.GetType().GetProperty(pi.Name + "Property", BindingFlags.Static | BindingFlags.Public | BindingFlags.FlattenHierarchy); if (dpProp != null && dpProp.PropertyType == typeof (DependencyProperty)) return true; else { var dpField = dobj.GetType().GetField(pi.Name + "Property", BindingFlags.Static | BindingFlags.Public | BindingFlags.FlattenHierarchy); if (dpField != null && dpField.FieldType == typeof (DependencyProperty) && dpField.IsInitOnly && dpField.IsStatic) return true; } } return false; } } </code></pre>

vigoo's software development blog

Agent patterns in Golem

Rust agents in Golem 1.4

Starting the project</h3> The only prerequisites to implement this example are:</p>

Library analysis</h3> We already seen how our library analysis agent</strong> will look like:</p>

Frontend</h2>

CORS</h3> We need to add CORS Preflight endpoints to our route to make the scripts work. In the current version of Golem this is a bit inconvenient, as we need to add them one by one for each endpoint we defined, for example:</p>

Golem 1.3's code-first TypeScript agents

Golem 1.3's new JavaScript engine

[Video] Missing Testing Features in Rust @ LambdaConf 2025

LambdaConf 2024-2025 - one year of Golem

[Video] Golem powered by WebAssembly @ Wasm I/O 2025

Durable Execution is not just for failures

Trying it out</h3> To demonstrate this, we can just invoke workers randomly from the 10000 we've created:</p> </p> Thanks to the durable execution model, every one of the 10000 workers react just as if it was running.</p>

Using MoonBit with Golem Cloud

List model</h3> Let's start by modelling our list</strong>. The edited "document" itself is just an array of strings:</p>

Building the components</h3> With this set, the whole application (with its two already written components) can be compiled by simply saying</p>

Worker to Worker communication</h2>

[Video] Golem and the WASM Component Model @ LambdaConf 2024

Zig and the WASM Component Model

Golem's Rust transaction API

Worker to Worker communication in Golem

desert part 1 - features

[Video] Beyond OpenAPI @ Functional Scala 2023

Type class derivation with ZIO Schema

Introduction</h2> Making the compiler to automatically derive</em> implementations of a type class for your custom algebraic data types is a common technique in programming languages. Haskell, for example, has built-in syntax for it:</p>

Alternatives</h2>

Generating a Rust client library for ZIO Http endpoints

Implementing an endpoint</h3> When developing a server</em>, the most important thing to do with an endpoint is to implement</strong> it. Implementing an endpoint looks like the following:</p>

Trying it out</h3>
To demonstrate this, we can just invoke workers randomly from the 10000 we've created:</p>
</p>
Thanks to the durable execution model, every one of the 10000 workers react just as if it was running.</p>