Duesee's Blog

(Avoid) implementing STARTTLS

Sun, 24 Nov 2024 13:37:57 +0100

(Avoid) Implementing STARTTLS

STARTTLS seems simple. It consists of a single message to switch to encryption. But as you zoom in, you start to see increasingly intricate issues that keep unfolding. It’s best to avoid it.

Sierpiński carpet. Infinite perimeter and zero area. Start with a square, split the square into 9 equal squares, remove the central square, and continue recursively.

Yet… someone needs it.

You start bargaining: You know that STARTTLS is a real-world attack target. And you know that it’s an easy target. You really want to avoid it. You try to make STARTTLS unattractive, convince your colleagues to repeat their talk about STARTTLS at FOSDEM’ 24, submit a follow-up talk to FOSDEM ‘25, too, and keep repeating that we should avoid it

By the way, the Call for Participation in the “Modern Email” FOSDEM ‘25 DevRoom is still open!

And yet… someone needs it.

As an open-source maintainer in the email space you will (sooner than later) find yourself in a position where someone is desperately asking for help because their email provider doesn’t support implicit TLS but only STARTTLS. If these servers are operated by smaller organizations, universities, etc., you should ask them to update their infrastructure. But, sooner than later, you may end up blocked by some unreasonable entity¹. So…

How to (avoid) implementing STARTTLS?

… and still making everyone happy?

STARTTLS consists of two phases: a plaintext and an encrypted phase. In the plaintext phase, there is always the possibility that data was meddled with by a network attacker.

Consider this trace:

S: * OK Hello, World!       // Who is greeting me here, actually?
S: * 100000000000000 exists // Are you serious?
C: A STARTTLS
S: A NO                     // Uhm... okay?
<----- TLS handshake ----->
C: B fetch 100000000000000 ...

Nothing before the TLS handshake can be trusted. And I mean nothing. Not the “OK”, not the “Hello, World!”, certainly not the “100000000000000 exists”, and also not the “NO” (which is a typical STARTTLS stripping attack). We must throw away every byte before the TLS handshake.

This raises the question, “Why should we expose our application to a potential adversary already?” and brings us to my suggestion: Let’s avoid (most of) STARTTLS. Let’s not parse anything before the TLS handshake and try to transition to TLS no matter what. Let’s do what openssl s_client -starttls imap does.

Minimal STARTTLS implementation (in Rust)

Our STARTTLS client implementation could be …

fn main() {
    // Start with plaintext.
    let stream = {
        let stream = TcpStream::connect("127.0.0.1:1143");
        let stream = do_starttls_prefix(stream);

        // We now have a TcpStream that expects a TLS handshake *exactly* as on port 993.
        stream
    };

    // Proceed with TLS.
    let stream = TlsStream::connect(stream);

    // Nit: The server won't send another `Greeting` after STARTTLS.
    let mut client = Client::new_without_greeting();

    // Continue as you would with implicit TLS (993).
    //...
}

/// Bring a STARTTLS connection to the point where TLS is expected.
async fn do_starttls_prefix(stream: TcpStream) -> TcpStream {
    let reader = BufReader::new(stream);
    let mut lines = reader.lines();

    // Receive greeting.
    // Note: Greeting is *always* a single line.
    let _ = lines.next_line().await.unwrap();

    // Send STARTTLS command.
    lines.get_mut().write(b"A STARTTLS\r\n").await.unwrap();

    // Receive (and discard) all lines up until we get a
    // command completion result for our "A STARTTLS" command.
    loop {
        let line = lines.next_line().await.unwrap().unwrap();
        if line.starts_with("A ") {
            break;
        }
    }

    lines.into_inner().into_inner()
}

While this example is specific to IMAP, you can do the same for SMTP, POP3, LDAP, etc. Think of STARTTLS as an “insecure prefix” that you barely need to overcome to make your server accept your beautiful TLS bytes. Treat STARTTLS as a guest. Greet it warmly and offer it what it needs, but keep it from taking over your space so that your routines stay intact.

FAQ

Q: Is the greeting indeed a single line?

Yes. None of the greeting ABNF rules …

Tree of all ABNF rules used by greeting.

… allow a newline. Further, “IMAP folklore” suggests that a Code must not contain literals. At least, all extensions I saw so far seem to be very careful about this “rule”.

Q: Is the loop required?

Yes. IMAP servers may send their * CAPABILITIES ... right after the greeting. The loop discards these lines.

Q: Why without_greeting?

STARTTLS is defined so that after the transition to TLS, there will not be a second greeting. This is a minor difference between a session started via implicit TLS (port 993) and a session that we primed towards TLS.

Q: Is .starts_with enough?

Probably yes. The implementation could be made more solid, e.g., by using a random token in case the server sends a literal. But… why should it?

Q: Is good error reporting still possible?

We lose precision in our errors. When we implement STARTTLS this way, we cannot detect a server that does not advertise STARTTLS or rejects our command. We will only see a TLS error. This is the price we pay for a reduced attack surface.

Microsoft supports IMAPS on port 993 but not SUBMISSIONS on port 465. This has an interesting historical background. Today, it’s unreasonable. ↩︎

The European Union must keep funding free software

Sat, 20 Jul 2024 15:37:57 +0200

The European Union must keep funding free software

Initially publishead by petites singularités. English translation provided by OW2.

If you want to sign the letter, please publish the letter on your website and complete the table ~~below~~ here.

Open Letter to the European Commission

Since 2020, Next Generation Internet (NGI) programmes, part of European Commission’s Horizon programme, fund free software in Europe using a cascade funding mechanism (see for example NLnet’s calls). This year, according to the Horizon Europe working draft detailing funding programmes for 2025, we notice that Next Generation Internet is not mentioned any more as part of Cluster 4.

NGI programmes have shown their strength and importance to supporting the European software infrastructure, as a generic funding instrument to fund digital commons and ensure their long-term sustainability. We find this transformation incomprehensible, moreover when NGI has proven efficient and economical to support free software as a whole, from the smallest to the most established initiatives. This ecosystem diversity backs the strength of European technological innovation, and maintaining the NGI initiative to provide structural support to software projects at the heart of worldwide innovation is key to enforce the sovereignty of a European infrastructure. Contrary to common perception, technical innovations often originate from European rather than North American programming communities, and are mostly initiated by small-scaled organizations.

Previous Cluster 4 allocated 27 million euros to:

“Human centric Internet aligned with values and principles commonly shared in Europe” ;
“A flourishing internet, based on common building blocks created within NGI, that enables better control of our digital life” ;
“A structured ecosystem of talented contributors driving the creation of new internet commons and the evolution of existing internet commons”.

In the name of these challenges, more than 500 projects received NGI funding in the first 5 years, backed by 18 organisations managing these European funding consortia.

NGI contributes to a vast ecosystem, as most of its budget is allocated to fund third parties by the means of open calls, to structure commons that cover the whole Internet scope - from hardware to application, operating systems, digital identities or data traffic supervision. This third-party funding is not renewed in the current program, leaving many projects short on resources for research and innovation in Europe.

Moreover, NGI allows exchanges and collaborations across all the Euro zone countries as well as “widening countries” [^1], currently both a success and an ongoing progress, likewise the Erasmus programme before us. NGI also contributes to opening and supporting longer relationships than strict project funding does. It encourages implementing projects funded as pilots, backing collaboration, identification and reuse of common elements across projects, interoperability in identification systems and beyond, and setting up development models that mix diverse scales and types of European funding schemes.

While the USA, China or Russia deploy huge public and private resources to develop software and infrastructure that massively capture private consumer data, the EU can’t afford this renunciation. Free and open source software, as supported by NGI since 2020, is by design the opposite of potential vectors for foreign interference. It lets us keep our data local and favors a community-wide economy and know-how, while allowing an international collaboration.

This is all the more essential in the current geopolitical context: the challenge of technological sovereignty is central, and free software allows to address it while acting for peace and sovereignty in the digital world as a whole.

About

Tue, 22 Aug 2023 00:00:00 +0000

Hello, World!

I’m Damian Poddebniak, a software engineer and IT security researcher who believes in free software, open access to knowledge, and a world with net-zero greenhouse gas emissions. I always enjoyed programming but fell in love with (the) Rust (community). Thus, expect to read a lot about Rust in this blog! However, I am also human and may write about other things.

Freelancing

I recently started a journey as a freelancing software engineer to work on projects and with clients that share my passion for open-source software.

Expertise

My core strengths encompass email protocols, applied cryptography, public-key infrastructures, and diligent software design in Rust. Generally, I enjoy writing parsers for complex protocols and designing usable, misuse-resistant APIs. Still, I am inquisitive and would love to work with geo-information and embedded systems to enter the renewable energy sector.

During my Ph.D. at the University of Applied Sciences in Münster, I co-authored several research papers on end-to-end encrypted email (IMF, MIME, S/MIME, OpenPGP, …), transport encryption (TLS, X.509, …), and web security (ALPN, SOP, …) that uncovered numerous security issues, leading to protocol revisions in Internet standards such as S/MIME, OpenPGP, and the IMAP protocol.

Beyond my technical expertise, I can translate complex technical topics into accessible, engaging content for a broader audience to promote projects effectively. I’m a transparent person, and, as far as I can tell, people generally enjoy working with me :-)

Clients and rates

I am particularly drawn to projects promoting open-source software, secure communication, and clients in the renewable energy sector. These are my dream clients, as leveraging technology in these spaces is crucial for our future.

My rate is ~800€ / day, and when you choose to work with me, you’re not just hiring a freelancer – you’re investing in a partnership that delivers value beyond the project’s scope. I am dedicated to ensuring every project concludes with a diligently designed, meticulously documented, easily maintainable software artifact.

You can reach me via [email protected]" >[email protected]. Pronouns: he/him/his.

Education

Dr.-Ing. in Computer Security, 2016-2021 University of Applied Sciences Münster, Münster, DE
M.Sc. in Computer Science, 2014-2016 University of Applied Sciences Münster, Münster, DE
B.Sc. in Computer Science, 2011-2014 University of Applied Sciences Münster, Münster, DE

A gentle introduction to IMAP

Thu, 20 Jul 2023 00:00:00 +0000

Hello, World!, again!

When asked for advice on what to do as a fresh Ph.D. student a few months ago, I said confidently: “Create a blog and publish regularly.” ¹ So… two and a half years, a move to Hamburg, an excursion as a cryptography engineer, and a few new hobbies later, here’s the next post!

Joking aside, the last blog post was crucial to me as it paved the way toward a grant from the NLnet Foundation! NLnet heavily contributes to an open information society and helps to fix the Internet. I recommend looking at what NLnet can do for you and thinking about what you can do for NLnet!

What motivated this blog post is that I’m just about to conclude the imap-codec project. And what is the best way to make people look at it? Exactly: A blog post about IMAP!

Today, I’ll tell you everything I think is helpful to get up to speed with IMAP. This blog post will be a “gentle” introduction. The next one will be about everything that makes IMAP hard to tame.

Introduction

State of Mind

IMAP is a lot. It’s a stateful protocol that doesn’t lend itself to a simple implementation and can be intimidating to someone who is used to HTTP. The syntax is branched, subtle, and has various issues. It doesn’t help either that there are more than 70 extensions, with any of them potentially breaking your understanding of how IMAP operates. Take your time.

It may seem pretentious to use complicated phrases for simple concepts – I sometimes also joke about the term “string interpolation.” But phrases like “command continuation request,” “command completion result,” “non-synchronizing literal,” etc. provide clarity.

Lastly (and I know this may sound sarcastic): Try to accept things as they are. Many things are/were there for a reason. And those that aren’t/weren’t can’t be changed anyway.

Piece of Mind

But there is good news! When using imap-codec, you don’t need to care about syntax. imap-codec is based on imap-types, and it promises that everything you can create is valid. Try to create something in imap-types! If it works, it’s allowed.

Thus, let’s annotate all messages with the corresponding Rust Debug-print. As we will see, this makes it obvious what is supported and what is not.

Maybe the most significant change since the last blog post is that we now have imap-types that position itself as a candidate for a “standard library” for IMAP in Rust. The similarity to http is on purpose :-)

Speaking IMAP

The most efficient, most secure, and recommended way to use IMAP is through TLS on port 993. Non-encrypted connections are traditionally made on port 143 with the option to do an inline transition to TLS. This is called STARTTLS and should be avoided.

We will use a non-encrypted connection on port 1143 to make our life easier because most operating systems require special permissions when handling port numbers below 1024.

To follow along, you can start imap-codec’s tokio-server demo using …

cd imap-codec
cargo run -p tokio-server -- 127.0.0.1:1143

… and connect to it using …

nc -C 127.0.0.1 1143

We use Netcat with the -C parameter so that all newlines (\n) are encoded as “Internet newlines” (\r\n) as required by IMAP.

Warning: Don’t use Netcat to connect to anything remotely important, as nothing is encrypted. If you really want to experiment with a real server, you could use …
openssl s_client -verify_return_error -crlf -brief -connect <host>:<port>
… instead and connect through TLS.

Greeting

In contrast to, e.g., HTTP, the server sends the first message in IMAP. Thus, you should already see something like …

* OK [CAPABILITY IMAP4REV1 AUTH=PLAIN] IMAP server ready

Let’s use a comment for “bytes on the wire,” denote the role, e.g., the client (C: ) or server (S: ), and add a Debug-print to show the type instance:

// S: * OK [CAPABILITY IMAP4REV1 AUTH=PLAIN] IMAP server ready
Greeting {
    kind: Ok,
    code: Some(Capability([Imap4Rev1, Auth(AuthMechanism(Plain))]+)),
    text: Text("IMAP server ready"),
}

The type already tells us a bit about Greetings. Generally, when you encounter a * , this means “untagged.” We have yet to talk about tags, but greetings are never tagged. Thus, there is no tag field.

The information we can extract from the greeting is a kind, (possibly) a code, and a human-readable text. The greeting kind is usually Ok and signals that the server will serve our connection. But it could as well be… well… look into GreetingKind :-)

The Code is optional and may carry additional data. Here, it already tells us what capabilities the server supports.

Note the + at the end of the capabilities vector. This signals we have a NonEmptyVec.

If you are like me, you might have wondered how applications should parse IMAP server ready. They don’t. It’s just a human-readable Text that could be shown to a user, e.g., in case of an error. Let’s always use ... for brevity.

Note: You can toy around with your messages using …
cargo run --example=parse_{greeting,command,response}
… that will parse and Debug-print your input.

Capability agreement (again?)

If the server doesn’t send its capabilities – or the client doesn’t understand “codes in greetings” – it needs to ask for them using the CAPABILITY command. Let’s pretend we didn’t already know the capabilities and ask again:

// C: A1 CAPABILITY
Command { tag: Tag("A1"), body: Capability }
// S: * CAPABILITY IMAP4REV1 AUTH=PLAIN
Data(Capability([Imap4Rev1, Auth(AuthMechanism(Plain))]+))
// S: A1 OK ...
Status(
Ok {
    tag: Some(Tag("A1")),
        code: None,
        text: Text("..."),
    },
)

Now is a good time to talk about Tags. Unlike SMTP and POP3, an IMAP client can send multiple commands immediately, and the server can answer them out of order. Thus, we need a mechanism to match the command/response pairs.

In the above trace, the client used A1 as the command tag, and the server responded with two responses, Data and a Status. Here, the second response reflects the command tag.

Remember what we said about terminology?

Looking closely, the status tag is optional. In fact, status responses are “command completion results” and say something about a command’s result when tagged. Untagged status responses serve as an alternative form of server data and don’t say anything about a command.

Let’s also stress again that the server can return multiple responses. It’s even more complicated: No response is really “bound” to a command and can be emitted by the server for unrelated reasons. The server can also emit responses despite no “in-flight” command.

The following trace could be perfectly valid:

Command { .. } // Tagged with A1  // Command
Data(_)                           // Server data
Data(_)                           // Server data 
Data(_)                           // Server data
Status(_)      // Untagged        // Server data
Data(_)                           // Server data
Status(_)      // Tagged with A1  // Command completion result

// No in-flight command, but we get more server data.

Data(_)                           // Server data
Status(_)      // Untagged        // Server data

It’s helpful to think of an IMAP server as something that can send (almost) any response at any time. You always need to be prepared to receive a response. If you are interested in something particular, e.g., a specific email, you can ask the server for more responses using a command. But again: Asking doesn’t mean you will immediately get it. The server may tell you about other emails before handing you the requested information (so to say).

Note: The IMAP standard describes what server data the client can expect to receive before the command completion result. It may receive more than that but should also receive the “REQUIRED untagged response(s).”

Of course, the server MUST NOT send made-up tags. A command completion result without a command to conclude doesn’t make sense (except for interesting attacks :-))

Authentication

Now, let’s look at a login sequence. I hope it has become easier to read by now.

// C: A1 LOGIN alice password
Command {
    tag: Tag("A1"),
    body: Login {
        username: Atom(AtomExt("alice")),
        password: Atom(AtomExt("password")),
    },
}
// S: A1 OK ...
Status(
    Ok {
        tag: Some(Tag("A1")),
        code: None,
        text: Text("..."),
    },
)

IMAP has multiple ways to authenticate. There is a LOGIN and AUTHENTICATE command with similar purposes. The AUTHENTICATE command makes IMAP usable with authentication mechanisms defined by “SASL.” We can almost ignore SASL here. However, the base IMAP protocol requires you to implement the SASL AUTH=PLAIN mechanisms which would look like this …

// C: A1 AUTHENTICATE PLAIN
Command {
    tag: Tag("A1"),
    body: Authenticate { mechanism: AuthMechanism(Plain) },
}

Where are the username and password, you ask? Let’s figure it out in the next blog post :-)

Note: AUTH=LOGIN is a more verbose, less efficient, non-standardized mechanism invented for unknown reasons.

Note: If you want to connect to Gmail through IMAP, you need AUTH=XOAUTH2 or AUTH=OAUTH2. This requires registering an App with Google to create the necessary tokens. It’s not trivial and took me an hour to do using a Python script I downloaded from the Internet.

~~Unfortunately, you can’t speak IMAP with Google by any other means. It’s good for security but complicates testing.~~

Update: As pointed out by @csb6 and @sir, it’s still possible to authenticate with a password. First, make sure to activate 2FA. (You can use a security key to keep your phone number private.) Then, in the 2FA settings, scroll down to the bottom and generate an “app password” for email. You can use this as a usual password for your gmail account.

Folder selection

We want to read some emails now, but not so fast! We first need to SELECT a folder …

// C: A1 SELECT INBOX
Command { tag: Tag("A1"), body: Select { mailbox: Inbox } }
// S: * 3 EXISTS
Data(Exists(3))
// S: * 3 RECENT
Data(Recent(3))
// S: * OK [UNSEEN 1] ...
Status(Ok { tag: None, code: Some(Unseen(1)), text: Text("...") })
// S: * OK [UIDNEXT 8] ...
Status(Ok { tag: None, code: Some(UidNext(8)), text: Text("...") })
// S: A1 OK ...
Status(Ok { tag: Some(Tag("A1")), code: None, text: Text("...") })

Great, we got a gazillion responses and a positive command completion result. This means we are in the INBOX now. And we already know how many messages the INBOX holds due to the Exists response.

Fetching of email data

Let’s now FETCH the first email:

// C: A1 FETCH 1 (BODY[])
Command {
    tag: Tag("A1"),
    body: Fetch {
        sequence_set: SequenceSet([Single(Value(1))]+),
        macro_or_item_names: MessageDataItemNames(
            [
                BodyExt {
                    section: None,
                    partial: None,
                    peek: false,
                },
            ],
        ),
        uid: false,
    },
}

Why does one line result in such a complicated type, you ask?

Back answer: Did you know that an IMAP server needs to parse and understand the complete structure of every email it holds to serve partial information? Do you know GraphQL? IMAP already had it!

Besides having surprising consequences on end-to-end encrypted email, partial fetching allows, e.g., to fetch only all subjects in the inbox. This is really important to write good clients. You don’t want your email client to download your whole INBOX (and all attachments) before being able to show you a list of “from,” and “subject.”

You can request subjects only. Or specific headers. Or a particular MIME part.

Or …

1337 bytes
starting at index 42
of the header
of the second MIME part
of the first MIME part
of the first email plus emails from sequence number 2 to the end
without marking them as read

… like …

//    Tag
//    |    Fetch by UID
//    |    |         Which emails?
//    |    |         |      What exactly?
//    |    |         |      |
//    /--\ /-------\ /---\  /----------------------------\
// C: ABC1 UID FETCH 1,2:* (BODY.PEEK[1.2.HEADER]<42.1337>)
//                          \-------/ \--------/  \/ \--/
//                          |         |           |  |
//                          |         |           |  Count
//                          |         |           Start
//                          |         Part
//                          Body (w/o marking as read)
Command {
    tag: Tag("ABC1"),
    body: Fetch {
        sequence_set: SequenceSet(
            [
                Single(Value(1)),
                Range(Value(2), Asterisk),
            ]+,
        ),
        macro_or_item_names: MessageDataItemNames(
            [
                BodyExt {
                    section: Some(Header(Some(Part([1, 2, ]+)))),
                    partial: Some((42, 1337)),
                    peek: true,
                },
            ],
        ),
        uid: true,
    },
}

You get the idea! Hopefully, you are convinced you don’t want to write this by hand or parse it yourself. Have I already told you about imap-c… just kidding :-)

It doesn’t end here …

Well, yes, this blog post does end here. But I will release another one very soon™. In the next one, I will show you everything I avoided mentioning here. We will cover literals, authentication, and why designing a sound IMAP library is hard.

See you soon!

Of course, having a blog as a Ph.D. student is about something other than publishing. It’s about regular writing and feedback. ↩︎

Type-Driven Development

Tue, 26 Jan 2021 00:00:00 +0000

Hello, World!

I was recently reminded about Cunningham’s Law …

“The best way to get the right answer on the internet is not to ask a question; it’s to post the wrong answer.”

– Ward Cunningham

… and it made me think.

During my Ph.D. I used Obsidian, an app to write and cross-reference Markdown documents, to sort my thoughts. However, I didn’t share most of it and never discussed it with a broader audience.

So I figured… why don’t I have a blog?

After wrestling with Zola for some hours and comically failing to configure a theme, I installed Hugo. No worries, Zola. I’ll try a second time in the future! Anyway… this is my blog now. Hosted under a HSTS preloaded .dev domain. Served by Caddy. Structured and styled by the pretty Stack theme.

In my first post, I will think aloud about some design decisions I made during the development of the imap-codec library.

Introduction

Motivation

imap-codec was born out of frustration. During a research project, I analyzed IMAP clients for certain behavior and had difficulties constructing syntactically valid IMAP messages. IMAP is… “branched.” Some syntax elements are required in one place and prohibited in another one. I often stared at hard-to-understand IMAP sessions. Sessions, where I was 100% sure that a response was correct and learned that it wasn’t.

I started imap-codec to gradually improve our tooling and to not make the same errors twice. And, somehow, I remained interested in it. But let me first introduce the domain we are working in.

Parsing and serialization

imap-codec is an IMAP parser and serializer. The task of a parser is to turn a sequence of bytes, an input, into a more useful data structure, an object. The parsed object provides easy access to the encoded information. The task of a serializer is to “flatten” a parsed object into a sequence of bytes again.

More concretely, imap-codec provides two parsers¹, command and response, which turn a sequence of bytes (&[u8]), to either a Command or a Response object. For the serialization part, Command and Response provide an encode method (via the Encode trait), which turns the parsed objects into bytes.

We will focus on the command part only because it makes the post easier to understand (and write.) But all thoughts apply to other parsers and types as well.

Bytes to command and vice versa (example)

When a user inputs the byte sequence …

A123 select "InBoX"\r\n

… to the command parser …

let (cmd, _) = command(b"A123 select \"InBoX\"\r\n")
                 .unwrap();

… it returns a …

Command {
  tag: Tag("A123"),
  body: CommandBody::Select {
    mailbox: Mailbox::Inbox,
  }
}

… object.

When cmd.encode(...) is called, it generates the byte sequence …

A123 SELECT INBOX\r\n

To eliminate a source of confusion: IMAP commands are prefixed by a tag. We used A123 but could have also used FooBar1337. Even… <script>alert`XSS`</script>. (Yes, you can call a JavaScript function with a mere template literal. No, I don’t know why. Yes, this can be useful in attacks, e.g., in the ALPACA Attack.)

There isn’t really that much to tell about the usage of imap-codec as a parser. You feed it some bytes and get a fully-parsed and verified-to-be-sane object (and a remainder.) No callbacks. No shotgun parsing. But there is more happening that is not obvious at first glance.

Misuse resistance, correctness, tightness, …

The term “misuse resistance” is better-known in cryptography but can be used more broadly. Basically, it means that a developer can’t use your library in the “wrong way.” Using a cryptographic library in “the wrong way” can lead to severe security issues. But many things can be misused with varying consequences. For example, a library (or application) can become perplexing when it is easy to misuse. It’s not just about security.

Currently, I describe imap-codec as misuse resistant, and mean that you can’t produce a syntactically invalid or inconsistent IMAP message – even when you try really hard. But there’s more to it. The API encodes domain knowledge about IMAP. In a role of an email client/server developer who is not necessarily an IMAP expert, you will (hopefully) have a hard time screwing up.

But wait. You’re not the one who has commit rights for imap-codec, are you? You’re not supposed to just change the internals of imap-codec. There are people for that! If someone screws up with imap-codec, it is more likely to be me. So… where is my safety net?

Where is my safety net?

Misuse resistance can happen “on different levels.” You can make your API misuse resistant to your consumers but hard to maintain internally. Sure, you can be careful, but I’m not. I don’t want to be! With that attitude… I better have a good strategy to not make stupid mistakes. Thus, I use Rust’s type system as much as possible and cross my fingers that a good public-facing API evolves from that. Concretely, all types in imap-codec, i.e., all structs and enums, should be designed, so that inconsistent state is not representable.

I know I might bend the term “misuse resistance” here. I could use “correct”, or “tight”, instead. But who said that instantiation of types cannot be resistant to misuse?

Leveraging Rust’s type system

The above example …

Command {
  tag: Tag("A123"),
  body: CommandBody::Select {
    mailbox: Mailbox::Inbox,
  }
}

… shows two instances where imap-codec uses Rust’s type system to enforce invariants on the type level.

First, note how the tag is wrapped in a Tag struct and consider what would happen if a Command could be constructed like this:

let cmd = Command {
  // `&str` instead of `Tag`
  tag: "A123 X", 
  body: CommandBody::Select {
    // `&str` instead of `Mailbox`
    mailbox: "InBoX",
  }
};

Serialization of cmd would likely produce:

A123 X SELECT "InBoX"\r\n
     ^
     |
     error: no such command "X"

Obviously, the tag must not contain any whitespace. But &str is not made to enforce that. Thus, imap-codec has a Tag, a newtype wrapper of &str enforcing additional rules. Especially (but not exclusively) that no whitespace is allowed.

Second, note that the string "InBoX" is not contained in the parsed Command. Instead, the mailbox became Mailbox::Inbox which is serialized as INBOX (without the quotes.)

Unlike other folder names, the Inbox in IMAP is not case-sensitive. If the selected mailbox were saved as a string, a programmer would need to remember to check all cases of “inbox”. The infamous …

if "inbox" == mailbox.to_lowercase() {
  // ...
}

The Mailbox struct unifies all Inboxes to Mailbox::Inbox. Concretely, it is defined as …

enum Mailbox {
  Inbox,
  Other(MailboxOther),
}

… and there is no way in imap-codec to create a Mailbox::Other(...) that semantically denotes the Inbox.

C’mon, how bad can it be?

How bad was the recent Firefox outage? I don’t know. But mishandling case sensitivity is a real source of bugs. But, more importantly, as domain specialists, we encode our domain knowledge into the type.

imap-codec uses a lot of these constructs. Atom, Tag, Text, … Even NonEmptyVec<T> and NonZeroU32!

IMAP section specifiers (example)

The IMAP RFC defines a “section specifier” syntax to fetch only specific parts of an email. Here are some examples from the IMAP RFC:

HEADER        // Fetch header
TEXT          // Fetch text
1             // Fetch part 1
1.HEADER      // Fetch header of part 1
1.TEXT        // ...
...
5.3.4.HEADER  // ...
...

How should we design a type, say, Section, for that?

Our first attempt …

struct Section {
  path: Vec<u32>,
  kind: Option<Kind>,
}

enum Kind {
  Header,
  Text,
}

… should work. All examples can be constructed, e.g., …

// HEADER
Section {
  path: vec![],
  kind: Some(Kind::Header),
}

// ...                                               

// 1
Section {
  path: vec![1],
  kind: None,
}                

// ...                                               

// 5.3.4.HEADER
Section {
  path: vec![5, 3, 4],
  Some(Kind::Header),
}  

// ...

However, there is a problem. There exists an invalid combination of path and kind:

Section {
  path: vec![],
  kind: None
}

When this is encoded, it will likely result in an empty byte sequence. However, this violates the IMAP specification for a section specifier. Now, we can be careful to not let this happen. But we can do better by redesigning the Section type to ensure that this assignment is not representable in the Section type.

We can merge …

struct Section {
  path: Vec<u32>,
  kind: Option<Kind>,
}

… and …

enum Kind {
  Header,
  Text,
}

… to a single …

enum Section {
  Part {
    path: NonEmptyVec<u32>
  }
  Header {
    path: Vec<u32>
  }
  Text {
    path: Vec<u32>
  }
}

Now, our Section is either 1) a Part, in which case the path can not be empty, or 2) a Header or Text, in which case the path can be empty.

Later, we also learn from the IMAP RFC that the numeric path must not contain zeros! Let’s fix that, too. Our final Section type then becomes …

enum Section {
  Part {
    path: NonEmptyVec<NonZeroU32>
  }
  Header {
    path: Vec<NonZeroU32>
  }
  Text {
    path: Vec<NonZeroU32>
  }
}

Ideally, all struct’s and enums in imap-codec should provide this certain type of quality that any arbitrary assignment of fields should make sense. I saw this property called tightness by Pablo Mansanet. A decent term, if you ask me. Quinn Wilton described it as Programming with Types. And there is also the term “Type-Driven Development” used (or even introduced?) by the creator of Idris, Edwin Brady. There is even a book titled “Learn Type-Driven Development” by Yawar Amin and Kamon Ayeva. To be honest… I didn’t know about Type-Driven Development before. I thought this is just how you are supposed to write Rust!

Tight types can easily be generated

A nice benefit of tight types is that you can simply [derive(Arbitrary)]. This allows to easily generate crazy – but valid – messages. This is useful for testing: You can encode(...) arbitrarily generated objects into bytes to test the serialization logic. Afterward, you can parse the serialized bytes to test the parsing logic. And you can compare results to test for correctness. Have a look at this beautiful IMAP command and how imap-codec is fuzzed!

Conclusion

Thinking about every type might seem like overkill at first. But it is really not too much work and seems to pay off. And more importantly, I am (somewhat) confident that imap-codec is not affected by injection attacks, which may lead to adventurous security issues. (More on that in some later blog post.)

Rust’s type system, especially algebraic data types, is useful for enforcing invariants on the type level. A good library design can use the type system to encode domain knowledge – think about how Inbox is case-insensitive. While mistakes can still happen in the “design phase” – e.g., a software developer misunderstood something and enforced the wrong thing – later modifications to the library are less likely to violate invariants. This programming strategy helped me work on imap-codec, and I hope that new contributors will also benefit from this safety net.

Also, I feel that if you are willing to spend time on a standard, better encode all you learn into the types you use. You will inevitably forget most details, and it is only a question of time that you break something because of that.

While the direction for imap-codec is (somewhat) clear to me, there is still work to do. Currently, imap-codec exposes (too?) many details. I have some thoughts on why this can either be negative or positive for this kind of library. But I haven’t gotten enough feedback and didn’t have the time to think it through.

Appendix

Parser correctness and malleability

You may have noted that imap-codec did not “recreate” what it parsed – A123 select "InBoX" vs. A123 SELECT INBOX. But, ideally, we want that …

$$ \texttt{parse}(\texttt{serialize}(\texttt{object})) = \texttt{object} $$

… and …

$$ \texttt{serialize}(\texttt{parse}(\texttt{input})) = \texttt{input} $$

… for any object and any input. (Excluding the error case.)

Or, in other words, we want parsing and serialization to be inverse to each other. In this case, we call a parser/serializer combination correct.

Let’s think for a moment about what this means in practice. First, we see that it is pretty easy to write a correct parser. Lo and behold …

struct Object(&[u8]);

fn parse(input: &[u8]) -> Object {
  Object(input)
}

fn serialize(object: Object) -> &[u8] {
  object.0
}

… voila!

This parser/serializer combination is correct. But it is not useful. We just wrapped the input in an Object newtype to obtain another domain to map our input to. This shows that the definition of Object – or generally the domain we map our input to – is key to usability and correctness.

imap-codec is not correct

imap-codec’s parsers are not injective because they can map different bytes, input1 and input2, to the same object. This means that the parse function is not a bijection, which means that it cannot have an inverse, which means it can not be correct.

We could archive correctness by preserving all information from the input – even the semantically irrelevant information such as the casing of the “select” command. This is maybe required in “format-preserving” parsing but less useful in protocols.

IMAP is malleable, and there is not much we can do about it

Arguing with my security hat on, a data format (or protocol syntax specification) should be non-malleable. Non-malleability means that each message has a unique representation. If this is not the case, you must preserve irrelevant information to archive correctness. Thus, correctness is probably not what you should want from a protocol that is malleable.

Why do you care?

Because fuzzing! It’s cooler when you have correctness!

I wanted to use fuzz-testing in imap-codec to …

let object = lib_fuzzer_magic();
assert_eq!(parse(encode(object)), object);

and

let input = lib_fuzzer_magic();
assert_eq!(encode(parse(input)), input);

… but I needed to cheat!

Instead of checking that imap-codec can recreates what it has parsed – which it can’t because of malleability – I parse the serialized bytes again and compare the parsed objects …

let input  = lib_fuzzer_magic();   

let object = parse(input);
let output = serialize(object);
let cheat  = parse(output);

// Does not work ...
// assert_eq!(input, output);

// Cheating ...
assert_eq!(object, cheat);

Despite uncovering a few bugs, I am not quite sure what I test here exactly … For example, this test would not catch if I would screw up and always return the same (wrong) object in the parse function.

A different parser is required in certain message flows. But command and response are the main entry points. ↩︎

Publications

Mon, 01 Jan 0001 00:00:00 +0000

Search

Mon, 01 Jan 0001 00:00:00 +0000

Software

Mon, 01 Jan 0001 00:00:00 +0000

Talks

Mon, 01 Jan 0001 00:00:00 +0000

Duesee's Blog

(Avoid) implementing STARTTLS

(Avoid) Implementing STARTTLS

Sierpiński carpet. Infinite perimeter and zero area. Start with a square, split the square into 9 equal squares, remove the central square, and continue recursively.

Yet… someone needs it.

And yet… someone needs it.

How to (avoid) implementing STARTTLS?

Minimal STARTTLS implementation (in Rust)

FAQ

Tree of all ABNF rules used by greeting.

The European Union must keep funding free software

The European Union must keep funding free software

Open Letter to the European Commission

About

Hello, World!

Freelancing

Expertise

Clients and rates

Education

A gentle introduction to IMAP

Hello, World!, again!

Introduction

State of Mind

Piece of Mind

Speaking IMAP

Greeting

Capability agreement (again?)

Authentication

Folder selection

Fetching of email data

It doesn’t end here …

Type-Driven Development

Hello, World!

Introduction

Motivation

Parsing and serialization

Bytes to command and vice versa (example)

Misuse resistance, correctness, tightness, …

Where is my safety net?

Leveraging Rust’s type system

IMAP section specifiers (example)

Tight types can easily be generated

Conclusion

Appendix

Parser correctness and malleability

imap-codec is not correct

IMAP is malleable, and there is not much we can do about it

Archives

Publications

Search

Software

Talks