Andrew's Blog

Sending Build Output to Vim

Mon, 01 Feb 2021 16:40:00 +0100

Vim is actually quite easy to build from source with a make install. You can edit src/Makefile to enable features and custom language extensions. Turns out, I hadn’t scrolled through that file in a while, since version 8.0.1295 comes with an interesting addition – the compilation flag --enable-autoservername.

Running the editor with a --servername parameter allows remote connections from a different instance using command-line flags or function calls. I’ve been using this functionality to build a custom testing framework in Ruby, but you have to knowingly launch it with that flag. Now, each instance can be available to connect with by default.

It’s not necessary, exactly – you could always make yourself a shell alias like alias vim="vim --servername VIM-$RANDOM". But the discovery gave me an idea of how to remove a tiny bit of friction from my Rust workflow, and I might end up using the same technique in the future. Read on to learn how to send your build output to a Vim instance for easier processing.

But why?

When using a large IDE, or even something smaller like VSCode, I think there’s a tendency to keep one instance running and open things in it. In fact, on my machine, running code <some-file> in the terminal opens the code in an existing instance, even though you can launch multiple ones.

Vim is a lot more decoupled than that. You can have one big Vim open holding tabs, terminal windows, git commands and a lot more. But it’s just one possible workflow and somewhat counter to what you get from the defaults. Personally, I tend to have one Vim focused on the code I’m currently working on, but I might also open other ones with notes or other projects to ~~copy-paste from~~ use as inspiration.

I also use separate terminals for running commands – tests, git, etc. I respect the flexibility of :terminal, but I guess I’m just too used to jumping between workspaces and I like keeping things separate. But this does make it slightly harder to debug builds. Usually, when Rust shows me compile errors or test failures, I’d double-click their locations and paste them into Vim. A plugin like vim-fetch helps a lot with this kind of thing.

It works, it’s fine, I’m used to it. But if I could directly send the errors over to my editor…

Finding the right instance

First, let’s decide where to send the output. There might be multiple instances, but I’m looking for the one in the same directory I’m running the command in:

# Let's get a list of all available instances:
vim_instances = `vim --serverlist`.lines.map(&:strip)
cwd           = FileUtils.pwd

servername = vim_instances.find do |candidate|
  # Use --remote-expr to get the results of `getcwd()` from each of the Vims:
  candidate_wd = `vim --servername #{candidate} --remote-expr 'getcwd()'`.strip

  # The one we're looking for should have a  current working directory that's
  # somewhere under the command's working directory:
  if candidate_wd.start_with?(cwd)
    candidate
  else
    nil
  end
end

The invocation to send commands is extractable, although I don’t usually need the output. Here’s what it might look like if we pulled it out to a function without capturing the output:

def send_to_vim(servername, command)
  system 'vim', '--servername', servername, '--remote-expr', command, out: '/dev/null'
end

This one is easier to manage in terms of escaping and quotes, too, since the system call takes care of quoting the individual arguments properly.

Running the cargo command

The cargo tool is extensible in the same way git is: If you create a command in $PATH named cargo-something, it becomes available as cargo something. This means it’s a good idea to handle both invocations at the start of the program:

if ARGV.first == 'vim'
  # then it was called as `cargo vim <command>` and not `cargo-vim <command>`,
  # so remove the "vim" part:
  ARGV.shift
end

if ARGV.count < 1
  STDERR.puts "USAGE: cargo vim <build|run|test> [args...]"
  exit 1
end

But once we have that, ARGV contains everything we need to run the real cargo command and get its output. It would be nice if I could just get the output of the previous command. I imagine myself running cargo build once, having it fail, and then just go “oh, let’s just take this elsewhere”, but I couldn’t find a good way of doing this. Instead, re-running should be fast enough the second time around, this time into a file:

Tempfile.open('cargo-vim') do |f|
  print "Running cargo #{ARGV.join(' ')} ... "

  # The output from `cargo` goes in the standard error stream, so we want to
  # redirect that one to the temporary file. Stdout we don't care about much.
  result = system 'cargo', *ARGV, err: f, out: '/dev/null'

  puts "DONE"

  # ...
end

In the real code I’ve also added some measurements and visual cues to get something slightly fancier:

% cargo vim build
Running cargo build ... DONE 123.23ms 🗸
% cargo vim build
Running cargo build ... DONE 348.68ms ❌

But the core of the code is rerunning the same command into a tempfile. That file’s path can then be used to populate the quickfix list if the result is falsey:

Tempfile.open('cargo-vim') do |f|
  # ...

  if result
    # No errors, clear and close the quickfix list
    send_to_vim(servername, "setqflist([])")
    send_to_vim(servername, "execute('cclose')")
  else
    # There's errors, populate the quickfix list
    send_to_vim(servername, "execute('silent compiler cargo')")
    send_to_vim(servername, "execute('silent cfile #{f.path}')")
    send_to_vim(servername, "execute('silent copen')")
    exit 1
  end
end

Vim can handle cargo build output, because the basic Rust configuration has a “compiler” definition for it. Running compiler cargo is enough to tell Vim how to parse the output of the build process in a useful way. The built-in cfile command loads up the given file path into the quickfix window as an “error report”.

The end result is a cargo vim build that produces something like this:

Was this really necessary?

You don’t need any of this if you’re fine with changing your mental patterns a little bit. You can easily have the “compiler” setup in ~/.vim/ftplugin/rust.vim:

compiler cargo
set makeprg=cargo\ test

That way, if you get a build error, you could just switch to Vim and run the built-in :make command. It’ll pretty much do the same thing.

It could be argued my cargo vim command is slightly more convenient in case you build up a more complicated command line, like if you invoke cargo test --test test_markdown, you can just tweak that invocation to cargo vim test etc etc rather than copy the extra bits to append to :make in Vim. Or there might be ENV vars that change the build process. Honestly, though, I just don’t like having the “running” part in Vim due to some form of mental compartmentalization. You do you.

What this kind of separate script would be nice for for is pre-processing of the error output before sending it over to Vim’s compiler setup. I might end up doing the same thing for RSpec, where importing test failures is trickier due to some tests being printed as spec/test_name_spec.rb[1:2:3:4]. That part at the end is an identifier for the test, based on where it is in the example hierarchy, and it’s not possible to translate it directly into a file:line location. I’ve got a proof-of-concept script that might give me that info to plug into a similar “send to Vim” tool. Plus, RSpec can record the last set of test failures in a file, so it might not even be necessary to re-run the suite.

Either way, no, none of this is “necessary”, but it was a fun little exploration for me. And maybe you’ve learned a thing or two about Vim’s quickfix list and its client-server interface.

From Vim to Presentation Slides

Sat, 05 Oct 2019 08:00:00 +0200

There’s a little-known Vim command called :TOhtml that does something quite surprising for a built-in – it converts the current Vim window into raw HTML. This includes syntax highlighting for code, and even line numbers. It can convert the full buffer or a range of selected lines.

While I’ve always considered it a fun trick, it was somewhat recently that I realized it has a very practical use – to paste syntax-highlighted code into presentation slides.

The Process

First off, HTML is a good start, but it needs to be parsed as Rich Text Format:

" Main entry point, the :Tortf command, translates the given range as (start,
" end) parameters for the function that does the real work. Defaults to the
" whole buffer.
command! -range=% Tortf call s:Tortf(<line1>, <line2>)

function! s:Tortf(start, end)
  " We need a temporary file to actually write the HTML to. The :TOhtml
  " command opens a buffer, but doesn't save it:
  "
  let filename = tempname().'.html'

  " We run the command with the given range and save it to the temporary file:
  "
  exe a:start.','.a:end.'TOhtml'
  exe 'write '.filename

  " The `libreoffice` command will figure out how to open an HTML file -- with
  " the "writer" tool:
  "
  call system('libreoffice '.filename.' &')

  " No need to stay in this buffer -- it's done its job, so we can safely get
  " rid of it:
  quit!
endfunction

This command-function pair can live in your .vimrc or in a file in the ~/.vim/plugin directory. Once I’ve set an appropriate light scheme, ran the :gui command to ensure I’ve got the right colors, I can execute the :Tortf command and get something like this:

Now I can copy that code and try to paste it in a LibreOffice Impress presentation. Instead of just using Ctrl+V, though, it’s better to “Paste special”. On the current LibreOffice version, that’s located under Edit > Paste Special > Paste Special…, or it can be triggered with Shift+Ctrl+V:

The normal paste tends to change fonts depending on what you’re pasting over. Random things get bolded, or colors change. This way tends to produce less surprises.

And that’s pretty much it.

Some Sensible Questions to Ask

Any problems with this method?

Oh, tons. When I copy a slide, its code ends up centered for some reason, so I need to select it and left-align. Sometimes, colors still get messed up. Editing the code works, but it requires some work to maintain the right syntax highlighting.

Some of this might be LibreOffice bugs, or it might be me not using it correctly. Let me know if you have any solutions I can try!

Why use LibreOffice at all then?

Frameworks like impress.js are pretty good for code, but I find them less flexible in terms of just positioning graphics and such around. Even with code, I’d occasionally like to position it weirdly or appearify chunks of code one by one, and the PowerPoint model of making presentations gives me that flexibility.

Keynote might be much better, but I’m not a Mac user, so I couldn’t say.

Would this method work with Keynote?

Probably yes, but I can’t be sure. You’d need to tweak some of the commands to launch Keynote and some other word processor instead of LibreOffice.

What about using screenshots instead?

It’s an option, but code-as-text means you can make small edits later. It also allows people to select the code from the slides afterwards, even if you export them to PDF.

Mind you, with the existence of tools like vim-silicon, screenshots are certainly a more interesting option nowadays.

Testing in Rust: Writing to Stdout

Mon, 05 Aug 2019 08:00:00 +0200

For the last two years, I’ve been one of the organizers of an elective Rust course in Sofia University. Last semester, one of the homework assignments we came up with was to build several kinds of “logger” structs with buffering, multiple outputs, and with tagged logging. (The full assignment is here, but all the instructions are in Bulgarian.)

It was a pretty good exercise on using Rc and RefCell that wasn’t a linked list or a similar data structure. The goal was to ensure a logger can be cloned, with the copies sharing both the same buffer and output.

Testing the code seemed easy at first glance, but I did run into a bit of a problem when simulating loggers printing to “the screen”. The solution is not exactly complicated, but I think it’s a good pattern to share.

The `Write` Trait and Simple Stubs

The common trait for output in Rust is std::io::Write. This is implemented for files, sockets, and, most importantly for testing purposes, for Vec<u8> and &mut Vec<u8>. This makes it very easy to write a basic test:

#[test]
fn test_mutable_reference() {
    // Prepare our output stub:
    let mut out = Vec::new();

    // Don't take ownership, so we can access it for the assertion:
    let mut logger = Logger::new(&mut out);
    logger.log("Some warning");
    logger.flush();

    // Easier to compare if we just convert the bytes to a string first:
    let string_output = String::from_utf8(out).unwrap();

    assert_eq!(string_output, "Some warning\n");
}

The logger implementation is a simplified version of the one in the task:

use std::io::{Write};

pub struct Logger<W: Write> {
    out: W,
}

impl<W: Write> Logger<W> {
    pub fn new(out: W) -> Self {
        Logger { out }
    }

    // Just write the message directly to the given output with a newline.
    pub fn log(&mut self, message: &str) {
        self.out.write(message.as_bytes()).unwrap();
        self.out.write(b"\n").unwrap();
    }

    // Not an interesting method, but could be if we added buffering.
    pub fn flush(&mut self) {
        self.out.flush().unwrap();
    }
}

There’s no timestamps, no Clone implementation, and it’s a terrible idea to use unwrap in there (more on that later), but it’s enough to demonstrate the problem, even though the first test passes just fine:

running 1 test
test test_mutable_reference ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

The issue shows up when we try to share the output vector somehow, so we can see how multiple loggers’ output interpolates:

#[test]
fn test_mutable_reference_in_two_places() {
    let mut out = Vec::new();

    // Does not compile:
    let mut logger1 = Logger::new(&mut out);
    let mut logger2 = Logger::new(&mut out);

    logger1.log("One");
    logger2.log("Two");
    logger1.log("Three");

    logger1.flush();
    logger2.flush();

    assert_eq!(String::from_utf8(out).unwrap(), "One\nTwo\nThree\n");
}

You can’t really compile this code, because you’d be taking two mutable references to the same vector, in the same scope. You could create the loggers in two separate scopes, but that’s not what I’d like to test. If I initialized the two loggers with std::io::stdout() in a main function, they’d print “One”, “Two”, and “Three” just fine.

Of course, the stdout() function returns separate Stdout structs that share a file descriptor, probably with some amount of locking. So, instead of using a simple Vec, we can do something fancier.

A More Robust Stub

The solution to sharing resources in Rust is usually Rc, with an added RefCell so we can allow mutability:

use std::io::{self, Write};
use std::rc::Rc;
use std::cell::RefCell;

#[derive(Clone)]
struct TestWriter {
    storage: Rc<RefCell<Vec<u8>>>,
}

impl TestWriter {
    // Creating a new `TestWriter` just means packaging an empty `Vec` in all
    // the wrappers.
    //
    fn new() -> Self {
        TestWriter { storage: Rc::new(RefCell::new(Vec::new())) }
    }

    // Once we're done writing to the buffer, we can pull it out of the `Rc` and
    // the `RefCell` and inspect its contents.
    //
    fn into_inner(self) -> Vec<u8> {
        Rc::try_unwrap(self.storage).unwrap().into_inner()
    }

    // It's easier to compare strings than byte vectors.
    //
    fn into_string(self) -> String {
        String::from_utf8(self.into_inner()).unwrap()
    }
}

This struct is perfectly cloneable and holds a simple mutable vector. All we need to do is implement the Write trait for it that delegates methods to the vector through a borrow_mut():

impl Write for TestWriter {
    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
        self.storage.borrow_mut().write(buf)
    }

    fn flush(&mut self) -> io::Result<()> {
        self.storage.borrow_mut().flush()
    }
}

With this, our “fake stdout” can be cloned between two loggers just fine:

#[test]
fn test_clonable_writer() {
    let out = TestWriter::new();

    {
        let mut logger1 = Logger::new(out.clone());
        let mut logger2 = Logger::new(out.clone());
        logger1.log("One");
        logger2.log("Two");
        logger1.log("Three");

        logger1.flush();
        logger2.flush();
    }

    // Ownership of `out` is never transferred, so we can access it:
    assert_eq!(out.into_string(), "One\nTwo\nThree\n");
}

If we wanted to test loggers in different threads, we could change the Rc<RefCell<Vec<u8>>> to Arc<Mutex<Vec<u8>>> and use lock().unwrap() instead of borrow_mut().

All the unwrap calls are not a problem, given that there’s no “user” to report errors to – if we’ve somehow messed up our test setup, it should probably blow up. We might make it a bit clearer what broke by using expect calls instead. For instance:

fn into_inner(self) -> Vec<u8> {
    Rc::try_unwrap(self.storage).
        expect("TestWriter: More than one Rc refers to the inner Vec").
        into_inner()
}

That said, it’s not ideal to keep using unwrap in the logger code itself.

Error Handling

It’s not exactly obvious what the best way to handle errors is, for a logger. For one thing, we want the logger to be the thing printing the errors to files or to standard output. Returning Result from every single log statement would also be kind of annoying to deal with. The idea we came up with for the homework task was to split things up a bit and create pairs of methods – one that is allowed to fail and one that logs the error to stderr:

// A testable method that might error out:
pub fn try_log(&mut self, message: &str) -> io::Result<()> {
    self.out.borrow_mut().write(message.as_bytes())?;
    self.out.borrow_mut().write(b"\n")?;
    Ok(())
}

// A simple wrapper that we expect just ignores stuff:
pub fn log(&mut self, message: &str) {
    if let Err(e) = self.try_log(message) {
        eprintln!("{}", e);
    }
}

In order to test it, we can just create another struct that implements Write by failing unconditionally:

struct ErroringWriter {}

impl Write for ErroringWriter {
    fn write(&mut self, _buf: &[u8]) -> io::Result<usize> {
        Err(io::Error::new(io::ErrorKind::Other, "Write Error!"))
    }

    fn flush(&mut self) -> io::Result<()> {
        Err(io::Error::new(io::ErrorKind::Other, "Flush Error!"))
    }
}

It returns the right kind of error, and a specific message, if we really wanted to test details. I’d personally be fine with a simpler assertion:

#[test]
fn test_erroring_io() {
    let out = ErroringWriter {};
    let mut logger = Logger::new(out);

    if let Ok(_) = logger.try_log("One") {
        assert!(false, "Expected try_log with an erroring writer to return an error")
    }

    // Should not panic:
    logger.log("Two");
    logger.flush();
}

If it’s not an error, there’s something wrong, and that’s probably good enough guidance for something this simple. If you want to go in the other direction, a more sophisticated mock might even have methods to toggle its erroring or non-erroring state with more Rc/RefCell juggling.

Extracting to a Separate File

Update (2023-04-02): A better way to organize the shared code can be found in The Rust Book. I don’t know if I missed it when I was writing this article, or if it’s new(ish), but it’s there.

Having the custom structs in the actual test files can get in the way. What we could do is extract a separate testing.rs file to stick the code in:

src
├── lib.rs
├── main.rs
└── testing.rs
tests
└── test_logger.rs

We’ll need the module actually exported in lib.rs:

pub mod testing;

pub struct Logger<W: Write> {
    // ...

And then we have to import it in the test code:

use logger::Logger;
use logger::testing::*;

#[test]
fn test_clonable_writer() {
    let out = TestWriter::new();
    // ...

It’s a bit annoying, since the test module is included with the “real” code. You’d think it’d be possible to avoid this with a #[cfg(test)], but at the time of writing, I couldn’t manage to do it – adding a #[cfg(test)] above the pub mod testing line doesn’t compile. We could, however, add #[cfg(test)] to the individual types and impls:

#[cfg(test)]
#[derive(Clone)]
pub struct TestWriter {
    storage: Arc<Mutex<Vec<u8>>>,
}

#[cfg(test)]
impl TestWriter {
    // ...

That way, the module itself is always compiled in, but the structs are only present in the binary that’s used for testing. Difficult to say if it’s worth the trouble – these mock types would only be imported in tests either way, so there’s little chance of naming conflicts or binary bloat.

Closing Thoughts and Further Reading

The standard library includes some powerful testing tools and a convenient test runner. Even so, it can be worth it to create custom tools to make tests simpler to write and easier to read. Extracting them to their own module means they can easily be shared between different test files. You might even decide to split them up additionally into, say, src/testing/io.rs, src/testing/assertions.rs and so on.

In the end, tests are code too – reusing common logic works just as well, and you can apply all the same engineering patterns you use in “real” code.

If you need more flexibility in testing I/O in particular, try std::io::Cursor. It works just like a Vec for reading and writing, but allows you to rewind it and seek into it, like you could with a real file.

For more ideas on mocking, you might also take a look at “Mocking in Rust with conditional compilation”.

Testing in Rust: Temporary Files

Fri, 01 Mar 2019 14:30:00 +0100

I recently wrote a tool to manipulate images embedded in mp3 tags: id3-image. To be able to make changes to the code with confidence, I needed tests. Rust comes with the very sensible Write trait that could have allowed me to mock any IO (a blog post for another day), but in this case, the project relied on the excellent id3 crate, so I wasn’t doing any file-writing myself.

Instead, I wanted to have an actual mp3 file and an actual image, apply my code to them, and inspect the results. This generally isn’t difficult to do, but I ran into a few unexpected gotchas, so I’ll summarize how I ended up implementing it.

First Attempt: a Tempdir

I’ve done this often in other projects: Create a temporary directory per-test, change into it, copy some test files, and then work on them as needed. When the test is done, the entire directory can be wiped out.

In Rust, I’d do this with a macro like so:

macro_rules! in_temp_dir {
    ($block:block) => {
        let tmpdir = tempfile::tempdir().unwrap();
        env::set_current_dir(&tmpdir).unwrap();

        $block;
    }
}

The tempfile::tempdir() function comes from the tempfile crate. After unwrapping, I get a TempDir that, when dropped, will remove the directory. Neat.

The next step is copying a test file from a predefined place. In my case, I put some files in tests/fixtures/, so:

macro_rules! fixture {
    ($filename:expr) => {
        let root_dir = &env::var("CARGO_MANIFEST_DIR").expect("$CARGO_MANIFEST_DIR");
        let mut source_path = PathBuf::from(root_dir);
        source_path.push("tests/fixtures");
        source_path.push($filename);

        fs::copy(source_path, $filename).unwrap();
    }
}

It doesn’t have to be a macro, but it means I can be a bit loose about the type of $filename, at least to start. The env var CARGO_MANIFEST_DIR is something I’m just using to get the root of my project. Not sure if there’s a better way, but I always run tests via cargo, so I feel like it should be reasonably reliable.

I could use these two tools to build a simple test that embeds an image and then checks if it worked:

#[test]
fn test_successful_jpeg_image_embedding() {
    in_temp_dir!({
        fixture!("no_image.mp3");
        fixture!("test.jpg");

        // No embedded pictures to begin with:
        let tag = id3::Tag::read_from_path("no_image.mp3").unwrap();
        assert!(tag.pictures().count() == 0);

        embed_image("no_image.mp3", "test.jpg").unwrap();

        // One embedded picture after we've run the function:
        let tag = id3::Tag::read_from_path("no_image.mp3").unwrap();
        assert!(tag.pictures().count() > 0);
    });
}

It’s a readable test, as long as you know what the helper macros do. And it works quite well. The problem shows up when I write more than one of these.

The problem

With several tests, this approach fails:

failures:

---- test_successful_png_image_embedding stdout ----
thread 'test_successful_png_image_embedding' panicked at 'called `Result::unwrap()` on an `Err` value: entity not found', src/libcore/result.rs:997:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

---- test_remove_and_add_image stdout ----
thread 'test_remove_and_add_image' panicked at 'called `Result::unwrap()` on an `Err` value: entity not found', src/libcore/result.rs:997:5

---- test_successful_image_embedding_in_a_file_that_already_has_an_image stdout ----
thread 'test_successful_image_embedding_in_a_file_that_already_has_an_image' panicked at 'called `Result::unwrap()` on an `Err` value: StringError("Error writing image to music file test.mp3: entity not found")', src/libcore/result.rs:997:5


failures:
    test_remove_and_add_image
    test_successful_image_embedding_in_a_file_that_already_has_an_image
    test_successful_png_image_embedding

test result: FAILED. 2 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out

Even more annoying, the failures are random – different tests fail at different times. Usually, when I have problems like this, it’s a timing issue – could it be that filesystem changes are not being persisted? Does adding a few sleeps help?

I’ll spare you the debugging session – the problem is more of a feature:

If you want to control the number of simultaneous running test cases, pass the
`--test-threads` option to the test binaries:

    cargo test -- --test-threads=1

Changing the directory per test doesn’t work, because several tests are setting it basically at the same time. And the current working directory is process-local, not thread-local, so there’s no way to ensure each test thread is isolated.

Files and Paths

So what to do? An option is to run with --test-threads=1. This works just fine, but it seems like a hassle to require users of the tool to run tests in a special way.

Instead of changing directories, we could allocate a temporary path for fixtures instead, and use that:

let song_file  = fixture!("no_image.mp3");
let image_file = fixture!("test.jpg");

The tempfile crate does have a tempfile() function that’s automatically deleted when we have no reference to it anymore. Unlike with tempdir() which returns a custom structure, this gives us a File whose deletion is handled by the operating system.

It doesn’t seem like a bad option. I’d expect I’d be able to do this:

// Spoiler: doesn't compile
embed_image(&song_file.path, &image_file.path).unwrap();

Unfortunately, no:

error[E0609]: no field `path` on type `std::fs::File`

The built-in File type doesn’t give us a way to get its path. As it turns out, not every file needs to have a real path. From a reddit thread:

With from_raw_fd you can have a unix pipe, a standard input/output stream or even a TCP/IP socket wrapped in a File.

Leave it to Rust to illuminate my misconceptions, again. There is a crate that could help with this particular case: filepath. Instead, I figured I’d use TempDir instead and package it up in a more reusable way.

A Reasonable Solution

Let’s start by making our own type, instead of passing around Files or PathBufs:

struct Fixture {
    path: PathBuf,
    source: PathBuf,
    _tempdir: TempDir,
}

The _tempdir field has an underscore, because its only purpose there is to keep ownership of it. That way it’s dropped when the Fixture is dropped. There’s separate source and path, because I’d like to separate creating a Fixture and copying the test file over (for purely pragmatic reasons that’ll be clear later).

Creating a new fixture is a mixture of the previous in_temp_dir! and fixture! macros:

impl Fixture {
    fn blank(fixture_filename: &str) -> Self {
        // First, figure out the right file in `tests/fixtures/`:
        let root_dir = &env::var("CARGO_MANIFEST_DIR").expect("$CARGO_MANIFEST_DIR");
        let mut source = PathBuf::from(root_dir);
        source.push("tests/fixtures");
        source.push(&fixture_filename);

        // The "real" path of the file is going to be under a temporary directory:
        let tempdir = tempfile::tempdir().unwrap();
        let mut path = PathBuf::from(&tempdir.path());
        path.push(&fixture_filename);

        Fixture { _tempdir: tempdir, source, path }
    }
}

So this creates a “blank” fixture – its directory exists and will be cleaned up, and it has a path, but there’s nothing there yet. In order to copy the file, I’ll add one more associated function:

impl Fixture {
    fn copy(fixture_filename: &str) -> Self {
        let fixture = Fixture::blank(fixture_filename);
        fs::copy(&fixture.source, &fixture.path).unwrap();
        fixture
    }
}

This will actually copy the file from the source path into the temporary directory where we’ll be changing it. To make the tests a bit more terse, why not add a Deref implementation and a helper method:

impl Deref for Fixture {
    type Target = Path;

    fn deref(&self) -> &Self::Target {
        self.path.deref()
    }
}

fn read_tag(path: &Path) -> id3::Tag {
    id3::Tag::read_from_path(path).unwrap()
}

Going overboard with derefs is generally discouraged, but this is code that will only be used for testing – we can be a bit more loose with it, throw unwraps around, that sort of thing. Whether it’s actually a good idea depends on personal preference, I feel. Here’s what the test looks like now:

#[test]
fn test_successful_jpeg_image_embedding() {
    let song  = Fixture::copy("no_image.mp3");
    let image = Fixture::copy("test.jpg");

    let tag = read_tag(&song);
    assert!(tag.pictures().count() == 0);

    embed_image(&song, &image).unwrap();

    let tag = read_tag(&song);
    assert!(tag.pictures().count() > 0);
}

I like this quite a lot. It might be that embed_image(&song.path, &image.path) would have been clearer, but if we change embed_image to accept a slightly different argument (a File? A custom type?), we could just Deref to a different thing. So there’s some added flexibility in saying “our custom test-only structure is auto-converted to whatever the code expects”.

As for Fixture::blank, here’s how I use it in a different test:

#[test]
fn test_extracting_a_jpg_image() {
    let song  = Fixture::copy("test.mp3");  // a copied fixture
    let image = Fixture::blank("test.jpg"); // a blank fixture

    // To start:
    //   - the song file has at least one embedded picture
    //   - the image path doesn't correspond to an existing file
    let tag = read_tag(&song);
    assert!(tag.pictures().count() > 0);
    assert!(!image.exists());

    extract_first_image(&song, &image).unwrap();

    // After we've run the code, the image should exist on the filesystem
    assert!(image.exists());
}

So, using TempDir here has the nice benefit that we don’t need to have a real file – we can just keep a placeholder in the system that can be used as an output.

Closing Thoughts

I’m sure there’s other ways to deal with filesystem testing, but this is one that I’m quite happy with and a pattern I’ll likely use elsewhere. If you’d like to see the full test suite, you can check the project out on github.

If you have ideas for improvements on the test suite (or on the code), feel free to open an issue on the repo to start a conversation.

Code Alignment

Sun, 13 Nov 2016 14:30:00 +0100

Okay, this is going to be a bit bikesheddy. It’s about a code style rule that some people love and some people hate, and I have an ✨opinion✨ about. It’s not a particularly strong opinion, as in, if the team I’m working with disagrees with me, I have no issue following the popular vote. It’s a small enough thing that it might not matter in the long term.

Let’s start by examining some code:

describe Registration do
  it "disallows duplicates" do
    create(:registration, uid: '123', provider: 'twitter')

    first  = build(:registration, provider: 'twitter',  uid: '123')
    second = build(:registration, provider: 'twitter',  uid: '234')
    third  = build(:registration, provider: 'facebook', uid: '123')

    expect(first).not_to be_valid

    expect(second).to be_valid
    expect(third).to be_valid
  end
end

The three variables, first, second, and third, are aligned by the = sign, and the method invocations on the right are aligned by commas. Here are the benefits I see with this:

On the left side, you can easily scan variable names. If the body of the code that uses these variables is complicated, you can index it visually – you look at the variable on the left side, you check what you have on the right. It’s very rare that I read variable names like normal text – left-to-right, top-to-bottom. I just scan them quickly and then consult the “index” as I’m reading the body of the code.
On the right side, we have some very homogeneous invocations of the build method. The thing we’re interested in the most are the differences between them (what is the provider, what is the uid?) and when they’re ordered like this, it’s easy to see them at a glance.

This might not be too obvious in a short piece of code, but with larger chunks of text, it gets more and more useful. Arguably, you don’t want to end up with large chunks of code, but it depends. For instance:

$icon-arrow-thick-right: "\e600";
$icon-arrow-thick-left: "\e601";
$icon-arrow-thick-up: "\e602";
$icon-arrow-thick-down: "\e603";
$icon-arrow-thick-b-right: "\e604";
$icon-arrow-thick-b-up: "\e605";
$icon-arrow-thick-b-left: "\e606";
$icon-arrow-thick-b-down: "\e607";
$icon-arrow-thin-right: "\e608";
$icon-arrow-thin-up: "\e609";
$icon-arrow-thin-left: "\e60a";

… and another 400 lines of the same. Compare to:

$icon-arrow-thick-right:   "\e600";
$icon-arrow-thick-left:    "\e601";
$icon-arrow-thick-up:      "\e602";
$icon-arrow-thick-down:    "\e603";
$icon-arrow-thick-b-right: "\e604";
$icon-arrow-thick-b-up:    "\e605";
$icon-arrow-thick-b-left:  "\e606";
$icon-arrow-thick-b-down:  "\e607";
$icon-arrow-thin-right:    "\e608";
$icon-arrow-thin-up:       "\e609";
$icon-arrow-thin-left:     "\e60a";

This isn’t exactly “code” – it’s a table of contents of sorts. As such, it’s super useful to have it aligned. Just imagine a book where the index is described without alignment and how annoying it would be to orient yourself by pages/chapters. Again, on the left side you have an easily-scannable list, on the right, you have some very homogeneous content where you can easily notice the differences between lines (so you can filter the similarities out).

CSS declarations are an interesting counter-example. You rarely want to scan “labels” on the left side. The fact that one CSS selector uses margin and padding isn’t as important as the values of those fields. And on the right side, you can have some very heterogenous content – pixels, pairs of numbers, quadruplets of numbers, text:

.something {
  margin-top: 20px;
  padding: 0 20px;
  text-align: right;
}

Probably not much value in aligning it. There are some exceptions, like:

dl {
  -webkit-margin-before: 4px;
  -webkit-margin-after:  4px;
          margin-before: 4px;
          margin-after:  4px;
}

But this one can be hard to maintain automatically. I might do it if it’s isolated in a mixin, simply because it would be read far more often that it would be updated.

Drawbacks

Code like this can be more difficult to edit. You need to have an editor tool to help you (like this, or this, or this), and I think many people don’t, and align the code manually. Which doesn’t seem like a great idea. I don’t feel it’s worth enough to waste energy on it if your and your teammates’ editors don’t have the tooling to do it easily.

Then there’s the issue with VCS logs. Every alignment change due to a new item means touching a lot of other items. Mind you, with git diff, there’s the -b/--ignore-space-change flag, so it might not be a huge problem in practice.

Another potential problem is large differences in lengths. For instance:

one                          = 1
two                          = 2
three                        = 3
forty_two_hundred_and_twenty = 4220

It’s now significantly harder to make the connection between one and 1, because they’re quite far away from each other. My solution in situations like this is simple:

one   = 1
two   = 2
three = 3

forty_two_hundred_and_twenty = 4220
thirty_five_thousand         = 35_000

You might want to ask yourself why some variables are a single word, and others are an entire phrase. I find this kind of inconsistency to the amount of detail can hurt comprehension quite a lot. Maybe there’s a name you could introduce for a new concept? It’s not a “solution” to the alignment problem, all I’m saying is, it might be a minor smell to think about.

I’ve also heard the argument that search-and-replace might end up being harder. I’m not convinced it’s a big issue, to be honest. Instead of looking for foobar =, you’d have to look for foobar\s\+=. It’s rare that only this one pattern will be enough to get things done anyway, and it’s very likely you’ll need manual intervention whatever set of patterns and replacements you come up with.

Is it worth it?

Depends. When you have a bunch of data with entries that have a very similar structure, I’d certainly want it to be aligned. Think the SCSS example above, or a database schema, or something like this:

payment_plans_for_select_box = [
  {label: 'Free',           value: 'free'},
  {label: 'Business',       value: 'business'},
  {label: 'Premium',        value: 'premium'},
  {label: 'SuperMega Plan', value: 'super_mega'},
]

I’d also go for it in the testing case, where you often have a setup phase with several invocations of the same functions with different parameters. For anything else, it depends, and it seems like mostly personal preference. The “similar enough structure” idea is my key guiding point, in all cases.

It’s tricky to apply, because it’s very contextual. How much is “similar enough”? Difficult to say. And it’s definitely not something that an automated tool can determine (at this time). It’s one of the reasons I dislike gofmt, it removes any alignment even when there would be a very noticeable benefit from it.

Update: Turns out, not exactly. Gofmt removes alignment from, say, multiple var lines, but if they’re grouped in a single var ( ) statement, it actually aligns them. Which makes sense, I suppose – it provides a way for the developer to communicate “this is a group with similarities”, and the tool responds to it.

Either way, whether you decide to apply this rule or not, it’s a good idea to know why, in more concrete terms than “it feels right/wrong”. I hope I’ve given you a possible answer to this question.

Building a Better gf Mapping

Wed, 09 Mar 2016 11:44:00 +0100

I recently started working with ember.js. It’s a nice framework, but, like most newish technologies, Vim support is minimal. So, as I usually do in such cases, I started working on some tools to help me out with navigation. Tim Pope’s projectionist helped a lot, but I wanted more, so I started building it up in what would later be published as ember_tools.

The biggest feature was the gf mapping (“go to file”), which was inspired by the one in vim-rails. Using gf in a rails project almost always does “the right thing” for whatever you can think of. So, I poked around in that one, tried to figure out how it was implemented. In the process, I learned a few things and had a bunch of pretty good ideas, which I’ll talk about in this blog post.

Some basics

Before I start with the interesting stuff, let me explain a few basics you might need to know in order to understand it. Feel free to skip this section if you’re feeling confident in your Vimscript skills.

First, the way you ordinarily customize the gf mapping is not by directly remapping it, but by modifying the “include expression”, or includeexpr. It’s an expression that gets evaluated every time you press gf or <c-w>f or anything in that family of mappings. In this particular scenario, things are implemented a bit differently, but it’s useful to know the concept.

A different setting that modifies some core Vim behaviour is iskeyword. It contains a comma-separated list of all (groups of) keys that are considered part of a “word”. This affects what pressing w does, it affects expand('<cword>'), syntax highlighting, all sorts of things. It’s generally pretty risky to try changing it from its default. Recent versions of Vim allow some finer-grained control over some of this behaviour, but this one setting remains a powerful and dangerous tool that we’re going to use in a very careful way later.

On a completely unrelated note, autocommands are sort of like Vim events. You can listen to an autocommand like BufWrite (when Vim writes the buffer to a file), or InsertLeave (when Vim exits insert mode) to run any old Vim command at the right time.

And the last concept that you’ll need to know about, is the execute command, often shortened to exe. It’s sort of like eval in most languages, but it doesn’t allow you to evaluate a random expression, only a command. For instance, if you have some sequence of keypresses that you’ve been building up, like

let keypress_sequence = number_of_steps_to_the_right . "l"

You can’t just run the sequence with normal! keypress_sequence. That would run the literal key sequence “keypress_sequence”. Instead, you can run exe "normal! ".keypress_sequence to evaluate the variable keypress_sequence to its string value and execute the normal! command with the right thing.

And, if you didn’t figure it out from the last paragraph, in Vimscript, the dot is used for string concatenation, so:

let a_hundred = "100"
let a_hundred_plus_one = a_hundred . " + 1"

echo a_hundred_plus_one
" -> 100 + 1

I can’t guarantee you won’t have to reach for :help in the following paragraphs, but hopefully this should be enough. And with that out of the way…

How it works in vim-rails

If you poke around enough in vim-rails’ source code, looking for how gf works, you’ll likely end up in a place that has the following line:

cmap <buffer><script><expr> <Plug><cfile> rails#cfile('delegate')

It maps the (fake) key sequence <Plug><cfile> to the expression rails#cfile('delegate'). Yes, <Plug> is a “key” of sorts. If you look at :help <Plug>, you’ll discover that it “can be used for an internal mapping, which is not to be matched with any key sequence”. If you follow some links, you’ll also find it’s “a special code that a typed key will never produce”.

In general, <cfile> is something that expands on the command line to the filename under the cursor, based on Vim’s built-in rules for recognizing a filename. What Tim Pope does here is create a new expression, <Plug><cfile>, which expands to whatever rails#cfile('delegate') returns.

That last function is the one that actually does the heavy lifting of figuring out how the stuff under the cursor maps to an actual file. So that, in vim-rails, calling :echo rails#cfile('delegate') with the cursor on the word “user”, would output “app/models/user.rb”.

The expression <Plug><cfile> is mapped to that value on the command-line with a cmap. The following two lines hook this up to the gf mapping:

nmap <buffer><silent> <Plug>RailsFind <SID>:find <Plug><cfile><CR>
" ... snip ...
nmap <buffer> gf <Plug>RailsFind

This creates a normal-mode mapping for <Plug>RailsFind, just in case somebody wants to use a different mapping for it. What it actually does is call the :find command with the right filename. Interestingly enough, the :find command can find all kinds of files in the path setting, so there might be some value in playing around with that one… But that’s not something I’ve experimented with yet.

The actual gf mapping just delegates to the plug-mapping. There’s a few other mappings around that area that set things up for <c-w>f and similar other mappings, but they work roughly the same way.

So why is this interesting for ember tools? Well, a tiny problem I happened to have is that my ember app was in a subdirectory of a rails app. This would activate vim-rails for my ember files, and that means activating rails’ gf. In order to avoid something like this happening, I added this little workaround to ember_tools:

" Override gf if rails sets it after us
autocmd User Rails cmap <buffer><expr> <Plug><cfile> ember_tools#Includeexpr()

Which I find pretty clever! See, thanks to Tim Pope’s <Plug><cfile> setup, I only need to override that one mapping to return the results of calling my own function. That function can, at any point, return rails#cfile('delegate') to fall back to the default vim-rails behaviour. All the other related mappings work just fine. It’s an interesting way to achieve this without a complicated combination of settings.

This also opens up some possibilities to extend vim-rails’ gf mapping! As powerful as it is, there’s a few improvements I wanted to do there as well… I started work on this in a script I called rails_extra. Currently, it’s a mess of functions from various of my plugins, and it’s not organized particularly well. It works wonders, though. I’ve made gf follow custom rspec matchers, look for factories, go to required assets in the asset pipeline. My favorite extension, though, is gf on translation keys, which I’ll talk about next.

Finding translations

Internationalization has always been a pain for me to keep track of. It’s a necessity, but it does complicate things a lot and I often wish I had better tooling for it. For ember, I use the ember-i18n project, and I hook up a helper to be able to do something like:

{{i18n 'some.nested.translation_key'}}

So how do I set things up so I can gf on the key and end up on the translation? The tricky bit here is not the actual file. If I can detect I’m on a translation key, I can easily say that the file to be returned is, in my case, ../config/en.yml. (In this particular project, the translations are the same ones defined for the rails project, which happens to contain my ember one.)

Detecting the key might look like this:

function! EmberGfTranslations()
  " ... snip ...
  if !ember_tools#search#UnderCursor('i18n [''"]\zs\k\+[''"]')
    return ''
  endif

  let translation_key = expand('<cword>')
  " ... snip ...
endfunction

I’m not going to explain ember_tools#search#UnderCursor, because it would take me another full blog post to go over all the weird logic in there. Suffice it to say, it works reasonably well for checking whether the cursor is somewhere on text that matches this pattern and moving said cursor to the beginning of it. In our case, due to the \zs, the beginning happens to be right on top of the translation key.

Note, by the way, that the pattern matches \k\+ (in quotes). The \k pattern matches “keyword” arguments, and, ordinarily, the . character is not one of them (it causes way more trouble than it’s worth). But in most ember identifiers I care for, ., -, and / are perfectly valid. Think of components, for instance – {{user/profile-picture}}. So I decided I can just do something like this:

let saved_iskeyword  = &iskeyword

let callbacks = [
      \ ...
      \ 'EmberGfImports',
      \ 'EmberGfTranslations',
      \ 'EmberGfRoutes',
      \ ...
      \ ]

for callback in callbacks
  try
    set iskeyword+=.,-,/
    " look for the right callback to invoke
  finally
    " whatever happens, just restore the old value of `iskeyword`
    let &iskeyword = saved_iskeyword
  endtry
endfor

Basically, I just add everything I need to iskeyword, and restore it afterwards. No harm, no foul.

Anyway, after storing the translation_key I found under ther cursor, I can split it by dots to get the list of sections in the rails yaml file. In the example, I’d have a list like ['some', 'nested', 'translation_key']. So what now?

The way the gf mapping works, we can only have our custom include expression return a file path. In our case, the file is easy – the big yaml file with translations, config/locales/en.yml. But how do we make Vim jump to the right place where the translation is found?

Well, the solution I came up with… I’m honestly kind of surprised that it works. Let me start a new section for that.

When you have a problem, just put :exe until it’s fixed

Imagine you want to set things up so that the next time a particular file is opened, a particular command is run, only once. It might look something like this:

autocmd BufEnter /some/file/path :echo "foo"
autocmd BufEnter /some/file/path :call ClearFileOpenCallback() " how to do this?

The first autocommand executes the thing we need, the second one clears that callback. How can we clear the callbacks? A simple way is to group them with a specific name. Here’s how that might look:

function SetCallbacks()
  augroup file_open_callback
    autocmd!

    " set up a bunch of callbacks
  augroup END
endfunction

function ClearCallbacks()
  augroup file_open_callback
    autocmd!
  augroup END
endfunction

By naming a group called file_open_callback, we can refer to this temporary autocommand group. The initial autocmd! in the beginning of the group clears everything that’s ever been defined in it. That’s why our “clear” function is just the group and the clearing command.

So the setup makes sense – one function sets a callback, which, after it’s done with its work, triggers the other function. Said function clears the callback – and that’s how you get a one-shot autocommand.

Now, the question is, how do we set up the autocommand so that it searches for the pattern we give it? Just put :exe until it works:

function! ember_tools#SetFileOpenCallback(filename, ...)
  " all the words to search for, provided as variable arguments:
  let searches = a:000

  " use absolute paths:
  let filename = fnamemodify(a:filename, ':p')

  augroup ember_tools_file_open_callback
    autocmd!

    " start by going to the top of the file:
    exe 'autocmd BufEnter '.filename.' normal! gg'

    " search for every pattern, one after the other:
    for pattern in searches
      exe 'autocmd BufEnter '.filename.' call search("'.escape(pattern, '"\').'")'
    endfor

    " clear all callbacks, so we leave it a one-shot thing:
    exe 'autocmd BufEnter '.filename.' call ember_tools#ClearFileOpenCallback()'
  augroup END
endfunction

function! ember_tools#ClearFileOpenCallback()
  augroup ember_tools_file_open_callback
    autocmd!
  augroup END
endfunction

The exe calls just evaluate down to autocommands keyed to that particular file. Each of the steps is triggered only on BufEnter of this one particular file, and they run in a sequence. Once they’re all called, we can clear the callbacks.

After getting the translation keys in an array, we can split up the keys and search for them in sequence with (something similar to):

:call ember_tools#SetFileOpenCallback('config/locales/en.yml', 'some', 'nested', 'translation_key')

This ends up creating a list of autocommands that looks like this:

autocmd BufEnter /path/to/config/locales/en.yml normal! gg
autocmd BufEnter /path/to/config/locales/en.yml call search("some")
autocmd BufEnter /path/to/config/locales/en.yml call search("nested")
autocmd BufEnter /path/to/config/locales/en.yml call search("translation_key")
autocmd BufEnter /path/to/config/locales/en.yml call ember_tools#ClearFileOpenCallback()

After that, we return config/locales/en.yml and the autocommands kick in, position the cursor in the right place, and then unset themselves. Not that complicated, even if it is kind of hacky. So the function that takes care of going to the translations ends up looking like this:

function! EmberGfTranslations()
  " ... snip ...
  if !ember_tools#search#UnderCursor('i18n [''"]\zs\k\+[''"]')
    return ''
  endif

  let translation_key = expand('<cword>')
  let translations_file = fnamemodify('config/locales/en.yml', ':p')

  " Prepare the arguments to the file open callback function:
  "   - The filename to fire on
  "   - The pieces of the key, searched one by one
  "
  let callback_args = [translations_file]
  call extend(callback_args, split(translation_key, '\.'))

  " The "call" function calls a function. Vimscript can be an odd language
  " sometimes.
  call call('ember_tools#SetFileOpenCallback', callback_args)

  " After we've set up the autocommands to fire when the file is opened, we
  " return the file path and let Vim take care of the actually opening it (in
  " a split, or tab, or whatever, depending on the mapping that was invoked)
  return translations_file
endfunction

There’s a few Vimscript-y weirdnesses in there, but hopefully, it should be readable enough as it is.

To Sum Up

The gf mapping is a useful abstraction to navigate your code, and extending it yourself is not super difficult to do. You take the text under the cursor, pick it apart with regexes and figure out what the path you’re looking for is. You do need some background knowledge, but the :help files are always nearby to help you out. If you’re working on a particular framework that doesn’t have special support for it, consider writing your own.

If you’ve never written Vimscript so far, it could be a nice introduction that’ll give you a good productivity boost. Hope this article has given you some basic tools to get you started, and some clever ideas to inspire you to keep going.

The New(ish) Facebook Authentication Process

Fri, 06 Mar 2015 15:18:00 +0100

Facebook changed their authentication process fairly recently. Instead of using a global user id, users would now get an app-specific id instead. This took me by surprise, and I had to jump through some hoops to get stuff to work with our setup. This is a short post that outlines what we did to work around the problem.

The Good Old Way

The way things worked in The Old Times, you had a facebook app, authenticated with it, got a user id and saved it to the database. Upon the next facebook authentication, you’d get the same id and find it in the database, logging the relevant user in. Usually, you’d maintain multiple different apps for production, development, staging, and whatever other domain you need, since apps were only limited to a single domain.

For rails developers, Omniauth helped a lot. With that, all you needed to do was hook up a link to the right url. Sending the user to it would redirect them to facebook, log them in there, and provide you the id in request.env['omniauth.auth'] under the key :uid.

The Changes

Of course, with app-local ids, this gets a bit more complicated. Having three different apps no longer works well, since you have different ids for the same users. Putting them in the database means users can only log in with the facebook app they used to sign up. If your development database is a clone of production, it’s no longer easy to do tests on your local machine.

The solution in this case is using test apps. A test app is created from a normal app, has different access credentials, but the same user ids as its original. It’s not public, so you can’t cheat the system by using it in production, but you can use it for testing purposes on a domain other than your main one.

Facebook for Business

But what if you have a legitimate need for multiple domains in production? At Knowable, we had both “knowable.org” and “joinpolymer.com”, since we had two separately branded products backed by the same database. “Polymer” was our new project, and we wanted to introduce social logins to it. That’s when we noticed the app id changes. We couldn’t use test apps for the purpose, because they’re not public. We couldn’t use the same facebook app for both domains, either. So what now?

The solution was “facebook for business”. This was probably created for a completely different purpose, but it also happens to be the offical way to work around this issue. If you create a “facebook business” and you claim both apps as belonging to it (you just need to be an admin to do that), you can request an additional field from facebook called token_for_business (documentation). This is a string that’s the same across all apps of that business, unique per user. This means that storing that in the database lets you use it for authentication alongside the id they give you.

As far as I know, there’s no way to pull this off easily in Omniauth. We needed to fetch it in a separate query to facebook using the koala gem:

def get_facebook_business_token(omniauth)
  graph = Koala::Facebook::API.new(omniauth['credentials']['token'])
  profile = graph.get_object('me', fields: 'token_for_business')
  profile.fetch('token_for_business')
end

Usually, it’d be enough to look for the user/authentication with the recommended Omniauth oneliner:

auth = find_by(provider: omniauth['provider'], uid: omniauth['uid'])

In case this didn’t work and the provider is “facebook”, you’d simply get the business token from facebook and try fetching the authentication with that as well:

business_token = get_facebook_business_token(omniauth)
auth = find_by(provider: 'facebook', facebook_business_token: business_token)

Not ideal, given that we need to put extra fields on the authentication record specifically for facebook, but it works fine for this one special case.

Impact

I’m slightly surprised that omniauth-facebook doesn’t provide something for this, but I’m assuming it’s a bit of an edge case to need two separate apps on two separate domains. And there’s no point attempting to fetch the business token for “normal” apps.

Another interesting thing that we discovered is that facebook no longer provides a username, either. And, a friend working in a different company told me they may even not provide an email, if the user happens to not have confirmed their email to facebook. It gets harder and harder to create a fully functional user out of facebook data, but I suppose their changes are mostly geared towards protecting users’ privacy. Hard to say how to handle these cases, though.

Small Refactorings

Tue, 29 Apr 2014 20:46:00 +0200

A common piece of advice for dealing with bad rails code is to avoid making huge rewrites all at once. Instead, make small fixes whenever you see a problem. An often-cited variant of this is the “boyscout rule” – leave the code better than you found it.

This is solid advice. Putting hacks and workarounds would only make things worse. Rewriting swaths of code for even small features would take a long amount of time and introduce risk. The best approach is to make small refactorings and build new features with new, better, code.

As with most good advice, this can still backfire. Ever seen an app with five different caching mechanisms and seven different event tracking approaches, all slightly different? This may have been a result of multiple developers going “this is horrible, I’ll fix it”. In the end, they created an even larger mess, despite their good intentions. How did this happen?

Let’s start with a different question.

What is a “good” refactoring?

In the ruby community, we care a lot about good code. We have blog posts, conference talks, and many books on the topic. The community still gets a lot of newcomers and we try to educate them on the importance of structured, well-tested code.

Unfortunately, even if you took two experienced rails developers and put them in front of bad code, they’d probably disagree on the solution.

Some would follow The Rails Way, trusting themselves to the framework and its well-structured conventions, others would insist on extracting the logic in service objects, separate engines, or microservices.
Some would partition the code into many extremely small classes, others would call this “overcomplicating” and opt for a more coarse approach.
Some would aggressively monitor and improve the performance of the app, others would scoff at the idea, claiming it’s a problem to be solved at a later time.

If you’ve identified with one half of the statements, I’m willing to bet there are rails developers out there that take the exact opposite stances and still create well-working apps. As much as we wish there was only one good way of doing things, that doesn’t seem to be the case. Much of our decisions stem not from reason, but from our backgrounds, previous languages, what worked for us and what didn’t. This becomes a particular problem in larger teams working on separate parts of the application.

Good developers doing bad things

Note that I’m still talking about “good” developers. By “good” in this case, I mean they genuinely believe in writing readable, maintainable code, they have good intentions and try to follow basic programming principles. Still, it’s not surprising for them to end up with a mess, especially when working with legacy code.

Unfinished business

Let’s look at a more concrete use case. A developer needs to make a change to existing functionality in the big old legacy app. The logic ends up being in one 300-line method call that mutates global variables, accesses information from constants set in initializers and responds to three optional boolean flags that contain double negatives in their usage (dont_add_extra_taxes: false). The developer shudders in horror, and decides to get rid of this monstrosity, so they create a brand new class for the changed functionality, and use that one instead.

This is the best solution at this point in time. Adding even more hacks to this large method is a terrible option. And rewriting it would require finding all the ways it’s being used and thinking of a good way to reimplement them. Which would take a long time and hold considerable risk of errors. It’s much better to get the immediate work out of the way and then find some time to take care of the rest of the refactoring.

The problem is, there are now two implementations of logic that’s only slightly different. The old method still exists. Even if the developer says they’ll “fix it later”, that “later” might never arrive. Depending on the development processes and company culture, it could be difficult to allocate time for this kind of refactoring. Especially since it doesn’t seem to hold any value – the functionality is there and it works, right? There’s no immediate rush. As a result, that developer might even leave the company before the time they get around to it. I’ve seen quite a few cases of code that was clearly meant to be part of a larger refactoring… By people no longer working there. Heck, I’m guilty of being that person myself.

So many people

Still, I said we’re talking about a big team, right? There’s lots of good developers in this imaginary company we’re looking at. Wouldn’t somebody else pick up the torch and finish the cleanup? They will, if they know about it and agree with it. If another developer has to make a change that ends up in that same terrible method, it’s not clear whether they’ll find the previous one’s attempt to reimplement it. Even if they do, it’s unlikely they’ll adapt this solution to fit their different use case, since that may involve quite some work as well. Not to mention what I previously talked about: different developers have different ideas of “good code”. They may even find the rewrite attempt and decide it’s not really that good and they can do a better job.

In any case, it’s easy to repeat the previous process: Just create a new class that handles this use case.

In the end, everybody that touches that code ends up reimplementing a small part of it, but without a coherent vision of where it should end up, we’re left with a lot of subtle duplication and the original code keeps chugging along untouched.

Limited scope

Consistency is key here. When we work on a particular feature, it’s easy to get tunnel vision and see everything in light of this one particular use case. Looking at the original code and all of its quirks, it’s really easy to dismiss them as irrelevant. And, for the limited scope we need it for, that may be true. But the dont_add_extra_taxes option was created because of some particular feature. Understanding all those features is the only way towards really rewriting the code.

This could, and should, happen gradually, since it would be practically impossible to figure it all out without trial and error. It often ends up, however, that “gradually” means “write this one use case and then drop it”. Continuing to reimplement everything may require re-rewriting the new code, or even deleting it and starting over. This is difficult to do, especially when it doesn’t feel like that work is needed at the time. When you repeat this process for multiple developers, all of them working on the code at different times and in different contexts, it’s no surprise that the end result is an incoherent mess.

Company culture

As a result of all of the above, some teams even end up being paranoid of “refactoring” (in quotes). When too many developers have tried to refactor the code and either broke production or made things worse in the long run, the atmosphere deteriorates. It gets more and more difficult to refactor and it often has to be done quickly and sloppily between tasks. This itself leads to more problems, which closes the downward spiral.

This is a difficult environment. A nice blog post that tackles the topic is “Refactoring: The Human Factor” by Andrzej Krzywda.

Solutions

So if small, gradual refactorings end up causing all these problems, which path should we take instead? Keep hacking up the old method, making it larger and even more complex? Stop working on features and devote a week or two for a complete rewrite of this one method?

Both of these sound ridiculous, and for good reason. Small refactorings are still our best tool to deal with these situations. I’d say we only need to be aware of the limitations of the approach and figure out how to mitigate them.

Planning

One of the simplest things you can do, before jumping into refactoring, is think. Our first instinct can often be wrong and inconsistent with the rest of the app. Once we’ve started down some particular path, it’s difficult to turn around and make different choices, so we often push forward, making things somehow fit in a kludgy way.

It would be best to avoid this from the very beginning. Think about the effect of the refactoring and how it’s going to fit with the system. Consider what new abstractions have to be created and how these will affect the architecture. Make drawings, random notes, mind maps, but get a good, clear picture of where you’re going before heading out.

Even so, it’s almost certain you’ll make mistakes. Be prepared to adjust your vision and iterate on the refactoring multiple times. The fact that it’s new code doesn’t automatically make it good, especially if it’s introducing new structure that may or may not work well with the existing one.

Communication

Another somewhat obvious way to avoid all these issues is to have a good, clear line of communication between developers. If you’re going deep into the bowels of bad code, it pays off to ask around for help. Maybe someone has already been there and started working on a replacement. If you can take that and extend it, or build something else with a consistent interface, you’re on the right track towards finally getting rid of the big bad method.

This can be hard to do, but if your team has no shared understanding of features, you’ll end up having trouble sooner or later. Share knowledge in standups, pair with people on tricky problems, ask for advice. Do your best to spread the knowledge around and it’ll pay off pretty soon.

Patience

Rash behaviour is especially common in newbie developers, possibly just out of university. They often attempt to apply their fresh knowledge to fix the codebase without taking the time to understand it well enough. Patience is key. It’s important not to go overboard with cleanup while you’re still unfamiliar with the code, unless it’s a simple fix or you’re pairing with another team member.

Of course, it’s also important not to wait too long. When you stare into the heart of legacy code, the legacy code stares back. Even if the convoluted logic was written this way for a good reason, that doesn’t make it good code. Understand what it does, then figure out how to replicate the functionality in a better way.

Persistence

The final trick is to persist. With rails apps, it’s usually not that difficult to understand any given piece of functionality. You can use a pen and paper to chart the interactions, you can put debug statements in strategic places, but eventually, if you persist, you’ll figure out what the code does and how it does it (and feel very clever as a result). The trick is finding the time and energy to do so.

Only several times in my career have I had the opportunity to really sit down and devote some undisturbed time to understand a large piece of code and rewrite it. These moments are rare and usually happen for odd reasons, but when they do, be sure to make good use of them. Otherwise, you could timebox yourself to, say, half an hour every day of trying to understand this one method. Put comments in it, describing what you know of it so far, or just write them down on paper. Build hypotheses, validate or reject them. With patience, you’ll eventually get to know every dark corner and you can then figure out an architecture that implements the same features in a better way.

It’s not easy, but you have to pull the trigger at some point. You can skirt around on the edges of the method, you can wrap the old code in a new interface, but it will never really be gone until you dig deep into it and figure it out. By all means, finish your feature in a separate class first, get that out of the way and move the ticket to “Done”. But be sure to create a “Delete bad code” ticket and genuinely make the act of deletion a goal, in your head. Only when you meet that goal can you really pat yourself on the back for doing an excellent refactoring job.

Conclusion

One of the best ways to get rid of legacy code is to make small improvements. New features or changes should not be done in the old code, but should be created in new classes, with clean interfaces.

However, this process relies on the assumption that you will eventually delete or fix the old code. Failing to do that is going to leave you with an ever-expanding app with no prospects of real improvement. It also doesn’t absolve you from thinking about the architecture of the new code and communicating with your teammates. Creating new components without a clear vision and without adapting them when needed only leads to more chaos.

For a nice parable that tackles a similar topic, take a look at The Codeless Code.

The "Back to School" idea

Sun, 20 Oct 2013 12:10:00 +0200

OHM was a place full of amazing ideas and clever hacks. In this atmosphere, a friend and I started talking about our education system and the age-old promise of “multimedia” in the classroom. It’s a popular topic these days, education. Startups like coursera are providing free online lessons, clever tools like geddit are trying to improve teacher-student interaction. What Vloo and me discussed was using presentation slides in class in order to improve engagement.

Something like this is already in use in my home country of Bulgaria. There are specialized classrooms equipped with computers and projectors, and every once in a while, the students have an “interactive class”, where they watch videos and perform quizzes. However, this is something that happens rarely and is considered a “special” lesson. I was more interested in turning any lesson into an actual lecture. Just as lecturers at tech conferences teach me about the latest, greatest web framework, it should be perfectly possible to use the same tools for a chemistry, or biology, or history lecture.

As we were describing what a lesson would look like, we realized we could actually prepare one. We dubbed the event “Back to School” and I started working on a chemistry lecture to present at the local hackerspace, initLab.

Slides Make Everything Better

The lecture itself was nothing amazing. I talked about elementary particles, covalent bonds, electron clouds (slides in Bulgarian). My lack of deep knowledge in the subject made me a bit uncomfortable and it showed, but it was alright as a whole. The more interesting part was the meta-lecture (slides in Bulgarian), in which I talked about my motivations and prompted a discussion. As expected, people had much to say. Us techies have this tendency to apply technical solutions to all sorts of problems. While it may be misguided sometimes, I think it’s ultimately a good thing. Education is one area that we can throw a lot of tech at, and the amount of startups in the area shows.

My simple idea is inspired greatly by lectures on programming. Go to any conference talk and you’ll see slides. Slides are the natural companion to words, the illustrations, the replacement for a chaotic whiteboard when we know what we’re talking about. Personally, I love this. Don’t get me wrong, I hate having to prepare slides, but I can see the value they bring to a lecture. In fact, I fully believe that, in the future, when everybody has a brain-installed computer, we’ll all have the ability to project ad-hoc holographic illustrations of whatever we’re talking about right now, making every interaction better.

But even if you don’t subscribe to weird future speculations, it’s hard to deny that the occasional illustration helps when trying to figure out something complicated. I tried to distill what is it that helped me out in this particular talk:

Summaries: A single word or phrase on a blank slide works wonders as a mental anchor. The lecturer may be going on and on about the details, which is why having the topic right in front of the audience helps in anchoring to the general idea.
Illustrations: Depending on what you’re learning, seeing a simple illustration of it could make all the difference. Imagination only goes so far when it comes to, say, the shapes of different electron orbitals.
Review: After every section, it’s easy to have a quick review of the main points with a set of bullets. It’s also easy to move back and forth: “Remember the Bohr model? Now that you know about electron orbitals, let’s talk about it again”. Not only is the repetition itself good for remembering, it’s also easier to make the connections between ideas that looked unrelated a moment ago.
Flow: Teachers follow some plan while talking, going through a particular string of ideas in a predetermined way. Having the plan set in the slides itself, it’s easy to stick to it and avoid distractions. And, with a good enough flow, it’s possible to suck the listener in, giving them a ride from idea to idea, building nice little bridges between them. I still remember fondly a series of algebra lessons in university, seemingly unrelated, culminating in the proof of a theorem that elegantly used every bit of knowledge we’d gained from the start of the semester. It was like watching the exciting reveal to a detective story.

The best thing about presentations is that it’s not even hard to make useful slides. Note that I don’t say “great” or even “good”, just “useful”. You don’t have to weave a complex tale with eye-catching illustrations and hilarious jokes to make a set of useful slides. There are a few simple rules I can think of that need to be adhered to:

Little text and/or illustrations per slide: You don’t want to write a novel on a slide, and you don’t want to overload it with imagery. One phrase per slide is good. One title and 4-5 one-phrase bullets in a slide is good. One illustration in a slide is good.
Step-by-step reveals: Making all bullets in a slide visible as you’re talking about them is a big no-no. People will read ahead and will start thinking about what comes next. In the meantime, they will have a hard time listening to you. Showing things gradually keeps the flow going and anchors the screen to your words.
Little or no effects: This one should be a given. Fancy zooms and slides have been out of fashion pretty much since they were invented. The only animation you need is “Appear”.

Naturally, a good presenter may break the rules, as long as they have a good reason. But for a novice, just sticking to these constraints means making a very usable slideshow with not a whole lot of effort.

Education

Globally, there’s a ton of complaints about the current education system and a lot of drive to innovate it. Above, I mentioned coursera and geddit. What they do is provide children and adults with better ways to actively pursue better education. Remember the “actively” bit there, I’ll be back to it in soon.

These two startups (most startups, I beleive) are somewhat oriented towards places that can afford this. For instance, geddit assumes that everyone in the classroom will have a smartphone or tablet. While there have been campaigns like One Laptop Per Child that aim to equip poorer classrooms with technology, we can’t forget that there are many schools that still don’t have the possibility of providing for every student. There is still hope, though.

Sugata Mitra’s “The Child-driven Education” talk demonstrates an experiment that tries to bring quality education to even the most remote regions by leveraging children’s natural curiosity. It uses very little technology (a single computer in a wall, one computer per 4 children), but due to the format of the study groups, this works out quite nicely.

Motivation

One problem that doesn’t seem to be addressed enough is the lack of motivation. Most startups provide amazing ways for people to learn new things, provided they actively allocate time and effort for it. Even Sugata Mitra’s experiments rely on children being motivated to learn, though it does attempt to “trick” them into learning by communicating with each other.

However, if children have no desire to learn, there’s not much that can be done. In my experience, school has always had a culture of “learning is uncool”. The cool kids skip class and smoke behind school, they mess with the teachers, they play pranks. Students that actually get good grades are stigmatized as “nerds” and are pushed to underperform in order to avoid being ostracized. Adding to that, teachers lose motivation when they’re constantly heckled and ignored. They may become bitter and resentful which serves even more to damage the learning environment. This is an especially big problem with young teachers who don’t have the inherent authority that comes with age.

In Bulgaria, student protection laws tie up teachers’ hands. A teacher may never order a student to leave the classroom, regardless of how they behave. A poor grade may only be written upon examination, but if the teacher tries to examine a student, they may play dumb and then dispute the grade, wasting half of the class’ time in the process. Aside from that, schools are stimulated to accept students regardless of their grades, since a lack of students will mean being financed with less money from the government. In the end, this means that it’s easy to just stroll through school without learning anything, provided you’re arrogant enough to “game the system”.

Solutions?

It’s really hard to motivate people. The only solution I can think of is finding ways to stimulate learning from an early age and fight the “too cool for school” culture. More and more people are becoming self-learners as adults, exploring online courses and youtube videos, and this is something that should be pushed to children as well. Educational shows like the old “Bill Nye the Science Guy” have potential for this, but if lessons at school are not engaging, only these shows on the side will always interest only a minority of kids.

The idea of using presentation slides is somewhat targeted to improve motivation. I don’t think that the classroom should turn to using gimmicks and games every single day in order to maintain interest. This would be difficult to maintain in the long term, and it’s unclear whether actual knowledge will be kept afterwards. I’ve had bad experiences in university with a lecturer trying to show mostly pretty pictures and fun videos during class, only to end up with a full class of people that lack understanding of even the simplest theorems, due to focusing only on the fun bits. But incrementally improving the classes themselves, making even the boring parts of the subject a bit more memorable with visual cues and repetition, could actually work, I believe. Especially if it’s done every single class in a consistent, unobtrusive way. We are creatures of habit. Having a single “special” class once a week is not going to turn bad habits around. Having every class be just a bit more engaging might.

Something Actionable

Whether this can actually be pulled off is another topic entirely. I’m not a teacher and all this is borne out of my experience as a student and assumptions as a developer. One way I can see this working out is if some techie decides to “pair program” with a teacher for a while. As in, the techie can discuss the lesson with the actual, professional teacher, build up some slides in collaboration with them, and then present them as the teacher is speaking. I can imagine this happening with young teachers, both because they still haven’t built up their methodology in dealing with children, but also because they’d be more inclined to trust a technical solution.

On my side, a lot of people seemed to be interested in the initLab lecture and there was a fair amount of discussion afterwards, which at least makes me positive that many people take these ideas at heart. One friend in particular is going to be leading HTML & web-dev lectures to high school kids, and I’ll follow up with him to see what his experience is in terms of engagement. I’d still really like it if we found a way to do “pair teaching” with some actual, real teachers and see how it goes, but I’m not sure if there would be enough support from schools and enough technical possibility to pull this off (a projector, a canvas).

In the end, I feel that my own part in this will be all talk. I’m far too easily distracted by a new idea for a vim plugin or a ruby gem to keep up a serious education-related campaign. But if you happen to be reading this and you happen to have a friend who’s a teacher, consider talking them about this, asking them what they think of it all. And let me know how it goes.

OHM 2013

Sun, 18 Aug 2013 11:02:00 +0200

I went to OHM (Observe, Hack, Make) this year, a hacker festival in the Netherlands. It was a fun week of camping, freedom talks, technology, retro-gaming and all sorts of other cool stuff. Since I was going there anyway, I decided to make a Vim event to spread word of the One True Editor.

It was a three-day Vim workshop and I’m pretty happy with how it went. The first day I talked about the very basics, things like moving around in a buffer and opening files. It was only an hour, which is not nearly enough to learn anything practical, but I tried to at least show the basic philosophy of Vim as I see it. My final summary was something along the lines of:

It uses semantic movements. Instead of pointing where the cursor should go, Vim can simply be told what to do, like “select a paragraph” or “delete the next two lines”.
It’s all text. The interface is completely textual, no animations, no fancy icons. This limitation, however, ends up for the better – there’s a lot of amazing UI-related plugins that build directory trees and fuzzy-finder dialogs with pure text.
It’s only text. Vim is at its best when it’s editing text, and for everything else, it works great with external programs.
It’s extensible. There’s an amazing amount of plugins out there, and writing some simple tools of your own is not difficult at all.

I also pointed the attendees to my usual sources of beginner knowledge:

The Vim subreddit. Filled with discussions on new plugins, workflows and Vim features.
Stack Overflow. Excellent place to ask Vim-related newbie questions.
And, of course, the good ol’ :help. Some of the best documentation on a piece of software I’ve seen so far.

The second day was something like a workflow show-and-tell. I went through some of my personal tricks, like how I navigate through a project, how I arrange windows as I’m working, how I deal with language-specific tools. I prompted everyone to share their own workflows on these topics and they didn’t disappoint. Before you know it, we were talking about registers, relativenumber and various plugins. Regrettably, I didn’t keep notes after the workshop, but I’m going to present the second day’s content to a few folks here in Berlin, so maybe I’ll make it a bit more organized and publish it afterwards.

The third day was the Vimscript day. I skimmed through some scripting basics, like if-clauses, for-loops and the likes. This part was mostly glossed over as “it works pretty much as you’d expect for simple cases”. I tried to focus on the more interesting parts like variable scopes, functions and commands. For a more workflow-related angle, I talked about how to think about separating the user interface part from the actual scripting part, reiterating some points from an old talk/blog post, “Lessons Learned From Building Splitjoin”. To get a feel of how it all fits together I wrote a simplified version of my simple :Grep command from this gist. I wanted to talk more about debugging and testing, but there wasn’t enough time. I still managed to squeeze in a demo of the Decho plugin, one of my favourite debugging tools.

All in all, it was a fun couple of days, and I hope everyone that came learned something useful. Next time, I should definitely try to keep some more detailed notes for sharing purposes.

Andrew's Blog

Sending Build Output to Vim

🔗But why?

🔗Finding the right instance

🔗Running the cargo command

🔗Was this really necessary?

From Vim to Presentation Slides

🔗The Process

🔗Some Sensible Questions to Ask

Any problems with this method?

Why use LibreOffice at all then?

Would this method work with Keynote?

What about using screenshots instead?

Testing in Rust: Writing to Stdout

🔗The Write Trait and Simple Stubs

🔗A More Robust Stub

🔗Error Handling

🔗Extracting to a Separate File

🔗Closing Thoughts and Further Reading

Testing in Rust: Temporary Files

🔗First Attempt: a Tempdir

🔗The problem

🔗Files and Paths

🔗A Reasonable Solution

🔗Closing Thoughts