RSS Feed

Fetching the latest release of a GitHub package with Cloudflare Workers

Fri, 17 Jul 2020 00:00:00 GMT

There's an example of this worker up at https://mod.jpwilliams.dev/github-release-version that can be used for any package like so: https://mod.jpwilliams.dev/github-release-version/jpwilliams/midi-mixer-releases.

When creating the website for MIDI Mixer, I needed to be able to list the latest version available to download without making any hefty requests and without updating the site every time I updated the software.

GitHub provides a wonderful API that's exceedingly easy to use, meaning we can mock up a Cloudflare Worker that returns the data we need with not a huge amount of effort.

A Basic Worker

The simplest Worker we can produce is the one that Cloudflare provides as its default: a "Hello World" worker. It looks like this:

addEventListener("fetch", (event) => {
  event.respondWith(handleRequest(event.request));
});

/**
 * Respond to the request
 * @param {Request} request
 */
async function handleRequest(request) {
  return new Response("hello world", { status: 200 });
}

We won't go in to Workers in-depth here, but they work very similarly (both in code and function) to a ServiceWorker. If you've never used Cloudflare Workers before, I'd strongly suggest checking out their Quick Start guide to get going, well, quickly.

For our purposes, we're first going to need to figure out how to to retrieve the data we need.

The GitHub API

GitHub allows you to explore pieces of their API with no credentials, meaning we can instantly see what information we can get. After a quick look on the GitHub docs site I found the Get the latest release endpoint. You can go and check out the response right now: https://api.github.com/repos/jpwilliams/midi-mixer-releases/releases/latest.

There's a lot of data there, but the key piece for us is the tag_name property. The tags I use for versioning on GitHub are semver-compatible version strings prefixed with v, so this is perfect to use as my version number.

Now we know what we're looking for, let's add a function to our worker that'll pull the data we need.

async function getVersion(username, repo) {
  const targetUrl = `https://api.github.com/repos/${username}/${repo}/releases/latest`;

  const headers = new Headers();
  headers.set("accept", "application/vnd.github.v3+json");
  headers.set("authorization", "token YOUR_GITHUB_TOKEN");
  headers.set("user-agent", "YOUR_GITHUB_USERNAME");

  try {
    const res = await fetch(targetUrl, { headers });
    const release = await res.json();
    if (!release)
      throw new Error(`No valid releases found for ${username}/${repo}`);

    return new Response(release.tag_name || "");
  } catch (err) {
    console.log(err);

    return new Response("");
  }
}

getVersion is a function that takes a username and a repo, and returns a response containing either the version string if it was successful or nothing (but still a 200 OK status) if it wasn't. You should handle failing cases depending on your use, but for me it made sense to just "silently" fail and return nothing.

There are a few pieces to pick apart here.

First, the headers. When calling the GitHub API from your browser, it realises this and lets you through (with a tight rate limit). When accessing it programmatically, however, GitHub needs some more love. For this to work you'll need to:

Set Accept to application/vnd.github.v3+json. This tells GitHub that you're intending to send an API request and are ready for the format it's delivered in.
Set Authorization to token YOUR_GITHUB_TOKEN. You can generate a token on your Personal Access Tokens page. It'll need either the repo or public_repo permission depending on your needs.
Set User-Agent to your username. GitHub actually requires this as a means to provide accountability for particular API calls.

Now we have our main code, we'll need to expand our "Hello World" handleRequest to a much nicer version.

Handling the request

handleRequest needs to grab the data that we need to be able to send to getVersion. In this Worker, I've done this using the URL path. We can access the path really easily by using the URL interface with the string URL we get in the request.

async function handleRequest(request) {
  const url = new URL(request.url);
  const [username, repo] = url.pathname.slice(1).split("/").slice(-2);

  if (!username || !repo) {
    new Response(
      "Invalid request. No username or repository specified. Format is /username/repo",
      {
        status: 400,
      }
    );
  }

  try {
    const res = await getVersion(username, repo);
    return res;
  } catch (err) {
    console.log(err);
    return new Response("");
  }
}

Awesome! We use a try/catch here to ensure that if anything goes wrong it's just logged and we return a silent failure. The only piece of code that might be a bit odd is how we extract username and repo.

Remember that a URL to access the current service I've deployed looks like this: https://mod.jpwilliams.dev/github-release-version/jpwilliams/midi-mixer-releases. If a request comes in to that URL, here's what happens to extract username and repo:

// Request to: https://mod.jpwilliams.dev/github-release-version/jpwilliams/midi-mixer-release
const url = new URL(request.url);
console.log(url.pathname); // /github-release-version/jpwilliams/midi-mixer-releases

url.pathname
  .slice(1) // github-release-version/jpwilliams/midi-mixer-releases
  .split("/") // ["github-release-version", "jpwilliams", "midi-mixer-releases"]
  .slice(-2); // ["jpwilliams", "midi-mixer-releases"]

That, along with destructuring gets us our values regardless of the base path (github-release-version in this case) it's deployed on!

Sorted.

With that, we're done! A really quick, really clean method of grabbing the plain-text version number of the latest release of any GitHub repository.

If you're using this in production (or with Wrangler), please don't put your GitHub API token in the code. Instead, use Secrets. For an example of this, see the repo below.

The complete code base for this with TypeScript and Wrangler included are available at jpwilliams/github-release-version. Check out MIDI Mixer too if you fancy seeing a live implementation!

Nullish Short-Circuit Assignment in TypeScript 4.0 (beta)

Thu, 02 Jul 2020 00:00:00 GMT

On June 26th 2020, TypeScript 4.0 Beta was announced. The update brings a myriad of useful changes, but one such addition is something you'll quickly get in to the habit of using every day: Nullish Short-Circuit Assignment.

// A function that adds a post.
// If the user in the post doesn't exist, add a fake `0` user instead.
// It's valid to submit the fake `0` user too.
interface Post {
  content: string;
  user?: number;
}

// Old
function addPost(post: Post) {
  if (typeof post.user !== 'number') {
    post.user = 0;
  }
}

// New
function addPost(post: Post) {
  post.user ??= 0;
}

You've used a += b before to add to a number or extend a string without having to write a = a + b. What's frustrating, though, is having to check if a variable is nullish (null or undefined) before assigning it. TypeScript 4.0 adds three new assignment operators: &&=, ||=, and ??=. Using these alongside the Nullish Coalescing added in TypeScript 3.7, even more boilerplate code can be stripped down!

In the grand scheme it's a small improvement, but the new code is definitely shorter and more readable.

What about a simpler example? In the block above we wanted to be able to parse 0 as valid input, so we couldn't just do a falsey check. What if we just wanted to use nullish coalescing (??) without the short-circuit assignment (??=)? That's pretty succint too, right?

function parseConfig(config) {
  config.setting = config.setting ?? "set";
}

This doesn't look too bad. We have to repeat config.setting which could lead to some spelling mistakes and therefore unintended results, but it's still pretty readable and concise. What this is vulnerable to, however, is setters. If you're not familiar, a setter is a function that binds an object property to a function to be called when there's an attempt to set that property. For example, I could ensure that setting the username property on my object updates the profileUrl field appropriately:

const obj = {
  set username(name) {
    this.profileUrl = `https://example.com/@${name}`;
  }
}

obj.username = "FoobarBaggins";
// obj.profileUrl === "https://example.com/@FoobarBaggins"

Because a setter is just a function, it could potentially perform a computationally-expensive task without the developer realising. In our Nullish Coalescing examples, performing obj.foo = obj.foo ?? "bar"; will always call the foo setter, regardless of if obj.foo was nullish or not. If we use a short-circuit (obj.foo ??= "bar";), the setter will only be called if obj.foo was nullish. Nice!

obj.username = obj.username ?? "FoobarBaggins"; // calls setter every time
obj.username ??= "FoobarBaggins"; // only calls setter if obj.username nullish

I highly recommend you check out the wonderful TypeScript 4.0 Beta announcement for all the nitty-gritty of the new features. You can use the beta today with npm i -D typescript@beta, though check if it's fully released when you're reading this!

How to unpack the return type of a Promise in TypeScript

Sun, 05 Jan 2020 00:00:00 GMT

TypeScript 4.5 (released on November 17th, 2021) introduced the Awaited type that solves this problem. If you can use Awaited, use that instead of the below method. - edited 1st December, 2021

While making @jpwilliams/distributed-promise I came across an interesting requirement: I needed to acquire the return type of an async function after it had been resolved. I needed to unwrap a promise.

For example, an async function that simply returns "foo" would have the return type Promise<string>. For my purposes, I needed just the string type.

It turns out that this is very possible since TypeScript 2.8 and the introduction of the infer keyword back in March 2018. For the requirements I had above, I ended up with the following type:

type AsyncReturnType<T extends (...args: any) => any> =
	T extends (...args: any) => Promise<infer U> ? U :
	T extends (...args: any) => infer U ? U :
	any

This could be used like so:

type T0 = AsyncReturnType<() => Promise<string>> // string

This type, given a function type, will return either the type of the resolved promise if a promise is returned or the straight return type of the function. This is really cool, but it confused the hell out of me at first glance, so let's step through.

First, this makes heavy use of infer, which we can use to extract types from matching signatures. A simple example of this would be making a type that takes any[] as input T and returns the any. We can do this in exactly the same way we did with the function signatures above:

type Unwrap<T> = T extends (infer U)[] ? U : T

There, we check if T matches the pattern something[] and use infer to extract that something. If it does, return our something, otherwise just return T. This brings us to the second major point: notice the ? and : ternary operators being used. Their use here is exactly the same as in regular ol' JavaScript. Cool, huh?

Now we have that under our belt, understanding the original type is a bit easier:

// create a generic type
type AsyncReturnType<T extends (...args: any) => any> =

	// if T matches this signature and returns a Promise, extract
	// U (the type of the resolved promise) and use that, or...
	T extends (...args: any) => Promise<infer U> ? U :

	// if T matches this signature and returns anything else,
	// extract the return value U and use that, or...
	T extends (...args: any) => infer U ? U :

	// if everything goes to hell, return an `any`
	any

Nice! Though I don't need it for my uses, a good catch-all might be to include the Promise type itself.

type Unwrap<T> =
	T extends Promise<infer U> ? U :
	T extends (...args: any) => Promise<infer U> ? U :
	T extends (...args: any) => infer U ? U :
	T

As mentioned above, this is used within @jpwilliams/distributed-promise, a small library that can be used to distribute promises across the network and multiple processes.

Using EventEmitters to resolve Promises from afar in Node.js

Wed, 11 Dec 2019 00:00:00 GMT

A common-but-powerful pattern I've come to frequent is the utilisation of EventEmitter alongside a Promise, enabling me to resolve said promise from an entirely different part of my code base.

To demonstrate, let's revisit the basics of both an EventEmitter and a Promise.

EventEmitter

An EventEmitter is an object which can be used to "emit" named events which, in turn, cause Function objects ("listeners") to be called. Here's a speedy example that will log "foo", "bar", and "baz":

const { EventEmitter } = require('events') // this is a core Node package

const emitter = new EventEmitter() // create our emitter
emitter.on('data', console.log) // console.log any data coming in from the 'data' event

// send data
emitter.emit('data', 'foo')
emitter.emit('data', 'bar')
emitter.emit('data', 'baz')

Now that we have that in place, as long as we pass that same emitter around, different parts of our application can communicate asynchronously and without relying on a direct connection.

Promise

For a Promise, let's look at the most recent way of defining them. Those used to newer versions of Node.js (and JavaScript in general) will know of async functions:

async function foo () {
	await doSomeWork()
	await doSomeMore()
	const data = await getUnorderedData()
	data.sort()

	return data
}

const data = await foo()

A function with the async prefix will always return a Promise and has the added benefit of allowing us to use the await keyword to "block" while some asynchronous work is completed.

For our purposes, though, we're going to use a different way of defining a Promise: with callbacks. Using new Promise, we can get two functions (resolve and reject) to use to, well, either resolve or reject the Promise.

function foo () {
	return new Promise((resolve, reject) => {
		...
	})
}

You can be cheeky here, too, and make the callback an async function to get the goodness of await while still having the tighter control of the two functions.

Combining the two

So how does this tie in with our EventEmitter? Well now that we have a function that resolves our Promise, it's really easy to use an emitter inside. We can use the once method to register a one-time listener that removes itself once it's invoked.

function waitForNextData () {
	return new Promise((resolve, reject) => {
		emitter.once('data', resolve)
	})
}

🤯 Now we could call waitForNextData which would return a Promise which would resolve once some new data came in via our emitter!

const data = await waitForNextData()

This is an incredibly simple combination of JavaScript's asynchronous toolkit, but provides some sneaky tactics to use across larger projects when direct communication between components is either difficult or ill-advised. It's used heavily in @jpwilliams/remit, a microservices toolkit, to manage incoming and outgoing messages which may be received in a place far different from where they were sent. Also @jpwilliams/waitgroup, a tiny version of Golang's WaitGroup with promises, which is a great mini example of this technique.

Modularising GraphQL for scale

Sun, 24 Nov 2019 00:00:00 GMT

When fiddling around with GraphQL projects, I've frequently found it difficult to know where things should go. Just like naming variables, naming and placing files is hard. The simplest method here is to bundle everything in to one file, but that quickly gets out of hand.

Separate files means separate logical groups of relevant data. There are plenty of ways to easily view directory trees, but it's not so easy to bundle all of the mental state of a 3,000 line file in to your head. What if our entire GraphQL server could be viewed neatly, like this:

On the left here, we see the directory tree for the types in this example GraphQL server. Each type has its own directory with a schema.graphql file defining it. For the simple Date type, we've only made two files - schema.graphql and resolvers.js. The files here are really simple:

// schema.graphql
scalar Date

// resolvers.js
const GraphQLDate = require('graphql-date')

module.exports = GraphQLDate

While we've separated those in to two files, with the code we'll be writing we could just as well place them in a single file, Date.js, in place of the entire Date folder:

// Date.js
const GraphQLDate = require('graphql-date')
const schema = `scalar Date`
const resolvers = GraphQLDate

module.exports = { schema, resolvers }

This means that we can keep exceedingly simple types in singular files where we can see the entirety of their functionality in a small number of lines. Larger types like Message, though, we can split up as we see fit.

This is perfectly doable, but there's a certain degree of mucking about that's required to get all of these schemas and resolvers in to place. Usually that means back to one massive file that requires a hundred others.

For ease of use, a better method of loading and managing GraphQL types, resolvers, and loaders is available as a package at @jpwilliams/graphql-modular-loader. An example of its use with apollographql/apollo-server is as follows:

const { ApolloServer } = require('apollo-server')
const { loader } = require('@jpwilliams/graphql-modular-loader')

const { typeDefs, resolvers } = loader('./types')

const server = new ApolloServer({ typeDefs, resolvers })

server.listen({
	port: process.env.PORT || 4000
}).then(({ url }) => {
	console.log(`🚀	Server ready at ${url}`)
})

But let's slow up a second. If we use that package and separate our entire GraphQL functionality in to individual files, what are a few things we could gain?

Ease of deletion. Deleting code is a big part of cleaning up. Sometimes that means entire sections of a deprecated API. Here, you'd delete the relevant folder and that's it. Done. Otherwise, you're digging through files hoping nothing secretly relies on what you've done.
Separation of concern. Building up the state of an application in your mind is a huge part of programming. If you can open a single file and only have to concentrate on that singular part, you're already half way to solving the problem.
"Bookmarked" code. Searching for the exact part you need of a large file is not a nice experience. Comments and IDEs help with this, but nothing beats having your code organised in to small, relevant files.

This is the aim of @jpwilliams/graphql-modular-loader. Personally, it's helped me keep many projects shockingly orderly.

The package supports every aspect of GraphQL that may need to be separated out: schemas, queries, mutations, subscriptions, field resolvers, and loaders. For a full look at the source code and up-to-date documentation, check out the jpwilliams/graphql-modular-loader GitHub repository.

It's all about trade-offs

Mon, 11 Jun 2018 00:00:00 GMT

I recently had a great discussion regarding WebSockets and HTTP/2 with a former colleague. During the discourse, they uttered one line so deeply-set in truth that it must be shared. Repeatedly.

"It's all about trade-offs."

As someone who has been developing web applications for a little while (and as somebody who repeatedly rewrites them), I consistently find myself doubting the quality of the system at hand. It could be more performant. It could be more readable. It could be more developer-friendly or use the better (read: newer) technology X. These niggling thoughts often cause me to cave in and needlessly redesign a working system. Sometimes it's better, sometimes it's not.

Development as a whole is a mixed field. Its obscure mix of creativity and science is what attracts so many, yet it's that same trait that muddies the waters, encouraging bikeshedding and frequently instilling doubt in our minds. But remember, it's all about trade-offs.

For me, system design - and therefore development - is about achieving a balance between two distinct states of nirvana: performance and simplicity. Fulfilling both of these states is very possible, but we're most often limited by one ugly constraint: time.

Performance covers how fast it runs. If it's a service that delivers a response when asked a question, how quickly does it respond? How many questions can it handle at once?

Simplicity can be read as how understandable the system is. Does the system work in a clean and intuitive way? How easily would somebody else understand the system? Building something that's easily understood takes a lot more time and consideration.

Time is the major constraint. Deadlines, whether arbitrary or not, are everywhere and can't be ignored. Often, an application or feature is wanted to perform X and making the application prettier in ways the end user won't see isn't budgeted for.

"Any fool can write code that a computer can understand. Good programmers write code that humans can understand." - Martin Fowler

As a whole, we already abide by these forces unknowingly: the vast majority of developers use high-level languages like JavaScript and PHP that facilitate the creation of human-readable code. A program written in a high-level language will often be less performant than one written in low-level languages like C or Assembly, but readability, ease of use and ease of change counts for a lot, so we sacrifice these performance gains. Ergo, we trade performance for time and simplicity.

Very few projects, applications or architectures provide the freedom to reach the peak of performance and simplicity simultaneously, so we face trade-offs. We use X technology because its more well-known, so easier to find help with, though it may be less performant. We use a nasty-looking regex to perform a particular search because it's blisteringly quick, though it's insanely difficult to adapt to changing requirements. We use microservices because they're great for defining service boundaries and ownership, though they're hell to deploy.

Dynamically-typed versus statically-typed, relational versus non-relational, object-oriented versus procedural, monoliths versus microservices - there are no right answers. If you can reasonably justify the balance of performance, complexity and time in your application then your choices are correct. In most cases, a solution is not invalid because it fails to maximise a single aspect of its potential; it is the sum of its aspects.