Sandstorm.io Blog

Sandstorm now belongs to Sandstorm.org

Kenton Varda — Sun, 14 Jan 2024 00:00:00 +0000

TL;DR: Primary maintainership of the Sandstorm project has changed, and a along with that, the project has moved to sandstorm.org.

The “New” Sandstorm Project

I’ll start with the point: I, Kenton Varda, am no longer the maintainer of the Sandstorm project. In fact, I haven’t really been for quite some time. Instead, a group of avid Sandstorm users, led by Jacob “ocdtrekkie” Weisz, have been doing a lot more work than me. Many have been involved for quite a few years now.

Yet until recently I have continued to hold the role of gatekeeper: I was the one who could push releases, merge changes, approve blog posts, etc., but was failing to do so. That doesn’t make sense.

So, I’m relinquishing that control. Sandstorm now belongs to the Sandstorm Community under Open Source Collective.

However, Jake and I agree that this change should be very explicit. People coming to the Sandstorm web site today are often confused about Sandstorm’s status, even thinking it is backed by a company – we need to make things clear. And, it doesn’t necessarily make sense for for people who who opted into auto-updates back when Sandstorm was a company to keep getting them from the community. Not only because those users may not be comfortable with it, but also because the community itself may not be. Sandstorm’s users included (and may still include – there’s no way for us to tell) companies, newspapers, educational institutions, research laboratories, and even government agencies. An auto-update that, say, breaks SAML login could do a lot of damage overnight. For the community to make progress, the stakes need to be a little lower.

To that end, the new project is going to live under a new domain, sandstorm.org. Hopefully, the .org TLD, along with some design changes, makes it clear that this is a community project, not a company. sandstorm.io will stay as it is, but with the banner you see above flagging it as historical. If and when sandstorm.org decides to push auto-updates, users will have to manually opt into them.

For my part, I continue to be a Sandstorm user, and as a user I do hope the project is able to make progress under new maintainership. Time will tell.

Sandstorm’s Backstory

For those curious how we got here, here’s the whole story from the beginning…

Long ago – circa 2014 to 2016 – Sandstorm was a startup, of which I was co-founder. We ran a successful crowdfunding campaign in 2014 and raised VC money in early 2015. We hired 5 people, making a team of 7. We had aimed for a series A round in 2016. We saw a lot of developer excitement for our product on Hacker News and the like, and we expected to follow in the footsteps of other startups who were able to raise a pre-revenue series A based on such excitement. Unfortunately, that did not happen. For whatever reason – and I could list several possible reasons, but I’ll never really know which it was – investors weren’t interested. This set off a panic where we lowered our sights to raising a “bridge round” while trying to produce actual revenue from enterprise sales. But this made things worse: we did not have any idea how enterprise sales worked. We made two sales ever, for a measly 4 figures. Investors ran away.

By the end of 2016, it was obvious the company was going to crash, so we started looking for an acquisition. Too late, of course – our potential acquirers knew it was a fire sale. There was a common saying in Silicon Valley startup circles at the time that in a pure acquihire, your company is worth $1M per engineer. We learned quickly what should have been obvious: this is a fanciful myth. You cannot block another company from hiring your employees, so why on Earth would they pay you – or rather, your investors – for the right? Just to be nice? Several companies were very interested in our engineers, but the best they offered was to acquire the company for $0 and help us wind it down, while making job offers to some or all of the employees.

One company (whom I won’t name, but you’ve heard of) actually told us: “We will add up all the salaries and stock grants we offer your employees, and that’s what you can say was the exit price. If you’d like, you can take a chunk of the equity and give it to your investors rather than the employees.” I found this extremely disturbing. Another company (you’ve heard of them too) interviewed the whole team and then informed us they were passing on the founders but hiring two employees.

Another company we talked to was Cloudflare. They too offered to acquire the company for $0, but they made good offers to the whole team without any funny business. Additionally, they had a project in mind for me. They said: “We know we want to let people run code on our edge network, but we haven’t figured out exactly how. We could give you ownership of this.” I was intrigued. And after paying myself $60k/year for several years, boy did that regular salary look nice.

But I still very much wanted Sandstorm to succeed. So I told Cloudflare, I’ll come work for you, but I want to keep Sandstorm the company, and be able to work on it as an open source project. They agreed.

So in early 2017 we announced that Sandstorm was moving to a community model and, shortly after, that most of the team joined Cloudflare.

For some time, I continued to work actively on Sandstorm, and a motley crew of community members helped out. But at the same time, by day, I was having the most fun of my career, having been given a greenfield project to hack on with no distractions. That project became Cloudflare Workers. Within six months we released a beta, and (coincidentally) exactly a year after I joined Cloudflare, we officially launched the product. The product has continued to grow on a startup-like trajectory ever since.

So, I was (and am) now essentially the founder of a startup-within-Cloudflare which was actually succeeding. It became hard to put energy into Sandstorm, the project that felt like a failure, even if I still believed in it. Sandstorm became increasingly a chore: Once a month I would review and merge pull requests, update dependencies, and push a new build. If I spent time on it beyond that, it was to shut down the paid hosting service (which never had enough users to make money) or switch sandcats.io to Let’s Encrypt when our previous contract with Globalsign expired.

Over time, even just basic maintenance became difficult. Sandstorm uses MongoDB for metadata storage, but it is stuck on version 2.6, which is ancient. I couldn’t update it because updating Mongo requires manual intervention, but Sandstorm is something that thousands of users have installed and expect to auto-update. Most of them aren’t even aware they are running Mongo. Maybe I could have written code that would automatically handle the migration? That sounded like more work than I was interested in. Eventually, though, the Mongo drivers used by Meteor were updated to a version that didn’t support 2.6. So now I couldn’t update Meteor, Sandstorm’s primary dependency. And we can’t update Node, because old Meteor doesn’t work on newer Node. (That, by the way, is largely because V8 made a change which permanently broke node-fibers, which Meteor deeply depended on. Oh the tangled web we weave…)

During all this time, the Sandstorm web site remained mostly unchanged, save for the occasional blog post. I think this has been misleading. The site was designed when Sandstorm was a company, by our full-time professional designer. We made some changes after the company shut down, but it still looks like a company’s web site. People seem to stumble across the page and believe it is backed by a full-time team rather than a few volunteers. From time to time it even shows up on the front page of Hacker News.

I have felt bad about this, but couldn’t quite figure out what to do. Redesign the web site to make it look less professional? I’m not quite sure even how to do that, but I’m pretty sure I don’t have time.

In late 2022, the one person who had been contributing code changes to Sandstorm with some regularity, Ian “zenhack” Denhardt, decided to embark on a total rewrite called Tempest, so contributions to the main project dried up. In early 2023, I gave up pushing monthly releases, since there seemed to be no point: no code changes had been made and no dependencies could be updated.

And then in mid-2023, tragically, Ian passed away in an accident. I’m no good at eulogies, but suffice to say it was a big loss for Sandstorm, Cap’n Proto, and other communities Ian was in.

Later in 2023, Jake wrote a blog post, originally to be posted on sandstorm.io, requiring my approval. The post was ambitious, and even though I wasn’t being asked to do anything except approve it, I felt that doing so implied a commitment on my part that I wasn’t prepared to make. Instead, we made the decision that I should take myself out of the loop. We came up with the plan to transfer leadership of the project. Sandstorm-the-company had long owned the domain sandstorm.org but not used it; I decided it made sense for this to become the new home of the community project and transferred it to them.

For reasons I can’t go into here, I was unable to dissolve Sandstorm the company until some time in 2022. (It turns out I really should have accepted the $0 offer from Cloudflare, to make it their problem, but alas.) But once I could finally do it, our investors and I agreed on a dissolution plan that allowed Sandstorm’s remaining IP to be transferred to a non-profit entity representing the community. Of course, the code is open source under the Apache license, so anyone can freely use that. But, the agreement also covers trademarks, domain names, etc. Open Source Collective is a qualifying non-profit.

And so, here we are. Sandstorm now belongs to Sandstorm.org.

Fundraising for the Sandstorm Community

Ian Denhardt — Thu, 17 Jun 2021 00:00:00 +0000

Hey Everyone!

Members of the Sandstorm community are doing some fundraising!

Specifically,

In the immediate term, we’re participating in FundOSS, a donation matching program that is a joint effort between Open Source Collective and GitCoin with a novel democratic funding model for allocating matching funds, designed to boost the impact of many small donations.
Longer term, we’re looking to promote an OpenCollective community organization, as a basis to support Sandstorm’s development financially.

The organizers are familiar faces, but we aren’t and haven’t ever been formally affiliated with Sandstorm, Inc; we’re just long-time community members looking to chart a path for the Sandstorm project going forward. We all miss the days when there were several people working on Sandstorm full-time, and want to find ways to support continued development of new features and additional app packaging efforts.

If you want to help us out here’s how you can:

Make a donation (even a small one) via FundOSS between now and June 24th.
If you want to support us in the long term, consider also making recurring donation to our OpenCollective organization.

Updating Tiny Tiny RSS, Moving Sandstorm's Security Model Forward.

Ian Denhardt — Sat, 08 Aug 2020 00:00:00 +0000

This past week I updated the Sandstorm package for Tiny Tiny RSS. In addition to updating to the latest upstream code, I’ve also made some improvements to the package’s integration with Sandstorm. In particular:

The package now uses Sandstorm’s scheduling API (merged earlier this year) to check for updates periodically even when the user isn’t actively using the app. This was a major pain point for some users, as it meant that if they didn’t check their feeds often enough, they could miss articles.
I’ve migrated the package away from using Sandstorm’s deprecated HackSession interface for fetching feeds, and over to using the Powerbox.

The latter of these is critical to enable Sandstorm’s goal of full network isolation for applications. It has always been the intent that apps running on a Sandstorm server should not be able to phone home or otherwise access the network without the user’s consent, but early on the team implemented some temporary hacks that apps could make use of until Sandstorm’s Powerbox API was actually available. Now it is, so it’s time to migrate existing code away from these legacy APIs. Once we’ve done that, we can remove the holes, and finally deliver on Sandstorm’s security promise of network isolation for server code.

In order to make this happen, I wrote a daemon that runs inside of the grain, intercepting HTTP traffic and making powerbox requests for access to the relevant hosts on-demand. The daemon is re-usable and could be used as a quick way to port other existing apps that have legitimate need to access a variety of different hosts at runtime, which are not known in advance.

Unfortunately, this transition does come with some growing pains, and there are a few things users should be aware of when upgrading.

First, since existing TTRSS grains will already have subscriptions to feeds that they haven’t requested access to, the first time you start the new version of TTRSS you’ll see a flood of requests for all of your existing feeds. This is annoying, but fortunately after this initial setup you’ll only need to grant access when adding new feeds.

Second, a more serious limitation of the current implementation is that it makes it impossible to add new feeds through mobile & desktop clients – at least without also having a browser window open somewhere. This is because, without the Sandstorm UI open, Sandstorm has no way of actually showing a user the powerbox dialog. We’d like to find ways to improve on this.

Finally, while the daemon is exciting for app packagers, in that it can make basic porting of apps not designed for Sandstorm much quicker, the integration is imperfect, resulting in a sub-par UX: as it stands, users will see an extra prompt (or two) when first subscribing to a feed.

Maintaining security while avoiding annoying extra prompts like this is one of the design goals of Sandstorm’s powerbox, but to fully take advantage of it will require more invasive changes to TTRSS; rather than prompting the user for a feed URL and then trying to fetch it (causing the daemon to request access), it should just ask Sandstorm for a feed directly. If you’re an enterprising PHP hacker and want to help make this happen, get in touch on the sandstorm-dev mailing list or the #sandstorm IRC channel on libera.chat.

Let's Encrypt support for Sandstorm and Sandcats

Kenton Varda — Sat, 13 Jun 2020 00:00:00 +0000

Sandstorm now has built-in support for fetching certificates from Let’s Encrypt. This applies both to Sandcats and to custom domains.

Let’s Encrypt with Sandcats

Backstory

Sandcats.io is Sandstorm’s free dynamic DNS and TLS (aka SSL) certificate service. Since 2015, Sandcats has been making it easier to set up self-hosted Sandstorm servers, by letting anyone claim a subdomain of sandcats.io on which to host their server, automatically configuring DNS and HTTPS.

Sandstorm requires a wildcard host so that it can create sandboxes by assigning random hostnames. So, the certificates issued by Sandcats.io have always been wildcard certs. Back in 2015, that was a challenge: at the time, Let’s Encrypt did not yet offer wildcard certs, so the only way to get one was to pay for it. Unfortunately, many CAs considered wildcards to be an “enterprise feature” and charged excessive amounts of money for them, often hundreds or even thousands of dollars.

So how were we able to offer wildcard certificates for free? Well, we negotiated a deal with GlobalSign. Recognizing that subdomains of Sandcats.io were unlikely to be enterprise customers, and recognizing a growth opportunity if Sandstorm took off, GlobalSign offered us a deal that worked. Sandstorm was a funded startup at the time, and we were happy to pay a few bucks per certificate-year and call it “customer acquisition cost”. We paid for thousands of certificate-years upfront, anticipating growth.

But, the growth we hoped for didn’t happen, and we only ever had about 500 self-hosted servers using Sandcats. In 2017, Sandstorm failed as a startup and reverted to being just an open source project. Having never seen the growth we were hoping for, we hadn’t even come close to using up the first block of certificates we had paid for from GlobalSign. Since we had already paid, we left the system running, happily issuing certificates.

Fast forward to 2020. Finally, our contract is running out. In fact, it did run out, in early March – embarrassingly, I had miscounted how much time we had left. However, GlobalSign was nice enough to give us a small extension, giving us time to migrate our users without disruption. And it turns out that, these days, Let’s Encrypt supports wildcards. So, moving to them is the obvious choice.

What’s New

Starting a few days ago, all Sandstorm servers using Sandcats.io are in the process of switching to Let’s Encrypt for future certificates. The process is designed to happen slowly so that we can address any problems that arise, but so far everything has been smooth. All servers should be transitioned by the end of the month.

If you’d like your server to start using Let’s Encrypt immediately, visit the TLS certificates admin page at /admin/certificates on your server, click the button to create an ACME account (you will be prompted to agree to Let’s Encrypt’s Terms of Service), and then click “Fetch Certificate Now”.

Note that at present, we have not yet updated the install flow, so newly-installed Sandstorm servers will still use a GlobalSign certificate initially and then will switch to Let’s Encrypt later. We’ll be updating the installer soon, and then we will decommission the GlobalSign flow.

Let’s Encrypt with non-Sandcats domains

To implement Let’s Encrypt support for Sandcats.io, it made sense for your self-hosted Sandstorm server to directly talk to Let’s Encrypt via the ACME protocol. This differs from the old GlobalSign flow, in which Sandstorm would talk to a central Sandcats.io server, which then talked to the GlobalSign API on your behalf. This was necessary for GlobalSign since Sandstorm the company was paying for the certificates, so our credentials were needed when talking to the API. But for Let’s Encrypt, the Sandcats.io central server only needs to set some DNS records to pass an ACME DNS-01 challenge; everything else happens on your own server.

Given this implementation, it was straightforward to support domains other than sandcats.io. I chose to use the ACME.js library to implement the ACME protocol, and it turns out this library already had a suite of plugins to support a variety of popular DNS providers. If your domain uses any of the DNS providers supported by ACME.js, then Sandstorm can now automatically obtain TLS certificates for your domain from Let’s Encrypt.

However, at present, initial setup is difficult, because of the chicken-and-egg problem: How do you access the admin UI to configure certificates, without a certificate? And than brings me to…

Help make it better!

Currently, there are two major problems with setting up Sandstorm TLS on your own domain:

The UI is very bad. I am a terrible designer, and I didn’t have much time to work on it. Especially bad is the fact that to configure any DNS plugin, currently, you need to enter a JSON blob into a textarea, where the JSON blob’s format is different for every plugin and documented in their respective READMEs as a JavaScript method parameter (not even JSON)… Only experienced programmers could possibly understand what to do. We should make the UI better. See issue #3300.
There is a chicken-and-egg problem at install time, as the current UI to configure TLS is accessed over HTTP – which implies that you already need a TLS certificate for it to be secure. We need a CLI configuration option as an alternative. See issue #3367.

If you’d like to help implement either of these, click on the issue links above and comment!

Announcing the release of vagrant-spk 1.0

Jacob Weisz — Sat, 22 Feb 2020 00:00:00 +0000

Hello! I’m Jacob Weisz, a member of the Sandstorm community, a long-time contributor, and the new maintainer of the vagrant-spk tool. I’m thrilled to announce the 1.0 release of vagrant-spk.

What’s vagrant-spk?

vagrant-spk is the premier tool for packaging Sandstorm apps. Unlike the spk tool built into Sandstorm, vagrant-spk creates a virtual environment within which to build your app. This provides a reasonable measure of reproducibility and maintainability, along with default templates (or “stacks”) of common configurations apps likely need to run.

What’s new?

All vagrant-spk stacks now work correctly with the Debian Stretch box we’ve currently standardized on. Most recently this includes reworked support for Node.
A new upgradevm feature has been added to make it easy to move to the latest Vagrantfile and global-setup file supported by vagrant-spk. Along with this we’ve improved detection for broken configurations.
We’ve changed vagrant-spk to use port 6090 so that it no longer conflicts with local Sandstorm installs using default ports. You can use upgradevm to switch your package to this.
The enter-grain feature, which has been broken for a long time, has now been fixed thanks to Adam Bliss and Ian Denhardt.

Why go to 1.0 now?

Historically, vagrant-spk was frequently released alongside Sandstorm releases and was regularly dependent on those Sandstorm releases for functionality. At the time, the convention was to release vagrant-spk with versioning complimentary to Sandstorm releases.

However, times have changed. vagrant-spk improvements are rarely strongly correlated with Sandstorm releases, and of course, both Sandstorm and vagrant-spk releases are less frequent in nature.

vagrant-spk is also now a mature tool, having been used to package a large portion of Sandstorm apps. The natural progression at this time is to move from 0.236 to 1.0. This release is fairly small because the priority for 1.0 is to deliver a polished, stable release that we can then iterate upon.

The future of vagrant-spk

We have a list of improvements and features we are beginning to think about for vagrant-spk’s future. As part of renewed community efforts to revive the Sandstorm project, we are planning a regular cadence of releases for vagrant-spk. We have already begun designating goals for vagrant-spk 1.1, which we’d like to deliver later this year.

Reviving Sandstorm

Ian Denhardt — Mon, 03 Feb 2020 00:00:00 +0000

Hi! I’m Ian Denhardt, a long-time Sandstorm user and community member. I have some exciting news to share. Let’s start by talking about some recent history.

It’s no secret that development on Sandstorm has been pretty sparse since Sandstorm-the-business shut down in 2017. The tone of that announcement was optimistic; while Sandstorm had failed as a business, as an open source project it had attracted an vibrant community, and there was hope that the project could continue to be successful outside of a for-profit setting.

Things didn’t go as smoothly as we’d hoped, however. While Sandstorm had had a healthy community of folks using it and building apps, there had been fewer contributions to Sandstorm itself from outside the company, and no one but Sandstorm employees had been doing major core development on a regular basis. Additionally, Kenton has had less time than he’d hoped to work on Sandstorm.

The project never quite died. It has continued to receive security and maintenance updates. Internationalization support was added, and several folks have stepped up to translate the UI into other languages. A few other features landed, and a few apps continued to see updates as well. But it would be more than fair to say that the project had stagnated, and while I and many others were still using it, it was clear that development wasn’t going anywhere fast.

That looks to be changing.

About two months ago, Lyre Calliope sent an email to the sandstorm-dev mailing list, starting a discussion on how to get the project moving again. Since then, I and others have been contributing code and documentation, and have had weekly “office hours” meetings. In spite of a full time job, moving across the country, and a new baby to take care of, Kenton has still managed find a bit of time to review and merge pull requests, and help plan. Here are some highlights of what’s been happening in terms of development:

Jacob Weisz has done a ton of much-need work triaging our issue tracker, and has been working on some updates to vagrant-spk, looking to get a 1.0 release out soon.
Adam Bliss updated our tooling for managing the docs website, to make review easier and so we can get updates out more quickly. As part of this, he’s updated the GitWeb app, and included support for static publishing. We’re calling the feature “GitWeb Pages”. He also fixed a long-standing bug in vagrant-spk, which will make the app development experience better.
Lauri Ojansivu has been working on internationalizing the mass transfer feature that was added recently, and translating it into Finnish.
I recently implemented support for apps scheduling background tasks, something that’s been a major pain point for developing certain types of applications. I’m working on a few follow-up tasks and related functionality, and put together a demo app that uses Sandstorm’s Powerbox API. To me, this is one of Sandstorm’s most exciting features, but having been implemented only shortly before the company shut down, it still has rough edges and the documentation is lacking. I plan to fix that.

As a community we’re once again very hopeful, and I personally am committed to spending what time I can making Sandstorm’s vision a reality. If you want to help, join us in the #sandstorm IRC channel on libera.chat, and sign up for the sandstorm-dev mailing list.

Sandstorm Oasis is Shutting Down

Kenton Varda — Sun, 15 Sep 2019 00:00:00 +0000

On December 31st, 2019, Sandstorm’s paid hosting service, Sandstorm Oasis, will begin winding down.

Only Oasis is affected. Other Sandstorm services, such as Sandcats.io, the app market, and automatic updates for self-hosted Sandstorm, will continue to operate.
No new monthly payments will be accepted starting January 1st, 2020. Users will be able to finish out their last billing period paid in December and ending in January. Once your subscription ends, your apps will not be able to start up.
Grain owners will continue to be able to download their data or transfer it to another Sandstorm server for at least another six months, until June 30, 2020. After that, data may become permanently unavailable.

It’s time to go Self-Hosted

If you’re an Oasis user, fear not! You can keep using Sandstorm on your own server, and you can easily transfer all your Oasis data to it.

Indeed, today, there’s almost no reason to prefer Oasis over a self-hosted Sandstorm server. Consider:

A similarly-priced server on Digital Ocean running Sandstorm will load apps much faster than Oasis does, while giving you 5x the storage space.
Users in Europe (of which Oasis has disproportionately many, even though we never really intended Oasis to be suitable for them) would be better-served by a European hosting provider, providing lower latency, and governed by European laws. Oasis is located in the United States.
Oasis is currently operated by one person (me). I do my best, but should something happen to me, Oasis could disappear suddenly. In contrast, your self-hosted server will never disappear no matter what I do.
Once Sandstorm is installed on your server, it’s almost entirely self-managing. Updates are installed automatically. TLS certificates are renewed automatically (with Sandcats.io). Modern VM hosts (like Digital Ocean) can perform automatic backups.

In order to make it extra-easy to transfer your data to a new server, I have added a new “Mass transfers” feature to Sandstorm. Find it by clicking the button at the top of your Grains list:

Then follow the on-screen directions to specify a destination server, review the list of grains to transfer, and then execute the transfer.

To recap:

Why shut down Oasis?

The story of the last few years

Sandstorm, as a company, mostly shut down two years ago. The company had run out of investor money while having achieved essentially no revenue, and no hard evidence that we’d ever achieve any. While Sandstorm was popular on Hacker News, that popularity never really converted into paying users. Meanwhile, in the market where we imagined we’d find real profits – enterprise software – we’d made no real progress whatsoever. In this state, we were unable to attract new investors, and we were unable to find a company to acquire the business.

The Sandstorm team was forced to look for new jobs. Most of us were hired by Cloudflare, though some chose to go elsewhere. Personally, I chose Cloudflare because I had always liked the technical culture I saw in their blog posts, and because I was interested in the project they wanted me to work on.

Originally, my plan had been to keep developing Sandstorm as an open source project. I felt – and still feel – that if only some rough edges could be smoothed out and some key missing features filled in, Sandstorm could really be a plausible replacement for the set of web services people use every day. I set a goal of getting myself off of Google services by replacing the key bits with Sandstorm apps – especially email. I thought that if I could really get that working, maybe we’d be in a position to relaunch the company.

I made some progress. On nights and weekends, I managed to clean up one of the hairiest rough edges of Sandstorm, fixing the identity system. I also rewrote the basics of how Sandstorm handles HTTP traffic, making it much faster and cleaner and removing JavaScript from a part of the system where it had no business being.

For a while, Oasis was costing far more to operate than it was taking in in revenue, with me making up the difference out-of-pocket. But, between the HTTP rewrite (which saved several machines), and discontinuing the Oasis free plan, I was able to bring things to the point where Oasis is mildly profitable, earning a few hundred dollars a month.

But, meanwhile, at my new job at Cloudflare, I am the lead engineer / architect of a project called Cloudflare Workers, a “serverless” platform that simultaneously deploys your code to 193 (and growing) locations around the world. Starting from scratch when I joined, I built a first prototype in a few months, had a public demo and beta customers shortly thereafter, and launched it to the world exactly (by coincidence!) a year after joining. Today, Cloudflare Workers handles something like a million times the traffic Sandstorm ever did. Meanwhile, the team has grown from just me to a literal bus-load of people. And we’re really just getting started.

As much as I love Sandstorm, it’s hard to come home from my successful day job to work on an unsuccessful side project. And so, I have been spending less and less time on Sandstorm. I still push updates every month to keep the dependencies fresh, but hadn’t worked on any new features in about a year and a half before adding mass transfers recently.

Meanwhile, without leadership, the community has mostly disbanded. The only app that gets regular updates anymore is Wekan, thanks to its maintainer Lauri “xet7” Ojansivu. Jake Weisz heroically continues to carry the Sandstorm flag, reviewing app submissions (mostly from Lauri), replying to questions and bug reports, and advocating Sandstorm around the internet. A couple others lurk on the mailing list and IRC. Most people have moved on.

Why not leave Oasis running?

Oasis is mildly profitable: it brings in about $1800 in revenue each month, while costing around $1400 each month between infrastructure, services, fees, and business upkeep (e.g. tax prep). Almost 200 people are paying for it and it appears most of them are in fact using it. As long as it isn’t losing money, why not let it be?

First, the obvious reason: It still takes time to operate. Once a month I have to spend my Saturday afternoon testing and pushing an update. Several times, changes in dependencies have broken things, requiring debugging time. In fact, Oasis’s cluster management and storage back-end (known as “Blackrock”) is still running a build from October 2018! For reasons I’ve been unable to determine, newer builds after that point start crashing under moderate load. I’m unable to reproduce such load in a test environment, so the only way to test potential fixes is to push out a full release, watch it fail, and roll it back. After several tries, I’ve mostly given up. Luckily, this component of Oasis has had no major changes and does not have any directly-exposed attack surface, so pinning the old version is mostly fine… but it’s a fragile position to be in.

On a related note, I am on call 24/7 for Oasis. It rarely breaks, but when it does, I have trouble fixing it in a timely fashion. For one example, in January, an unexplained Google Cloud hiccup forced me to transfer Oasis to another zone, which it wasn’t designed for (whoops, yeah, it’s not multi-homed, we never got that far). It was down for hours. Luckily it was a weekend and I was at home, or it could have been days. In another incident, I discovered that GMail had been routing all my monitoring alerts (and e-mail to support@, security@, contact@, etc.) directly to spam for months.

But, more important than the time burden on me is that I no longer feel good about charging money for this product. Almost all the app packages are from 2015-2016; many of those apps have had significant updates in their standalone versions since then which are missing on Sandstorm. Apps load super-slowly on Oasis. Many have significant missing features vs. their stand-alone versions, due to not having adapted to Sandstorm’s security model. And the Sandstorm UI itself remains woefully incomplete and janky. I constantly worry that most of the people paying for Oasis signed up by mistake and never noticed it on their credit card statements – that may sound far-fetched, but in fact I have had at least a few complaints from people who did just that (which I then refunded). I worry that it seems like we have European customers and I wonder if they realize Sandstorm is located in the US and may not comply with relevant European regulations. I feel embarrassed that people who haven’t read the blog assume the product is supported by full-time staff. Would Oasis still be profitable if it were only used by people who fully understand the state of the company? I’m not sure.

Finally, Oasis today provides almost no advantages over self-hosting. The price of virtual servers has come down to the point where self-hosting Sandstorm on an equivalently-priced server will give you a much better experience than Oasis can. Sandstorm was always supposed to be about owning your own server anyway. In fact, in retrospect, I think we never should have created Oasis, but should instead have focused entirely on self-hosting all along.

What’s next?

Sandstorm will continue to exist as an open source project. I personally plan to transfer my Oasis grains to a self-hosted server, and keep using it. I have to admit, building the mass transfer feature was kind of fun – I’d forgotten how little time it takes to build significant features in Meteor. And I’m still interested in self-hosting my email, if I can cobble together a decent UX. Maybe I’ll be inspired to build something on Sandstorm… we’ll see.

However, after the shutdown of Oasis, the project should be understood to be a hobby project, not a business. People should no longer rely on me, working in my spare time, to safeguard your data or keep it accessible.

Results of discontinuing the free plan

Kenton Varda — Sun, 28 Oct 2018 00:00:00 +0000

Recently we made the tough decision to end Sandstorm Oasis’s free plan. This change has now been made and the dust is settled. (If you’re affected and are unsure how to export your data, see my previous blog post.)

Today, for the purpose of transparency, I wanted to show you the results of this change.

First, I’m happy to report that revenue increased more than I expected:

Timeline:

August 27: MRR is $828. I announced decision to end the free plan on our blog and Twitter.
September 2: MRR is still $828. I updated Oasis UI to add prominent warnings for free users.
October 17: MRR has reached $1104. I flip the switch to turn off the free plan.
October 28: MRR is now $1428.

Second, server resources were reduced. In particular, utilization of Oasis’s “worker” machines (where users’ apps actually run) dropped in half:

We previously had four worker VMs, each of which was a GCE “n1-highmem-2” machine costing $60.50 per month. We were able to cut two of those machines for a savings of $121.

Sandstorm currently runs 13 other VMs – six others to operate Oasis itself, three to run our web site and app market, two for Sandcats.io DNS, one for monitoring and one for metrics aggregation. It’s likely that we could further consolidate some of these, although these machines run smaller-sized instances with bespoke purposes meaning it will take a lot more work for comparatively smaller savings.

In August I estimated that the cost to continue operating Sandstorm (including servers and corporate maintenance) at about $1560 per month. With the server reduction, we’re now at $1439 per month – just barely above the $1428 in revenue. So, we went from a $700/month deficit ($8400/year) to break-even by making this change.

How to download your Oasis data

Kenton Varda — Thu, 18 Oct 2018 00:00:00 +0000

We announced in August that Oasis’s free plan would be discontinued on October 14. As I explained then, we were forced to do this as Oasis costs $1500 per month to run, but was only earning $828 per month in revenue. I have been personally paying the remaining $700 every month to subsidize free users of the service, and I am unable to continue doing so. By limiting the service to only paying users, Oasis will be able to break even.

For the past month or so, the Sandstorm UI has featured a prominent message warning that the free plan was going away:

Unfortunately, we were unable to send an e-mail warning, as there are well over 100,000 free accounts on Oasis, the vast majority of which are abandoned. A mass e-mail would likely have landed us on spam blocklists (and rightly so, as we’d be annoying a lot of people).

I had hoped that the message in the UI was prominent enough that all active users would notice it. However, it appears that some people did not see the message, and now find themselves unexpectedly unable to open their apps. I’m very sorry for this!

Your data is still there

You can still download your data. To do so, open a grain and click the “download backup” button in the top bar, which looks like a down-arrow:

This will give you a ZIP file that contains all the data from the grain.

In some cases, you will be able to open this data easily. However, many apps store their data in custom formats that you may have difficulty opening. In these cases, it’s best to use the app itself to access the data, which means you need Sandstorm. One option is to install Sandstorm on your own server, but not everyone has a server available for this.

As a hack, another option is to use the Sandstorm Demo itself to get temporary access to your grains. The Sandstorm Demo gives you a temporary account which you can use to run any app on the Sandstorm app market, for free. The only catch is that your demo account will be deleted after one hour – but this should be enough time to upload your grain, open it, and copy out data. In the worst case, after your demo expires, you can always start another demo to extract more data.

To start a demo, open an incognito window or log out of Oasis, so that Oasis doesn’t know you have an account already. Then, visit demo.sandstorm.io or click the button on the Sandstorm.io home page:

Once the demo has started, make sure to install the app that your grain was created with. Then, visit the “Grains” tab, click “Restore backup…”, and choose the backup zip that you downloaded before.

You now have a (temporary) functioning version of your grain.

Oasis free plan will be discontinued October 14

Kenton Varda — Mon, 27 Aug 2018 00:00:00 +0000

Update: This change has now taken effect. Looking for help downloading your data from Oasis? See our latest blog post »

Starting October 14, 2018, Sandstorm Oasis’s “free” plan will be discontinued. Users will still be able to log in for free to access apps and grains owned by other, paying users, but free users will not be able to install their own apps nor create grains. Existing grains owned by free users will remain available for download via the “download backup” function, allowing you to transfer the data to a self-hosted Sandstorm server or manually extract it. However, the grains will no longer be able to start on Oasis unless you upgrade to a paid account.

Why do this?

Unfortunately, Oasis does not make enough money to support itself.

As of this writing, Oasis hosts 2642 monthly active users. This number is actually up in the last few months, despite Sandstorm having put no effort at all into user growth since we changed gears in early 2017.

However, only 88 of those users have a paid subscription, generating a total of $828 in monthly revenue. Unfortunately, the bare cost of operating Oasis currently comes to about $960 per month, mostly to pay for servers. Additionally, to keep Sandstorm the corporation in existence as a vehicle to operate Oasis costs around an additional $600 per month (for things like taxes, tax preparation, banking fees, etc.). Although these fees don’t technically go to keeping the service running, dissolving Sandstorm the company would almost certainly require shutting down Oasis, and so to keep Oasis running we must pay these fees.

All told, this means Oasis is running a deficit of $700 per month, which I personally pay out-of-pocket.

I do not want to shut down Oasis, because I know a lot of people depend on it and use it every day. But, it doesn’t make sense for me personally to pay for compute for every person who signs up. I need each user to pay their share.

What are the options for existing users?

If you currently have a free account with data on Sandstorm Oasis, you will need to do one of the following:

Upgrade to a paid plan on Oasis.
Switch to a self-hosted Sandstorm server. You can transfer your grains from Oasis to the new server by downloading a backup of each grain (down-arrow icon in the top bar when the grain is open) and uploading it to the new server. You won’t need to pay Sandstorm anything in this case, but by moving your grains off Oasis, you can help us shut down some servers to save money.
Download your data and extract it manually. Sandstorm grain backups are simple zip files containing all of the data the app stored. The format is different from each app – it may be a JSON file, a SQLite database, a MongoDB, etc. You may need special tools and knowledge to extract the data, but it’s all there.
Do nothing. Your data will remain intact and can be downloaded at any time. If you aren’t actively using your Oasis account, then you need not take action at this time.

FAQ

Have you considered a crowdfunding campaign?

Crowdfunding campaigns are harder than they look. Sandstorm ran one early in its life, successfully raising about $60,000. During that time, I had to spend every day for a month and a half going out and selling people on it. We lined up press articles. We had two paid employees pumping out apps. We got on Hacker News several times.

These days, I have a day job leading development of Cloudflare Workers and related projects. Workers is taking off quickly and there’s tons to do to make it better. Alas, I just don’t have time to coordinate another crowdfunding campaign for Sandstorm on the side.

Can I donate to Sandstorm via Patreon or something?

The best way to “donate” to Sandstorm is to sign up for a paid Oasis account. This is already set up, and the payment processing fees we pay to Stripe are much lower than what Patreon and the like would charge. All revenue from Oasis goes directly to paying for operation costs.

Will Oasis shut down eventually?

If Oasis remains non-profitable after this change, we’ll eventually have to shut it down. If it pays for itself, I’d like to keep it running, but I cannot make any guarantees. I will, of course, provide plenty of advance warning before shutting it down entirely.