This post serves as a permalink to accompany my talk from EmberConf 2021.

Mho

The demo of the service-worker-based build tool is available here with instructions to run both the prebuilt binary (currently compiled only for OSX, sorry!) as well as how to rebuild everything yourself.

It's demo quality. There are pieces of it that are still scaffolded. I implemented enough features of the Embroider v2 spec to get realistic results, proving that we can, for example, crawl all the addon metadata and combine their public assets. I didn't implement the full spec yet, so a few things are handled in app-specific ways. I do think the results are representative, and I intend to keep developing mho until it's a production-quality build tool.

Standalone Webpack

To run an Ember app with standalone webpack, here's a sketch of what your webpack config needs to look like. Thanks to Kris Selden for sharing his: https://gist.github.com/krisselden/7a988d65a7481dac1450dc3ae934f692

As of yesterday, ember-auto-import can use ECMA dynamic import() to lazily load any dependency into your Ember app (or addon).

To demonstrate how it works, here's a screencast where I build an app that fetches some real data from my city and plots it using Highcharts. The code for Highcharts is only loaded when needed, using dynamic import().

The demo app's source is available here.

Photo credit Guillaume Bolduc

This is my contribution to the Ember 2018 Call for Posts.

When it comes to community goals for this year, I want to highlight two major themes:

1. Taking all our great ongoing work over the finish line.
2. Letting our newer contributors shine and lead.

Stick the Landing

Our first priority should be to finish shipping the hugely valuable features that our community has been building for the past twelve to eighteen months. It's a really big list! And the bulk of it is getting so close to done. I'm including things like:

• angle bracket component invocation syntax
• @tracked properties
• no more automatic wrapping component element
• full support for ES Class syntax throughout the framework
• out-of-the-box rehydration support for all Fastboot apps
• a more pluggable build system that enables greater community experimentation
• module unification application layout
• seamless interoperability with any ESM-formatted package on NPM (probably the number one request by every other #ember2018 post I've read!)
• CSS Blocks as a first-class option for style-management in every Ember app

Many of these things can already be tried out today, because the implementations are nearing completion. If we keep executing, a year from now we will take all these features for granted, and that will be wonderful.

Let our newer contributors shine and lead

There are plenty of folks like me who have been part of the Ember community for six years. That is like two centuries in Javascript years. We have a lot of accumulated experience and that is a key part of why we've been able to build durable software that continues to evolve while bringing even the biggest, most ambitious apps along.

But we also have an energetic crop of newer contributors who have been absolutely killing it lately. I don't want to try to name names because I will miss some people, because there is more activity than I can track. They are a big reason Ember's future looks very bright to me.

Our priority as a community should be making sure someone who has been contributing great work for half a year feels as much sense of ownership as someone who has been around since Sproutcore 1.6rc2. Our newer contributors are also some of the best ambassadors for teaching and spreading Ember, because they have the zeal of the newly converted, and they don't have as much Curse of Knowledge as some of us Elder programmers.

So if you have been contributing -- for a few months or a few years, through code, docs, events, answering questions, outreach, or the addon ecosystem -- and you ever feel like you're stymied and don't know how to make things happen, please get in touch. If you're thinking questions like:

• how can I lead people to consensus on this new feature?
• how can I help implement this new things everybody agrees they want but nobody has built yet?
• am I really ready to give a big conference talk?
• how can we make the process for contributing to the docs / the guides / the website / the addon ecosystem more open and accessible?

this is an invitation to chat about it. I want to help you succeed and I urge everyone else who is perceived as an oldie with community authority to do the same.

Photo credit rawpixel

When Google set out to measure what makes their best software development teams successful, they found a strong correlation with psychological safety:

A shared belief held by members of a team that the team is safe for interpersonal risk-taking... A sense of confidence that the team will not embarrass, reject or punish someone for speaking up.

I find it interesting that this lends explanatory power to two much earlier ideas that came out of the famous Toyota Production System: "Five Whys" and "Stop-the-line". Both of them are directly relevant to building a software team with more psychological safety.

Five Whys

When a problem happens, it's tempting to accept the first explanation.

Q. Why is our site down!?

A. Because Ed accidentally deleted the production database!

The idea of Five Whys is to force yourself to keep asking questions until you uncover deeper causes.

Q2. Why did he delete it?

A. Because he typed the wrong command.

Q3. Why was he typing that command at all?

A. He was trying to re-deploy the staging environment, and accidentally hit production instead. He was typing commands because our deploy process is manual.

Q4. Why is it manual?

A. Because we're under-prioritizing site-reliability work. Let's use this incident to explain to the business why we need to spend time on automated deployment.

Q5. Why was he using credentials sufficient to delete the production database?

A. Because we use the same credentials in staging and production. I guess we should change that.

Five Whys has a direct benefit (finding and fixing deeper problems). But it has an even more powerful indirect benefit of improving your team's psychological safety. Mistakes are treated as gaps in process, technology, or training. Not as personal failings.

Stop-the-line

Another widely-admired part of the Toyota Production System is the rule that any person on the assembly line is obligated to push the Big Red Button that stops the entire assembly line if they see anything wrong. And the whole team will gather to identify the root cause before starting things up again.

When it was first introduced, this ran directly counter to dogma. Stopping the line is incredibly expensive, you're supposed to do everything you can to keep it running.

But iterative process improvements are like compound interest — over time they make an overwhelming difference that more than pays for stopped assembly lines (or paused feature development work).

It is your obligation to push the button. You never get punished for it, and you never get told you shouldn't have. Even if you turn out to be wrong, the fact that you perceived a problem where there wasn't one was itself a problem, pointing to the need for better design or better training.

In both software and manufacturing, stop-the-line leads to improvements that are otherwise hard to make because it isolates specific problems precisely when they are occurring. There is no better time to fix the underlying cause.

In software, stop-the-line doesn't involve pausing an assembly line. Instead, it means pausing work on your currently assigned feature in order to solve another problem that you hit along the way. If you hit the bug, others are likely to hit it too. So it's worth fixing, or at least worth talking to the rest of the team about it.

A culture of stop-the-line directly fosters psychological safety. Everyone — even the most junior members of the team — should feel safe pointing out problems, without fearing that they might be ridiculed for being wrong or accused of criticizing their peers.

Necessary but not sufficient

There's more to psychologically safe teams than just these two principles. But I think they're worth talking about because they're relatively easy to teach and understand, and they may catalyze introspection that uncovers other cultural/interpersonal problems.

They're also a litmus test for a junior team member who's wondering "am I bad at this, or is this a toxic environment?". If your organization can't or won't honestly engage in this kind of self-examination, it's not you, it's them.

Image credit Creative Common by-nc-nd Toyota UK

“A lot of beginners under-appreciate the degree to which googling things is part of a programmer’s job,” says @steveklabnik #wednesdaywisdom pic.twitter.com/FnXo3rBfPb

— Glitch (@glitch) May 31, 2017

That quote is true, and it's a specific example of a more general principal: get comfortable with not being able to fit everything in your head simultaneously.

Beginners often try to hold whole programs or whole problems in their head. And they start to unconsciously believe that getting better at programming involves being able to hold bigger programs and bigger problems in their heads.

This is a trap.

I suspect that people with incredible working memory are actually handicapped at learning programming, because they can get through most practice problems without ever learning the critical skill of paging things in and out of your mind.

You literally need to be OK with forgetting things in order to free up space, and trust that you'll have a good system for relearning what you forgot next time you need it. Sometimes that involves Googling, sometimes it's a well-crafted search through your code or documentation.

This ability to retrieve information on demand is the underlying motivation for all the things we do to manage complexity. Well-structured code allows you to forget about most of it most of the time, and learn or re-learn individual pieces on demand.

Photo credit mimitalks via Flickr, CC by-nc-nd.

Lots of software projects struggle when they try to implement search-related features. I was reminded of this by @trek's reaction to this Algolia marketing tweet:

Development teams often delay building search due to lack of confidence in getting it right & fear that it will take longer than expected.

— Algolia (@algolia) May 11, 2017

Stop reading my diary please. https://t.co/cnWlInPm83

— Trek Glowacki (@trek) May 11, 2017

I have heard nothing but good things about Algolia's product, and if you're one of those teams that has been putting off the "Add Search" item on your TODO list, outsourcing it to them is probably not a bad idea.

But instead of treating search as something you can tack on later, what if you make search the foundational primitive for your whole application? That is the approach we're taking in Cardstack.

In a trivial sense, every screen of your application is already driven by "search". An SQL query, a GraphQL query, a JSONAPI GET: these are all specialized forms of search. But when the requirements change and you need to tune any of them for relevance, personalization, fuzzy matches, relatedness, or additional heterogeneous types, you're left painstakingly hand-coding new behaviors. These searches are brittle because the data access patterns and the resulting UI patterns are all hard-coded directly into the application.

In contrast, a search-first application:

1. Handles every data fetch via a search engine: a system that is good at both structured and fuzzy queries, optimized for low-latency. (Examples include Elasticsearch and Algolia.)

2. Allows the search results to drive the resulting UI. Every piece of content returned by the search is expected to have a corresponding UI component that is appropriate for displaying in the current context.

This results in fluid applications that are better at uniting disparate pieces of data into holistic experiences. And it leads us toward the big-picture vision of an open card ecosystem — once the whole experience is driven by search and families of UI components with standardized external APIs, there's no reason all the components need to come from one vendor/application/author. The user can remix micro-applications together into more than the sum of their parts.

There is a popular trend among some Javascript developers to argue that humans should stop writing CSS and HTML (and HTML-based templates). They argue everything should instead be written as Javascript, to reduce complexity and present a more unified developer experience.

And certainly many people who try this approach like it. It works well for a certain kind of team. But this approach over-optimizes for the present, and discounts the future. It's going to get leapfrogged by technologies that are already starting to land.

As the web platform matures as an application runtime, it's starting to become a true compilation target.

Q: Sure, almost everybody transpiles something these days, how is this news?

A: There is a critical tipping point that we haven't hit quite yet. A true compilation target is one that you don't need to understand to use.

Today, everyone doing practical work in compile-to-Javscript languages needs to know Javascript too. The tooling simply isn't mature enough and the abstractions leak too much. Even if you write in some other language, you need to think in Javascript to debug anything.

We have seen this all before. There was a time when you couldn't be a professional programmer without knowing C. Your higher-level language was implemented in C, and it would break a lot, and you would need to debug the C layer. Further back, you couldn't be a programmer if you didn't know assembly, because C compilers were too leaky an abstraction, and debuggers were not good enough to debug-while-thinking-in-C, so you had to debug-while-thinking-in-assembly.

But eventually, layers mature and become mostly ignorable. You don't need to know assembly today to be productive in most parts of programming. You don't need to know C. And we are on the cusp of the day when many web programmers won't need to know Javascript.

The reason for this is that both the runtime capabilities (in the form of WebAssembly and ASM.js) and the develop & debug facilities are advancing quickly.

Every time this process has repeated in the past, the old guard who were already experts resisted the change, and denigrated the newcomers who had the gall to call themselves programmers even though they didn't know assembly! But the newcomers win in the end, because there are so many more of them, and the nexus of activity moves up the stack.

It's the nature of software to make last year's cutting-edge problem into next year's moderately-hard problem into something a precocious child can do a few years after that. Realize there are many of us out there working toward the point where a kid in a weekend can reproduce your $X million app by plugging together reusable components. It's not a question of if, but when.

And we're going to succeed because we control all the language semantics and choose how they map into underlying Javscript (or webassembly) semantics. That ability to choose gives us immense leverage over the future.

How does this relate back to whether HTML and CSS belong in Javascript? The bet to make is that we're going to see more use of specialized languages. And HTML and CSS are the grandaddy specialized languages that have enough social consensus and capital investment to be the seeds of the next generation. A project like the Glimmer Virtual Machine is really a new programming language being born, one that has standards-based HTML baked into its DNA.

When we succeed, it will not be because we turned everyone into Javascript ninjas. It will be because we made it radically easier to compose applications. Fully-general programming languages are irreducibly complex. You can only go so far in automatically helping people with them before you summon demons like the halting problem. Specialized languages can put a velvet rope between most users and the deep abyss of computational complexity, and present gradual onramps for users who have the need and desire to begin exploring those nether regions.

If you plant your flag on "it's just Javascript semantics", you don't have that flexibility. You cannot evolve as far or as fast.

The asteroid is coming.

Photo credit: Daniel Mennerich via Visualhunt / CC BY-NC-SA

Many people have expressed an interest in contributing to Cardstack and have been clamoring for some docs to help them get started. This is the first of three posts I intend to ship in the near term to unlock some of that potential energy:

1. Cardstack Architecture Notes. This post. A high-level overview — breath over depth — that points out many of the key concepts and their motivations.
2. Roadmap. Where are we headed, in terms of features that don't exist yet but are needed.
3. Working demo app.

For a more general introduction to Cardstack and the motivations behind the project, you may want to watch this talk I gave at EmberConf:

Orientation: Relationship to Ember & Glimmer

Glimmer is a fast, small UI component rendering engine for the web. It gives us declarative templates that are carefully designed to be a superset of HTML that's appropriate for rich, dynamic applications. Glimmer's internals are fascinating, though not critical to this post. (It has an optimizing compiler and two-phase bytecode virtual machine that's as fast as virtual-dom implementations on initial render while remaining faster than them on update.)

Ember is a batteries-included framework for building ambitious web applications. It is built on top of Glimmer (which was originally extracted from Ember's fourth-generation rendering engine). Ember's greatest strength is its welcoming community that champions shared solutions and stability without stagnation, which has enabled a deep ecosystem of shared code to grow and flourish. Ember is an antidote to "Javascript fatigue".

Cardstack is built on top of Ember, and so in one sense is just a family of Ember Addons. But Cardstack differs from Ember by also offering standardized server-side code. This allows Cardstack plugins to do more out-of-the-box than Ember addons can normally do.

My goal is to bridge two worlds:

• Cardstack should be useful to people who are already writing Ember apps who want to incrementally adopt its features. They are the early-adopter audience who can get the most out of it today and who can help push the project to maturity.
• But we are also consciously building toward the point where a new, much bigger audience can adopt Cardstack: organizations that don't have the experience, budget, or appetite for risk that's needed today to build custom in-browser applications. People who would have otherwise chosen from the earlier generation of content management systems, because those systems do more out-of-the-box.

This goal dictates parts of the high-level Cardstack architecture: it's implemented as a family of separate packages, so that it can be incrementally adopted and the individual packages can be immediately useful in the Ember ecosystem. And it's designed with the ultimate goal of letting people ship ambitious applications to production without writing a line of Javascript.

Source Code

The source for all the Cardstack packages lives in the cardstack/cardstack repo. Each package is published to npm separately, under the @cardstack organization.

Several related Ember addons that are not Cardstack-specific have been spun out as completely independent repos, such as ember-toolbars. This follows the general rule: purely client-side concerns fall within the sphere of Ember, and so they ship as plain old Ember addons. Concerns that span client and server are Cardstack plugins, the core set of which lives within the cardstack/cardstack repo.

Hub, Plugins, Features

The main server entry point is the @cardstack/hub package. The Hub is not a framework for writing server apps, it is a server app. You configure and customize the Hub by adding plugins.

Cardstack plugins are npm packages that have "cardstack-plugin" in their keywords list and a cardstack-plugin section in the package.json file. By default, their Cardstack-specific code is located in their top-level directory, but this can be customized by setting the src property in the cardstack-plugin section of package.json (this is important because many Cardstack plugins are also Ember Addons, and it's nice to be able to use a completely stock Ember Addon layout and move the Cardstack-specific bits into a subdirectory).

Each Cardstack plugin can provide one or more Features. Examples of Features include:

• field types
• constraints
• writers
• indexers
• searchers
• authenticators
• middleware

Running the Hub

To start up, the Hub needs seed models. These contain the minimal set of configuration needed to access the rest of its data. For an example, see the blueprint included in @cardstack/git.

In addition to being a standalone node package, the Hub is also an Ember Addon that registers development-mode and test-mode middleware with ember-cli, so that the Hub runs automatically when you do ember serve or ember test. When run this way, it mounts itself under the URL /cardstack within ember-cli's web server, and it conventionally loads seed models from cardstack/seeds/development.js or cardstack/seeds/test.js.

In production, you can start the Hub server directly via @cardstack/hub's bin/server.js, and pass the location of the production seed models as the only argument.

Don't actually deploy the Hub to production right now, this is pre-alpha software and not all authorization features are fully implemented.

Plugin Loading

Plugins must be activated before they will take effect. Eventually this will be controlled via admin UI, but for right now you do it by adding a plugin-configs model whose module attribute contains the name of the plugin's npm package. For example, assuming you are running the Hub via ember-cli in development:

POST http://localhost:4200/cardstack/plugin-configs
Content-Type: application/vnd.api+json
{
  type: 'plugin-configs',
  attributes: {
    module: '@cardstack/mobiledoc'
  }
}

Data Sources

The Hub is not the authoritative storage location for any of your data. It has no database of its own, at least not in the way you may be accustomed to. What it has is a fast cache & search index, powered internally by Elasticsearch. It uses plugins to index, search, and write to whatever set of upstream data sources you need.

This gives us the ability to present an idiomatic, JSONAPI-compliant, performant API with uniform query semantics, even when the data comes from disparate legacy system, third-party services, or hosted databases of your choice.

Data source plugins are implemented via some combination of

• an indexer, which ingests upstream data and emits JSONAPI documents for the Hub to cache and serve. A good example is the indexer in @cardstack/git.
• a writer, which writes changes directly to the upstream data source (after which an indexer can pick them up -- we always go round-trip to avoid split-brain synchronization problems). A good example is the writer in @cardstack/git.
• a searcher which can do data-source-specific deep searching of data that you don't want to fully index. A good example is the searcher in @cardstack/github-auth, which lets you choose to outsource your user models to GitHub.

Schema

The Hub needs to understand your content's schema in order to properly index it, expose endpoints for reading and writing it, validate it, etc. Within the Hub, content, schema, and configuration are all represented as JSONAPI documents.

Consumers of the Hub's web-facing JSONAPI interface don't really need to make a distinction between what is content vs schema. However, authors of data source plugins need to be aware that schema is defined as models that alter how other models will be indexed. For example, if you add a new field to a content-type model, that alters the schema (and potentially requires some content to be reindexed). Whereas if you create a new article model, that is just a content change.

The list of types that are treated as schema is located here in the source. The @cardstack/hub package also contains the bootstrap schema, without which we would have no way to getting started (what fields are allowed when POSTing a new content-type? The bootstrap schema is where that information comes from.) The bootstrap schema contains built-in types like:

• content-types You might create content-types like "events" and "articles". Each of these can map to a DS.Model class within ember-data.
• fields You might create "title" and "body" fields that you will want to attach to your "articles" content type. Each of these can map to DS.attr, DS.belongsTo, or DS.hasMany in ember-data.
• constraints You might attach a "length less than 40 characters" constraint to your "title" field.
• default-values You may want to create a "now" default value that you can attach to a "last-edited-date" field.
• grants You may want to express a rule that users in a particular team may edit or read certain content types or fields.

There is no requirement that the schema for a content-type and the content-type itself are stored in the same data source. You can keep all your shema models in a data source like @cardstack/git, even though some of the types are stored in a different data source. Alternatively, data source plugins are free to emit schema models if they want to -- this allows dynamic discovery of upstream schema.

JSONAPI

The @cardstack/jsonapi plugin contains Hub middleware that exposes endpoints for all your content and schema models. The intent is that you shouldn't need to implement custom middleware just to hold business logic -- business logic can be done via a combination of plugins that provide authenticators, grants, constraints, default values, and writers.

Authentication

The @cardstack/authentication package provides support for clientside sessions (via an ember-simple-auth authenticator) along with a Hub middleware that provides the server-side endpoints for establishing sessions.

Activating @cardstack/authentication opts your app into Cardstack-controlled sessions, but it doesn't implement any particular strategy. For that you also need to activate a plugin that offers an authenticator feature and configure it as an authentication source. An example is @cardstack/github-auth, which uses GitHub OAuth login.

Authentication plugins shouldn't necessarily dictate anything about your user model or where it's stored. The @cardstack/authentication test suite has illustrative examples of the ways you can rewrite upstream users to link them with your own user content-types.

Authorization

Authorization is built into @cardstack/hub, since it tends to be a cross-cutting concern. We have a fairly complete implementation of authentication for all write operations at both the content-type and field level. The upcoming roadmap post will go into more detail on what's needed for read-operation grants and mappings between users and groups.

Clientside Tools

The @cardstack/tools package contains the generic clientside components for rendering and tracking Hub-managed content within an Ember app and making it editable as appropriate.

The cardstack-content component is the entry point for rendering any piece of Cardstack content. It accepts a content model and a format:

{{cardstack-content content=model format="page"}}

It then uses naming conventions to find an appropriate component template to use. In the above example, if the model is of type "article", we would look for a cardstack/article-page component.

Within that component template, you would use the cs-field component to display the fields from the content, either using their default renderers:

{{cs-field content "title"}}

Or by accessing the field's value directly and customizing how it's handled:

{{#cs-field content "avatarURL" as |url|}}
  <img src={{url}} />
{{/cs-field}}

In either case, during editing the presence of cardstack-content and cs-field lets the tools discover which parts of the page belongs to which piece of content and where its fields are rendered.

An upcoming goal is to use a compile-time template transformation to simplify {{cs-field content "foo"}} to {{foo}}, which would work because we can do schema-dependent compilation.

Models

The roadmap post will cover this in more detail, but the general idea is that we want to generate Ember Data models directly from the Hub's schema so you don't need to hand-code them. This is not implemented today, and when experimenting with Cardstack you will generally need to create your own models (and probably adapters and serializers).

Routing

The @cardstack/routing package lets you delegate a subsection of your Ember app's routes to Cardstack's conventions. It provides routes for all content types, pleasant error handling for missing content, and branch query parameter support that lets your preview different versions of your site.

It's fine to not use @cardstack/routing, particularly as you're learning and debugging. It's not especially configurable yet and you may have more luck making your own routes, peeking into @cardstack/routing for ideas. But of course the longer-term goal is to not require people to hand-write routes, so this package will be important.

Field Type Plugins

Field type plugins provide

• Ember components for rendering and editing
• server-side functions for validation and search indexing
• client-side functions that control how to display placeholder content for empty states

The most comprehensive example is @cardstack/mobiledoc. It provides three components. Their names follow Cardstack conventions that allow them to be discovered by @cardstack/tools:

• field-editors/mobiledoc-editor implements the component that appears within the Cardstack sidebar to represent a mobiledoc field.
• inline-field-editors/mobiledoc-editor implements the component that is rendered inline on top of your field's content while editing it. This is optional, not every field type needs this.
• field-renderers/mobiledoc-renderer provides the default rendering output for mobiledoc fields.

It also includes app/fields/mobiledoc.js, which customizes the way empty mobiledoc fields appear.

And it includes a cardstack field feature cardstack/field.js that provides the server-side methods used by the Hub to customize how mobiledoc fields are validating, indexed, and searched.

Search

The general strategy for assembling many pieces of content into a particular page is to provide a @cardstack/search package that provides:

• components like cardstack-search that accept a query and yield content models
• a field type implementation for query, with editor components that let end users build and tune search queries.

Branching

Cardstack uses the concept of branches to allow multiple versions of a site to coexist at once. This enables powerful workflows.

While the concept and name "branch" definitely comes from Git, it's important to note that other data sources that don't happen to be Git can also be configured to support multiple branches of your site. For example, a PostgreSQL plugin should allow you to provide different database configurations that correspond to site branches like "dev", "staging", and "production".

The intent is that code, content, and schema can all be changed on one branch and will take effect immediately when viewing that branch's version of the site.

There are necessarily some exceptions, and a given server has a "controlling branch" that is more important than the others. The controlling branch is where the Hub reads configuration that controls authorization decisions, the base set of data sources that are enabled, etc. Attempting to do all these things per-branch gets too difficult to understand and manage.