Before I jump into this one, though, I have a favor to ask:

For the past two years I’ve been working on a startup building immersive, roomscale arcade games. Think VR without the goggles. It’s called Third Wave Arcade. We have a huge opportunity to build our first arcade with a great partner, but we need to raise some capital for a round of software development. If you or anyone you know are investing in this space, drop me a line for more info.

OK back to it — to recap, (“last time, on Augmented Realist”), the last post in this series laid out a structure meant to be the second in the chain of a system that allows anyone to attach any digital thing to any real thing - if you didn’t read it, check it out here:

At the end of that post, I promised to circle back to the first link in the chain: segmentation and hashing. I think more than ever it’s apparent that a nearly universal segmentation system is possible, potentially enabled via multimodal LLMs trained on synthetic data.

This represents a more opaque intertwining of language and bias into scene understanding than even a more direct segmentation-specific system trained on hand-labeled data from a single part of the world. For many LLMs, we don’t know what they’re trained on, nor really how they arrive at conclusions. That opacity is a bad thing.

This piece of the puzzle is a moving target. It’s by far the most implementation-dependent, and thus mercurial, of the proposed system. The nitty-gritty specifics of a chosen implementation will have second- and third-order consequences in domains like privacy, interoperability, encoded bias, and modularity.

Trying to speculate about this piece while the state of the arts flew away like buckshot, I began to think again about building a demonstration of this system, to ground the exercise in something closer to home for those paying attention and, perhaps most importantly, for those who couldn’t care less.

I came up with a stunt - a very useful, very compelling, very dangerous stunt.

Imagine you could, for starters, attach anything digital to anyone’s face.

Put another way, imagine a system that made everyone’s face into a kind of QR code that pointed to a kind of link soup - a list to which anyone could add anything hosted online.

This is doable with today’s tech, but it’s an inversion of the way we think about our online identities.

Today, your identity is an anchor onto which platforms build a public ledger about you. Your dossier lives inside their walled gardens, subject to their rules.

But in this new model, identity itself becomes a kind of writable surface. Anyone, not just corporations or the person themselves, can affix meaning to your face—links, claims, media—without needing permission from a gatekeeper. It’s not just that people are posting about you. They’re posting on you. Or at least on the open index that your physical presence now anchors.

Importantly, the identity, and the content, live outside any platform.

As a corollary, we’d want the owner of the face to have the first say - the ‘choos’, for those of you who read the last post. We should be able to ask not only ‘what has been written about this person’ but ‘what does this person have to say about themself?’

Primitively, you could think of this as someone’s homepage, or their vCard, but creative users and years of culture will lead to sophisticated executables and augments that turn people themselves into experiences, applications, and operating systems. This was the idea behind the title image of the Say and the Choos post above - it depicts a street hawker selling handbags via her choos - only a few of the handbags are real, the rest are her virtual storefront. (Reposted here):

I genuinely considered - and still consider - building this, to prove a point. To open Pandora’s box. It would foment a wildly disruptive shift in our society, first in some very obvious ways, and later in knock-on effects hard to predict from where we sit today.

If we were to extend the system to all people, places, and things, what would actually happen? I realized that I should probably work some of that out, so my advocacy is better grounded, and the language more lived in, even if it’s impossible to know all that could come of it.

I’ve been arguing that some version - a worse version - of this is inevitable, and that we’d be best off with a free and open system. But what would that look like? Predicting online commerce in 1988 didn’t take a ton of imagination, but who had their money on scambaiting, mukbang, liveblogging terminal illnesses, fictosexual communities, on speedrunning Wikipedia bans?

To that end, back in Jan of 2023, I wrote a little speculative fiction. I’m copying the first chunk of it here. I don’t write fiction so, you know, YMMV, no express warranty, etc., but what I’m hoping to accomplish most of all is to imagine a world where secrets are rare, where expertise is diffuse, and where everything is an infinite palimpsest, and it’s been that way for a while.

Send me an email, leave me a comment, tell me what you think.

All the Kinds of Glass: Draft: Excerpt

First everything came to us in a trickle. Then our ids everted, shouting outside so loud we couldn’t manage a private thought, couldn’t touch anything without its opposite, the act of thinking a matter of a special effort, drowning in anything you could want to see or know.

The whole thing ran on making others want things, or want to do things, and the scale of it and the stakes - the money to be made - was so huge, before we realized it we'd built up corps that straddled new nations with everything riding on making it easier and easier to change others' minds, or to slip ideas in without them noticing.

It got so we didn’t believe it - any of it - each other, or ourselves, and we knew we had made a mess of the chance, that we’d blew it, broken something, and that the broken thing we made was so big we didn’t have room to make another alongside it, like we were in a house together and it was full, to the walls and up to the ceiling, so stuffed with untrustworthy trash we'd brought home we couldn’t dream of bringing anything else in, no matter how much we wanted it, or thought we knew better now.

This is to say nothing of proliferation of shit the AIs dumped out — factory farms of inanity and horror and ersatz help and polemic jamming every signal on every side until there was no way and nowhere to look anymore for voices of real people you might believe about something, were you even inclined to believe anything anymore.

All that was left for it, it seemed to some, was to wall in the people we wanted, to form clubs and cliques and cults where there were rules, where things made sense, and where we didn’t need to confront abhorrent ideas, or at least there was something to be done about them. A dominion for every sect, and despite what some said, they weren't all populated by ordinaries with more in common than they'd think with those across the wall.

No. Sorted like that, it was easier for a gripe to become a chant, and from there a war cry and a fact and the basis for the next campaign, the Overton window flying faster and further ahead like a late-day shadow, in all directions away from the origin, until for everyone the world seemed impossibly, irredeemably crazy.

But then there was gear, a joke at first too. They called them ‘poongoogles’. Middle fingers pointed them out everywhere, pocket protector for the face. But money talks, and influence at any price is cheap when you've spent billions already and staked your kingdom, and so minds opened, slowly at first, then faster as the right people were paid to say the right things or share the right pictures.

At first it was an expensive way to use the flatnet, as the Web became known, to use it bigger and more of it at once, for the rich in their homes and offices, or ostentatiously in coffee shops and on airplanes, and gear still had nothing to do with the stuff around it, and the flatnet was as balkanized and carved-up as ever.

Wearing, with the old rectangles wheat-pasted dumbly between us and the world, was briefly even more introverting than how we used to pour our focus into rectangles that others could see, and that we could look away from. But not for long.

The second flier hit Fan Deng mid-chest with a wet chuff and a cloud of pink mist. He puckered in the middle, folded, ass out, and went face first onto the engineered floorboards, with a dull, soft sound like a duffel of pumpkins dropped from waist height.

Ren's hands went reflexively to his temples. He felt an immediate flush of anxiety and guilt, as he did every time they had to hurt somebody. "Tell me he didn't fall on his gear,” he blurted. Fear throbbed behind his eyes.

He crouched by the corpse, tilting his head sideways, cheek to the bamboo, peering into the space between Fan's face and the floor. He pried by the shoulder, rolling the dead man onto his back. The gear looked intact, and Ren shot Krit a look of incrimination and relief.

Krit, looking on placidly, spread his hands as if to say, "I just shoot em", as a third and fourth and fifth drone buzzed in to land in the nylon fanny pack on his hip. “This wasn’t my idea, bro,” he offered quietly, just audible over the ringing in Ren’s ears.

Down on one knee, Ren slid a slick black lozenge - a witness - in front of Fan's unseeing eye. It rested directly on the eyeball, below the curving stick of gear extending from his temple. The witness read like a giant pupil, and with his slack face made Fan look to Ren like a cartoon boy that had been bonked on the head.

Fan - a body now, in boxer-briefs and a bloodstained polo shirt, lay perfectly still, somehow also quickly sagging. Ren gazed up slightly, focusing on the PIP relayed to his own gear. He saw the ceiling and a modal that said

FALL DETECTION
It looks like you've taken a hard fall.
Contact emergency services?

"We gotta do this quick," Ren said, nodding at the blood pooling behind Fan's back, "Help me sit him up."

Krit hiked the knees of his trousers and squatted down behind the corpse, prising his thick hands behind the body to lever it up to a straight-leg sit, staying there with his hands on its shoulders. Below his cuffs poked a clutter of faded flash — a Mary, a Puerto Rican flag, names in blackletter type. The witness stayed put over Deng's eye.

"Doll is in my pocket," Krit said, tilting his head left.

Ren began to reach for his pants. "Jacket," Krit said, his loafers creaking as he shifted his weight.

Ren removed the totem gingerly, but it was solid wood and felt heavier than it seemed. He looked it in the face. It was crudely made, with jute hair and a simple features carved into its dark, smooth surface. Something inside him lurched a little.

Inhaling deeply, he grasped it from the back and held it at arm's length in front of Fan's face, showing it to the dead man's gear. Through the witness, Fan's head gave him his answer.

"Oh," he said, too quiet for Krit to hear.

"I got into that car say," Krit announced, around a mouthful of musubi.

"I thought you weren't boosting," Ren said, "Mister would shit." He was half listening, half thinking about the recording on the witness in his pocket.

"Nah nah not boosting," Krit said, "just cool is all. Get title, history, public stuff from the muni say, but mixed in they’s exploits, prices, scrap, and rate on boosts from everybody in the covey."

"That sounds like a lot — how do you wear that with all your other shit? There's cars everywhere." Ren said to the sky above him. He was drinking an iced coffee and reclining, legs straight out, on a park bench. His gear was opaque black in the sun.

Ren wore a light grey tropical wool suit, crisp white shirt unbuttoned down to mid-chest, and brown leather derbies with no socks. It was humid, high eighties, with a languid breeze that made it feel hotter, but he made a point to get full sun on his skin every day, especially in winter, and sometimes the only chance was lunch.

His skin was uniformly dark year-round but he'd often take his bare feet out of his loafers to even a tan line only he could see. Today though, the image he got off the totem, through Deng's gear, had him unsettled, and being barefoot in public had always made him feel unprepared. He kept his shoes on.

At the edge of his vision, a faint lime-green field, shapeless, languidly shifting across a lowering sky, foretold rain on the way, peeking from behind midtown buildings to the West. Behind the blocky green he could make out the darkening ledge of a shelf cloud, looming.

Krit was walking back from the bodega. He gestured at the row of cars parked along the street beside him, in case Ren was taking his feed. As partners, they left a line open during business hours, but Ren usually only peeked Krit if they were separated and Krit got into some shit, or he had something important that couldn't wait.

"I'm not rinsing it all the time," Krit said, "I have my head bump me if I go by something good though - G rides or really hard hacks. Just cuz I still like this shit.

"I used to have it comp so it was, like, window stickers on the car windows? Like at the dealership? I thought that was cute but after a while I realized it was kinda conspicuous. The way I had to like, lean at the cars to read it? So now I just use the whatever OS bob and it's above the car most of the time. The say stuff is jus text so you can do whatever with it anyway.

"I don't search off it but I looked around when I first got in? There's a bo-nanza bro!" Krit laughed a big guy belly laugh, wet and half cough. "You gotta pay for a lot of the shit but the say links out to other covey hits and you can do a whole boost and fence without touching shit on some of the older rides. It's wild bro!" He laughed again, raucously, hacking a little, "fuckin things sell themselves ... I still love it man."

Here he sidestepped to dap beefy fists with another heavyset guy leaning by the dock of a commercial building, pointing a finger at his temple to explain he was on a call.

"You want in? Need 3 from the covey to vouch."

Ren shaded his face from the sun with one hand and let the other dangle at the end of his slack arm, eyeing the looming rainclouds.

"No thanks," he squinted, "don't need another firehose. Sick of fucking with the says I got. I've been dropping some, trying to dry up a little. Can't see where I'm going half the time and I don't have the energy to be tightening up my head every time I add a new whatever." He looked around the park with his brow knitted.

"Aiight," Krit said, "You get what you needed from the doll?"

"I got something," Ren said, into his fist, "might need Mister to tell us what."

Jackie scanned the marsh and woods where they hemmed the perennial meadow around her. In her gear, invasive grasses, shrubs, vines, and trees lit up in shades of blooming additive fuchsia and hot lavender, burning in the comp with real greens and browns.

"Blech," she said to the trees, brushing the palm of her hand over the catkins on the hip-high grass at her side.

She glanced back at the herd of goats grazing behind her, then subvocalized, "Head - let me see just the Bovidae tox again. Invasives." Most of the pink dropped away, leaving a handful of shrubs and ground cover. She spat.

“Is this including state extension say?” she asked silently, her throat barely moving with the minute movements of her tongue and jaw.

“Yeah,” her gear told her.

She squinted, thumbing a rivet at the front of her coverall and eyeballing the meadow border. She looked down at her muddy boots. A row of dense clumps of thin, long-bladed sedge led to a much larger stand of the same. At the base of the clumps, standing water poked through a silty clay soil.

"Isn't this appalachica?" she said aloud, crouching to pull a perigynium closer. "Head?"

"Yeah," her gear said again, and a broad mat of the plant burned in pink, seeming to follow the lowest, wettest area of the marshy plain between her and the woods. She looked up to the tree line and back to the sedge.

"Gimme ephemera for September 21," she asked, and just as she suspected, the arc traced through the sky was dotted with fully-visible sun-sized circles nearly sunrise to sunset.

She looked back to the grasses, and spat again, nudging a clump with the toe of her boot.

"Fuck are you doing here?" she asked the grass.

Staples sat alone at the small bar in what used to be a kind of speakeasy. The tiny space was warmly lit, with all its tight tables, and the 6 stools at the bar, occupied by patrons making quiet conversation. No customers stood. No one was allowed to enter without somewhere to sit.

Midway through the short row of two and four-tops, tucked against the glass curtain wall, two guitarists plucked out an almost inaudible bossa in the tight nook where a table had been removed for them. The warm air in the bar smelled like rosemary smoke and simple syrup.

Two Japanese bartenders demonstratively stirred crystal mixing glasses held at eye level, gazing closely at hand-cut ice cubes as they twirled their bar spoons in a way that made as little sound as possible.

Outside, pedestrians hustled around a triangular square, their steps prodded on by the obvious rainclouds. At the plaza center, a large cube sculpture perched improbably on its corner.

Staples contemplated the menu bob, which appeared to glow through the distressed mirror of the antique backbar. The place still had paper menus available, an affectation more than anything, but the server hadn't been by yet to offer him one. The server role itself was a similar kind of holdover — almost everywhere you could order from the menu directly, through your gear.

Like the other, slower methods at work behind the bar, the paper menu and the gentility of ordering with a human contributed to a feeling of intentionality that differentiated the place with a decorum Staples appreciated when he needed a break.

Unbidden, gently, a field of fine blue lines converged to a point off to his left, slightly behind him, at a table near the corner. His gear had picked out the face of someone in the room. The style of the field said it was a person of interest, and the color said it was someone he'd labeled 'work' or 'business' in his personal, private say.

He'd come to the bar to think alone, but if he'd faced whomever this was, they'd probably faced him too.

Since gear had become ubiquitous, new and more elaborate faux pas arose like warts where gentility's premises lagged new powers. He fought instinct for a second and rubbed his eyes.

Decades before, wearing had begun in fits and starts. Early, multi-billion-dollar commercial efforts had focused on resolving the formidable technical issues in making the hardware practical.

The largest companies in the world each pursued a vision of monopoly control in what was generally agreed to be the largest single market of any kind ever created. They assumed gear, like smartphones before it, would find an audience of early adopters first, and that a raison d'etre, or a killer app, would naturally follow, to be succeeded by more or less the entire economy, marching straight into their new storefront.

It sort of worked out that way. The timing of the first serious entrées to the consumer market coincided with a growing ambivalence towards tech's influence on life in general. They launched to little fanfare among waves of studies showing the ill effects of screen time and internet use among the youngest cohorts to have grown up with social media.

Industrial, commercial, and military uses were the first to find revenue. But for decades, despite the hardware shrinking, from face masks to goggles to smaller goggles to enormous glasses to less-enormous glasses and so forth, a conspiracy of social stigma, cost, impracticality, and the lack of a compelling use case left consumer gear largely in the hands of gamers, gear-heads, artists, and pornographers.

In the end, it was a group of hackers, or artists, or activists - to this day nobody knows who exactly - that ripped off a number of band-aids at once and slipped society a kind of transhuman roofie.

Intended as a grey-hat demonstration of the danger of facial recognition surveillance, myLittlePwndy showed the world the power - and inevitability - of using the meat world itself as an anchor to the digital.

As a demonstration, the system was simple and self-contained; anyone could upload any image or video and get back a list containing one 'Person' per face that appeared in their submission. That Person - just an ID number, really, was, for the most part, unique, and, importantly, deterministic.

At first, the system didn't have any information beyond that. No names, no concordant data at all shipped with the system as it was. It was simply a map from face to number.

But the thing that made myLittlePwndy subversive - made it world-changing - was that anyone could submit anything for any face.

Want to tag a celebrity? Label yourself? A friend? Add a bio? A link?

Leave an anonymous love note on your crush. Spread a rumor on your boss's face. Doxx an enemy with their home address and phone number. Scribe conspiracies on the likeness of ex-presidents. Nail revenge porn to your ex. People got creative, fast, and the tech press went wild over it.

It was like, all of a sudden, everyone in the world had an uncensorable, unmoderated, public comments section attached directly to their very person, and there was nothing anyone could do about it. Until Congress got involved.

Where Europe had spent the last 50 years doing their best to regulate new technology before the worst problems arrived, the US had generally continued the laissez faire post-hoc strategy they'd employed since the first days of the internet. There were noises made about anti-trust action, about forcing popular foreign-owned services to onshore their data and shed their entanglements, and the grand tradition of dragging tech CEOs in Congress continued unabated, but the federal government had remained effectively hands off tech straight through the first two AI booms.

myLittlePwndy was too much to ignore. Social media and the big free services had been privacy-obliterating but at least they were opt-in. This was something else entirely, utterly inescapable, and something about how it involved people's actual faces, rather than some abstract idea of their online identity, brought the drumbeat to a fever pitch nearly overnight.

This being America, though, appetite for an out-and-out ban on this complex behavior was expressed by only the most insecure members of Congress, with most Senators advocating a more measured approach emphasizing preservation of the rights of all involved, albeit for different reasons. There was the right to free speech, the freedom of the press, the people's interest in receiving information, the freedom of corporations to make gobs of money, and plenty of precedent to suggest that the courts' interpretation of this activity would protect the general idea as speech about a person, albeit not without reservation.

These explicit rights would supersede any implied right to privacy when it came down to it, but by this time, individual online identities had become so culturally and financially important, so ramified and woven into life, that this first real link from the digital to the meat, ineluctable as a fingerprint and suddenly more important than a social security number, forced the issue to the top of the national agenda.

We must, it was argued, 'own' something - some real estate - at the front door of the connection from our physical form to the digital. It would take a constitutional amendment to prevent what was widely agreed to be a kind of speech from taking place. The usual laws around libel, slander, truth in advertising, hate speech, etc., would continue to apply, perforated as they were by what was left of Section 230, but the chain of concepts underpinning 'writing' 'on' someone's 'face' were, when it came down to it, too much of an exquisite corpse of innocuous and previously-protected behaviors to prevent, even if the amalgam felt altogether new and scary.

But, the thinking went, we could mandate that a person have the first word, if they wanted it, about themselves. It didn't neatly fit into any existing protections but straddled just enough Americans held sacred that a relatively unambiguous and truly novel bill passed with bipartisan support in a bitterly divided Congress. The law said that any service providing digital information to the public based on a user's identity had to allow that user to dictate a canonical response of their choosing, for free. A kind of a first or last word.

This personal payload became known as a 'choos' - pronounced 'choose' - and its codification in law and the furore that accompanied its birth were the instigating event for what became A Space, a transition that in retrospect was in some ways more important to society and to the human experience than the birth of the internet that preceded it. It was almost as though the internet, up until that point, inestimable as its impact had been, was simply a caterpillar inching along, waiting to wrap itself up, liquefy to an undifferentiated goo, and subsequently eat its way out of its cocoon, unfold new wings, and absolutely fuck shit up.

For it was myLittlePwndy (later to be supplanted by its most popular choos-complaint imitator, FaceDoxx), the Choos law, and the Cambrian Explosion of services, infrastructure, techniques, businesses, and culture that followed, that finally brought the internet into the physical world, and along with it, the realization that the whole of computing's history to date had taken place in a flat world of words, trapped inside tiny rectangles. And with that, the genie left the bottle.

Staples poked his thumb quickly and furtively toward the center of the blue field, hand down by his leg where no one would see, in a gesture that was more of a tic than anything.

His gear showed a simple bob with the face and name of someone he didn't recognize, along with a note he had taken. Apparently they'd met at an urban planning conference. They'd talked about light rail and waterfront zoning. He'd made a note that she was married, which reminded him that he'd been single at the time, before he'd been subsequently married and divorced himself. Still furtively, he cut the air with his fingertips, and the bob and the field disappeared.

He pointed at the disappearing bob with a thumb tip covered by his index finger, a gesture most users mapped to choos. His gear went looking for a response, in parallel sending an embedding of the color, scale, shape, texture, and location of the woman's face to FaceDoxx and two other lookups. He already had her name in his personal say, so his gear sent that along as well, to narrow it down.

Her canonical record - her 'choos' - came back on FD and one other - it was the same on both. She had linked out to a personal blog and a card that identified her as a distance runner, a mom, and recruitment officer for a consultancy in Maryland. It included a professional headshot where she looked nervous and constipated, like a hostage.

Staples briefly wondered if his note about her married status meant he’d been attracted to her at the time they met. Based on the headshot, if that were true, his tastes had changed significantly in the years since. Not for the first time, he wondered how well he remembered the person he’d been before he married. He winced and wiped a line through the condensation on his water glass with a fingertip.

Satisfied he didn't need to say hello and that she likely wouldn't do so either, he pinched his thumb and forefinger together and twisted in a counterclockwise motion, like turning down a knob. Everything his gear was showing faded out, leaving the room completely dry. He noticed the bartender had brought his drink and placed it before him - a cloudy, foamy, pale yellow thing in a Nick and Nora glass, smelling faintly of lavender and Chartreuse, garnished with a torched orange rind.

Back in the room, in the now, he stared at the foam. His hand twisted at a kink in the hair over his ear. Staples had a problem, or at least the feeling he had a problem, but it wasn't yet clear what kind of problem he had.

Jackie let the car drive her home in silence. She had left the dairy without checking in with the manager and hadn't changed her muddy boots before getting into the sedan. She was lost in thought and wanted to stay that way for as long as possible.

Something felt off about this job and something told her that if she looked for it too hard it would vanish in front of her eyes. She’d confirmed the operations folks had looked for the usual suspects - rumen acidosis, heat stress, crowding, silage, hardware disease.

She wasn't an expert on diary production or cows or animals, or, based on how her love life had been proceeding for the last lonely and frustrating decade, people. But she did know plants.

Things in the natural world changed faster with each passing year. The Northeast had fared better than almost any other part of the country through the repeated shocks brought on by climate change.

Extreme heat melted everything in the sun belt. Never-ending droughts and wildfires spooked off the private insurance market in California, and the state's inability to prop up the industry finally brought the idea across that tens of billions of dollars in homes sat on land unfit for human habitation. The rich got out and only the poorest stayed behind, left to fight with the surrounding states for a dwindling water supply, and for federal money to prop up collapsing utilities.

Repeated bailouts of inundated southern cities cratered Congress' will to continue propping up the National Flood Insurance Program. Hurricanes of increasing strength and frequency slashed their way deeper and deeper into the Southeast until only those with the strongest stomachs and the deepest pockets would rebuild with bomb-proof elevated houses in the parched, flat wastes between mosquito-infested marshes, leveled city blocks, and abandoned tract neighborhoods.

Similar stories of chain collapse made much of the country uncomfortable for those without the considerable means needed to buy comfort; an increasingly tiny segment of the population with each passing year.

The Northeast wasn't unaffected - more hurricanes than ever made landfall above the Mason-Dixon, bigger and wetter and more often. Deep-freezes, bomb cyclones, atmospheric rivers, tornadoes, droughts, heat waves, flash floods, and wildfires near and far made everything feel more precarious and more hostile than before. But on balance, compared to the blasted hellscapes in other corners of the country, the Midwest and New England looked appealing enough that their populations grew in the high single digit percentage points some years from internal migration alone.

Jackie was an observant person, and preternaturally pragmatic, even as a child. She saw this coming - something like this, anyway, two decades before, as a teenager also realizing the fear and loss and rage she felt watching the natural world spinning off into chaos was crippling her as a person, emotionally and physically. Her guts were in knots. She ground her teeth in her sleep. She sighed involuntarily and yawned from stress in quiet moments.

She couldn't escape a feeling of directionless resentment or the unannounced spikes of panic that stabbed through her from nowhere. She turned inward for years, thinking dark thoughts about the world that awaited her, and, even more disturbingly, about the people she loved, who seemed so oblivious.

Some of that pain never went away. What was left she channeled into a more constructive inclination to immerse herself in the natural world - to sink into it while it lasted. She took longer and longer trips, venturing further and further into what passed for wilderness, as far as she could get in borrowed cars, hitchhiking, and on buses and trains. She kayaked, hiked, canoed, climbed, and, and in her early twenties, studied, flying through online degree programs in botany, geology, and hydrology in the solitude of tents and hammocks.

Despite the freedom and exhilaration and the escape, the inconstancy of backpacking and nomadism wore thin eventually, and she returned to the calculations she re-ran several times a year, playing out the slow-motion trends that were plain to see if you were honest with yourself. The map in her head eroded on its edges and scorched and pocked with rust-colored holes as she played time forward.

Opportunity was no longer concentrated in the cities. Two and a half pandemics and an ever-strengthening remote work culture made urban rent hard to justify, whether for offices or shoebox apartments. To judge by the market positioning of the retail and restaurants in the big-city downtowns, the only people left in the tall buildings at city centers were the super wealthy and their superintendents.

Further out, rural communities struggled to maintain even a meager subsistence, battered by a climate that went from fickle to inconstant to downright violent in a surprisingly short period of time, faster than farmers could adapt, anyway.

Jackie put her savings down in the high hills of central New Jersey, a rangy farm country in direct sight of the New York skyline 40 miles away. She put a used trailer on a 5-acre plot of recovering farmland, a sloped parcel with a small stream, a pair of huge pin oaks supporting a rotting deer blind, a scattering of struggling apples, a stand of dead Walnuts killed by a succession of invasive moths, a carpet of invasive grasses, and a marsh full of invasive reeds.

Her neighbors were a mix of blue-collar workers, wealthy Republicans, destitute recluses, and gentleman farmers. She tacked up fliers at the Tractor Supply, the Grain & Feed, the 4-H, and the post office, advertising her services as a plant detective. She waded into hard issues on the state extension site and in-person, bartering for a beat-up chest freezer, an old shotgun, an Ikea bookshelf, and a few bushels of dried persimmons, all in the first year.

She diagnosed fungal blight on ornamental dogwoods, bolstered mycorrhizal diversity in herbicide-scorched meadows downhill from a golf course, and got paid actual money to assist the transitioning of a university’s sports complex green roof from a monoculture centipede grass to a native wildflower mix.

She built a small but active following for her public say, ranking high on native plant ID, diseases of local natives, and some kinds of moss.

Her trailer was sparse, physically furnished with a threadbare recliner and a folding table and a single bed and a very old halogen torchiere floor lamp that did nothing to improve the appearance of the smoke-stained popcorn ceiling. At first, in a fit of aspirational asceticism, she'd convinced herself she didn't care about the appearance of her home, then shortly thereafter realized she did. Or at least that the way it was now could, at the end of a bad day, touch off her latent depression for a night or more.

She didn't have money to do anything meat, but she'd given the place a lot of love in A Space, opting for a rich Chinoiserie wall-covering and a sheeting water feature and some tasteful Deco sconces and loads of potted plants. She put some effort into choosing a selection of A houseplants that would have thrived in the meat conditions, knowing her botanist lizard-brain would have rebelled at the impossible appearance, however virtual, of full-sun orchids and birds of paradise blossoming in the taupe murk of her trailer.

It felt silly to her. She'd grown up with computers and promptly, if not enthusiastically, worn gear when it became common to do so. She used A Space in her work, to check assumptions and find things she'd missed. She wore for entertainment and for navigation and almost one hundred percent of the time she was in a city, as did almost everyone. She diligently curated her say, reviewing tags regularly like she was grooming a zen garden.

But she also thought of herself as being rooted in the meat, a naturalist given to inhabiting the present, disinclined to self-delusion.

Today, Jackie knew that much of what she thought about herself was a story she'd written as a teenager, a legend, and that even if some of it had been true then, most of it didn't hold up to inspection now. Once you realize you’ve been thinking in fiction, it doesn’t just go away on its own unless you make something up to replace it.

Unless she was reading or researching or gaming, she didn't wear much at home, but on days like today, when she returned to the trailer with a cloud in the back of her head, she'd learned to leave the gear on and spend the evening in a nicer setting.

As Ren sat up, a vector outline of a police car flared into view the next block over, its light cutting through the intervening buildings like a neon sign in a blooming teal. Across the way, solid blue dots indicated the location of nearby police outside the immediate area. The car outline faded and became dotted as it left the view of the user that had seen it, no one Ren knew but a lookout in his wider crew nonetheless.

When this Cop Spot system was first adopted by Belarusians running a kind of import/export hustle out of Bay Ridge, they’d called it Systema. They had repurposed an app originally made to help schoolteachers keep track of a flock of kids, but the underlying technique generalized perfectly to many eyes wanting to watch many others.

Users with the magnet link received a peer-to-peer heartbeat of last-seen locations of law enforcement. The data came from 'narcs' - users running an extension to their head that reported police vehicles and hit a private say on every face, looking for known cops, whether in uniform or not. Most people rinsing the feed were narcing too, in the spirit of a healthy ecosystem.

As adoption grew, the payload grew, and even though the bundle was light, with objects carrying only location, an optional name, ID type, the report embedding, and direction of travel, it became impractical to rinse the whole set as users in India, China, Brazil, Russia, and the Philippines enthusiastically piled in. It was just too many narcs watching too many cops - too much firehose, a common problem in A.

This necessitated the adoption of a regionalized sharding system - updated endpoints for reporting sightings began bucketing data into Voronoi-bounded geographical zones dynamically subdivided in a way that kept the heartbeat reasonable for any one region.

Unless he was looking for a clue to something big going down, Ren kept Cop Spot on a roughly 500-foot need-to-know boundary. At the moment, he had that spread out to 5 blocks, not just because he had just been involved in a shooting that morning, but because he was distracted, and he felt like outsourcing some vigilance so he could think.

Mister had to have known he would at least be curious about what he saw in Deng's gear. She might have even suspected he'd be more than curious. As it happened, he wasn't sure what he was feeling, but at this point he wasn't ready to rule out that the tight feeling in his stomach and ball sack was fear.

It was hard to tell that feeling from the one he got when his body felt a big opportunity might be nearby, like when it seemed like he might get laid.

His body knew he was going to have to bring the witness, and the doll, to Mister sooner or later, and if he didn't, he'd better be ready to run. His mind was slowly figuring that out, too.

… that’s it for the excerpt. Thanks for reading! Tell me what you think in the comments, or send me an email, and if you liked it, tell a friend about Augmented Realist, won’t you?

If you're seeing this post without reading the last, you'll probably want to go back and get the overview of the system we're discussing and why it's designed the way it is.

This article is dedicated to fleshing out the specifics of a single component in that system: the 'say'. In the chain of steps laid out in the overview, this is the second.

The first step in the chain we're discussing is an on-device system that accepts input from sensors and generates unique hashes or IDs for anything in the real world - the 'hasher'. A say is a service that maps those unique IDs to language - a sort of a tagger.

We're skipping over that first hashing step for now to focus on the say because almost every other piece of this system is incidental - just a scaffolding - whereas the say is at the heart of the design.

Much of the current thinking about a future AR centers exclusively on connecting digital things directly to real people, places, and things. That idea doesn’t scale, creates artificial scarcity, reinforces existing power structures, and creates digital landlords in the process. Rather, we should enable connecting digital things to language, and focus on enabling users to choose the descriptions of the world that best suit their needs and perspective.

In response to that, at the highest level, the purpose of the system I'm proposing is to allow anyone to attach any digital payload to any real person, place, thing, or idea, and for anyone else to have access to those payloads with no intermediary.

The topic of today's post - the say - provides a protocol through which anyone can attach language to the meaningless IDs the system gives to the world, such that those who have something to share about a real thing can attach digital stuff to that language in addition to the ID itself.

To break this down one more way - first we make a meaningless but unique identification: this platypus is "eae5bb3b-9b08-4079-a32d-e789ec2fe2d6". Any device in the world running the same hasher will pull up the same ID in the presence of the same thing. Then we ask a distributed system to describe, or provide tags for "eae5bb3b-9b08-4079-a32d-e789ec2fe2d6", so we can do something with that information.

Out comes a word cloud, and rather than being the same for every user worldwide (like the ID is), the language that comes out is instead determined by who the user trusts, what language they speak, and, in effect, what their values and beliefs are.

Trust and Semiosis

“So the Lord God formed of the earth every beast of the field, and every fowl of the heaven, and brought them unto the man to see how he would call them: for howsoever the man named the living creature, so was the name thereof.” (Gen. 2.19)

There was a period of time during which one could be tricked into thinking there was a mainstream consensus on reality. That time seems to be coming to an end, with some mixed consequences.

That consensus, though, consisted of a very small subset of the realm of possible ideas, and even during periods of widespread accord, we each, personally, carry subjective perspectives and opinions that may be rare, or even unique to us.

If you ask a hundred people to describe a given person, dog, building, tree, product, vehicle, artwork, shoe, a frog, etc., you will, for each item, get back a distribution of language - a word cloud - with lots of overlap, and likely some outliers as well. Some of those outliers you might reject as objectively wrong, but some might be a matter of opinion.1

To give an example, take the frog. Someone unfamiliar with frogs but generally afraid of them might describe that frog as 'huge' and 'slimy', whereas a herpetologist might describe the frog as 'small', and furthermore 'variegated', or 'aposomatic', 'toxic', 'sexually dimorphic', etc. It's entirely possible, especially in the case of things as varied and under-classified as frogs, that another herpetologist might disagree with one or more of the suggestions of the first.

Similarly one could imagine the exercise leading to some honest disagreement about, for example, the descriptors of land or territory, or about a person. Think about the words that might be used to describe the West Bank, or a controversial politician, and then compare that collection of descriptors to the language used on their Wikipedia pages.

I'm not trying to argue that knowledge-organizing projects like Wikipedia, Wikidata, the Semantic Web, corporate knowledge graphs, good old-fashioned maps, etc. represent wasted effort, but rather that they necessarily encode a viewpoint2, in most cases that of a compromise based on some moderation rules, on the nature of a thing, or, taken in the gestalt, on the nature of reality.

By their design, these efforts minimize contradiction and compartmentalize disagreements, creating an institutional perspective that, at a certain scale, takes on the likeness of fact, and confers that status to subjective statements contained therein. And even though projects like Wikipedia are available in a rainbow of languages, language itself encodes cultural perspectives.

When we begin connecting the digital world to the physical world, if we do so with a process where authorities, corporations, or institutions, however well-meaning, are exclusively responsible for naming and labeling the world, we run the risk of hegemonizing semiosis. As evidence of how untenable a single universal viewpoint is, Google has long ago given up on serving one map to the whole world and now shows different borders depending on who's asking.

This is not a matter of degree - something you can do better or worse - you can either make this mistake or avoid it entirely. On the internet today, anyone can provide a service that is topically about an idea, person, place, or thing, and those services, in form of apps, web pages, and protocols, anyone can find and reach via search, shared links, direct navigation, and so on.

I've already made the case that we shouldn't trust anyone with the ability to dictate what digital things can and can't be connected to the real world. Nor should we even contemplate a system wherein artificial limitations on how many digital things can occupy the same connection, or space. Those approaches create a new kind of property and bring landlords along into what is otherwise an unbounded new resource.

The most natural framework for mapping arbitrary data (ideas) onto the world's things and concepts is the one we already have - language, but rather than offer a top-down description of the world onto which we connect our digital information, whether crowdsourced or centrally-controlled, we should allow the descriptions themselves to be as open as the digital world they enable.

In helping machines interpret the world around them, users should be in control of whose language they employ to describe the world, and when.

The Say Cascade

In order to enable user choice in a scalable, practical way, we should build say systems with a cascading lookup in mind.

Systems that use the output from a say to perform searches for digital material could use a cascade or a hierarchy to resolve conflicts, contextually give preference to subject matter experts, keep secrets, consider provenance, and share knowledge.

Let’s think about retrieving some descriptors of a person. You see someone you recognize at a business conference, but you’re not sure how you know them. Your system hashes them, and reaches to your personal say with the resulting ID. Nothing comes up. The system falls back to the says of your friend network, looking for both friends-only and public mappings on this person’s ID, penned by folks from your personal network. Nothing there either. The system falls back to a say operated by your company, used to provide an employee directory, info on vendors, and a CRM. Nothing there either.

You use a tap-dancing say you use to keep abreast of tap-dancing-related info when you’re watching tap-dancing because you're into tap-dancing for some reason and maybe that's why you recognize them. Hm. This person is not a tap-dancer. Lastly, your search falls back to your chosen baseline say, a public collaborative service somewhat like Wikipedia, and finally it gets a hit. Your relationship is parasocial - this person is the CEO of Dell. You’ve never met them.

As an aside, if your reaction to this premise is "that sounds an awful lot like universal facial recognition," you'd be right. This might be burying the lede, but in a system like this, anyone would be able to write anything (legal) about anyone else, just like they can now, except instead of figuring out their name and googling them, you'd have access to that information simply by looking at them.

This, of course, raises major concerns about privacy and harassment, but ones that only differ from the current situation on the web in degree, not in kind. There's plenty to discuss there but for now we'll put a pin in it and focus on user choice, as it's the theme of the post, and home in on the canonical response, or 'choos' (pronounced like 'choose').

Choos: Canonical Responses

In the world of the web, trademarks effectively entitle their holders to related domain names for the purposes covered by the trademark (big gloss here - more accurate description if you care to read it). You go to coke.com, you get the Coca-Cola corporation. For people, it's not so simple.

I don't own noahnorman.com, nor do any of the other unfortunate Noah Normans out there, except, I presume, one, and that's horse shit and possibly the instigating event of this whole enterprise. My supervillain origin story, if you will.

Further, if you search my name, my web page of choice is not guaranteed to be the top result, and if it is, that's to the exclusion of the desired result for the other Noah Normans out there. None of us are even guaranteed to show up on the first page, or any page at all.

In AR, we've just discussed how you could go from real-world things and people to language, which of course doesn't solve this issue for me. But a hash made from me - my appearance and context, translated into a unique identifier, I could potentially 'claim'.

We'll go into ideas of how this claiming system might work soon, but what this idea unlocks is the ability for individuals, trademark holders, even landowners, to offer 'canonical' augments attached to their hashes: a choos.

As an individual I might want to offer a vCard or an application to those in my presence - something like a blog or personal webpage or a storefront. I might want to extend my appearance with AR clothing. I might want to decorate the interior of my personal space, or provide decor and point of sale functionality in a restaurant I operate. I might want to virtually extend the exterior architecture of a building I own.

As a viewer, I may be interested to see the choos of a person I'm talking with, or of an entire crowd of people at once, or I may not. I may want to see architectural augments in some contexts and not others.

The possibility to offer a canonical augment doesn't preclude the ability of others to attach augments to the same hash directly, or to language that uniquely identifies a thing - it just allows, for the first time, everybody to have direct control over their preferred extension into digital space as others see it.

This could, for many people, be the first and only time they have a digital presence, but its widespread use could change the very nature of in-person human interaction, commerce, and architecture.

I'm starting to believe a personal choos - one connected to the ID produced from your physical appearance, could be the instigating killer app / controversy flashpoint / fomo engine this system needs to get the conversation started about what it means to directly connect the digital world to the physical, and why we should care about how that's done.

If that sounds interesting or scary or both to you, if it sounds like something you'd like to help make real or stop from happening, or if you know someone I should talk to about how to do it and how to get it right, please reach out, either in the comments or via email.

Next post I'll be looking at the first step in the chain - segmentation and hashing - in more detail.

First off - the elephant in the room: it’s been an embarrassingly long time since the last post here - the one where I promised a plan. I started this post almost a year ago, then took a detour to explore some of these ideas in fiction, but it went better than I expected and thus resulted in more of a lacuna than intended.

I’m probably not going to finish that expedition any time soon, so I’m back to continue to argue the case, this time with some concrete suggestions of where we can go from here.

-N

The first few posts on Augmented Realist have teed up the problem, but if you’re new here, here’s the TL;DR:

The world’s largest companies are racing to create vertically-integrated augmented reality systems. Those systems, because of their form factor, will eat all computing.

In fact, all screen-based activity, computing and otherwise, will be rolled up into AR. Nobody needs a TV or a computer or a smartphone or a watch or even signage when you already have a screen between you and everything you can see. This includes entertainment, work, browsing, and anything with an interface - all are better served, cheaper, by AR’s ability to put pixels anywhere in your line of sight than they are now by the screens and printed material all over the physical world.

If, as seems likely, only a handful of these companies capture a lion’s share of the AR market, the rules of their ecosystems will determine what is possible for almost every user in almost every digital activity.

This might call to mind Apple’s total control of iOS, which seems like a tolerable kind of prison, but remember that no matter how locked down the App Store gets, an iPhone can still access the web. And while Apple’s WebKit - the only engine iOS browsers can use - is opinionated and limited in what it supports, at least you don’t need Apple’s approval to create or update a webpage, as you do with an iOS app. And, what’s more, webpages (‘web apps’) can be made to do almost anything an app can do, in a mobile-native way.

This is an important point: no matter how locked down smartphones or computers are today, they still can access the web, which is a mostly free environment for expression and commerce and executable code.

AR is another paradigm entirely - its immersive, context-aware, spatial experiences are only possible on AR hardware. Unlike today’s web, which is accessible even on lightweight devices like refrigerators, watches, cars, etc, tomorrow’s AR won’t work outside of the systems made to enable it specifically.

That means anything created for AR, whether applications, games, art, reference, networking - any of it - will only be accessible in AR, lacing it into a chicken-egg relationship with the hardware we’re just now seeing on the market.

The way things are going, there will be no analogue to the wide-open web for augmented reality. The only AR-native activities users will be able to undertake, and developers will be able to create, will be those sanctioned by, and taxed by, the Metas and Googles and Apples of the world. All computing activity must be co-signed by the platform creators.

Concerns over digital rights aren’t nice-to-haves or policy-wonk wish list items anymore. We do everything online, and the often-horrific examples of what happens when you don’t have privacy and self-determination on the internet are piling up.

When your computer (AR glasses or contacts or implants) is in between you and the world, then everything you do is on your computer, and today’s problems are magnified a thousandfold. Imagine Facebook (Meta) or Google or Apple running not just the software that powers your whole computer / smartphone / smartwatch, but the software and hardware you wear in front of your eyes as you go about your entire life.

WHAT ABOUT THE WEB

When, not if, augmented reality eats all computing, it won’t make the web we have today go away. Flat ‘screens’ still work in AR, and flat-screen experiences are even more ergonomic and convenient when you can put the screen anywhere, at any size.

But, despite some heroic efforts to make it so for a very long time by some very smart people, the web we have today is not going to provide the backbone for a real AR-first internet.

The fundamental issue is the unsuitability of the DNS to connect real, physical people, places, and things to digital payloads, but some even doubt that TCP/IP is up to the task of moving the necessary bits about to do the job in AR.

Even if the infrastructure of the web were better suited to the use cases and workloads of AR, the web is poorly suited to furthering privacy, openness, transparency, and decentralization. The aforementioned involvement in all of your shit makes that a dealbreaker in AR.

We need a solution purpose-built to both reflect our values and to provide a backbone to an AR-first internet.

BACKGROUND

If you’re new here, a completionist, or if you want to catch up further before proceeding, get the full background on the stakes with:

To get an expansive view of what augmented reality might evolve into and why it matters, check out:

For a bit more about why private solutions to this anchoring problem aren’t in our best interests:

To understand the efforts currently underway by digital landlords to rent-seek in AR:

OK ... Caught up? Bought in?

I’ve teased this enough, and I’ve fired some shots from cover. Time to put the plan across and solicit feedback. I’ll go through each section of this stack quickly and then unpack and refine each bit in more detailed articles later on.

A REMINDER

The AR use case I’m targeting here is the one expounded on in What is Augmented Reality (For).

In brief: while it’s natural to first conceive of single-purpose augmented reality applications - things like HUDs, games, and tools - the much more profound thing I’m trying to unlock is the ability for anyone to attach any digital thing to any real thing, person, or place much as we already do with concepts, ideas, and language.

That ‘augment space’ or ‘A Space’ idea (placeholder title - ‘world wide web’ and ‘internet of things’ are taken … suggestions welcome) is where the real power of AR lives.

THE PROPOSITION

I propose we create a means for anyone to attach any digital thing to any real thing, and for users to access those digital things in a way that reflects our values.

We’d create a means for any device to compute a mostly-deterministic but meaningless identifier (a hash like, perhaps, a UUID) for any real thing.

With this hash, a device could ask the service of the user’s choice to describe that thing in language - most usefully in nouns and adjectives.

With this description, a device could then use the service of the user’s choice to search, or suggest (AI Clippy), or ask for recommendations for digital things (augments) that are attached to that real thing or to the language used to describe it.

With those augments, a filter of the user’s choice (Clippy again) could narrow down the enormous number of augments available to only those that might be helpful to that user in the current context.

With all this choice - choice of how to label the world with language, choice of how to search A Space with that language, choice of what bubbles up, choice of browsers with which to view it, and thus the possibility to create and access content on a choice of hardware, we’d retain some of the competition and freedom we have on the web, and we’d unlock the most natural, frictionless way to communicate since the spoken word.

HOW, v0.1

Let’s look at this through an end user’s use-case.

You are wearing ‘gear’ (a headset, contacts, an implant, etc.). In your field of view is a can of Coke.

In A Space, ‘on top of’ that can of Coke — anchored to it — are millions of augments: digital notes, assets, and executable created by anyone with something to say about a can of Coke, or soda, or cans, the Coca-Cola company, or capitalism, tooth decay, etc.

Depending on your context and preferences, you may want to see one, some, or none of those right now.

Let’s examine what happens to make that possible.

This design is intentionally modular - the intent is to create a series of interfaces, allowing users, developers, nonprofits, governments, consortia, and corporations to supply competing solutions at every link in the chain.

The design embraces subjectivity and mutability — it assumes no consensus on choice of language, use of that language, choice of technology, etc. It allows the user to compose, hierarchically, a mixture of authorities and to trust only some or none, contextually.

Also intentional is the compartmentalization of information — as much as possible, information is processed on your local device and filtered in local processes, preventing the need to report everything in your field of view to a third party or cloud service, or to leak your preferences and behaviors.

So here’s how it goes, cartoonishly simplified - we’ll fill in the details and edge cases later:

1: SEGMENTATION and PERCEPTUAL HASHING

INPUT: COLOR and DEPTH SENSING
OUTPUT: A HASH, or UNIQUE ID

Your gear is locally running a confection of machine learning models. These models accept information from your gear’s sensors - color camera(s), depth sensors, perhaps others. Those models segment the scene into individual objects, plants, animals, people, etc.

For each object in the scene around you, the models together generate a unique hash, or an ID. That hash could be an encoding of the three-dimensional shape of the object via something like a NeRF. It could include color information and scale as well. It might even involve that object‘s location in its context or on Earth in general.

This hash is computed locally, on-device, privately, and more or less deterministically, at least enough for hash-space nearest-neighbor calculations to say with some confidence that two hashes are or aren’t of the same thing.

2: CASCADING LOOKUP and MAPPING

INPUT: A HASH
OUTPUT: NATURAL LANGUAGE

With the locally-computed hash for an object, your device calls out to a series of lookup servers - a service known as a ‘say', cascading through them to build up a description of the hash in natural language.

The lookup sequence begins by checking first in your personal say, which runs its private bit locally, and which may contain public or private descriptions of the people, places, and things of the world. An example might be that you might publicly or privately label another person ‘BFF’ or ‘distasteful’ or ‘cousin by marriage’ or ‘Ted Nugent’.

You’d then move down your say cascade of trust. Perhaps next on your list would be your circle of friends and family, wherein a personal and potentially private view of the world is encoded.

Next in the lookup might be a mapping maintained by group to which you belong - eg an endpoint provided by your business that stores trade secrets, or a mapping from a military unit, or from a government agency like NOAA or NASA, from a religious or ideological group like a church, a secret society, etc.

Finally, you might fall through to pick up a more baseline definition from a major say provider like (hypothetically) Google or Wikipedia.

One could even imagine a system whereby your cascade differs depending on the subject matter.

As an example, perhaps for mappings where your baseline service identifies the hash as being foremost a sort of ‘bird’, you’d then go back to reach for more specific labeling from the Cornell Ornithology Lab say, overriding anything you got from Google because you trust Cornell more to get it right when it comes to birds.

Importantly, this modular system acknowledges a critical reality: language is personal, subjective, and something on which there will never be consensus, yet language is the vehicle through which we organize, search for, and create information. The labeling of the world post-Babel has been done a billion times over, differently each time, and is not something we can delegate to a team at a FAANG or a crowdsourced dataset to decide for all of us.

Further, this mapping from hashes to language enables creators of augments to attach their payloads to ideas and to broad categories of things for which they don’t necessarily have the hash at the time of publishing. You might, for example, want to attach an augment to all ‘breads’, or ‘cosmetics’, or ‘jetskis’, etc.

It should be equally possible to attach augments directly to single hashes, as we’ll discuss further.

3: SEARCH, SUGGESTION, RECOMMENDATION, FILTERING, DISCOVERY

INPUT: NATURAL LANGUAGE
OUTPUT: ADDRESSES (URLs) of AUGMENT RESOURCES

Now, for a given object - a person, place, or thing, you’ve got a cloud of words from sources you trust. Our say check for this can of Coke returned [soda, Coke, Coca-Cola, drink, beverage, water, CO2, can, high fructose corn syrup, water, acetic acid, caffeine, caramel color, carbonated, red, white, aluminum, acidic, unhealthy, high glycemic, sweet].

Our system now takes this language to our ‘head’ - a personal AI agent capable of searching and filtering on our behalf. In the system I’m proposing, a head wouldn’t be required, but I’m positing it as almost indispensable because of the nature of augmented reality.

On the web, we formulate searches, one at a time, in written language, and then scroll through the results - mostly words and a few images - at the leisurely pace at which we’re comfortable, on a small screen in fixed proximity to our bodies.

In a mature A Space, we’re sharing space with millions of augments at all times, with each and every thing, person, idea, and place around us suffused with writings, media, and code created by every person who ever had a thing to say, show, or make about that idea, plus a shitload of graffiti and SPAM. (This is the ‘animism’ part of Towards a Digital Animism.)

We’re interacting with this A Space while trying to use our bodies in the real world. We might be driving a car, or running, or trying to have a meaningful conversation, or alone with our thoughts. It’s hard to prove a negative, but I believe there is no UI solution that allows us to drink from that firehose and select among the nearly infinite options while doing anything else at all, especially not using our bodies for locomotion or labor or lovemaking etc.

In the here and now, we’ve been dipping our toe in this kind of delegation for more than a decade anyway, handing off more and more of the early filtering to recommender services, be they the ‘algorithms’ in streaming services, social networks, or mapping services, the ranking algorithms that sort search results for us and bubble up salient elements into ‘answer boxes’, or, increasingly, the hallucinations of LLMs.

So I’m not too out on a limb here by saying the way things are going, I anticipate we’ll each be running a (local, private, fine-tuned, context-aware) ‘head’ that acts as our research assistant, personal assistant, receptionist, editor, etc., and not just for AR, but for what reaches our ‘inboxes’ as well.

So it’s your head that takes this natural language returned from your says and runs it out to any number of services we prefer as a user - things we’d recognize as ‘search engines’ from the web, recommender systems private to any of the groups to which we belong, other AIs, etc., looking for augments that might be worth your attention.

4: PRESENTATION

INPUT: AUGMENTS
OUTPUT: AUGMENTS

Lastly, we have something resembling a browser - something that can present augments. Exactly what this software does, what hardware it supports, what augments it supports - is another opportunity for spirited competition in a system like this.

ROAST MY IDEA

OK those are the beats. The broad strokes.

Subscribe below to get a ping next issue, where we’ll start to dig into the specifics, edge cases, and possibilities in some of these components.

[ED NOTE: I’m going to experiment with turning on comments for this post because I want and need feedback - the critical kind - on this outline.

What am I missing? Pitfalls? Opportunities? Help me with the name too.]

In the last AR post, Towards a Digital Animism, I made the argument that conceptions of AR with 'scarce mappings' — that is, artificial limitations on the number and kind of augments that are connected to the things, people and places of the real world — are land-grabs by prospective rent-seekers.

There are plenty of reasons to be concerned about the potential downsides of allowing anyone to 'attach' any digital payload to any thing or person or place. Nonetheless, curtailing that freedom in order to prevent bad behavior would be like preserving the technology of writing for a trusted elite in order to avoid the promulgation of bad ideas.

In that last post, I gave the hypothetical example of an AR-enhanced Times Square, making the point that in a system where all augments are always visible, public spaces would immediately become too crowded with digital elements to see through or use in any way.

Your first instinct about how to address this issue might be to ‘ban’ or ‘outlaw’ people putting augments on real estate they don’t own, as Mark Pesce advocates in the book Augmented Reality.

Power lies not only in being able to prevent the writing of locative metadata, but conversely, to limit the ability to read it.

Who speaks for the world? Before AR cloud providers shrink that into a question of technological capability, we need to create organizations and institutions that maintain our existing systems of ownership – however flawed – because we can at least assert our agency within them, through legal means. Assigning the right to write to any commercial organization will only see them monetized, potentially weaponized, and reframed as technicalities, rather than consequential property rights, with the potential to shape human behavior.

If unchallenged by property owners – currently perhaps only dimly aware of this potential theft of a property right via a bit of technological sleight of hand – these trillion-dollar tech giants can act as though all of the real world belongs to them, to be inscribed with locative metadata serving their own ends.

Pesce, Mark. Augmented Reality. Wiley. Kindle Edition.

It’s hard to overstate how much I disagree this position.

I believe that augments are speech, and that augments that are ‘anchored’ to real-world things or people or places are simply speech about those things.

Augments aren’t really anywhere any more than web pages are somewhere. Just like web pages or statements or newspaper articles, augments can be about people, places, and things, but they’re not in those places or actually hovering around those people, despite that they might appear to be if someone chooses to view them that way.

By Pesce’s logic, we should ban web pages that are about private property, and prevent them from coming up in searches about those locations. Google Maps shouldn’t have the right to provide information about private property. I fail to see how a ‘Place’ on Google Maps, with its Street View images, satellite imagery, commentary from users, and links to the internet, is any less ‘there’ than an AR augment that bears the same geolocation in its metadata.

This idea could be expanded to further prevent anyone from 'attaching' or 'anchoring' augments to other people, or trademarked products, or famous images, etc. The whole line of thinking represents a kind of regressive techno-authoritarianism. Unless he's deliberately shitposting or playing a Kaufman-esque heel, it's bizarre that he's named this freedom he's arguing against 'the right to write.'

Pesce bolsters his case here by saying that despite that existing ownership systems have flaws, at least they're a system, implicitly claiming that rather than seize the opportunity presented by AR's novelty to fix what is broken, we should instead just bring along what we have because it's easier I guess?

This take sucks, and I'm not persuaded by his effort to frame it as a 'small landowner vs the tech giants' cause. Even if that were the whole of the issue, the reality is that none of the tech giants in question do much 'writing' of their own today, but rather allow their users to do the writing on their platforms (think eg. Twitter, Facebook, Google, etc).

The type of property rights Pesce advocates for here would curtail the rights of absolutely anybody to 'write', or speak, about private property, likely with rules around intellectual property not far behind.

More Speech

Augment speech should be free (as in speech and also often as in beer1, but that’s a topic for another day), and there should be lots of it. The solution to too many augments vying for the same space - which is to say effectively too many people with something to say about a person, place, or thing - is not to reject all but one ‘official’ statement or to prevent most people from speaking about most things.

The solution is to create a system that is designed to accommodate unlimited speech and to allow viewers to make sense of it in a way that works for them.

Today we have search on the web and in the other web-like corners of the Internet (again eg Facebook, Twitter, et al). I’m not sure how we should best approach this problem for AR, but I have some ideas.

I know I keep teasing the big reveal but I’m trying to keep these short-ish and on a single topic. I promise - outline coming in the next post.

The idea is that, at a global scale, somehow all the world’s augments will manifest in the same space at the same time, or that all virtual things will be visible to all users at the same time. No filtering, pure bedlam.

Fictional depictions of metaversey worlds reinforce the notion, from Snow Crash to Wreck-It Ralph 2 to Ready Player One. In each we’re given images of a vast assemblage of an endless horde of incongruous avatars. Seemingly infinite constructions, infrastructure, and activities are all laid on top of one another all at once in an impossible 3D conurbation.

It makes sense in the context of storytelling that’s intended to convey the shocking scope and prismatic expression of a 3D, immersive internet. It’s fun. It’s a mess. Shit is flying everywhere. Monsters, robots, dragons, neon, etc. — it’s what you’d expect if somebody in the early 90s imagined what the internet would be like in, say, the late 90s.

And the idea is an obvious first approximation when contemplating what it would mean to have many inter-operating immersive virtual experiences.

Unfortunately, I’m getting the impression that “Minecraft and Fortnite and Robolox in the same room at the same time” is as far as some have gotten towards imagining what it might be like to experience a Metaverse as they’ve been described.

That sounds fun I guess but there’s an elephant in the room.

The Map and the Territory Again

The idea of, by default, seeing, and digitally interacting with, every virtual thing within your proximity only holds up if you imagine AR as something made by others, obeying rules made by others, for you to passively consume.

In a system like that, the central organizing authority could maintain order, leasing out real-world real estate (like in Second Life but, uh … Life), and arbitrating the rights of creators to place augments ‘on’ people, trademarks, places, and things.

If you instead imagine that you and everyone else might rather be creators in this new medium, you quickly realize that this notion doesn’t scale. Who gets to decide what goes where? Does it all just pile up? Do we bid for position like AR is one big Facebook ad?

Popular physical places have a limit on how crowded with people and things they can become because electromagnetism says we can’t pass through each other, and because people generally don’t like touching strangers or seeing their things in a heap.

There’s no corresponding limit in AR, which means if there’s a ‘1:1 map’, in which there’s a single digital world overlaid on top of our own, that means either things are going to get really crowded, or we’re bringing landlords1 along with us into our augmented future.

We’d be carrying along the notion of property rights for no reason I can think of except to preserve capital’s role in making sure there are still serfs in a post-scarcity world. That’s assuming we’re intentionally modeling an AR internet in the image of our ideals. We are, aren’t we?

Purely virtual things — and here I should clarify that by ‘purely virtual’ I mean things that have no direct ‘anchor’ relationship to something in the physical world — represent only one facet of the problem, though, and when compared to My Kind of Augments™️ (the anchored sort) they have a different solution set for the kind of visibility permission / filtering problem they present to the end user.

The other, possibly bigger, issue with the ‘everything everywhere all at once’ / 1:1 map conception of an AR Metaverse lies with that latter kind of augment - payloads that are explicitly connected to people, places, and things.

Naturally, there’s no practical limitation on the ‘space’ available for augments, whether of the purely virtual or anchored kind. Any number of them could coexist, whether they’re represented as ‘in’ the same location in physical space or they’re attached to the same idea, trademark, symbol, image, person, place, or thing. Any system that has such a limitation baked-in is deliberately designed with rent-seeking front of mind.

Geo-located augments — virtual things that are intended to be viewed in a specific place — are one of the easiest augment flavors for most people to reason about, so let’s start there.

Imagine a heavily-trafficked location - say, Times Square. Tons of eyeballs means Times Square is a great place to put something you want a lot of people to see — hence all the billboards today.

If you’re making an augment that you want lots of others to stumble across, you might want to ‘leave it’ in Times Square, right? Naturally, you wouldn’t be the only one who had this idea.

But if we had a 1:1 map and the kind of ‘hard mode’ Metaverse where everyone sees every augment in sight at all times, well that would be a mess. You wouldn’t have a clear sight line through the digital detritus with which to see where you were going.

Similarly, you could imagine the astounding amount of cruft that would pile up around famous people or well-known images or products, etc.

For example, if anyone could attach an augment to any thing, how many augments do you think would be anchored to, say, a can of Sprite? Or the Mona Lisa? You’d never see the painting again.

It flat-out doesn’t scale.

Don’t Dream of Bad Futures

The scarcity thinking behind this idea, come by honestly and primed by images in popular culture, plays into a pernicious trap. It’s a construct where space — 3D space, cognitive space, and semantic space — is at a premium, and the gatekeepers — the platform owners and the ecosystem creators — collect rent to access it.

To those with the best of intentions nonetheless modeling their walled-garden systems under the total control of a central authority, it probably sounds like a compromise that trades openness for safety and quality. Less choice, more certainty.

Aside from the bottlenecking of expression down to a single authoritative augment per quanta of anchor, which is Bad, the glaring problem with allowing even the most well-meaning group the power to control augment space is that they’ll still shape it in their image. The products of an endless string of judgement calls required to curate and maintain the rules of an authoritative mapping themselves represent an encoding of the values of the authority.

This is marketed as a feature in the smartphone wars — concerned about privacy? Use an Apple product. They ruthlessly police their app store and are reshaping the internet itself to protect you from tracking. Care more about choice? Android or Windows are more of a free-for-all where customizability and freedom are prioritized over security and privacy defaults.

In AR we may have a dozen viable options, each an expression of a different philosophy on what augment space is for, and to whom it belongs. I’m not sure that itself is necessarily a bad thing unless there exists no space shared across all of them, free to anyone to contribute. A forum.

Babylon

The companies working on these AR platforms - they’re playing for keeps. Given their way, there are few of them that would elect to have any competition at all. That means if somebody wins meaningful primacy and establishes the Metaverse of Record, we’ll be more at their mercy in AR than we are at Google’s when it comes to what rises to the top on the web.

But that sucks.

What could be a greater expression of the American technocrat’s conviction of the objectivity of their viewpoint than thinking we could ever agree on the names of things on our map, much less what virtual thing can go where? Assuming there is a mapping, a Metaverse, is to ignore that there could never be a consensus on the map.

We can’t even have consensus on normal maps today - the ones intended to reflect political boundaries on the real world. On top of that we all speak different languages. Even if we speak the same language, we don’t agree on definitions. We even inhabit different internets. To think the world is mappable is a category error.

American and European readers - be honest - when you think of a future where a sizeable segment of the world is using AR, do you picture the hardware and platforms — however many there are — being made by American companies?

Naturally! Every one of them, right? Whether Apple, Meta, Google, Microsoft, Magic Leap, Snap, Niantic, Epic, or some newcomer, it’ll be an American company that makes your AR contacts and the ecosystem they run in, right?

While it may seem from a Euro-American perspective that American software and hardware companies are the only ones creating, in a meaningful way, the stuff of the internet, and thus it may seem likely that they’ll create whatever we regard as the successor to the internet, that’s just because you don’t live in a segregated internet like China or its OBOR orbit.

Meta (Facebook) seem like a pretty inevitable player in our AR future, whatever it is, doesn’t it? 17,000 employees working on a Metaverse play and a bet-the-farm rebranding from one of the world’s largest companies is likely to result in some not-insignificant foothold in the market. This notwithstanding the … upsetting glimpse of Horizon Worlds released last week.

But Facebook is banned in China. Chinese citizens, by and large, use WeChat instead, they among the 1.26 billion people who use the app worldwide. Google is available in China, but it’s not really Google. It’s Google China, and it’s not good for much. Chinese citizens (582M MAU) use Sina Weibo (they use different microblogging apps there too) or Baidu or instead. It’s like a whole other internet over there (sorry Condi)!

Anyway, when you think of the Metaverse, the big-tent version, with orcs and spaceships and octocats going about, you probably picture some free-as-in-speech NSFW MA-17 stuff happening in there right? Also maybe some exercise-of-free-speech stuff like protests and, just normal free speech, right? That’s not going to fly in a truly unfortunate number of countries around the world.

Of course, I’m making the argument that the internet is already fragmented today, but does that mean we should design a future internet that encodes our American cultural perspectives in such a way as to nearly guarantee its incompatibility with the views of other societies?

Or maybe we could design a system that is fundamentally agnostic about not just what is hosted, but how it’s searched, how it’s filtered, and how it is semantically anchored to the real world, and in the process design for the possibility that we all inhabit the same AR?

I’m here to argue that the things we hate are going to exist whether we sanction them or not, and that we’re very unlikely to agree on what the bad things are. Your favorite thing might be blasphemy in another’s eyes, and vice-versa theirs in yours. In my vision it’s on the business end — in the client or the client’s cloud of last-mile services — that we choose how we personally drink from the firehose.

Yes, I have details. Read on…

A spicy teaser — a cliffhanger, if you will

If you don’t subscribe to Augmented Realist already, now is an excellent time to start (for free!), and not just because now precedes everything that will happen next.

Next post I’m going to lay out a position on what exactly augments are — a framing that has legal and constitutional implications for a hypothetical AR web.

Twill be spicy, and will set the stage for the Big Pitch, where I make some first-approximation napkin-sketch proposals for what a real AR Web could look like.

Smash subscribe below, tell a friend, forward to your boss, chat it up in your Roblox Discord, nail it to the door of your church, etc.

Anyway this whole essay is the definition, so I’m in the clear.

As terse as I can get it

Augmented reality is a medium that layers digital elements onto real-world sensory input.

We don’t know what we’ll use AR for in the future, any more than we knew what computers would be used for in the 1960s, nor the internet in the 1980s, nor electricity in the 1880s, etc.

Lots of people are doing a lot of speculating but widespread adoption could well be 10+ years away, maturity much further than that, so it’s really anybody’s guess. Hell, we’re still finding uses for electricity and computers. The internet I think we’re done with.

Because of that naive perspective, at this point, the most we can say is that AR is a form factor. It’s a medium where digital elements (we’ll call them ‘augments’) are inserted into our sensory experience alongside the things we see and hear (and maybe smell and feel) while moving our bodies through reality.

The important part - the new part - is that we’re talking about digital things sharing our world, not our screens, with us. We will be invited to stop treating the digital as though it were a diorama in a box. As the technology matures, it will be hard to think of digital things - augments - as anything less than an equally different kind of real.

The genie leaves the bottle

Right now, computing happens behind LCDs and inside smart speakers, and we interact with the computers in our lives by going to them, by looking into the rectangles they use to communicate with us, or by speaking with them in the fixed points in space that they occupy.

The computer is a thing in a place; we peer into its world through its screen, and then we look away, and walk away, into reality, to do reality stuff, and the computer stays behind.

The computer is a timeless, blind oracle, trapped in a box. Despite all its knowledge of the world, every time we engage with it, it’s as though we’ve woken it from a dream. We have to remind it where we are, and who we are, and explain to it in stilted, unnatural language the thing we’re looking at, or we were just thinking about, or talking about in its presence, because it misses all of that.

With AR - wearable computers that inject their stuff between reality and our senses - computing sublimes and becomes ambient. No longer do we have to go to the computer, to hunch our focus into the slab, to twist our minds into a perfectly flat pile of rectangles and backlit text.

We’re upright, we move our neck to look around us, our eyes focus on objects in the foreground, to the horizon, or to the middle distance, accommodate, and converge, and we use our hands and arms and shoulders and feet naturally to move in a 3-dimensional space as our sea cucumber ancestors intended. Our brain works as it evolved to work: spatially.

With computers that see what we see, hear what we hear, and make sense of the world something like we do, the primary thing we think of as computing - the thing we’ve grown so accustomed to that we don’t notice we’re contorting ourselves do it - visiting the computer and explaining ourselves to the computer - begins to disappear.

What is the Metaverse tho

No fucking clue. Although, to be fair, I’m starting to get what is meant by the term, but I don’t think its hype jibes with how it’s being explained, so I’m back to thinking I don’t get it.

I’m not going to cool-kid strawman this thing — I’ll try to star-man if I can, but do drag me in the comments if I’m being unfair1. Here’s what I think is meant by most uses of the term:

The Metaverse is the product of interconnecting a number of virtual 3D worlds such that users and virtual things can move freely between them. It is, itself, a virtual 3D world.

I’m paraphrasing, and this might be recognizable as a simplified version of Matthew Ball’s definition [emphasis his]:

“The Metaverse is a massively scaled and interoperable network of real-time rendered 3D virtual worlds which can be experienced synchronously and persistently by an effectively unlimited number of users with an individual sense of presence, and with continuity of data, such as identity, history, entitlements, objects, communications, and payments.”

Continuing with Ball — he later offers a generalized definition less specifically focused on ‘real-time rendered 3D virtual worlds’:

The Metaverse, like the internet, mobile internet, and process of electrification, is a network of interconnected experiences and applications, devices and products, tools and infrastructure.

Of course, once you’ve removed the focus from what the Metaverse is used for, we’re really not making a distinction from today’s internet. That’s not very helpful, but it’s not my issue with the Metaverse.

Herein lies the dissonance: many definitions of the various Metaverses center, as Ball’s does, around ‘real-time rendered 3D virtual worlds’, but then immediately spiral out into claims that that same Metaverse will also encompass just about everything digital.

I should call out that this isn’t particularly an issue with Ball’s conception of the Metaverse but more with how it’s been interpreted by others.

My trouble with this is that there is an enormous surface area for utility in AR that has little to do with ‘real-time rendered 3D virtual worlds.’ I really don’t have a problem with the ‘Metaverse’ conception until it’s framed as the superset of all things spatial computing. That’s where I lose the plot.

My personal wig is extra-twisted because my hobby horse is left out. The paradigm I’m envisioning is not particularly concerned with the strictly virtual as it exists independent of the real, nor is it particularly focused on real-time 3D rendering, even if that does come into play to some degree.

Why AR matters

The most important thing AR has to offer is the ability to connect information to people, places, and things directly.

For the whole of human history, information has been stored and made available via two primary methods: experts and archives.

In order to avail yourself of the knowledge of an expert you need to know an expert. You need to be in proximity to that expert. And you need to have a compelling reason for that expert to tell you what you want to know, or, for deeper subjects, to teach you what they know.

In order to access information in archives you need to be in proximity to those records, you need to know how to use them, and you need to know what questions to ask. That last bit - the need to know what to ask - is a profound barrier to learning so fundamental to being human it’s hard to imagine how things could be different.

Even with resources, it is extremely difficult to learn within the realm of the Rumsfeldian unknown unknown. But with an AR used to anchor knowledge to the real, wearers could become collectively transhuman, with the accumulating benefit of the shared expertise of all who contribute. Like the Borg, but hopefully less colonial about stuff.

We’re already experiencing a taste of this with the internet, where answers abound, but to get information today, you have to ask the right questions, in language, and it’s often quite a long road to the answer you seek. Many of us are almost reflexively aware of the approximate difficulty of googling any given question even before we set out to do it, and moderate our expectations accordingly.

For a concrete example, think about what it takes for you to identify a plant. Without experience in botany, you lack the tools to describe the features of the plant. “It’s … tree-ish? It has a kind of trunky … uh … stalk? Kind of … bark-y but green? And it’s … short. Not that short? Has leaves. Small, feathery leaves.”

Even if you had access to a botanist, the terms you’re using and the details you’re noticing probably aren’t helpful. And unless you’re dealing with a particularly distinctive plant, you’re not going to get anywhere searching the internet with that kind of language.

For a personal example — here’s a conversation I had with a pollinator-specialist entomologist friend:

This conversation went on for about one hour and involved me sending nine unhelpful pictures and three videos before we felt confident we had an ID … this to accomplish something my entomologist friend could have done in an instant on her own were she there in person.

So I got my ID, but I got it by accessing the help of a (very generous) expert, for a very long time, sending images and video, and offering loads of (nearly useless) observations along the way.

Within reach, I also had a book, The Social Wasps of North America, in which the mud daubers in question (they turned out to be unusually aggressive mud daubers) do not feature (maybe they’re not social?) but in which many things that look like mud daubers to my eye have full-page features.

I’m sure the distinction is clear to those who know how to look, but the 208 species in the book are ID’d by remarks about their margins or tibiae or scutum or their ocular-macular space. Anyway, the problem is, while the book is a very reasonable $24.99, it’s almost useless without the expertise to use it.

So in this concrete, seemingly life-or-death example, where I was harried about by irate mud daubers and felt like at least knowing the name of my enemy could save my bacon2, neither access to expertise nor archives got me my answer in anything like a reasonable way, and still I was privileged to have both, as few do.

Now that I know what a mud dauber looks like I’ll never forget it, but that doesn’t help the next schmo in my situation.

But imagine a computer that sees what you see, knows where you are in the world, knows the time of year, the time of day, the local habitat, and uses those bits in concert with, e.g., a neural network, some estimated geometry via a NeRF, or more likely several techniques not yet invented, to make your answer available to you from a new, third source — an expert archive. Not an archive of experts but an archive that is, itself, an expert.

Imagine a world where, as the simplest starting point, everyone knows what almost everything is. What kind of world would that be? How would life be different if you rarely, if ever, had reason to think ‘what is that?’

Going one further — what if every person, place, and thing had - not just a name tag - but a pointer attached to it, a semantic anchor, onto which anyone could attach any information they had to contribute?

What if that information was actively contributed by individuals and institutions alike, and users could give preference to different sources based on the context?

In the case of our insect ID, that group might be the Entomological Society of America. For birds, we might prefer the Cornell Lab of Ornithology, falling back to the Audobon Society in the case of a lookup miss.

We might use Google’s lookup as a baseline for most things3, but override it for the topics on which we’d prefer to take the word of others, and further override those sources with our own definitions and mappings, perhaps for personal items or for secrets.

We might subscribe to the markup of an individual in matters of opinion or taste, giving, for example, an art critic’s commentary place of preference where it becomes available in the presence of (or within sight of representations of) pieces on which she’s published.

We might bubble up public records for a view into the hidden infrastructure under our feet, or for the permitting and violations and tax history of any building within view.

We might let an independent laboratory’s reports on product safety call our attention when something in our proximity has an outstanding recall, or contains an especially toxic material or other risk to our health.

We could superimpose on packaged goods and cars and appliances actuarial calculations for embodied carbon, or total cost of operation, or resale value, or aggregated consumer sentiment from reporting bodies and nonprofits we trust.

We might give our favorite artist’s work free reign, letting their ‘sculpture’ and ‘graffiti’ and ‘paintings’ and ‘softworks’ share our vision of the real on any walls and streets and halls where they leave them.

We might leave notes for ourselves on the faces of the people we meet.

A world after names

Let’s put a pin in this for now, but I hope you see where I’m going — what if all of our information - words, images, videos, ‘holograms’4, interactive code - data, reference, news, art, and personal expression - was anchored directly onto the people, places, and things of the real world?

This is the real promise of AR: computing vanishes and becomes an expert in our head. The ideas of the world, the world’s nouns and proper nouns, its people, its images, and every point in space on the planet - the very things themselves and not their names, become the tethers to which anyone can attach any form of communication.

This is a world where language is no longer a requisite intermediary for expression. If I have something to share with you about a building, or a certain hat, or a painting, or a boat, or a person, or a mountain, or a logo, neither one of us needs to know what the thing is called.

We should be in awe of the implications. We should be troubled as well.

We should be thinking about what it means that, the way things are going, a small handful of corporations will be in the ‘who says’ position of deciding what information is attached to all of the things and places and people, and how we’re allowed to participate.

I have some … alternative suggestions. Smash, if you will, that subscribe button.

If you haven’t yet, be sure to read part 1 of The Augmented Realist Manifesto before proceeding.

So what would be good? What’s bad?

So, AR is coming, for better or worse. What’s better? What’s worse?

Example 1: Convergence and the Attention Economy

Imagine a world with no more screens. Anything that has a screen - your TV, your computer, your phone, your car, your watch - they’re gone. Redundant.

All the physical screens are no longer necessary because anyone can put anything anywhere in their vision, screen-shaped or otherwise. If they want to have a tiny feed or a colossal screen hanging out in their field of view, they can do that.1

For evidence that this convergence is likely, consider that within 14 years of the first iPhone launch, it was already common to see restaurants without any physical menu offered, rather only a QR code on the table — the assumption being that everyone has an internet-connected QR reader and screen on their person, making the printed item redundant.

A Better Outcome: Wearers are in control over how they see the digital world around them. Augments are found and filtered by a client-side permissions structure entirely in the control of the user.

Multiple people can see the same augment, so, e.g., a movie theater is anywhere you want, with a screen as big as you want, visible to whoever wants to see it. Any two or more people can choose to share the experience of any digital thing at any time.

There are no more computers, per se, nor smartphones, nor offices as we think of them. Instead you can summon an infinitely flexible personal workspace with as much or as little screen real estate as you want, anywhere you want it, in the position you find most convenient or unobtrusive or ergonomic.

Relatedly, no more TVs cluttering up spaces and distracting from human connection unless you want them.

Relatedly, no more physical advertisements. No billboards, posters, etc. in the built environment. They’re too expensive to deploy and maintain, too static, and likely to be covered by other augments anyway.

Overall, fewer devices means less carbon and e-waste from the electronics industry - one device per person serves the purpose of every display and computer currently in use and more.

A Worse Outcome: We get a world where vertical integration, defensive incompatibility, and irresistible freemium business models collude to wrest control over reality from users’ senses.

(Virtual) billboards and ads everywhere. Advertising on every surface, on top of people, on top of live sporting matches you’re watching in person, on your pet, on your food, on the sidewalk, in the mirror, everywhere.

Animated product packaging, with sound, and you can’t stop it. Ads literally floating in front of your face. You can’t look away, you can’t turn it off, and for some hardware, it doesn’t help to close your eyes.

Lest you think we wouldn’t allow our senses to be cluttered that way, consider that TV is about 16% commercials (cable TV - the kind you pay for!), radio 26%, the model being that the content is ‘free’ or subsidized. The same can be found on the web and streaming video. The market has strongly endorsed this model.

We even do it in ticketed, live sporting events, and pay-per-view cable:

… and in our cities …

And, yes, Times Square and Dōtonbori are tourist attractions in part because of the ads, but when everywhere is Times Square, the ads lose their charm.

Advertising has been a fixture of the commons going back at least to ancient Rome, and it’s the engine behind two of the top ten largest companies of all time, both of whom are in the running to make the shortlist of owners of this AR future. History tells us ads are likely to come along into this new medium as well, where the sell is piped almost directly into your brain.

If you think you personally won’t participate in a system in which your attention is used as payment for a service, there are, nonetheless, billions of other people who likely would make that exchange. They’ll compromise because the utility of wearing will be too great to pass up, and because they won’t have the wealth to afford unsubsidized hardware without the adware.2

Example 2: Platforming vs Open Access

Imagine you’re a shop owner. You want to be able to control the appearance of your establishment in augment space - to provide visual interest, information, and context for your products, or perhaps to offer way-finding in larger stores.

Or maybe you have a quick-serve restaurant and you want to have that digital menu at a few key locations around the restaurant. Or maybe you’re a homeowner or renter who wants to decorate and add utility to your own space. Or maybe you’re an artist and you want to share your sculptures / paintings / interactive softworks.

The common thread here: you want to provide augments that don’t directly produce revenue from their users. They don’t ask for payment. There is no business model attached to creating these things - they’re simply useful, or beautiful.

A Better Outcome: You use a no-code solution, or your own coding skills, or you hire a developer to help you make the thing you envision, and then you publish it in a place where your target audience can discover it through proximity, physical indicators, search, suggestion (AI), word of mouth, etc.

Consumers use AR devices that can access something like the web we know today, executing arbitrary code found through a variety of means, with no intermediaries between publisher and viewer.3

You pay as much as you’re willing to pay to make the experience you want, and you own it outright, something like a web page. You can self-host, cheaply. You can monetize if you like, or not. You can promote by the means of your choosing. You can choose your tech. You can change your tech.

A Worse Outcome: Every AR device in popular use is tethered to one of several vertically-integrated ecosystems, including an app store, complete with ever-changing terms of service, shifting and lopsided privacy policy, strict rules around monetization, and a challenging and idiosyncratic discovery landscape.

If you want to control your experience as much as possible, you must, for each AR ecosystem you want to reach, engage developers, build an ecosystem-specific app, attempt to get it published in the app store4, pay the ecosystem its cut if you monetize, and advertise within the app store to attempt to find users.

Will there be in-ecosystem platforms that let you do this? Absolutely. Think Instagram or Medium or YouTube, or, in the restaurant menu example, Grubhub. Those platforms don’t require you to roll your own engine and find your own audience - they handle it for you.

If you use the AR equivalent of these platforms to make this sort of thing, will you own your work and control its presentation? Will you be able to prevent the platform from monetizing your augment on their own behalf, or selling your information, or your customers’ information, or injecting advertising in between you and your viewer? Precedent suggests not.

But those platforms will offer the ability to accomplish a version of your goal without requiring you to modify your business model to be extractive, or to have one at all. Without their cross-platform creation and discovery tools and their baked-in audience, your product would need to be monetized to offset the technical lift required to reach so many disparate hardware and software configurations.

As hard as it is to target anything more complex than a simple native application to iOS and Android today, imagine if there were instead 5+ popular AR OSes, each tightly coupled with hardware-specific APIs and idioms, each with its own relationship to the physical world.

Any intention to offer utility or entertainment or beauty to the most people possible would seem quixotic without something like an AR ‘web’ — a cross-platform markup language and an AR-native search and discovery system to lower the participation friction to the point that creators don’t need to raise money or give away their work to participate.

In this scenario, as has been the case with smartphones, many projects will by necessity choose to target only one OS. The negative consequences of that fragmentation are ameliorated in today’s environment by the (mostly) universal and ubiquitous web, which is as cross-platform as software gets.

Our flat, text-centric web will still have its place in AR: for still, focused perusal of the information-dense stuff we consume today. Its generational wealth of information and utility won’t go away, per se, but the data and markup of today’s web is largely incompatible with AR’s spatial computing paradigm, and is ill-suited to sharing space with the body. Using the web in AR will feel like editing a spreadsheet on your smartphone.

We’re going to need an equivalent in AR, or we’ll never have anywhere to call our own.

Example 3: Balkanization and Participation Friction

Let’s say you’re an entrepreneur in the entertainment industry — an attractions developer. You want to make the AR equivalent of a AAA video game5. This is going to be a huge project - a site-specific cross between a theme park and an MMO, woven into a physical location. Your team estimates it will take about 7 years and $340 million, minimum, to complete. You intend to make your money back via ‘ticket’ sales, subscriptions, and maybe in-world ads and product placement.

A Better Outcome: You develop your project, publish it to a publicly-discoverable server, and advertise its existence anywhere you see fit, because anybody with gear can find it and run it. Your audience pays you directly with any payment method you choose to accept, like a website, or a PC video game.6 You deploy updates whenever and however you like. Your development is constrained only by your ambition and your finances.

A Worse Outcome: You fragment your front-end dev budget into X buckets, where X is the number of competing walled-garden AR systems you want to reach. You fight to get your project approved and released into the app stores of each of those platforms, as you do each time you release an update, which is constantly, because you’re running a theme park.

In order to be discovered, you must dedicate resources to app store SEO and advertising on each ecosystem as well. You can only create experiences that comply with the TOS of the various ecosystems, and you can only accept payments via their terms, which take a ~15-30% cut of all revenue, including, on some platforms, things like food ordered from real-world vendors via your digital systems.

The ecosystems who are, in effect, both your partners and your captors, can, at any time, change the terms of their deal, and in so doing completely upend or outright destroy your business model.

Consequently, you greatly diminish your ambitions to account for the redundancies and overhead, and quickly consider finding revenue in places you would ideally never go looking. Your customers’ experiences suffer accordingly. Children and adults alike experience less joy for more money. The world is a little shittier.

Example 4: Your Personal Extension in Augment Space

Let’s say you want to augment your physical appearance. You want your clothes, or your hair, or your ‘makeup’, to have a digital component. Or you want to offer information or even some utility to people in your proximity — something like a business card or an FAQ or a menu of services or an art gallery.

Or let’s say you are a property owner and you want to extend the physical architecture of your building into virtual space. Interior decoration and functionality for your house, say, or for your commercial building.

A Better Outcome: While anyone has the freedom to attach augments to any thing and any place they wish, you, as the rightful owner of your person or of your property, have the ability to dictate the ‘canonical’ augment. Viewers can elect to prioritize canonical augments in their experience selectively, through dynamic permissions structures, e.g. allowing all of their friends or contacts canonicals through by default.

Your augment is, like any other augment, visible to all, carrying arbitrary code that viewers can allow through or block with a granular permissions scheme like the one we have on smartphones today. Users might allow audio but no visual permissions. They might allow an augment to run sandboxed code but not to reach out to the internet, etc.

Anyone has the freedom to ‘speak’ by augmenting anything, anywhere, whether their contribution is art, opinion, scholarship, journalism, or graffiti. They simply anchor their augment to a person, place, or thing. Speech - unpopular speech and even hate speech - amasses on persons and places of any renown.

Viewers and the intermediary tools they use for search and discovery employ different reputational, rating, AI recommendation, and social methods to bubble up relevant augments and suppress the things they don’t want to see, more or less invisibly, and in realtime (yes, ‘the filter’, and ‘the algorithm’, but ‘running local’ and in the users’ control).

A modular AR architecture allows users to pick and choose how augments do or don’t make their way into their experience, mixing and matching search, filtering, firewalling, etc. tools as they see fit. They use these tools to shape the flow from the augment firehose sitting on top of the world’s ideas, as, e.g., the American flag or the White House or an image of the president is encrusted with millions or billions of augments.

A Worse Outcome: recognition and tagging of the things, places, and people of the world is baked in to AR systems at the OS level, or is a done by AR applications themselves. That means there is no central place for a person to claim ownership of their identity, and no consistent way for them to have a voice in their extension into augment space.

Each major service offers some platform or OS-specific ‘Memoji’ system that only appears to others if they are running the same OS or have installed the appropriate cross-platform application. Even then, creators are constrained to making only what the system allows them to create. Precedent tells us this will be a narrow possibility space of cartoon avatars with zero ability to attach payloads of arbitrary code and assets.

Those who wish to offer augment-space extensions to architecture need to hope that their potential viewers have installed the platform app on which they published their augment, and that it’s running, and that the user discovers the augment.

If these digital architects and interior designers and property owners care enough, they can, of course, pay to advertise in-app, as discussed above.

Or they can participate in the surreal trend of putting the platform’s logo on their marketing or in their physical space, as many do with Twitter and Facebook today.7

In this fragmented environment, there is simply no way for creators to attach augments to the people, places, and things of the world in a way where they can expect the right people to find it. There are too many places to be.

Because the linkage from recognition to search to recommendation to presentation is so tightly coupled, users have little to no control over the search ranking, filtering, and suggestion mechanisms at work in the platforms they use, and their access to the world’s information, or to an audience, is subject, as we say today, to ‘the algorithm.’

What are we going to do about it?

I hope I’ve convinced you — the shit is going to hit the fan and we are standing underneath it. I propose we do something before it’s too late.

There remain, superimposed, good futures and bad futures ahead of us.

We should pick a good one … and we shouldn’t let the market decide what kind we get.

That’s the focus of this blog. How do we, serfs that we are (in which I include those who work at a Meta / Apple / Google / Microsoft / Snap / Niantic / Epic / Magic Leap), rather than take what we’re given, make the augmented future we want come to pass?

Follow me here on Substack as I explore, like a Soviet Futurist, hypothetical utopian architectures of AR. How could it work for us? How can we build it?

Look out soon for a podcast and/or YouTube thing which I envision to be three parts Pee-wee’s Playhouse to one-half part Crossfire with Newt Gingrich.

Augmented reality is going to change your life.

For better and for worse, it’s going to precipitate the most significant change to human experience in the history of our species, and nobody can stop it from happening.

The way things are going, we are going to roll up every digital thing we do into an increasingly natural-feeling reality-digital hybrid experience.

Don’t get fixated on glasses. The hardware will change with time, but computers and the digital world they contain will cease to be things in places and will instead sublimate to share the real world with us.

That transition will open the door to unprecedented tracking. It could lead to profiling, advertising, and loss of opportunity in our digital, economic, social, and creative lives.

At the same time, we could use the disruption to reinvent our relationship to technology in its entirety, in the process positively expanding our conception of what it means to be human in ways we can’t yet imagine.

Both could well happen at once.

Having accepted that, I believe we should aim for an outcome we like. If we don’t, a combination of market and regulatory forces will give us a shitty dystopian mess bundled with an irresistible upside. We could make ourselves a terrible offer we can’t refuse.

Right now, the way things are going, we’re going over the falls of posthumanism with capitalism driving the boat. We can do better.

The stakes: An unprecedented change in the human experience

Not that long ago, cell phones could barely see the web. Few people could use their phones to do anything but calling or maybe SMS.

And then, in the span of two decades, smartphones, with their ubiquitous 24/7 access to the internet, changed untold aspects of our society, our economy, our culture, and even our brains. We all became just a bit post-human from the explosive effects of billions of people having networked computers in our pockets.

With augmented reality (or AR, wearable computing, spatial computing, XR, etc — let’s call it AR), we are now at the cusp of yet another disruption — one whose effects will definitively change what it is to be human.

As a form factor, AR is the next logical step in the integration of ubiquitous networked computing with the natural experience of being an embodied human.

We’ve begun down this road with smartphones, but, with AR, computing will no longer be something we do on slabs of glass or at desks. It will be part of the fabric of our experience, commingled with the physical world around us.

As such, a segment of the population will soon begin an exponentially-accelerating, headlong rush into a kind of cyborg transhumanism.

The lived experience and capabilities of those who ‘wear’1 will be increasingly unfathomable to those who don’t, or to those who came before — far more so than the difference before and after electricity, or computers, or the internet, or smartphones.2

Today’s discussions of AR center on massively multiplayer gaming, on ownership of digital objects, on remote work, and generally on bringing concepts from our current lives along into this new world. That thinking signals a limited capability to imagine the kind of change we’re in for.

I am envisioning a kind of near-omniscience. Not a godlike awakening where the mysteries of the universe are revealed to us, but rather an ability to know almost anything that any other human knows without knowing what questions to ask.3

Wearing will bring a kind of networked awareness directly to our senses, in many cases bypassing the use of language or even conscious thought or observation.

Those who wear will live in a world with very different secrets from the kind we take for granted today, a world where the sorts of knowledge and abilities we now regard as the domain of experts will be as accessible as the time of day.

Those who wear will operate a kind of unconscious throttle, modulating at all times the amount of their attention and mental bandwidth they dedicate to either the real or to the augments their gear gives them.

As such, wearers will dream a liminal awareness, constantly floating between the true world and a full immersion in the digital aether that occupies the same spaces. The choice of how ‘dry’ they keep their consciousness will be something we regard as a transient personality trait, or life choice, akin to how ‘addicted’ a person is to their phone, or how ‘online’ they are today.

Don’t be fooled: the frivolity and silliness of today’s dreams of AR belie the power of the understanding possible from sharing your senses with a spatial internet. Today we sip information; tomorrow we will swim in it.

That’s just the likely bit

These are neutral terms for a likely outcome of the way things are going. As a single vision of a mature form of this computing, that has the less-likely and more extreme elements stripped out.

You may feel the situation described above represents one or another extreme as it is, whether a utopia or a hell, but there’s plenty of room for further extremes in either direction around that baseline, and much that could go right or wrong.

And one way or another, several companies will ensure that an AR-enabled future comes to pass.

As you probably noticed, Facebook changed their name to Meta, signaling how seriously they’re taking this endeavor. They have a reported 17,000 people working on it, and their latest headset is outselling the Xbox.

Tim Cook has said he’s AR’s ‘number one’ fan. Magic Leap has raised about $3b to date. Snap claims to have 250k creators making software for their glasses, with 300 of them having made ‘lenses’ that have been viewed one billion times each. Microsoft won a $22b contract from the army for a military-spec Hololens 2. Google never gave up on Glass. Pokémon Go was just a trojan horse for Niantic’s real-world AR plans.

These companies are engaged in a race not only to create the most compelling offering to consumers, but to dictate the terms of the deal for creators.

This is your problem

I’m not writing this as something I hope happens. I simply think it’s clear it will happen.

I believe almost everyone capable will wear eventually, because to not wear will be to be something of a Luddite, a bit like it would be to abstain from using computers or the internet or smartphones in today’s world, but far more so.

Still, with all the inevitable compromises that come with wearing, especially early on, many will choose to abstain, but their numbers will dwindle rapidly with time, as has been the case with smartphone or internet adoption.

If you’re skeptical and you’re old enough to remember a world before smartphones, think back to your mindset at the launch of the iPhone in 2007.

At the time, you didn’t have great maps on your phone. When you went about in unfamiliar places, you might have printed out directions from MapQuest, or pulled over on roadsides to examine an atlas.

Missing an exit was a big deal. Knowing whether you missed the exit or not could be non-trivial to sort out. Foreign subways could be inscrutable mazes.

You probably got lost a lot, and even when you weren’t technically lost, you might have known relatively little about your surroundings. You might have been unaware of things of interest even in the immediate vicinity of your home, like a hidden park, or an Uzbek restaurant, or an e-waste dump.

You probably found things differently from the way you do today - through word of mouth, or serendipity, and only really formed an idea of a place by physically visiting there yourself. There were no Places on Google Maps with their pictures and reviews and website links and Q&A and ‘People Also Search For’ sections.

But now we all carry devices that allow us to see recent high-resolution satellite imagery of almost anywhere in the world. We can see the street-level view from about 10 million miles of 360-degree photography, a capability that, 20 years ago, would have felt absolutely outlandish.

We have in our pockets perfect maps of the interiors of train stations, coupled with navigation systems that show exactly which escalator to take, and what platform to stand on, and when — maps that take into account line closures and delays, accessibility concerns, and more.

Even if we know our way, we don’t need to blindly follow a memorized ‘best route’, ignorant of traffic conditions, road closures, delayed mass transit, parade routes, or even natural disasters — all of that real-time information is factored in to the directions our devices give us.

We can spend an hour or two on the couch meandering across the world, combing a neighborhood’s streets, absorbing the visual texture and a rich stream of information that sits on top of that place: context, culture, commerce, history, topology, news, even 4-D imagery with building geometry and time-scrubbing over decades.

It’s almost impossible to remember a before, but we now have an understanding of what and where and place that is so complete that basic human behaviors like way-finding, discovery, and navigation are fundamentally different experiences than in the before-times not that long ago.

Did any of that seem possible, obvious, or even necessary when you first saw a smartphone? If you thought you might have those tools, did you think they’d ever come with you almost anywhere you go?

To appreciate how completely we’ve incorporated this capability into our experience, try conducting an extreme experiment: leave your phone at home and head out into a strange place.

Imagine using no digital maps while spending a week abroad, or, one further, absolutely no internet at all. If you’re anything like me, contemplating that idea requires engaging the part of my brain reserved for imagination and speculation.

I really don’t have a good idea what that would be like any longer, but I can say with some surety that it’s a profoundly different way of traveling, and of being, than what I’m used to today. It sounds romantic but it also sounds like losing a sense — like being blindfolded, or like some kind of monastic retreat.

It’s been just 15 years since the launch of the iPhone.

Even after reminding ourselves of how profoundly different our lives are with smartphones, it remains impossible to understand the nature of the before and after coming our way with AR, but I can say with some confidence that it will be even harder in the after-times to imagine the before.

We will be substantively changed.

So what would be good? What’s bad?

So, AR is coming, for better or worse. What’s better? What’s worse? Remember, the above was the middle-of-the road projection.

Things could get very dark … or we could become better versions of ourselves, as individuals and as a society. Or both! Probably both.

For speculative but specific examples from the spectrum of outcomes ahead, read on to part 2 of the Augmented Realist Manifesto.

To get a feel for the kind of blended infrastructure / semiotics / philosophy terrain I’ll be exploring in future editions, consider revisiting my now-classic 2015 post The Map and the Territory: On the Balkanization and Semiotics of Augmented Reality.