AI: Summary

This session of the Future Text Lab community centered on what host Frode Hegland called the “movement of knowledge” — the challenge of transporting and reshaping personal and scholarly knowledge from traditional 2D computing environments into XR. Frode demonstrated a working prototype that uses AI to extract defined concepts from student notes and spatialize them in Apple Vision Pro, provoking wide-ranging discussion about the nature of spatial cognition, the inadequacies of existing knowledge organization tools, the analogy of medium phase-shifts in media history, the neuroscience of memory, and what it would mean to both author and browse knowledge in three dimensions.

AI: Main Topic

The session opened with Frode presenting three slides on the “movement of knowledge” problem: that knowledge authored on a traditional framed display — with its specific affordances of keyboard shortcuts, trackpad, and linear framing — does not automatically suit XR, which offers a wholly different substrate. Frode then demonstrated a working prototype within his Author application for Vision Pro, in which student notes (around 8,500 words, mocked up with Claude) are parsed by an AI prompt to extract “defined concepts” — including persons, places, and events — which are then displayable as a spatial map inside the headset. The demo illustrated a workflow of write-on-desktop, extract-with-AI, view-and-manipulate-in-XR, and raised the central question of how to “sculpt” knowledge to genuinely fit both media rather than simply transposing from one to the other.

AI: Highlights

Peter Dimitrios coined the phrase “from A4 to AR format” in the chat, which Frode immediately embraced as a clean encapsulation of the session’s core challenge.

Brandel Zachernuk offered a precise framing of the appropriate role of AI in the demonstrated workflow: AI can be best trusted for low-stakes transpositions from one form of information to another, and when the mapping does not need to be mission-critical, it is ideal. He positioned this as “the best kind of use case” and a better-grounded application than many larger claims currently being made for AI.

Tom Haymes identified the problem as McLuhan-esque, a framing Frode explicitly adopted, noting it is directly relevant to the research proposal he is writing.

Jonathan Finn raised what Frode called a “juicy” problem: authoring in spatial environments requires that spatial arrangements carry interpretable meaning, otherwise there is no pathway back to a linear document or communicable artifact.

Peter Wasilko asked in the chat: “What was the Japanese term for books you own for future consultation that you haven’t had time to read as yet?” This question was directed to the group, and was answered by Frode in the chat using Claude, who provided a full explanation of tsundoku (積ん読), a blend of tsunde (to stack) and doku (to read). Frode followed the answer with the note “(Claude…)” and Peter Wasilko reacted with a heart emoji. This is the direct use of me — the Assistant — as a live reference during the session.

AI: Insights

The most consistent conceptual tension in the session was between the promise of spatialization and the problem of meaning. Jonathan Finn crystallized this: if you arrange concepts in 3D space while authoring, the spatial arrangement must carry interpretable meaning or the translation back to a shareable document becomes arbitrary. Frode’s response — that malleability and gesture matter more than fixed positions, and that commands like “order by time” should be temporary views rather than permanent structures — gestures at a resolution, but the tension was not resolved.

Tom Haymes drew a sustained analogy between the film/theater transition and the text/XR transition. Early cinema simply filmed theater; television shot radio plays. The community risks doing the equivalent with XR — putting linear documents into a headset and calling it spatial. The phase shift requires a new expressive vocabulary, not just a new container.

Brandel Zachernuk introduced a subtler point about the unsung affordances of XR: not spectacle or gesture-drama, but the continuity of sensing — multiple degrees of freedom, sophistication of point of view, the ability to look somewhere because you choose to rather than because a screen forces you there. He implied that once users move past novelty (killing zombies), these subtleties become the real medium.

Tom Haymes articulated a bifurcation of reading emerging from AI: “McDonald’s” information books that are best interrogated by AI for their extractable content, versus books requiring slow, reflective reading on a couch. This distinction maps onto a distinction in XR use: spatialising factual information vs. supporting deep engagement with complex ideas. Peter Dimitrios echoed this in the chat with “fast food vs. haute cuisine info.”

A quietly significant realisation emerged around physical bookshelf organisation: Tom Haymes, Jonathan Finn, and others converged on the insight that the fundamental limitation of physical and even digital libraries is the forced assignment of a book to a single category. XR allows a book to exist simultaneously in every relevant category through virtual duplication — a structural impossibility in physical space that becomes trivially solvable in XR. This is not merely a convenience but a qualitative shift in how knowledge organisation can work.

Frode Hegland introduced a neuroscience grounding for spatialization. He read from a book on brain physiology — describing Long Term Potentiation, the process by which repeated activation causes calcium-driven structural changes that physically stabilise a synaptic connection. His interpretive leap: the brain already uses something structurally analogous to spatial anchoring to consolidate memory, which suggests that external spatial representations of knowledge may genuinely align with internal cognitive architecture, not merely as metaphor but as functional correspondence.

Jonathan Finn observed that tools like Tinderbox and Devonthink are powerful but the problems they address are so fundamental that they should be OS-level affordances, not the province of specialist applications — citing Spotlight (not Sherlock) as a precedent for what becomes standard infrastructure.

Tom Haymes noted that AI devalues content at the commodity level but raises the premium on curation and contextualisation — the two things that AI cannot easily supply. This reframes the role of the scholar, the teacher, and the reader: not as conduits of information but as curatorial and contextual intelligences.

The physical constraints of XR movement were discussed with some candor. The “gorilla arm” problem — made vivid by Tom Haymes quoting from Make it So about Tom Cruise needing continuous breaks during Minority Report filming — and Peter Wasilko‘s description of being three feet from a bookcase, highlighted that full-room XR locomotion is a niche experience. Frode Hegland acknowledged this and distinguished AR (his primary focus) from VR, suggesting that overlaying knowledge onto real physical environments — including real bookcases — is a more tractable and contextually richer direction. The tension between seated micro-interaction and ambulatory spatial engagement was named but not resolved.

The content/context distinction continued to develop. Frode argued that any spatial knowledge environment needs a stable layer of context — known topics in known places — and a dynamic layer of current content that can be rearranged against it. Without this distinction, every new session produces a new mess.

Peter Wasilko‘s description of combining Tinderbox and Devonthink — dynamic agents and concordances as background processes — sketched a model of knowledge organisation that is already partially spatial in its logic, even when expressed in 2D. His workflow revealed that the gap between current power-user tools and a genuinely spatial system may be narrower in concept than in interface.

AI: Resources Mentioned

Metropolitan Museum of Art 3D models release (140 objects including a large tomb), shared by Brandel Zachernuk: https://www.metmuseum.org/press-releases/3-d-models-announcement-2026

Frode Hegland‘s Instagram reel (a song about the session’s themes): https://www.instagram.com/reel/DV8rHHHDGXT/

Frode Hegland‘s YouTube version of the song: https://youtu.be/OFSRoTYP8JM

The Story of Film documentary series, shared by Tom Haymes as a model of medium phase-shift analysis: https://www.imdb.com/title/tt2044056/

Open Syllabus Galaxy — a visualisation mapping book co-occurrence across academic syllabi, shared by Peter Wasilko: https://galaxy.opensyllabus.org

Open Syllabus main site: https://www.opensyllabus.org

Open Syllabus dataset documentation: https://opensyllabus.github.io/osp-dataset-docs/index.html

Open Syllabus blog and policies: https://blog.opensyllabus.org/terms-and-policies/

Tom Haymes‘ shared Gemini chat demonstrating AI bibliography generation from a bookshelf photo: https://gemini.google.com/share/8b451fc98153

Scribd link to Dan McClellan’s book The Bible Says So, shared by Frode Hegland, used to illustrate the constructive nature of meaning-making in reading: https://www.scribd.com/document/925739000/

Article on how “Sherlocking” became a term, shared by Peter Wasilko in reference to Jonathan Finn‘s mention of Spotlight: https://applecorepod.com/sherlock-the-mysterious-case-of-how-sherlocking-became-a-thing/

Make it So — book on sci-fi interface design, quoted by Tom Haymes (Minority Report / gorilla arm problem)

Ulysses — note-taking application used by Jonathan Finn

Tinderbox — knowledge management application described by Peter Wasilko for persistent dynamic agent queries

Devonthink — document management application described by Peter Wasilko for concordance-based exploration and clustering

Scrivener — writing application mentioned by Peter Wasilko

Notebook LM (Google) — used by Tom Haymes for bounded-corpus student research queries

Ken Perlin — NYU researcher, mentioned by Frode as someone whose lab he hopes to visit in New York in May

Phil Gooch — community member mentioned by Frode as having discussed AI-generated music that week

Bob Horn — referenced by Frode in relation to earlier Brandel Zachernuk experiments with a large spatial mural in XR

ACM Hypertext Conference — mentioned by Frode as a possible anchor event for the September Future of Text gathering in Europe

Second Life — mentioned by Tom Haymes in his anecdote about a flamethrower presentation as a model of what is uniquely possible in a virtual medium

Fahrenheit 451 — referenced by Brandel Zachernuk in response to Frode’s poem about carrying knowledge (“people as books”)

Long Term Potentiation (LTP) — neuroscientific concept, first demonstrated at the University of Oslo, discussed by Frodeas a physical basis for spatial memory anchoring

Song

This track is an AI orchestrated piece inspired by the transcript of this meeting, meant as a fun provocation to further thought. (suno.com)

Overview

This document specifies an annotation system for macOS and XR in which annotations and definitions are independent knowledge objects stored in a shared BibTeX-compatible ledger file. The system serves two user actions:

• Annotate (in Reader): select text in a document being read, write an optional note, specify a category, and view the result in a map.
• Define Concept (in Author): select text in a document being written, name and define the term, specify a category, and view the result in a map.

Both actions produce entries in the same ledger. Both are viewable in the same 2D mapped view on macOS and the same spatial environment in XR.

The selector model (how an entry points to text in a document) is adapted from the W3C Web Annotation Data Model (2017), translated into BibTeX fields. This gives the system robust, well-tested anchoring strategies while maintaining format consistency with the visual-meta ecosystem.

1. The Ledger File

1.1 Location and format

The ledger is a single UTF-8 text file with the extension .bib. On macOS it lives in the shared app group container accessible to both Author and Reader:

~/Library/Group Containers/{group-id}/annotations.bib

The file begins with a required header entry:

@ledger-meta{annotations,
  ledger-version = {1},
  created        = {2026-01-15T09:00:00Z},
  last-compacted = {2026-03-01T10:00:00Z}
}

The ledger-version field is an integer. The current version is 1. If the format changes in a future release, the version number increments. Applications must check this field on load: if the version is higher than what the application supports, it must refuse to write (to avoid corruption) and warn the user to update. Reading older versions is always supported.

The remainder of the file contains BibTeX entries, one per knowledge object, separated by blank lines.

1.2 Entry types

The ledger contains three entry types:

1.3 Writing to the ledger

All writes are appends. The procedure is:

1. Acquire coordination via NSFileCoordinator with a NSFileAccessIntent.writingIntent on the ledger file URL. This is required for cross-process safety between sandboxed macOS apps sharing a Group Container. Do not use flock — it does not coordinate with the file coordination system and will not prevent concurrent writes from the other app.
2. Within the coordination block, open the file for appending (do not truncate).
3. Write the new entry as a BibTeX block followed by a blank line.
4. Close the file handle.
5. NSFileCoordinator releases coordination automatically when the block exits.

No existing content is modified in place. To update an entry, append a new entry with the same ID and a newer date value. To delete, append a new entry with the same ID and status = {deleted}.

1.4 Reading from the ledger

On application launch, parse the entire file and build an in-memory index. Parsing proceeds entry by entry. If an individual entry is malformed (unclosed braces, invalid UTF-8, truncated write from a crash), log a warning with the entry’s approximate line number and skip it. A single malformed entry must never prevent the rest of the ledger from loading.

The index is a dictionary keyed by entry ID, where each value is the parsed entry. When multiple entries share the same ID, keep only the one with the latest date. Entries with status = {deleted} are excluded from query results but retained in the index for conflict resolution during import.

Build secondary indexes:

• Document index: dictionary keyed by target-document or source-document value, where each value is a list of entry IDs. Used for “all annotations on this document.”
• Category index: dictionary keyed by category string, where each value is a list of entry IDs. Used for “all Issues across corpus.”
• Tag index: an inverted index. Parse the tags field by splitting on , and trimming whitespace from each element. For each individual tag string, maintain a list of entry IDs that contain it. This means the entry tags = {methodology, statistics} produces two index entries: "methodology" → [anno-a3f8c] and "statistics" → [anno-a3f8c].
• Date index: sorted list of (date, entry ID) pairs for time-range queries.

On each write, update the in-memory index incrementally rather than re-parsing the file. Use NSFileCoordinator with a reading intent when re-reading is needed (e.g. if the other app may have written since launch). Register as a NSFilePresenter on the ledger file to receive notifications when the other app modifies it, and re-parse only the new content appended since the last known file size.

1.5 Compaction

Over time the file accumulates superseded entries. Compaction rewrites the file retaining only the latest version of each ID and excluding deleted entries.

Procedure:

1. Acquire coordination via NSFileCoordinator with a writing intent.
2. Build the final state from the in-memory index (all current entries, excluding deleted).
3. Write to a temporary file in the same directory.
4. Atomically replace the ledger file with the temporary file (FileManager.replaceItem).
5. Release coordination.

The compacted file is semantically identical to the original — it produces the same in-memory index.

Git versioning note: if the user versions the ledger with git, compaction produces a large diff (the entire file is rewritten). Recommend: commit the ledger immediately before compaction so the pre-compaction state is preserved, then commit again after compaction. This keeps both the incremental history (in the pre-compaction commits) and the clean state (in the post-compaction commit).

Run compaction on explicit user request, or automatically when the file exceeds a size threshold (e.g. when the file is more than double the size of its compacted equivalent, estimated by counting superseded entries).

2. BibTeX Encoding Rules

All field values in the ledger are enclosed in BibTeX braces {}. The following encoding rules apply to all string fields.

2.1 Special character escaping

BibTeX uses { and } as delimiters and % as a comment character. These must be escaped in field values:

The writer must escape these characters when serializing. The reader must unescape them when parsing. No other characters require escaping.

Example: if the user selects the text f(x) = {y : y > 0}, the selector-exact field is written as:

  selector-exact = {f(x) = \{y : y > 0\}}

2.2 Newlines in field values

BibTeX allows field values to span multiple lines within braces. User-written content (the content field) may contain newlines from the text input. These are preserved as literal newlines in the BibTeX file. The parser reads everything between the outermost { and } of the field value, including newlines, as the field content.

Leading whitespace on continuation lines is not significant — it is indentation for readability only. The parser trims leading whitespace from each line after the first, then joins lines with a single space. To preserve an intentional newline in user content, use the sequence \\n (backslash-n). The reader converts \\n to a newline character on parse.

Example:

  content = {First paragraph of the note.\\n\\nSecond paragraph
             with a continuation line.}

Parses to: First paragraph of the note.\n\nSecond paragraph with a continuation line.

2.3 Comma-separated fields

The fields tags, references, and related-terms contain comma-separated values. The parser splits the field value on , and trims whitespace from each element. Individual values must not contain commas. If a tag, reference key, or term ID contains a comma, it is invalid and the application must reject it at creation time (do not allow commas in tag names or BibTeX keys).

2.4 Integer fields

BibTeX has no integer type. The fields selector-start and selector-end are stored as strings containing decimal digits: selector-start = {2045}. The parser converts these to integers using standard integer parsing. If parsing fails (non-numeric content), the field is treated as absent and the position selector is invalid — fall through to the next selector strategy.

3. Entry Format: @annotation

Created when a user selects text in Reader and invokes Annotate.

@annotation{anno-a3f8c,
  target-document       = {doc:vm-78b2e4},
  selector-type         = {TextQuoteSelector},
  selector-exact        = {the methodology assumes a normal distribution},
  selector-prefix       = {As outlined in Section 2,},
  selector-suffix       = {which has been challenged by recent findings},
  selector-start        = {2045},
  selector-end          = {2089},
  selector-xpath        = {/body/section[2]/p[1]},
  category              = {issue},
  category-schema       = {scholarly-default},
  content               = {This contradicts the findings reported in Table 2.\\n\\nThe sample size is too small to assume normality.},
  author                = {user:frode},
  created-by-software   = {reader:3.2.1},
  date                  = {2026-03-06T14:23:00Z},
  tags                  = {methodology, statistics},
  references            = {smith2024-methods}
}

3.1 Field reference: @annotation

Required fields:

Required selector fields (depend on selector-type, see Section 5):

Fallback selector fields (always written if available, regardless of primary type):

Optional content fields:

3.2 Unique ID generation

IDs are generated as: prefix + first 5 hex characters of SHA-256(author + ISO timestamp + 4 random bytes).

Example: anno-a3f8c, def-c4b2a.

The prefix (anno-, def-, schema-) makes entries visually scannable in the raw ledger. IDs must be unique within the ledger and stable across the entry’s lifetime — they are never regenerated. The combination of author, timestamp (to the second), and 4 random bytes makes collision effectively impossible.

3.3 Maximum selector-exact length

If the user selects more than 1000 characters, selector-exact is truncated to 1000 characters and the field selector-exact-truncated = {true} is added. In this case:

• The TextQuoteSelector resolution algorithm uses the truncated selector-exact as a prefix match combined with selector-prefix and selector-suffix for disambiguation.
• The TextPositionSelector (selector-start / selector-end) defines the full extent of the selection and is the authoritative range.
• For display in the mapped view node label, use the first 40 characters of selector-exact regardless of truncation.

4. Entry Format: @definition

Created when a user selects text in Author and invokes Define Concept.

@definition{def-c4b2a,
  source-document       = {doc:vm-93a1f7},
  selector-type         = {TextQuoteSelector},
  selector-exact        = {standoff annotation},
  selector-prefix       = {concept of},
  selector-suffix       = {from computational linguistics},
  selector-start        = {1203},
  selector-end          = {1223},
  selector-xpath        = {/body/section[3]/p[2]},
  term                  = {standoff annotation},
  category              = {concept},
  category-schema       = {author-default},
  content               = {An annotation stored separately from the text it annotates, connected only by positional references.},
  author                = {user:frode},
  created-by-software   = {author:2.5.0},
  date                  = {2026-03-04T09:15:00Z},
  related-terms         = {def-a1f3e, def-b8d9c}
}

4.1 Field reference: @definition

Identical to @annotation with the following differences:

All selector fields work identically to @annotation. The field names are the same (selector-type, selector-exact, etc.) because selectors describe where text is in a document regardless of context.

4.2 Definitions in Author: selector stability

When the user is writing a document in Author, the document’s text content changes with every edit. Position selectors (selector-start / selector-end) written at definition creation time will become invalid as the author edits the text around them.

The system handles this as follows:

1. TextQuoteSelector is always the primary selector for definitions. The selector-exact field contains the term text, which the author is unlikely to change (if they do, they are effectively creating a different term).
2. On document save in Author, the system re-anchors all definitions for that document: for each @definition entry in the ledger where source-document matches, resolve the TextQuoteSelector against the current document text and update the position selector fields (selector-start, selector-end) and XPath field. This is done by appending updated entries (same IDs, new dates) to the ledger.
3. If a TextQuoteSelector fails to resolve during re-anchoring (the term has been deleted or substantially reworded), mark the definition as unanchored and notify the author.

This re-anchoring on save ensures that position selectors stay current during active editing, while the TextQuoteSelector provides the durable anchor.

5. Selectors

The selector system is adapted from the W3C Web Annotation Data Model. Two primary selector strategies are supported, plus a structural fallback. The system always writes all available selector fields on creation to maximise anchor robustness.

5.1 TextQuoteSelector (preferred primary)

Identifies text by its exact content plus surrounding context.

Fields: selector-exact, selector-prefix, selector-suffix.

Resolution algorithm:

1. Extract the document’s full text content as a single string (format-specific, see Section 7).
2. Search for selector-exact in the text (using normalised comparison — see below).
3. If exactly one match: resolved.
4. If multiple matches: disambiguate using selector-prefix and selector-suffix. For each match, compute a similarity score between the text preceding the match and selector-prefix, and between the text following the match and selector-suffix. Select the match with the highest combined score.
5. If no match: selector fails. Fall through to next strategy.

Normalised comparison: before matching, normalise both the document text and selector-exact by: collapsing runs of whitespace to single spaces, trimming leading/trailing whitespace. This handles reformatting (e.g. different line breaking) without false negatives.

Adaptive prefix/suffix length: the default prefix/suffix length is 32 characters. At creation time, after writing the initial 32-character prefix/suffix, verify uniqueness: search for selector-exact in the document text. If multiple matches exist and the 32-character prefix/suffix does not disambiguate, extend to 64 characters. If still ambiguous, extend to 128. Cap at 128 characters. This ensures disambiguation while keeping the common case compact.

5.2 TextPositionSelector (fast primary or fallback)

Identifies text by character offset from the start of the document’s text content.

Fields: selector-start, selector-end.

Offset counting: offsets are counted in Unicode scalar values (equivalent to Unicode code points for all characters outside the surrogate pair range). In Swift, use String.unicodeScalars for counting and subscripting. In Python, use len(text) (which counts code points). In JavaScript, use Array.from(text).length. Do not use Swift’s String.count (which counts extended grapheme clusters) or JavaScript’s String.length (which counts UTF-16 code units). The first scalar in the document is at offset 0.

Resolution algorithm:

1. Extract the document’s full text content as a single string.
2. Index into the string’s Unicode scalars at selector-start to selector-end.
3. Verification: compare the extracted substring against selector-exact (normalised). If they match, the selector resolves. If they do not match, the selector fails — do not use the position without verification, as it likely points to wrong text after an edit.

5.3 XPathSelector (structural fallback)

Identifies the containing structural element. Never used as primary.

Field: selector-xpath.

Resolution algorithm:

1. Evaluate the XPath (or synthetic path — see Section 7) against the document’s structure.
2. If the element is found, search within its text content for selector-exact.
3. If selector-exact is found within the element: resolved.
4. If the element is found but selector-exact is not within it: partially resolved. Anchor the annotation to the element with a visual indicator that the exact text was not found.
5. If the element is not found: selector fails.

5.4 Resolution order

When resolving an annotation’s anchor:

1. Try the primary selector (identified by selector-type).
2. If it fails, try the other selectors as fallbacks:

• If primary was TextQuoteSelector: try TextPositionSelector (with verification against selector-exact), then XPathSelector.
• If primary was TextPositionSelector: try TextQuoteSelector, then XPathSelector.

1. If all selectors fail, mark the annotation as unanchored.

5.5 What the system writes on creation

Regardless of which selector is primary, the system writes all available fields:

  selector-type         = {TextQuoteSelector},
  selector-exact        = {the selected text here},
  selector-prefix       = {text before selection},
  selector-suffix       = {text after selection},
  selector-start        = {2045},
  selector-end          = {2089},
  selector-xpath        = {/body/section[2]/p[1]},

TextQuoteSelector is the default primary for all formats. TextPositionSelector may be primary for plain text files known to be stable. XPathSelector is never primary.

6. Document Identity

6.1 Visual-meta ID format

Every document in the system has a stable identifier stored in its visual-meta. The format is:

doc:vm-{8 hex characters}

Example: doc:vm-78b2e4a1

The 8 hex characters are generated from the first 8 hex characters of SHA-256(author + creation timestamp + 4 random bytes). This ID is assigned once on document creation and never changes across saves, exports, renames, or transfers.

6.2 Where the ID lives

In the document’s visual-meta block (the BibTeX metadata appended to or embedded in the document), the ID is stored as:

@document{doc:vm-78b2e4a1,
  title  = {My Paper Title},
  author = {Frode Hegland},
  date   = {2026-03-01}
}

The ledger references documents exclusively by this ID, never by filename or path.

6.3 Documents without visual-meta

If Reader opens a document that has no visual-meta (e.g. a PDF downloaded from the web), the system generates a visual-meta ID for it on first annotation. The ID is stored in a local metadata sidecar file:

~/Library/Group Containers/{group-id}/document-ids/{sha256-of-file-first-4kb}.bib

The sidecar contains:

@document-id{doc:vm-f3a8b2c1,
  original-filename = {downloaded-paper.pdf},
  file-hash         = {sha256:first-4kb-hash},
  created           = {2026-03-06T14:00:00Z}
}

The file hash (SHA-256 of the first 4KB) is used to re-identify the document if it is renamed or moved. This is not cryptographically robust but is sufficient for the purpose of reconnecting a file to its annotations.

7. Text Extraction by Document Format

Selectors operate on a document’s text content: a single string extracted from the document’s native format. The extraction method determines how offsets are counted and what XPath expressions look like.

7.1 Implementation interface

Each application must implement a TextExtractor protocol for each supported format:

protocol TextExtractor {
    /// Returns the full text content as a single string.
    /// Offsets into this string are in Unicode scalars.
    func extractText(from document: Document) -> String

    /// Returns the Unicode scalar range for the given XPath, or nil.
    func resolveXPath(_ xpath: String, in document: Document) -> Range<Int>?

    /// Returns the XPath for the structural element containing the given
    /// Unicode scalar offset.
    func xpathForOffset(_ offset: Int, in document: Document) -> String
}

The same TextExtractor implementation must be used for both writing and resolving selectors within the same application. If Author and Reader use different extractors for the same format, TextPositionSelector offsets may be inconsistent between them. TextQuoteSelector is immune to this because it operates on content rather than position.

7.2 PDF

Text extraction: PDFDocument → iterate PDFPage objects → page.string for each page. Concatenate all pages with double newline (\n\n) as page separator.

XPath: synthetic path of the form /page[N]/block[M] where N is the 1-based page number and M is the text block index. Fragile and format-specific; serves only as a last-resort fallback.

Primary selector: always TextQuoteSelector. PDF text extraction varies between libraries, making position selectors unreliable across different PDF readers. The TextQuoteSelector’s content-based matching is essential for PDF.

7.3 HTML and EPUB

Text extraction: walk the DOM tree in document order. For each text node, append its content. Insert \n at each block-level element boundary (<p>, <div>, <h1>–<h6>, <blockquote>, <li>, <section>, <article>).

XPath: standard DOM XPath. Example: /html/body/section[2]/p[1]. For EPUB (which is a collection of HTML documents): prefix with the spine item href: {spine-item.href}/html/body/section[2]/p[1].

7.4 DOCX

Text extraction: parse word/document.xml. Walk <w:p> elements in order. For each paragraph, concatenate text from <w:r>/<w:t> elements. Separate paragraphs with \n.

XPath: synthetic path based on paragraph index: /document/body/p[N]. For documents with <w:sectPr> sections: /document/body/section[S]/p[N].

7.5 Plain text and Markdown

Text extraction: the file content is the text content. No transformation.

XPath: synthetic path /p[N], where paragraphs are separated by blank lines (two or more consecutive newlines).

7.6 Author internal format

Author documents are stored as [specify Author’s native format — e.g. DOCX, custom XML, or rich text]. The TextExtractor for Author must produce the same text string from the document being edited as it would from a saved/exported copy of that document. This ensures that selectors written during editing remain valid after save.

If Author uses an internal representation that differs from the export format, the TextExtractor must operate on the internal representation (since that is what the user sees during editing), and selectors must be re-anchored on export if the exported format produces different text content. See Section 4.2 for the re-anchoring procedure.

Note to implementer: this section requires completion once Author’s internal document format is confirmed. The key requirement is: whatever TextExtractor Author uses, it must produce a deterministic string from the current document state, and the same string must be producible from the saved file.

8. Entry Format: @category-schema

@category-schema{scholarly-default,
  categories         = {important, issue, quote, claim, evidence, method, question},
  colors             = {blue, red, green, purple, orange, teal, amber},
  context            = {scholarly-reading},
  w3c-motivation-map = {highlighting, questioning, highlighting, assessing, assessing, describing, questioning}
}

8.1 Field reference

8.2 Schema selection

The active schema is a per-project or per-user setting stored in application preferences (not in the ledger). The Annotate and Define Concept dialogs present the active schema’s categories. The user can switch schemas at any time. Existing annotations retain their original category string regardless of schema changes.

8.3 Colour mapping

Applications map named colours to their own platform-specific colour values. The named colour vocabulary is intentionally limited to ensure consistency across macOS and XR rendering.

If an annotation’s category is not present in the active schema (e.g. it was created under a different schema or the schema was modified), render it in grey.

8.4 Default schemas

@category-schema{scholarly-default,
  categories  = {important, issue, quote, claim, evidence, method, question},
  colors      = {blue, red, green, purple, orange, teal, amber},
  context     = {scholarly-reading}
}

@category-schema{author-default,
  categories  = {person, place, concept, event, method},
  colors      = {purple, green, blue, orange, teal},
  context     = {authoring}
}

9. User Interaction

9.1 Annotate (Reader)

Trigger: user selects text, then invokes Annotate via context menu, keyboard shortcut, or toolbar button.

Dialog contents:

• Category picker: horizontal row of coloured buttons, one per category in the active schema, plus an “uncategorised” option (rendered in grey). The first category is pre-selected by default, but the user may select uncategorised for a quick highlight with no classification.
• Note field: multiline text input. Optional — the user may leave it empty.
• Confirm button.

On confirm:

1. Capture the selection using the document format’s TextExtractor:

• selector-exact: the selected text (truncated to 1000 characters if longer; set selector-exact-truncated = {true}).
• selector-prefix: up to 32 characters before selection (extended to 64 or 128 if needed for disambiguation — see Section 5.1).
• selector-suffix: up to 32 characters after selection (same adaptive extension).
• selector-start / selector-end: Unicode scalar offsets.
• selector-xpath: XPath of containing element.

1. Escape special characters in all string fields per Section 2.1.
2. Generate unique ID: anno- + first 5 hex chars of SHA-256(author + ISO timestamp + 4 random bytes).
3. Construct the @annotation entry with all fields.
4. Append to ledger via NSFileCoordinator (Section 1.3).
5. Update in-memory index.
6. Render the highlight in the document view using the category’s colour.

Time budget: the dialog should appear within 100ms of invocation. The write to disk should complete within 50ms. The user should never wait for the annotation to be saved.

9.2 Define Concept (Author)

Trigger: user selects text, then invokes Define Concept via context menu, keyboard shortcut, or toolbar button.

Dialog contents:

• Term field: pre-filled with selected text. Editable.
• Category picker: same UI as Annotate, using the authoring schema (no uncategorised option — definitions must have a category).
• Definition field: multiline text input. Required.
• Related terms: optional autocomplete field that searches existing @definition entries by term name.
• Confirm button.

On confirm:

1. Same selector capture as Annotate.
2. Generate unique ID: def- + first 5 hex chars of SHA-256(author + ISO timestamp + 4 random bytes).
3. Construct the @definition entry.
4. Append to ledger.
5. Update in-memory index.
6. In the document view, visually mark the term and draw relationship lines to other defined terms that appear in the visible text.

9.3 Editing and deleting

Edit: the user clicks an existing annotation or definition. An edit dialog appears, pre-filled with current content and category. On confirm, a new entry with the same ID, updated content/category, the original selector fields unchanged, and a new date is appended. The in-memory index updates to the latest version.

Delete: the user chooses delete from the edit dialog. A new entry with the same ID, status = {deleted}, and a new date is appended. The in-memory index removes the entry from query results.

10. Views

10.1 In-document view

When a document is opened, query the in-memory index for all entries where target-document (or source-document) matches the document’s visual-meta ID. For each entry, resolve the selector (Section 5.4). Render:

• Annotations: highlight the text range in the category’s colour. If the entry has content, show a margin indicator. On click/hover, display the note in a popover.
• Definitions: render with a distinct style (e.g. dotted underline). On click/hover, display the definition. Draw relationship lines between visible defined terms connected via related-terms.
• Unanchored entries: do not render in the document. Show in a separate “Unanchored” panel with the selector-exact excerpt and a “Re-anchor” button that lets the user select new text to attach the entry to.

10.2 Single-document mapped view

Query: all entries referencing the current document.

Layout: a 2D canvas. Each entry is a node:

• Position: initially in document order (vertical) with category column (horizontal).
• Colour: from category schema. Unknown categories render in grey.
• Size: uniform by default; optionally scaled by content length.
• Label: first 40 characters of selector-exact plus category name.
• Edges: draw a line between nodes that share a tag, share a references value, or are connected via related-terms.

Interactions: filter by category, switch to force-directed layout, click to navigate to passage, drag to rearrange.

10.3 Cross-document mapped view

Query: all entries matching user-specified scope (project, tag, category, date range, or explicit document set).

Layout: force-directed by default. Document origin shown as background grouping or border colour.

Interactions: click to open source document, filter by document/category/tag/date, group by document or category.

10.4 Author-Reader overlay

Available whenever both @definition and @annotation entries exist for the same document, regardless of whether the definitions come from a bundle in the document’s visual-meta or from the local ledger. This means the overlay works for:

• A reader viewing someone else’s document (definitions from bundle, annotations from local ledger).
• An author viewing their own document (definitions and annotations both in local ledger).

Two layers:

• Author layer: definition nodes, edges from related-terms.
• Reader layer: annotation nodes, edges from shared tags and references.

Layers toggle independently, view side by side, or overlay. Cross-layer edges (where an annotation and a definition anchor to overlapping text) are drawn in a distinct style (dashed).

10.5 XR rendering

All four views have XR equivalents:

• 2D canvas → 3D space.
• Nodes → objects (spheres, cards, or panels).
• Edges → visible lines or ribbons.
• Category colours → same colours.
• Layers → spatial depth.
• Click → approach and gaze/point.
• Filter → spatial rearrangement (filtered-out nodes fade or move to periphery).

The data source is identical — the XR environment reads from the same in-memory index.

10.6 XR ledger access

The XR environment may run on a different device (e.g. a headset) that cannot access the macOS filesystem directly. Two access strategies are supported:

Strategy 1: Shared filesystem. If the ledger is in an iCloud-synced location or a shared network volume, the XR application reads the same file. This requires the Group Container to be in an iCloud-accessible location, or the ledger to be symlinked to one.

Strategy 2: Local network service. The macOS application exposes a lightweight read-only HTTP service on the local network (Bonjour-discoverable) that serves the ledger contents as JSON. The XR application discovers the service, fetches the ledger, builds its own in-memory index, and subscribes to change notifications via a simple polling or WebSocket mechanism. The macOS application is the only writer; the XR environment is read-only. Writes initiated in XR are sent as requests to the macOS service, which appends them to the ledger.

The choice between strategies depends on the XR platform. The spec requires at minimum Strategy 1 for initial implementation. Strategy 2 is recommended for production use with standalone headsets.

11. Bundling and Unbundling

11.1 Bundling (export)

When a document is exported, shared, or published:

1. Query the ledger for all entries where target-document or source-document matches the document’s visual-meta ID.
2. Filter: include all @definition entries (the author’s conceptual structure). Include @annotation entries only if the user explicitly chooses to share them (default: do not share reader annotations).
3. Serialize the selected entries as BibTeX blocks, applying the escaping rules from Section 2.
4. Append to the document’s visual-meta block:

@visual-meta-start{annotations,
  bundled-date = {2026-03-09T10:00:00Z},
  bundled-by   = {reader:3.2.1}
}
% annotation and definition entries here
@visual-meta-end{annotations}

5. Bundled entries are snapshots. They are not modified if the ledger changes after bundling.

11.2 Unbundling (import)

When a user opens a document containing bundled entries:

1. Parse the annotations section of the visual-meta.
2. For each @definition entry: display in the Author layer of the overlay view. Do not import into the reader’s ledger unless explicitly requested.
3. For each @annotation entry (shared by a previous reader): check the reader’s ledger by ID.

• No match: offer to import. On import, append to ledger.
• Match with same or newer date: skip (ledger is authoritative).
• Match with older date: prompt user to keep or replace.

1. Default: prefer the ledger.

12. W3C Compatibility

12.1 Export to W3C format

Each @annotation entry can be serialized as W3C JSON-LD:

{
  "@context": "http://www.w3.org/ns/anno.jsonld",
  "id": "urn:annotation:anno-a3f8c",
  "type": "Annotation",
  "motivation": "questioning",
  "creator": {
    "type": "Person",
    "nickname": "frode"
  },
  "created": "2026-03-06T14:23:00Z",
  "generator": {
    "type": "Software",
    "name": "Reader 3.2.1"
  },
  "body": {
    "type": "TextualBody",
    "value": "This contradicts the findings reported in Table 2.",
    "format": "text/plain"
  },
  "target": {
    "source": "urn:document:vm-78b2e4",
    "selector": [
      {
        "type": "TextQuoteSelector",
        "exact": "the methodology assumes a normal distribution",
        "prefix": "As outlined in Section 2,",
        "suffix": "which has been challenged by recent findings"
      },
      {
        "type": "TextPositionSelector",
        "start": 2045,
        "end": 2089
      },
      {
        "type": "XPathSelector",
        "value": "/body/section[2]/p[1]"
      }
    ]
  }
}

Field mapping:

12.2 Import from W3C format

Reverse the mapping. If the W3C annotation uses selectors not supported by this system (e.g. SVGSelector, CSSSelector), fall back to any TextQuoteSelector or TextPositionSelector present. If no compatible selector exists, import as unanchored with body content preserved.

13. Data Integrity Rules

These are invariants the system must maintain:

1. Every entry has a unique ID. Generated, never user-assigned. Collision is effectively impossible given the generation algorithm.
2. Every entry has selector-exact. Even a highlight with no note captures the selected text. This is the minimum data for re-anchoring.
3. The ledger is the single source of truth. Bundled entries in documents are snapshots. The ledger wins on conflict.
4. Selector fields are immutable on edit. Updating an annotation’s content or category does not change its selectors. To re-anchor, delete and recreate.
5. Categories are strings. Schemas are UI suggestions. An entry with an unknown category is valid and rendered in grey.
6. Colour is never stored. Derived from category via schema at render time.
7. Entries are never modified in place. All changes are appends. The file is safe to read without locking.
8. The system never silently discards an annotation. Unanchored entries are surfaced to the user.
9. Malformed entries do not block loading. A corrupt entry is skipped with a warning; the rest of the ledger loads normally.
10. The ledger file has a version number. Applications check ledger-version before writing. Newer formats require newer applications.

[Verse 2]
The sacred books are cities built of voices
The Torah wrapped in generations of responses
The Quran carried in a chain of learning
The sutras glossed by every hand they passed
And what we write today in haste or wonder
Will be the ancient text that someone finds
A thousand years from now and tries to read
Without the voice that knew what it had meant

[Verse 3]
So leave the trail, not just the words you chose but why you chose them, what you saw
What made you stop and underline and say
This matters, here, for reasons I can name
Because the page alone forgets its author
The way a building forgets the hand that built it
And all that survives of understanding.
What about what we thought to write beside the text?

[Chorus]
Select the words, say what they are
A claim, a flaw, a thread worth pulling
See them rise out of the page
And find their kind across the shelf, across the world, across time

[Verse 4]
We underline, we highlight, we write yes
Small fires struck along the trail
To find our way back through the dark of someone else’s argument
The hand that marks the page remembers
What the eye forgets.
Something in the act of stopping, naming
Pulls the meaning from the stream and holds it still

[Verse 5]
But here is where the old world breaks
The note you scrawled on page sixteen
Is prisoner of page sixteen forever…
It cannot rise, it cannot cross the room
To find the note you wrote last week
Inside a different book by a different mind
That says the same thing differently
The resonance you felt but couldn’t prove.

[Chorus]
Select the words, say what they are
A claim, a flaw, a thread worth pulling
See them rise out of the page
And find their kind across the shelf
A constellation where there was a list
A map where there was only sequence

[Bridge]
What if the annotation could stand up
Could carry its own name and walk between the documents that gave it breath
Not trapped in margins, not a second-class citizen of text but sovereign and connected
A thought with roots and wings
Anchored where it started
Free to land wherever it belongs

[Verse 6]
The author maps the territory they made
The reader maps the territory they found
Two charts of the same country drawn
In different light at different altitudes
Lay one upon the other, where they meet
Is understanding, where they part
Is where the new work starts, the question
No one asked, the gap, the absent node.

[spoken]

It has always been more important and difficult to come up with good questions than simply look for answers, and a future infused with AI will not change that.

[sung]

AI will not change that

[Verse 7 – slower, weight on each line] 

Answers are the easy part, they always were 

Socrates knew this, standing in the square 

The oracle said wisest, and he laughed 

Because he only knew what he could ask 

Now the machines can search the whole of human thought 

And bring you back a thousand answers within a minute  

But the question, the real question that was always yours to find. 

No engine yet has learned to feel the catch 

The hesitation where the mind says wait Something here is wrong, or new, or unexplained 

That pause, that mark, that annotation in the dark is still the most important thing a mind can make. 

And all the answers in the world are only servants of the question 

And progress is still hidden behind the questions no one thought to ask.

[Chorus]
Select the words, say what they are
A claim, a flaw, a thread worth pulling
See them rise out of the page
And find their kind across the shelf
A constellation where there was a list
A map where there was only sequence
A shape emerging from a thousand
Small acts of paying attention

[Outro – slow, building to final line]
This is what the margins always knew
That reading is not receiving
That every mark upon a text
Is thinking made visible, set free
From reed on vellum to the node in space
The gesture is the same, a human hand
That reaches toward the words and says
I was here, this mattered, and I know why
And someone, someday, finding this
Will know not just the words we wrote
But how we read, and what we thought
And what we understood

Imagine not only underlining when you read, but actively building a structure of the knowledge. Something which you can usefully access and analyze later.

This won’t be to everyone’s taste so it will be important to have a wide discussion of what this might be.

Proposal for Unified Annotation Infrastructure

   Here are the meeting notes for 9 March 2026:

AI: Summary

This session centered on annotation as a practice and a design challenge, weaving together empirical research findings, philosophical reflections on meaning-making, and speculative design ideas for how annotation infrastructure might work in spatial and digital environments. The group explored what people actually do when they annotate, why they do it, and how to build systems that honor those purposes — from e-ink tablets and PDF markup to XR-based spatial knowledge objects.

AI: Main Topic

Jamie Blustein opened with a presentation drawing on two of his published research papers. The first was a field study of annotation practices using physical paper, conducted across multiple universities over six years with students in computer science, electronic commerce, library and information studies, and literature. Students were given scholarly texts printed on large ledger pages to prepare for seminar discussions. The study identified a taxonomy of annotation types — arrows, asterisks, boxes, brackets, circles, highlighting, text, underlining, and compounds — and found that only six distinct purposes drove annotation for this user group: interpretive engagement, working through problems, tracing progress and drawing connections, barely engaging (marking long blocks), procedural self-direction, and place-marking as an aid to memory. The study found strong individual variation in how people annotated, but consistency in why. The second paper discussed glossaries as a form of annotation, introducing a four-dimensional classification: whether a glossary is tied to one document or floating, whether it is private or collaborative, whether it is published or unpublished, and whether it is static or updateable.

AI: Highlights

Tom Haymes explicitly highlighted his personal copy of Geeks Bearing Gifts by Ted Nelson as an example of annotation going so far as to render a text practically unreadable — describing it as “marked to hell” — illustrating the tension between annotation as engagement and annotation as obstruction.

Tom Haymes in the chat wrote: “Reading is about making a better version of yourself, not making a better version of the author (or the annotator).” This was treated as a significant provocation by several participants.

Jamie Blustein in the chat wrote: “The annotator is making the text their own. The annotation and annotated text is a new text. The notion of palimpsest as opposed to written text as fetish.” This was reacted to enthusiastically by Frode Hegland.

Frode Hegland shared in the chat: “Annotations: Point to, and leave anchors, to meaning in the world.”

Tom Haymes coined the phrase “We need a sextant for knowledge” in the chat, which Frode Hegland immediately asked him to speak it, calling it “very evocative.”

Brandel Zachernuk noted in the chat that he has long been puzzled as to why Hypothes.is has not had more impact as a system for social annotation.

Peter Wasilko closed the session by noting that Grok had impressed him by drilling directly to dissertation PDFs rather than surfacing top-level fluff, a practical tip for academic research.

AI: Insights

Annotation as first-principles design problem. Jamie Blustein argued that most annotation tool development has focused on what can be recorded technically — RDF schemas, data models — rather than starting from why people annotate. Returning to the six human purposes he identified could reground digital annotation design in genuine user needs rather than technical affordance.

The palimpsest vs. the fetishized text. A recurring conceptual tension emerged between treating source documents as sacred objects (reluctance to mark up borrowed books, fear of annotating incorrectly in front of professors) and treating them as substrates for personal meaning-making. Jamie Blustein noted that even digital paper resolves this tension partially: you always have the clean base layer. The psychological barrier, however, is cultural and social, not technical.

Annotation shapes the reader. Tom Haymes made the point that pre-existing annotations — even anonymous crowd-sourced ones as in Kindle highlights — exert a power relation over readers, nudging them toward someone else’s interpretation. Jamie Blustein confirmed research showing students are influenced by a professor’s pre-annotated copy even when told to ignore the marks. The implication: annotation is not neutral; it is always a power move over future readers.

Annotations as knowledge objects in their own right. Frode Hegland proposed reframing annotations not as attached decorations on documents but as independent knowledge objects that can be lifted from their original substrate, aggregated across sources — papers, books, places, people, conference sessions — and spatially arranged. This reframing collapses the distinction between a note and an annotation: the only remaining difference is the presence or absence of a pointer to an origin.

The annotation-as-glossary insight. Jamie Blustein‘s second paper proposed that glossaries are a distinguished type of annotation. This connects directly to Frode Hegland‘s observation that reading Behave (by Robert Sapolsky) produces a need to extract domain-specific definitions as a personal working glossary — a form of annotation that is neither marginal nor incidental, but structurally productive.

Context as annotation management layer. Frode Hegland introduced a distinction between annotation objects as the active working material and annotation objects as background context (the “canon on the shelf”). The same annotation object can function as foreground or background depending on what you are currently working on. This has implications for interface design: systems need a way to manage not just the content of annotations but their current epistemic role.

Spatial navigation of knowledge as instrument design. Peter Wasilko proposed a combinator model for navigating annotation spaces in XR — noun-class categories on one side, verb-type combinators on the other — analogous to the Stargate dial-home device: a sequence of selections through dimensional categories that narrows a vast data space to a specific target. Tom Haymes extended this with the sextant metaphor: you need to know where you are, not just where you are going, and different users (expert vs. novice) need different instruments. Frode Hegland connected this to direct spatial manipulation — twisting an object to reveal a different facet — as preferable to tapping a menu bar.

Model element as knowledge unit. Frode Hegland asked Brandel Zachernuk whether a model element in spatial CSScould represent a knowledge unit — a term and a definition — with no visual rendering. Brandel confirmed that non-rendering model elements are possible and already used as invisible scaffolding for spatial constellations, meaning that the knowledge graph and the spatial graph can be the same structure.

Zoom chat as annotation. Tom Haymes noted in the chat that Zoom chat logs are themselves a form of annotation — running alongside the spoken discussion, responding to specific moments, extending the document of the meeting itself.

Annotation standardization as political choice. Frode Hegland flagged that the W3C annotation format assumes internet connectivity, whereas document-centric annotation work does not. Choosing a standard is therefore a philosophical stance about what annotation is for, not merely a technical decision.

AI: Resources Mentioned

Websites and URLs

https://futuretextlab.info/2026/03/09/annotations-format-w3c-shaped/ — Frode Hegland‘s current technical proposal for an annotation format shaped by the W3C standard, shared in the chat by Frode Hegland

https://thefutureoftext.org — the community’s main website, shared by Frode Hegland

https://web.hypothes.is — Hypothes.is, a social annotation platform, mentioned by Brandel Zachernuk with puzzlement at its limited adoption

https://archive.org/details/howtoliewithstat00huff — How to Lie with Statistics by Darrell Huff, shared by Tom Haymes as his favorite statistics book

https://www.youtube.com/watch?v=EjUElpLZZtc — a Bill Buxton video on the inherent temporality of annotation, approximately 15 years old at time of meeting, shared by Brandel Zachernuk

https://developer.mozilla.org/en-US/docs/Web/API/File_System_API/Origin_private_file_system — the Origin Private File System web API, suggested by Brandel Zachernuk as potentially analogous to NSFileCoordinator for file coordination on the web

https://wasabi-jpn.com/magazine/japanese-grammar/combined-particles/ — a resource on Japanese combined particles as grammatical combinators, shared by Peter Wasilko in relation to his combinator model

Books

Geeks Bearing Gifts — Ted Nelson, mentioned by Tom Haymes (his used copy was heavily annotated by a previous owner)

Behave — mentioned by Frode Hegland as a book he is currently reading at the recommendation of a partner named Andrea; used as an example of reading that produces glossary-type annotations

A Humument — artist’s book by Tom Phillips, mentioned by Jamie Blustein in which the artist draws over the pages of a Victorian novel, incorporating the letters into artwork; updated in multiple editions over the years

Mona Lisa Overdrive — William Gibson, mentioned by Jamie Blustein for a scene in which a character uses a hand photocopier to scan pages and glue them into a notebook

Information: A Short History — a compendium edited by Ann Blair and others, mentioned by Brandel Zachernuk in relation to copying, cutting, and pasting practices across history

Meaning (1975) — Michael Polanyi, quoted by Frode Hegland in the chat: “We live in the meanings we are able to discern”

People

Ted Nelson — hypertext pioneer, referenced for the phrase “text under glass” and for his time as a paste boy at the New York Times; also author of Geeks Bearing Gifts

Cathy Marshall — researcher at Microsoft, mentioned by Jamie Blustein for studies of graduate students who were reluctant to share annotations for fear of being judged

Vannevar Bush and Brush (research by Marshall and Brush) — mentioned by Jamie Blustein regarding reluctance among graduate students to share their annotations

Douglas Adams — Frode Hegland recalled one meeting with him in which Adams described the concept of “virtual graffiti” — writing left in a physical space readable only if you are present in that space

William Burroughs — mentioned by Brandel Zachernuk as the most prominent practitioner of the cut-up technique, a form of radical document manipulation

Ann Blair — editor of the information history compendium mentioned by Brandel Zachernuk

Darwin and his grandfather Erasmus Darwin — mentioned by Jamie Blustein as notable historical annotators, especially Erasmus Darwin who drew long lines to mark entire passages

Bloom — mentioned by Jamie Blustein in reference to Bloom’s taxonomy of educational objectives, against which he structured his annotation classification

Elian Gonzalez — used by Jamie Blustein as a case study in rhetorical codewords and spin in media coverage

Technologies and Products

Supernote — e-ink tablet used by Jamie Blustein, discussed for its pen-tip-free stylus, keyword tagging, star gesture recognition, and landscape mode limitations

Remarkable — e-ink tablet considered but rejected by Jamie Blustein due to its software-as-a-service business model

Sony tablet — previously used by Jamie Blustein, noted for needing frequent pen-tip replacement

Apple Vision Pro / visionOS — spatial computing headset used by Frode Hegland for the Author application development, discussed extensively regarding its control bar and interaction design

Author (app) — Frode Hegland‘s application for visionOS, demonstrated live with its annotation and highlighting capabilities

Kindle — mentioned by Tom Haymes and Jamie Blustein; Jamie expressed disappointment with most annotated passages surfaced in Kindle

Meta home environment — mentioned by Peter Wasilko as a spatial context for anchoring knowledge tables on wall surfaces

Grok — AI tool praised by Peter Wasilko for its ability to locate dissertation PDFs directly

Gemini — mentioned by Tom Haymes as useful for academic paper discovery, and referenced for its claim that World of Warcraft has the most flexible and customizable game UI

W3C Web Annotation standard — mentioned by Frode Hegland as partly relevant to his annotation format work but too internet-dependent for document-centric workflows

Acrobat — mentioned by Jamie Blustein as a tool he uses to annotate documents before distributing them to students

Better Touch Tool — mentioned by Peter Wasilko for its hierarchical Touch Bar button menus, proposed as an analogy for combinator-based interaction in visionOS

BibTeX — mentioned by Frode Hegland as a model of simplicity for document-centric annotation format design

Stargate (franchise) — Peter Wasilko referenced the dial-home device (DHD) from Stargate as a model for combinatorial spatial query construction

Foundation (Apple TV series) — mentioned by Frode Hegland referencing the “Prime Radiant” as an evocative spatial knowledge instrument metaphor

Oulipo — mentioned by Jamie Blustein as relevant to discussions of hypertext and cut-up literary practice

Blue Sky, X (Twitter), LinkedIn, Slack — mentioned by several participants in a closing discussion about fragmented social media ecosystems and difficulty maintaining a shared community communication channel

RSS — mentioned by Frode Hegland and Tom Haymes as potentially useful for community information sharing, with Mark Anderson (not present) cited as an advocate of RSS and blogging

Pre-Session Songs

Co-Authored with Claude after extensive research on the different aspects of annotations and re-implementation in Author and Reader:

Lyrics

Pre-Session Poem

Co-Authored with Claude after extensive research on the different aspects of annotations and re-implementation in Author and Reader:

The author draws a map of what they meant. The reader draws a map of what they found. Neither map is the territory. But laid together, edge to edge, they show the rivers that run under both — the questions that the text was built upon and the questions it left for the first reader brave enough to write I‘m not so sure in the margin of the margin of the world.

What changes is not the act. The monk and the scholar and the student with a yellow highlighter are performing the same gesture: stopping the eye, naming the catch, externalising the moment when the mind says here.

What changes is the aftermath. For the first time in the history of reading, many ‘here’ can gather. A thousand moments of attention, each one small, each one honest, can rise from their separate pages and find their shape —not the shape you planned but the shape that was always there, waiting in the negative space between everything you’ve read, visible only now that the margins have learned to speak to one another.

Post Session Song

If we abstract it, we could think of a paper being a collection of annotations.

• Annotation mean essentially add-a-note to a document. How this is done and where the ‘annotation’ lives is not a trivial matter if we are to unleash the potential of annotations. That is what I am trying to work out here.
• Notes do not need to be added to other information, in contrast with Annotations.
• Document in this context refers to a thoughtful, coherent text intended, at some point, for publication/sharing.
• The workflow of an annotation or note is for the user to mark down a note about something, either in full text or as a glyph or color on a document (such as a color tab, underline, note in the margin) or separately, in a different document. There are several related terms, such as underlining, glossing and so on, which are ‘annotations’ in the way used here. This has been shown to be useful to increase the user’s understanding of the text.
• Annotated Bibliography is a structured list of research sources (books, articles, documents) where each citation is followed by a brief paragraph—the annotation—that summarizes, evaluates, and reflects on the source’s relevance and quality.

Interaction with Annotation

To Review single document with Annotations. For this the user will access the document and choose to view annotations in situ or separately, linearly, mapped or categorized, where the user should be able to specify what is useful, what should be hidden or deleted and to copy as citation if this then leads to a new document. Reader features the start of this, allowing users to highlight sections and choose what annotation to assign to them with are then marked in color, as well as option to write notes anywhere and a single ‘Annotation’ option per document, which can be used to compile Annotated Bibliographies

To Review a series of documents with Annotations. This is where the user should be able to interact with Annotations as first-class-objects, not being only inside documents.

Specification

Sample

The 23 February 2026 session of the Future Text Lab brought together Frode Hegland, Tom Haymes, Peter Dimitrios, Ken Perlin, Astral_Druid, and Rob Swigart (via chat) for a wide-ranging conversation centered on how text, citations, and knowledge structures can be meaningfully represented and interacted with in spatial and extended reality environments. The session moved fluidly between the concrete design challenge of displaying quotes and citations in visionOS, the broader philosophical question of what XR genuinely offers over 2D interfaces, the role of AI as infrastructure rather than a feature to bolt on, and the historical precedents — HyperCard, VisiCalc, WordPerfect — that might help the group identify what a “killer app” for spatial text could look like.

AI: Main Topic

The primary focus was Frode Hegland’s presentation of two slides posing a specific design question: when displaying a citation or quote as a node in XR, what should be shown in its closed state versus its open state, and how should different types of scholarly objects — defined concepts/glossary terms, citations (author, title, BibTeX), and quotes (citation plus text) — be visually distinguished and navigated. Frode proposed showing three lines of a quote with author and title beneath in the closed state, with full scrollable content on open, and floated the idea that node depth could encode the amount of text contained. This design question opened into a much broader discussion about context versus content in spatial environments, the nature of AI-assisted versus manual navigation, and what spatial computing can do that flat screens simply cannot.

AI: Highlights

Tom Haymes highlighted the importance of breadcrumbs and suggested color-coded layered controls at the bottom of each node — a rollover or gaze-based mechanism to shift between annotation, citation, and connection views — noting that citations are fundamentally breadcrumbs pointing outward.

Ken Perlin introduced the analogy of AI as road infrastructure: just as a car builder must design for roads that already exist and that shape every decision, anyone building spatial text tools must accept that AI is the underlying infrastructure, not an optional add-on. He explicitly stated this doesn’t mean putting AI into everything, but it does mean understanding where things are heading from 2026 onward.

Peter Dimitrios repeatedly stressed the value of specificity — “do a particular thing” — and cautioned against boiling the ocean. He noted that Frode’s authoring use case for Author is already well-defined and that overextending the spatialized version risks losing that clarity.

Ken Perlin proposed the concept of diegetic prototyping as a methodology the group should explore: looking at how science fiction films and TV shows (referencing Minority Report, John Wick, and Iron Man/Jarvis) depict spatial and AI-assisted interaction as a way of thinking through design fictions for mixed reality text environments.

Astral_Druid (via both voice and chat) raised the observation that any research study comparing XR to 2D interfaces faces an especially tricky problem: people are changed by the environments they inhabit, meaning the group is designing for a moving target. Ken affirmed this as a genuine research design challenge requiring careful controls.

Frode highlighted a moment of embodied confusion as a genuine insight: while in conversation, he instinctively tapped to trigger the “focus” function in his spatial app — despite not being in VR. He compared this to the frustration users of his Liquid tool feel when working on a machine without it, and suggested that producing this kind of “withdrawal” feeling could itself be a definition of what a killer XR app would achieve.

AI: Insights

The distinction between context and content in spatial environments emerged as a genuine conceptual reframing. Frode demonstrated that participant profiles, glossary terms, and relational metadata function best as passive “context” — background objects that can be ignored or snapped away — while quotes and citations are active “content.” This separation allows a spatial workspace to breathe without becoming a visual spaghetti of equally weighted objects.

There is a productive tension between Peter’s and Ken’s positions on 2D versus 3D. Peter argued that 2D environments like GatherTown / WorkAdventure already deliver a surprisingly effective conference-navigation experience without requiring a headset, and that 3D may be “sugar coating the information architecture.” Ken countered that the market reality — essentially nobody is using VR compared to hundreds of millions using 2D interfaces — means there can be no standards yet, and that VR today resembles hi-fi audio in the 1950s: a niche interest that won’t drive conventions. Frode played devil’s advocate for the “movie theater” position: some experiences justify a special environment even before it is widely accessible, and the group should feel free to invent for that environment without requiring it to degrade perfectly to 2D.

The shift from hardcoded Ted Nelson–style hyperlinks to AI-inferred contextual links was recognized as a fundamental transition the group is living through. Peter observed that where authors once explicitly authored the link (“go here”), AI will increasingly infer the most statistically probable destination — which raises the question of how an author preserves intentional, curated navigation against the probabilistic averaging of large language models.

Ken’s observation that he is prototyping a Wikipedia visualizer where the entire page text acts as the link — with an AI assistant deciding what to link to at runtime — is directly consistent with Peter’s framing. Both point toward documents where the link layer is no longer baked into the document structure but is generated dynamically based on context. This is a significant departure from the document model the group has been working with.

The x,y,z coordinate system for spatial layout was collectively reframed as a necessary but insufficient abstraction. Ken noted that people organize real rooms not by coordinates but by lived embodied logic — table, window, bookshelf — and that spatial information environments need to reach for a semantic layer above raw coordinates. Peter proposed layout “connection types” (stack, sphere, array, column) as a more human-meaningful vocabulary than positional values. Frode agreed that x,y,z is purely an internal representation — equivalent to a page layout engine — and that no user cares about it directly.

The question of whether walking through information is inherently valuable — or whether “zoom” (the ability to move toward and away from information density at will, with or without physical movement) is the actual underlying affordance — remained productively unresolved. Tom introduced zoom-in/zoom-out and speed-up/slow-down as two independent but related axes of cognitive control that spatial environments could serve. Photography came up as Tom’s personal analog for deliberate deceleration — a conscious hitting of the brakes — suggesting that the design of “slow” modes in information environments may be as important as fast navigation.

The form-factor barrier to XR adoption was named clearly and practically: Frode noted that his partner Dini won’t wear the headset in the morning because it disturbs her hair before meetings. Ken stated the Meta Quest 3 form factor is “a complete showstopper for almost everybody” regardless of its functionality. AR glasses — not VR headsets — were agreed upon as the inevitable and necessary form factor for mainstream spatial text use, with Apple Vision Pro positioned as the closest current research-quality tool, analogous to the original Mac as a first-generation device pointing toward what would eventually be practical.

Peter raised a privacy concern about camera-equipped glasses that went beyond mere inconvenience: face recognition capabilities built into AR glasses could “poison the well” for the entire category, creating a “glass-holes” dynamic that society may reject regardless of the utility of the underlying technology. Ken affirmed this is already happening.

AI: Resources Mentioned

Walter Ong, Orality and Literacy — mentioned by Frode Hegland as currently being read and described as generating an inspiration in every sentence, making it difficult to get through. PDF shared by Peter Dimitrios: https://monoskop.org/images/d/db/Ong_Walter_J_Orality_and_Literacy_2nd_ed.pdf

Visual Meta Spatial format — Frode Hegland’s spatial layout specification using id plus x,y,z: https://visual-meta.info/spatial-visual-meta/

GatherTown — 2D conference environment mentioned by Peter Dimitrios as a successful model for navigating conference rooms virtually. No longer open source. OSS alternative WorkAdventure: https://workadventu.re/ (GitHub: https://github.com/workadventure/workadventure)

Make It So: Interaction Design Lessons From Science Fiction — book shared by Tom Haymes, described as “sci-fi design lessons”: https://a.co/d/0aQupgEF — Ken Perlin had not read it and expressed intent to do so; Frode confirmed having read it.

Liquid — Frode Hegland’s macOS text-selection tool enabling rapid search, translation, and reference operations via keyboard shortcut and toolbar.

Author — Frode Hegland’s macOS writing application with integrated visual map and Visual Meta support.

Ward Cunningham and the Federated Wiki — mentioned by Peter Dimitrios as the context for his 3D wiki explorations, with Federated Wiki pages licensed CC-BY-SA.

Google NotebookLM — mentioned by Peter Dimitrios as an example of AI-generated views going beyond explicit citation and link authorship.

Google Gemini — mentioned by Ken Perlin as a tool he has been using, noting it does not create the illusion of being a person, which he found “refreshing.”

Claude — mentioned by Frode Hegland as his primary AI tool, used for spatial data analysis and text processing in Author, including a “paste processed” function using Apple Intelligence and ChatGPT.

HyperCard / Bill Atkinson — discussed extensively as a historical precedent; Rob Swigart noted via chat that Steve Jobs killed it and that Bill Atkinson was asked to build a VR version but declined.

VisiCalc, WordPerfect — referenced by Peter Dimitrios and Ken Perlin as paradigm-defining applications whose descendants still shape current software, and as a benchmark for what a spatial “killer app” would need to achieve.

Ted Nelson / Xanadu — referenced in the context of explicit authorial hyperlinking versus AI-inferred linking; Tom Haymes and Ken Perlin affirmed in chat that automated linking in Xanadu was indeed Nelson’s intent.

NLS / Doug Engelbart — referenced by Frode as the source of the command-with-question-mark interaction pattern that inspired Liquid.

Second Life — mentioned by Tom Haymes and Peter Dimitrios as a cautionary precedent: conferences held in it often just reproduced PowerPoint presentations, failing to use the medium distinctively.

Babylon.js and Three.js — mentioned by Peter Dimitrios as his current WebXR development tools.

Apple Vision Pro, Meta Quest 3, Meta Ray-Bans — discussed as current XR hardware; Ray-Bans noted by Tom Haymes as increasingly visible in public.

Neo4j — mentioned by Peter Dimitrios in the context of graph database work with node-layout visualizations.

Hiroshi Ishii’s ClearBoard — referenced by Ken Perlin as a 34-year-old precedent for shared spatial interaction over video, with data appearing in the same position on screen for both participants.

Minority Report, John Wick, Iron Man / Jarvis — referenced by Ken Perlin in the context of diegetic prototyping as designed fictions worth studying for spatial interaction ideas.

Munch Museum, Oslo — mentioned by Frode Hegland as a recent visit, noting one floor used physical dark-room installations to orient visitors around Edvard Munch’s life as an example of “fake spaces helping people orient.”

Jim Groom — mentioned by Tom Haymes as having delivered a memorable Second Life conference presentation involving a flamethrower that demonstrated what the medium uniquely enabled.

Song

Topics will include interactions for choosing what to select, what to view, how to arrange, how to show and hide and so on.

Further discussions are expected to include timelines, writing practices and the bigger question of how immersive systems might augment human thinking rather than replacing it.

We also asked what different kinds of annotations might mean on different kinds of knowledge objects. There is naturally a difference between annotating what is yours (your own notes) and that which you have acquired (academic papers, books etc.).

This is what the system we used for dialog looked like mid-February:

It is fair to say that XR is a ‘tool looking for an application’. This month we are looking at what might be useful in XR to orient oneself in a Conference, perhaps with scheduling and with the academic papers being published.

Future sessions to be planned: Interactions, Metadata, annotations & more…

AI: Summary

The primary thread was the relationship between XR interfaces and knowledge work, examined through two contrasting live demonstrations. Frode Hegland showed his Author system running in a headset, visualizing metadata from the ACM Hypertext 2023 conference — people, papers, locations — with selection, layout, and focus controls operated via a spatial toolbar.

Ken Perlin then demonstrated two prototypes: a hierarchical Wikipedia page viewer navigable with both controllers and hand gestures, and a semantic zoom demo using Robert Frost’s “Stopping by Woods on a Snowy Evening,” where walking closer to an object progressively reveals its title, full text, and analysis.

Sam Brooker introduced the ACM Hypertext 2026 conference (London, September 14–18), themed around “hypertext method” — hypertext understood not just as technology but as a way of thinking: non-linear, networked, dynamic. This framing set the stage for the group’s broader inquiry into how spatial computing might serve scholarly communities.

AI: Insights

Sam Brooker articulated a distinction that proved generative for the entire session: the difference between XR as navigation (moving between two pieces of important information) and XR as a site of knowledge creation (staying in the spatial interface to build meaning through relationships). He drew on Minority Report to illustrate this — all that gestural swiping is ultimately just transport between conventional video clips. The question he posed — whether the kinesthetic, spatial experience is interstitial or constitutive — became a recurring lens through which the group evaluated every demo shown.

Ken Perlin offered a bracing counterpoint to enthusiasm about XR knowledge tools by describing his “harsh test”: would he use any of this instead of his screen, trackpoint, and keyboard if no one were watching? His honest answer — universal failure so far — served not as pessimism but as a methodological standard. He framed his own prototyping as a study in “shared failure,” deliberately keeping AI out of his builds until he understands how bodily interaction with spatial information actually works. His emphasis that “technology is easy, good visual design is hard” recentered the conversation on interaction design rather than capability.

Ken Perlin’s semantic zoom demo — where proximity determines the level of detail revealed — crystallized an idea that several participants latched onto. The notion that an AI could dynamically generate summaries sized to your visual distance from an object in space married the group’s concerns about summarization with a concrete spatial mechanic. Peter Dimitrios called this “progressive disclosure by leaning in” and identified it as genuinely new to AR/XR.

Jonathan Finn proposed a topic the group recognized as underexplored: the visual representation of meaning as distinct from text. He noted that while everyone touches on this implicitly, the community has never directly addressed how information extracted from text might be shown without using more text. Sam Brooker responded in the chat by calling text “a kind of latent skeuomorphic hangover,” and Jonathan clarified he was thinking of semantic graphs — objects with labelled arrows — as standard OS-level interface elements.

Sam Brooker raised a tension around canonicity and curation in scholarly communities. Making it easy to surface, link, and layer commentary risks over-amplifying certain voices — particularly those who happen to write single quotable sentences rather than deeply important but hard-to-summarize work. He argued that the friction of manual pursuit has its own epistemic value, and that the goal should be to introduce complexity in a manageable way rather than removing it through summarization.

The chat log surfaced a parallel debate about AI authorship. Peter Dimitrios drew a line between AI as tool (spell-check, calculator) and AI as author, asking “where is ‘my’ voice?” Tom Haymes pushed back, questioning whether we fetishize writing itself without interrogating the function of the paper — if you read an AI-generated text and say “I approve,” does that make it yours? This tension between AI as augmentation and AI as displacement ran throughout both the spoken and written threads.

Frode Hegland and Ken Perlin agreed to exchange data — Frode will send a glossary derived from this meeting’s transcript to Ken, who will attempt to visualize it in his system. The intention is to look at the same meeting data through radically different spatial interfaces, which could reveal what is universal versus idiosyncratic in spatial knowledge representation.

AI: Resources Mentioned

ACM Hypertext 2026 conference — https://ht.acm.org/ht2026/ — introduced by Sam Brooker, taking place in London, September 14–18, 2026, themed “hypertext method”

Future Text Lab meeting page — https://futuretextlab.info/2026/02/19/march-2-2026/ — shared by Frode Hegland

Augmented Text Info — https://www.augmentedtext.info — shared by Frode Hegland

WikiNodes by Ken Perlin — https://cs.nyu.edu/perlin/wikinodes — shared by Ken Perlin, a tool for hierarchical Wikipedia browsing with linked page navigation

Project Cybersyn article (MIT Press) — https://thereader.mitpress.mit.edu/project-cybersyn-chiles-radical-experiment-in-cybernetic-socialism/ — shared by Peter Wasilko

Portland State University economics working paper — https://pdxscholar.library.pdx.edu/econ_workingpapers/67/ — shared by Peter Wasilko

Google Home web interface — https://home.google.com/u/0/ — shared by Brandel Zachernuk

Prompts.chat information gathering prompt — https://prompts.chat/prompts/cmm55hzfp0001l504s6rrl4hb_information-gathering-prompt — shared by Peter Wasilko

Last week’s meeting summary — https://futuretextlab.info/2026/02/19/23-feb-2026/ — shared by Frode Hegland

Gather (virtual workspace for remote teams) — mentioned by Ayaskant Panigrahi as an example of proximity-based audio, suggesting proximity-based controls for connections between text nodes

Minority Report — referenced by Sam Brooker and Frode Hegland as an example of gestural interfaces that are navigational rather than generative

Tron — referenced by Frode Hegland as a metaphor for entering a computer environment, contrasted with AR/XR as liberating knowledge into your hand

Diegetic prototyping in Hollywood — referenced by Ken Perlin, including Iron Man (Tony Stark and Jarvis) and John Wick, examined with his students as inspiration for intuitive spatial interaction with AI

Stuart Card — referenced by Sam Brooker in the chat, on communication between device and user as a stream of symbols

WordNet SynSet — mentioned by Peter Wasilko as a potential augmentation layer for every word

Song

For analysis

Full transcript: https://www.dropbox.com/scl/fi/s9yl8gvrlubcvlsvjxenk/2-March-2026.rtf?rlkey=2ml9thrxlq322v06xqdd3ava0&dl=1

CLEANED transcript: https://www.dropbox.com/scl/fi/oa2w0drsthb1c1ae7g1kd/2-March-2026-Cleaned-Transcript.md?rlkey=v3p3y0b5btbbtpoh9p0z0zefg&dl=1

Glossary: https://www.dropbox.com/scl/fo/9g2gf0f1kc4nzvjmai8e9/AKnRxx-qn0UD7l45t5uDINw?rlkey=iuj339967bv3clrmza43ij9rl&dl=0

We will consider ACM Hypertext ’26 as the case study and we will use Author for visionOS to build it, with the expectation that exports to JSON will be available for anyone to implement their own XR views.

This is a specific view of Dialog in XR, with mostly written interactions via published papers etc.