Learn it by watching
A short film per subject, each narrated and captioned, each built from documents a reader can open here. Watch one, then open the documents it showed.
Each film takes one subject apart using real documents: what a document is made of, and what a vocabulary is. Each has captions on the player and the whole narration written out beneath it, so a film can be read as easily as watched.
Anatomy of a document
A unimsg document taken apart, sigil by sigil: the header and comments, blocks and keys, text, dates, numbers and units, symbols, references, hashes, bytes and extensions, and how a table is laid out so a person can read it. Then the exact bytes it becomes.
Transcript
- Welcome to unimsg! In this first episode, we're taking a document apart, sigil by sigil.
- A unimsg document is plain text.
- You can write it by hand, and read it with your own eyes.
- And a machine turns it into exactly one sequence of bytes. Not roughly one. Exactly one.
- So let's read one together.
- Every document starts with a header. The percent sign says here comes the format; zero is the version, and the label says what this is. It never makes it into the bytes.
- Two dashes start a comment, for humans only; machines skip it. Comments never reach the bytes, so fixing a typo in one can't change the document's hash.
- Now the document itself. A key, then its value, is a pair. Wrap pairs in braces and you get a block: a map. Each key appears once.
- That at sign makes an identifier: a name you give a thing, so the rest of the document can point at it. In the bytes it's a tagged name, and the at sign isn't stored.
- Text is the easy one. Double quotes, and whatever's inside is kept exactly as written: UTF-8, no normalising, no surprises. In the bytes, a plain text string.
- Dates and times need no quotes, and the precision you write is the precision you keep. Two thousand twenty-six, ten, means the month, not a day.
- Three flavours of number. A whole number is an integer. Add a point and it's an exact decimal, so eighty point zero zero keeps its zeros. End it with an f, and it's a binary float. Nothing is a float by accident.
- A bare word before a value is a qualifier. The classic use is a unit: kg, two point four, is two point four kilograms. In the bytes, the word travels with the value.
- True, false and null are values, written as plain words. True is a single byte in the output. For floats, there's also not-a-number, infinity, and minus infinity.
- A colon makes a symbol: a label from a set of choices, like packed, draft, or open. A name, not free text. You'll see symbols everywhere once we meet vocabularies.
- The arrow is a reference. It points at something named with an at sign, here the sender, one line up. And reading never follows it. A reference is a pointer, never a fetch.
- A hash sign starts a content hash: an algorithm, a colon, then the digest in hex. We've shortened this one to fit. A hash names content by what it holds.
- The tilde brings in raw bytes, never text. Hex takes hex digits, b64 takes padded base sixty-four, and each has exactly one spelling, so two texts can never mean the same bytes.
- An exclamation mark is an extension: something the format doesn't define itself. A name, then a value. A reader that doesn't know it can still keep it, intact.
- The caret starts a typed array: lots of numbers of one type, packed tight. Here, thirty-two bit floats. Add a shape like two by three for a grid, and null for a missing value.
- Last but not least, the pipe. Square brackets hold a list; pipes make it a table, with a header row naming the columns, and one row per record. Let's zoom in.
- Five more glyphs are held in reserve: ampersand, dollar, star, question mark and backslash. Writing one is an error, so a future sigil can never change what an existing document means.
- Tables are where unimsg gets really readable.
- The header row names the columns, once. Each name is a key.
- Every pipe starts a row, and each row is one record.
- A cell can be any value: text, integers, decimals, symbols. Watch how each column keeps its own kind.
- And those spaces? They only line the columns up, so you can scan down them. They change nothing in the data.
- Underneath, a table is just a list of maps, one map for each row.
- Row by row, it's the same data, written two ways.
- Write it either way. The bytes come out identical.
- Now let's look underneath, at the actual bytes. Each line here is a whole document with a single pair.
- A one means a map with one pair. Then six three starts a text of three bytes: Q, T, Y. And zero three is the integer three.
- True is just one byte: F five. And the key is a text of two bytes, so it starts with six two.
- Price, nineteen ninety-nine, becomes C four, which is tag four: an exact decimal. Then eight two opens a list of two items: minus two, and one thousand nine hundred ninety-nine. Exact, never a float.
- Now the best bit. Write the keys in either order, and you get the same bytes, because the bytes sort the keys by their encoding.
- So spacing, comments, and key order never reach the bytes.
- Which means a hash of the bytes names the content, and nothing else.
- And here's every sigil on one sheet: what it looks like, what it means, and what it becomes in the bytes. Take your time with this one.
- That's every sigil, and the whole shape of a document.
- Next time: vocabularies, which say what a document's shape means. See you there!
Vocabularies
What a vocabulary is, and how a document is checked against one. Then vocabularies at work, each through a real document: artwork, a language's grammar, a spreadsheet, a building and a family tree. A vocabulary can also carry the way its documents are shown, and a building's three dimensions are stored in the document, calculated from it, and drawn in a fly-around.
Transcript
- Welcome back! Last time, we read a document, sigil by sigil. This time, we find out what a document actually means.
- Last time, we learned how every value is written: text, numbers, symbols, bytes, tables.
- But the format itself says nothing about what a key means. Is size a width in pixels? Or a shirt size?
- That's the job of a vocabulary. It says what a document's keys mean, so a person and a program can agree about a document neither of them wrote.
- Let's see how it works.
- A document says what it's written in with one line: conforms, an arrow, and the vocabulary's name. And remember, a reference is just a pointer, so nothing is fetched. It simply names the rules.
- Then a gate reads each block of the document against the vocabulary it declares. This file holds two, an animation and a graphic, and both conform.
- Now invent a key the vocabulary doesn't define: colour, gold. Sounds harmless. But the gate refuses it, and tells you exactly where: graphic dot colour is not a key this vocabulary defines here. The animation beside it is untouched.
- A vocabulary is itself a unimsg document. Look at its first line: it conforms to the vocabulary for vocabularies, which, in turn, is written in itself.
- One line tells you a lot. Consumer says who reads documents in this vocabulary: a person, a program, or both.
- Then the heart of it: a table of keys. Each row is a key the vocabulary defines, whether it's required, optional or conditional, and the rule its value must follow.
- The requirement column says whether a key has to be there. Required and missing? Refused. Optional? Fine, and it means nothing. Conditional? Required when a stated condition holds.
- The rule column says what the value must look like: text of at least one character; one of a closed list of symbols; or a reference to another table that describes the shape.
- And every way a document can be refused is listed: an id, and who decides it. Either the contract, which is mechanical, or a program check, for rules too subtle for a table.
- Why does a vocabulary care who's reading? Because a person can meet a category they've never seen and think about it. A program can't. So for a program, every set it acts on is closed, and for a person, a document can declare its own.
- A vocabulary can carry more than rules. It can carry a default view: how any document written in it is shown. This block lives inside the building vocabulary itself. It says: over the building vocabulary, emit an interface. A vocabulary and its default view are one deliverable.
- A view is declarative. It starts from the document, here its representations, can join in what it needs, and says which fields to show, and in what order. No code. A default view may not be code, so it can be read, diffed and checked like any other document.
- So when a viewer is handed a building document that nobody wrote a view for, it finds the vocabulary the document declares, finds the view that vocabulary carries, and draws it. Asked to explain itself, the viewer says exactly that: this subject, this vocabulary, this view, found in the vocabulary.
- And it's honest when there isn't one. When a vocabulary carries no default view, the viewer says so, and says the fix: the vocabulary can carry one, or the document can carry its own. Here is what it printed for one, on the day this was recorded. Resolution runs from a view beside the data, to the vocabulary's, to a last resort that can draw any document by its shape.
- Now let's meet five vocabularies at work. Each one is shown through a real document written against it.
- First, the graphic vocabulary: artwork, stored exactly. This is a real sticker, drawn straight from the document on the left.
- The outline of the smile is two typed arrays, like the ones from part one. Ops says what to do: zero is move, two is a cubic curve, and five closes the shape. A move, two curves, and a close.
- And coords holds every number those commands need, as integers. Each is divided by the document's precision, so minus twenty-four thousand means exactly minus twenty-four. There's no floating point to drift between machines.
- A fill is a paint. Here it's a colour literal. It could also be a gradient, by reference, or a theme token, or nothing at all.
- Every primitive has a role and a name. Names matter: it's how an animation, like the film you'll see at the end, says which thing to move.
- Next, the language vocabulary, which describes a whole grammar. This is Classical Arabic. Look at how it handles categories. It doesn't fix a list: the document declares them.
- Number has singular, plural and dual. And the dual is declared whether or not this language has one, because the paradigm has to be able to say that a dual cell is impossible, rather than just leaving it empty.
- Here are the independent pronouns, drawn from the document's own cells: person down the side, number across, and the form in the standard Arabist transcription.
- And look at this empty cell: the first person dual. The document doesn't leave it blank. A gaps table says: impossible, because Classical Arabic has no dedicated first-person dual; the plural serves for two speakers.
- That's the gaps table, written in the document. Not missing: impossible, with the reason beside it.
- And the document is honest about what it leaves out. A coverage table marks morphology, syntax and the lexicon as absent, not just omitted, so you can tell a first pass from a thin description.
- The interface vocabulary does the opposite. Its kernel is closed, because its reader is a code generator, and a code generator silently drops a control it doesn't recognise.
- Same repository, same mechanisms, opposite answers.
- The difference is who's reading.
- Now a spreadsheet, as a document. A column has a name, a type, a unit, and its values. The unit is one word: pounds sterling.
- A formula isn't a string to be parsed; it's a closed tree. Here: op sum, over the column named net. And right beside it sits the value that formula resolved to: 275 pounds 85.
- Add up the five net amounts and you get exactly that. So the document carries its own answer, and the gate agrees.
- Change the value without the formula, or the formula without the value, and the gate refuses it: carries 275.86, and its formula resolves to 275.85. A document can't quietly disagree with itself.
- The building vocabulary describes physical spaces, the boundaries they share, and the fabric around them. This is a two-storey studio house.
- Spaces are the rooms. Each has a label, a kind, the storeys it sits on, and a representation: its floor outline. The stairwell is one space on two storeys, with one identity.
- A boundary is shared between two spaces. The studio and the hall are separated by one boundary, and the wall that fills it is an element.
- An opening belongs to exactly one boundary. So a doorway is a fact about the boundary, not a hole in a wall that somebody has to guess.
- Since every room is a polygon, we can draw the plan straight from the document: the studio, the hall, and the entry forecourt, with the walls drawn from the same data.
- The studio and the hall share one boundary, and it's this wall.
- And two openings: the door through the shared boundary, and the entry from the forecourt.
- Now, how is the whole three-dimensional building stored? Every coordinate lives in one frame, and the document doesn't describe that frame itself. It pins another file, by name and by hash, so there's exactly one frame, and it can't change underneath.
- That file declares it: the unit is metres, the frame is right-handed, and z is up, to a precision of one millimetre. So there's no guessing what a number means, or which way is up.
- Storeys are just elevations along that up axis: the ground at zero, and the upper storey at three point two metres.
- A shape is a polygon: a ring of corners, each an x, y, z triple of exact decimals. This is the base of the ground slab: sixteen metres by ten, sitting a quarter metre below the ground storey. Any further ring would be a hole.
- Its area is arithmetic on the corners: sixteen times ten, a hundred and sixty square metres, exactly. No mesh, and no rounding.
- A solid is an extrusion: that polygon, swept along a vector. Up a quarter of a metre, and it's the slab: a hundred and sixty times nought point two five, forty cubic metres.
- Start with floors. A floor is a ring of corners, so its area is arithmetic: the shoelace formula, on exact decimals. The ground studio is 55.68 square metres, and the hall 92.16. Exact, not rounded.
- Solids are just as direct. The ground slab is that 16 by 10 base swept a quarter metre: 40 cubic metres. The front wall is its outer ring, less four window holes, swept 0.2 deep: 74.26 square metres of wall, so 14.852 cubic metres.
- The stair has 16 treads, each 0.2 metres higher than the last, and the top one reaches 3.2 metres: exactly the upper storey's elevation. The document agrees with itself.
- Add up every extrusion in the document, and the solid fabric totals 192.9632 cubic metres.
- And questions about space are arithmetic too. A point inside the partition wall is inside it. A point in the doorway, 1.2 metres wide and 2.2 high, is outside it. Same rule, exact answers.
- And here is the whole building, drawn by the building vocabulary's own default view, straight from those polygons and extrusions. There's no mesh file and no model, just the document. The camera is the only thing added.
- Slice off the roof, and the upper storey appears: the studio, and the gallery running round the stairwell void.
- Cut down to the ground storey: the hall, and the studio, the partition between them with its doorway, and the sixteen risers of the stair.
- So every question about this building, where is the door, how much is in the slab, how high is the stair, is arithmetic on the document. The building is the document.
- One more rule I love. If something isn't written, that means: not recorded.
- It never means false, or zero, or not there in the world. A document can't claim what it didn't say.
- Genealogy has one big idea: a family tree isn't a set of facts, it's an argument. What the records say, what the researcher concluded, and the distance between them.
- Here's one parent-of relation, William to George. Look at the basis: presumed birth. The file never said George was William's biological son. It's a default, and the document says that it's a presumption.
- Every relation carries its evidence: which record says it, the words it used, whether it supports the claim, whether the record is original, who told it, and how directly.
- And a confidence. Here it's: possible, and no more, because a tree imported from a file only states facts. It says nothing about what they rest on, and the document refuses to pretend otherwise.
- Drawn from those relations, here's the family: William and Eleanor, their children George and Margaret, and George's son Arthur. Every line is a claim with its evidence.
- And here's someone the document knows about but can't show: Margaret's child, outside this clipping. The coverage block counts it as unresolved. A gap is never mistaken for nothing there.
- And here's the best example of all: the film you're watching. This video is made of vocabularies. The artwork, every word and rectangle on screen, is a graphic document, written against the very vocabulary we met a few minutes ago.
- The timeline is a choreography: what should happen, to which performer, when, for how long, and with what ease. A table of actions, like the tables from part one. It declares no frame rate at all, because an intent doesn't have one.
- And a tool bakes that intent into an animation: one sample for every tick, twenty-five a second, nothing in between. These are the real numbers behind a title fading in. Three vocabularies, each with its own gate, and the result is the video.
- So that's a vocabulary: a document that says what other documents mean, with tables a gate can check, refusals it lists, and a default view that shows them.
- That's vocabularies, from the inside.
- Next time: how a document is shown. See you there!
Presentation layers
How a document is shown. A presentation layer is a document too: a view says which document it draws, which vocabulary it understands, and what it makes. Three real documents carry their own: a family tree, a building with its gardens, and a code repository. A view can live in a vocabulary, beside the data, or on its own, and the viewer takes the first that draws. The universal viewer reads views and never asks what a document is about, so it can draw a document nobody has written yet; a default view is not a replacement for a specialised application.
Transcript
- Welcome back! In part one, we took a document apart. In part two, we found out what a document means. Now, the question everybody asks next: how do you actually look at one?
- A unimsg document is data: keys and values, tables and bytes.
- But nobody wants to read a family tree as keys and values. They want to see the family. Or walk around the building. Or browse the repository.
- So who decides how a document is shown? In unimsg, a presentation layer does. And here's the twist: a presentation layer is itself a document.
- Let's meet three of the best: a family tree, a building, and a code repository. Then we'll find out how one viewer can draw all of them, and documents that nobody has written yet.
- This is the UniMSG viewer, and this is a document it's been handed: King Charles, his parents, and the two generations below him, with the people they married. The viewer draws it as a family tree. A card for every person, with a portrait where there is one, and a marker for every marriage.
- Point at someone, and they're selected. Open the panel, and everyone in the family is listed, by name.
- Pick a name from the list. Drag the tree, and it follows.
- And one button fits the whole family to the window.
- So where does this drawing come from? Look inside the file. It holds four blocks. The first is the family itself: the people, the events, and the relations between them.
- The second is a view: how to draw that family. The file carries its own instructions for being drawn.
- And the last two are libraries the view draws with: pieces of interface, like panels and lists, and pieces of drawing, like cards and marriage markers. A document can carry everything it needs to be drawn.
- Here's that view, and it's a block in the same file. Like every block, it says what it's written in: the presentation vocabulary.
- Of names the one document it draws: the Windsor family. Over names the vocabulary it understands: genealogy. So every path it writes is checked against what genealogy defines.
- Emits says what it makes. And there are only three choices: an interface, a picture, or a document. This view makes an interface.
- Then come the projections. A projection turns the document into rows for the view to draw. This one is called descendants. It starts at the persons, and looks each name up in the persons.
- And it walks. Follow the parent-of relations, from parent down to child. Every person is visited once, and arrives as a row, with their generation, and their spouse beside them.
- Joins fetch what each row needs from elsewhere in the document: a portrait, a name, the dates they lived. Then the view says how to draw the rows. None of it is code. It's all declared, so it can be read, compared and checked like any other document.
- Next, a building. This is the Royal Exhibition Building in Melbourne, and the gardens around it. The viewer draws it as a three-dimensional model, from the polygons and the heights written in the document.
- Turn it, and it turns. This is the Great Hall, the dome, and the wings either side.
- Choose a storey, and the model is cut at that height. This is the Great Hall floor: the plan of the hall, seen from above.
- And the plans tab draws the same building as flat plans.
- Here's the interesting part. How are plans drawn? The document carries the recipe. This is an entry from a drawing library inside the file. It's called footprints.
- For each row, take every shape the row lists.
- Cut each shape with a slab, from one height to another, and look down along the up axis. What's left is a footprint.
- And draw each footprint as a shape, filled with the row's tint if it has one, and a default paint if it doesn't.
- Read the words: repeat, section, shape. Nothing in this recipe knows what a building is. That's what lets a document carry its own drawings, and any viewer draw them.
- The third is a code repository. This is Lantern, a small link checker, drawn like the front page of a code host: its files, and its latest commit.
- Open a folder, and its contents are listed. The readme is shown underneath.
- The commits tab is the history, newest first, with the merges marked.
- And the refs tab lists the branches and the tags.
- To draw that history, the view starts at the repository's refs: its branches and its tags.
- It follows every commit to its parents, so each commit that can be reached is visited once.
- And it orders them: newest first, by when they were committed. With a tie-break on the commit's own name, so two commits made in the same second always land in the same order, on every machine.
- That's the whole recipe for a history. The view then draws those rows as a table.
- A family, a building, a repository. Three very different subjects.
- But look at what each view said. Which document it draws. Which vocabulary it understands. What it makes. And the rows it draws from.
- The viewer needed to know none of the three. It read each view, and drew what the view said.
- So where does a view live? It can sit in three places.
- Inside the vocabulary itself. It has no of, so it isn't about any one document. It's what a document of this kind looks like when nobody said otherwise: a default.
- Beside the data, in the same document. It draws this one document. That's what the family tree, the building and the repository all did.
- Or as a document of its own, pinned by its hash: shared by many documents, and able to outlive any of them.
- So when the viewer is handed a document, how does it choose? It works down a list, and takes the first view that draws.
- First, a view beside the data, in the document itself. Then the view its vocabulary carries. Then the document as it stands, if it already is an interface, a picture or a document, because then it already is what a view would make. And last, the general view.
- If a view can't be drawn, because a path doesn't resolve, or it needs something the viewer lacks, the viewer moves on to the next. And it says which one it passed over.
- That last step is what makes the viewer universal. Here's a document that names no vocabulary and carries no view: the planets of the solar system. No presentation layer drew this document, so it's drawn by the general view: every value it holds, laid out by its shape.
- Open a section, and it opens. Nothing is guessed about what it means. The viewer just shows what's there.
- There's an obvious way to build a viewer. Write a family-tree screen. Write a building screen. Write a repository screen.
- That works, until the next kind of document turns up. Then somebody has to write another screen, and ship another viewer.
- unimsg turns that around.
- The viewer knows no kinds of document at all. It has three layers. The top layer is the views. They're documents, written by anyone, at any time.
- The middle layer is the reader. It takes a view and a document, and works out a finished drawing, with every coordinate settled.
- And the bottom layer is the surface. It draws what it's given: text, paths, images, and triangles. It decides nothing.
- Only the top layer ever grows. A new way of drawing something is a new document. And the reader is generated once, from a single source, so every platform reads a view the same way.
- What the reader understands is closed, and small. Three things a viewer draws: an interface, a picture, or a document.
- And a closed set of marks: a shape, a path, some text, a rectangle, an image, a group, a repeat, a body, a set of cells.
- Look at how every one is named. By geometry, never by subject. There's no person in this list, no wall, no commit. A mark named for its subject would need a new viewer for every new subject.
- That's why a viewer can draw a view that was written after the viewer was. Everything a view says comes down to marks the viewer already reads.
- And that's the rule that keeps it universal. The viewer never asks what a document is about.
- It asks what the document says it's written in, and which views came with it.
- A document that names no vocabulary isn't guessed at. It's drawn by its shape.
- So here's the proof. A document nobody has drawn before: a table from a school bake sale.
- It does one thing the viewer cares about. It says what it's written in: the spreadsheet vocabulary.
- Then the data: what each stall sold, the price in pounds, and the takings, which is a formula.
- And a total, with the formula that makes it.
- The gate agrees: it conforms to the spreadsheet vocabulary.
- Now we hand it to the viewer. A viewer that has never seen this file, and was told nothing about it.
- It reads one line, finds the vocabulary, finds the view that vocabulary carries, and draws it. There's the table, and the total. And at the top, the viewer says which view drew it: the view in the spreadsheet vocabulary.
- Nothing was installed. Nothing was rebuilt. The document was written, and it could be seen.
- One thing is worth saying clearly. A default view is not a replacement for a real application.
- If you're building a spreadsheet application, you wouldn't use the default view. You'd build a specific, capable editor for the document: formulas you can change, charts, and everything people expect from the tool.
- A presentation layer solves a different problem. It gives every document a way to be seen, even by software that knows nothing about its kind.
- So a messaging app that's passed a spreadsheet can show it, using the vocabulary's default view, without becoming a spreadsheet app. The same goes for a family tree, a building, or a repository.
- The specialised application and the universal viewer read the very same document. One is built for a kind of document. The other is for any.
- So that's presentation layers. A view says what to draw, and how. It can live in the vocabulary, beside the data, or on its own. The viewer works down the list, and says which view drew. Because all it ever reads is views, it can draw a document nobody has written yet. And a specialised application can still do far more with the documents it was built for.
- That's presentation layers, from the inside.
- See you next time!
The episodes, as a document: learning/episodes.umsg raw.