The Simple Text Blind People Cannot Read

I wanted to use Pi. Pi is an agent harness with a TUI, and I could not use it.

That was not surprising, because I have never been able to use any TUI. Every single one I have tried has been difficult or impossible with a screen reader. What was surprising is that this used to make no sense to me. A TUI is text. In a window. That is the whole thing. I am reading text in a window right now. Why is that text readable and the other text is not? This became a real problem for me when Claude Code first came out.

As you know, curiosity killed the cat, and I’m a Jazz cat. So I had to find out.

The Text Is Not Text

Here is what actually happens when an application draws a TUI.

The application does not hand anyone a document. It writes bytes and terminal control sequences through a PTY. Move the cursor here. Write these characters. Move the cursor back. Overwrite that cell. Redraw this region.

The terminal emulator reads those instructions and maintains a mutable screen grid. That grid is the only artifact at the end of the pipe. It is a rectangle of cells, and each cell holds whatever the last instruction put in it.

It’s like a bro on coke at a party having 6 conversations at once.

# ESC [2;1H moves the cursor to row 2, column 1.
printf '\033[2;1HLoading...'

# Move back to the same cell and overwrite what was there.
printf '\033[2;1HDone      '

The progress bar that redrew forty times leaves one final row, and every step that got you there is gone.

So by the time a screen reader arrives and asks the accessibility API what is on screen, the thing it needs, what happened, in what order, and what it meant, was already destroyed. This happens faster than the accessibility API can say Bazinga.

This is why “just add accessibility to the TUI” does not work. You are asking the last layer to reconstruct information the first layer threw away.

How Do You Fix The Terminal?!

All I wanted to try was this agent harness. Instead I had to find a way to make some pixels cooperate. It’s always the darn pixels.

That is annoying but it is also the actual shape of the problem. Pi’s TUI was not the bug. Pi could be perfect and I still could not read it, because the destruction happens between Pi and my ears no matter what Pi does.

So: Ghostty fork .

Ghostty is a terminal emulator, for the non-devs reading this. The terminal state lives in Zig. That state gets projected into a native Swift window for rendering. My fork takes that same Zig terminal state and creates a second projection: plain text, for the accessibility API.

pub fn createAccessibilityContext(
  self: *Surface,
  alloc: Allocator,
) !*AccessibilityContext {
  self.renderer_state.mutex.lock();
  defer self.renderer_state.mutex.unlock();

  const context = try self.io.terminal.screens.active
      .createAccessibilityContext(alloc);
  const accessibility_change =
      self.renderer_state.accessibility_change;
  context.dirty_start_row =
      accessibility_change.dirty_start_row;
  context.dirty_end_row =
      accessibility_change.dirty_end_row;
  context.dirty_count = accessibility_change.dirty_count;
  context.change_generation =
      accessibility_change.generation;
  context.alternate_screen =
      self.io.terminal.screens.active_key == .alternate;
  return context;
}

To make an analogy, if you had to deterministically answer whether a dish has cloves, you could taste the dish and figure it out, or you could just go ask the chef. The Swift window is downstream. It is a picture of the grid. Building accessibility from it means you are one more layer away from the thing that actually knew what was going on, and you inherit every bit of history the grid already ate.

// Both paths are wired to the same terminal state.
.renderer_state = .{
  .mutex = mutex,
  .terminal = &self.io.terminal,
},

// Render projection.
self.renderer.updateFrame(
  self.state,
  self.flags.cursor_blink_visible,
);

// Accessibility projection.
const context = try self.io.terminal.screens.active
  .createAccessibilityContext(alloc);

Time To Eat The Pi.

With an accessible terminal underneath, I could work on Pi itself. Fork here .

The naive move would be to take Pi’s visual elements and read them out. I tried. It was not reliable. Same reason as before: Pi’s visual elements are what Pi decided to draw, not what Pi knows.

But Pi knows a lot. It just knows it in three different places, and they are three different kinds of information:

Transcript session blocks, from the Agent Session. User messages, tool calls, tool results. These are already clean. They exist as structured records because Pi needs them as structured records. Nobody had to reconstruct anything. I just had to stop reading the screen and read these instead.

Lifecycle events, also from the Agent Session. The agent started. The tool is running. The turn ended.

Ephemeral announcements. The things that are only ever a visual state: spinners, transient status, the stuff that exists to be looked at. These are the only ones that have to be converted, because they were never text to begin with.

// Transcript block
{
"version": 1,
"sessionId": "session-1",
"sequence": 12,
"type": "block",
"block": {
  "id": "message:assistant:1720000000",
  "role": "assistant-message",
  "label": "Assistant response",
  "text": "I found the problem."
}
}

// Lifecycle event
{
"version": 1,
"sessionId": "session-1",
"sequence": 13,
"type": "activity",
"state": "busy"
}

// Ephemeral announcement
{
"version": 1,
"sessionId": "session-1",
"sequence": 14,
"type": "announcement",
"id": "tool-start:call-42",
"text": "Running read",
"priority": "polite"
}

func receiveSemanticAccessibilityPayload(_ encodedPayload: Data) {
  guard encodedPayload.count <= 32 * 1024,
        let decodedPayload = Data(base64Encoded: encodedPayload),
        let event = try? JSONDecoder().decode(
          SemanticAccessibilityEvent.self,
          from: decodedPayload),
        event.version == 1 else {
      traceAccessibilityCue("semanticAccessibility invalidPayload")
      return
  }

  if event.type == "reset" {
      semanticAccessibilityState.reset(
          sessionId: event.sessionId,
          sequence: event.sequence)
      semanticAccessibilityLayoutDidChange()
      return
  }

  guard semanticAccessibilityState.active,
        event.sessionId == semanticAccessibilityState.sessionId,
        event.sequence > semanticAccessibilityState.lastSequence else {
      return
  }
  semanticAccessibilityState.lastSequence = event.sequence

  switch event.type {
  case "block":
      guard let block = event.block else { return }
      semanticAccessibilityState.upsert(block)
      semanticAccessibilityLayoutDidChange()

  case "activity":
      let suppressesTerminalOutput = event.state == "busy"
      semanticAccessibilityState.suppressesTerminalOutput =
          suppressesTerminalOutput
      resetAccessibilityFloodForSemanticActivity()
      if !suppressesTerminalOutput {
          _ = refreshAccessibilityProjectionAfterFlood()
      }

  case "announcement":
      guard accessibilityPipelineEnabled,
            let id = event.id,
            let text = event.text,
            semanticAccessibilityState.shouldAnnounce(id) else {
          return
      }
      announceAccessibility(
          text,
          priority: event.priority == "immediate" ? .high : .medium)

  case "close":
      semanticAccessibilityState.clear()
      resetAccessibilityFloodForSemanticActivity()
      semanticAccessibilityLayoutDidChange()

  default:
      traceAccessibilityCue(
          "semanticAccessibility unknownType=\(event.type)")
  }
}

What This Unlocks

Since I’ve been learning programming by taking on these projects, a big part is to always understand what happened. Over here you might think, oh well, this is so accessibility specific. It goes further than that though. The big principle I learnt here is that you should let the part of the program doing the compute be the source of truth whenever possible. This comes up in places far outside accessibility.

Take the music notation app I’m working on. The core is in, drum roll, Rust. The UI and all the macOS and hardware-specific stuff is in Swift. So the split is similar to Ghostty’s, but let’s not let Jared or Andrew Kelley read this blog post. I asked the clanker to make me a renderer in Swift that would display the score to the user. This score looked mostly alright, and let me tell you, mostly alright on a digital score basically means horrible to any working musician. When I opened up the adapter file to see what was going on, my brain did a somersault while I, audibly and alone in my room, went “Haaan?!”

The Swift part wasn’t taking the computed data and just using it. It was doing this crazy translation acrobatics, lines and lines of code, re-deriving what Rust had already worked out cleanly. A downstream layer re-litigating what the layer that actually knew had already settled. After I came down from the shock, a reaction I’ve only recently learnt to have, I took a few deep breaths, drank a seltzer, and converted this to a much thinner file that just takes DTOs and enums and hands them to Swift.

Rust:  The ledger in the borrow checker says it's memory safe.
       So no Arcs required, bro.

Swift: Really? For me? Oh Rusty, you shouldn't have.
       You make my life so easy.

Rust:  What can I say, my memory is just too sharp.

Swift: Don't worry, I'll make a nice native Apple UI and everyone
       will think I'm doing all of this work myself, and no one
       will ever realise you did all the hard stuff while I just
       looked good and had our Apple parents' blessing to run
       havoc on the OS.

Lil Swift proceeds to crayon all over the wall.

That thinner adapter is the whole lesson in one file. It takes what Rust already knows and translates it into the types the screen wants. Whichever side owns the compute owns the truth. Everyone else transports.

At its core, a lot of accessibility is about designing IO. The output is almost always harder than the input, but that’s the name of the game.

Don’t be blinded by your pixels.