Can You Get A Pixel To Talk?

Eshaan Sood,7/16/2026

So many software engineers I know started out because they wanted to make games. One of my favourite games when I was growing up was Roller Coaster Tycoon. There was something magical about bringing that game to life. Now many years later, fully blind, I’ve found myself sitting here and my screen is speaking out things to me like “Bank left.” I press the N key. “Straight track.” I press the G key. “Coaster is closeable. Shortest route in 6 pieces.”

Real OpenRCT2 coaster logic. Fully blind.

So how does this work?

OpenRCT2 is open source and available. So I knew I had the pieces I needed in terms of just the game logic and the source code available to me. The second piece, and this is going to come back, pixels don’t talk. This one line is what you need to remember if you ever do any accessibility work yourself.

I took the Hyper-Twister out of OpenRCT2. Real pieces, real placement rules, real ratings math, lifted out intact, and I built the accessibility architecture around code that was already there and did not care about me.

The other thing you should know before any of this: I don’t know any cpp or Swift. I mean any is not true, I know like a few sentences here and there. Nothing logical. So I could not check a single line of this by reading it.

There are some youtube tutorials where they’ll tell you you can just go to your agent and ask it “Hey I want accessibility in this game.” Thats not how this works. With Roller Coaster Tycoon specifically, the game engine is in C++, the visual projection is in SDL, and my OS is Mac OS, which means the accessibility API I actually want lives over in Objective-C land: AppKit and NSAccessibility.

The reason a lot of people get tripped up with accessibility is because of screen readers. You actually in a lot of ways have nothing to do with the screen when dealing with accessibility. The windows open source screen reader Non-Visual Desktop Access is a better name in the sense its more honest to what it actually does. I bring this up because someones first instinct might be to take the SDL layer and make it communicate with the screen reader.

But my first step was to get SDL out of the way.

Can The Overlay Hold Focus?

The first spike was just macOS windowing.

A transparent Swift NSPanel sits above an SDL-rendered window. The panel can become key without becoming the main window. VoiceOver can navigate native NSTextField elements in the panel. SDL keeps rendering underneath.

That sounds boring. It was not.

It meant the accessible surface could be native AppKit instead of some fake screen-reader layer trapped inside SDL. VoiceOver gets real controls. SDL gets to remain a renderer. Nobody has to ask pixels to explain themselves. Just remember, you don’t ever want the pixels to do the talking.

Who Owns The Keypress?

Nothing forced SDL out of the input path.

The obvious implementation is to let SDL keep handling SDL_KEYDOWN while Swift also handles overlay commands. It works in the demo. You press N, the selected piece changes, everyone claps. Corporate is asking you to ship this so they can make their big DMA claim faster than you can say Blind Bilbo Baggins Bet Burning Butter Because Bilbo Became Better.

Then later the same keypress gets applied twice, or key repeat behaves differently, or a modifier leaks through, or the overlay is focused but SDL still accepts gameplay input. Congratulations, you now have a haunted keyboard. And every screen reader user on the planet is collectively groaning about how companies are evil because they don’t test accessibility.

SDL still gets polled for window and quit events. It still renders. It still owns pixels and audio. But gameplay input events are disabled with SDL_EventState(..., SDL_IGNORE). Swift owns the keypress. Swift submits semantic commands.

Can C++ Talk Back?

So yes you can make C++ and Swift talk. We all know that. Thats fine. But can it emit stuff well enough for the Apple accessibility API to work reliably with a tiny bridge? My friend, no time like the present to find out.

Methodology correction 2

Second trap. The agent wanted to keep the bridge thin: take the keypress, pass it through, let Swift figure out what it meant. I liked thin. Thin is right. But it wanted thin in the wrong place: a bridge that moves keys instead of meaning, and then somebody downstream has to reconstruct what the user actually did. That is how you get a Swift layer inventing its own story.

So I started at the far end and walked it back. What does the user need to hear? What does Swift need to say it? What does Swift have to hand C++ to get that back? Something like:

<user>   key press N
<swift>  lil guy wants to place the next track
<cpp>    placed it. here's a snapshot of what just happened.
       deal with this for the user please, Swift
<swift>  yeah yeah, I know, I'm the superior accessibility language.
       yo user, here's what C++ did to the game, in plain english
<user>   look Ma, I'm making a coaster

I handed it this design, minus the part where Swift is smug about being the superior accessibility language. It came back as code. And here is the strange part: I could see the design in the syntax even though I cannot read the syntax. I knew what shape I had asked for, so I could tell whether the shape had survived.

The bridge stayed thin. The commands got meaningful. Those are different axes and the agent collapsed them. That is why the API is semantic state and not keydown events. That is why there is a suggestion field in it at all.


extern "C" int BuilderSubmitCommand(int command);

Swift sees that through @_silgen_name:


@_silgen_name("BuilderSubmitCommand")
func BuilderSubmitCommand(_ command: Int32) -> Int32

So when I press N, Swift maps that to a BuilderCommand.nextPiece, passes the integer across the C boundary, and then gets out of the way.


private func submit(_ command: BuilderCommand) {
    commandInFlight = true
    stateChangedDuringCommand = false
    _ = BuilderSubmitCommand(command.rawValue)
    commandInFlight = false
 
    if stateChangedDuringCommand {
        announceLatestState()
    } else {
        announce(command.fallbackAnnouncement)
    }
}

On the C++ side, that integer becomes a real builder command:


extern "C" int BuilderSubmitCommand(int command)
{
    if (command < static_cast<int>(Command::nextPiece) ||
        command > static_cast<int>(Command::toggleSound))
    {
        return 0;
    }
 
    submitCommand(static_cast<Command>(command));
    return 1;
}

That submitCommand function is where the actual game-ish work happens. It does not tell SDL “a key was pressed.” It changes the model.

nextPiece updates gState.selectedPiece. place builds a preview segment, checks whether the endpoint is legal, pushes it into gState.segments, moves the cursor, recalculates cost and ratings, and updates the completion state. undo removes the last segment and rebuilds the occupied endpoint set. Rotation changes gState.direction. Bank, slope and lift change the parameters for the next segment.

After each meaningful change, C++ calls emitState().


void emitState()
{
    if (gHeadlessValidation) {
        return;
    }
 
    const auto json = snapshotJson();
    SwiftBuilderStateChanged(json.c_str());
}

snapshotJson() is the important part. It turns the current C++ source of truth into a semantic accessibility payload:


{
  "coaster": "Hyper-Twister",
  "piece": "Right banked turn",
  "direction": "south",
  "bank": "right bank",
  "slope": "level",
  "lift": "off",
  "segments": 5,
  "cost": 136,
  "cursor": "x 6, y -1, height 1",
  "completionStatus": "Closeable",
  "completionSuggestion": "place straight track down",
  "message": "Placed Right banked turn. Now facing south."
}

Then Swift receives that JSON through an exported callback:


@_cdecl("SwiftBuilderStateChanged")
public func SwiftBuilderStateChanged(_ json: UnsafePointer<CChar>?) {
    guard let json else { return }
    BuilderOverlayController.shared.receiveState(json: String(cString: json))
}

Swift parses it, updates native AppKit fields, and lets VoiceOver inspect those fields:


func receiveState(json: String) {
    guard let data = json.data(using: .utf8),
          let object = try? JSONSerialization.jsonObject(with: data),
          let dict = object as? [String: Any]
    else {
        return
    }
 
    latestState = dict
    updateFields(dict)
 
    if commandInFlight {
        stateChangedDuringCommand = true
    } else {
        announceLatestState()
    }
}

That commandInFlight flag is small but important. It tells Swift whether the state update happened as a direct response to a command it just submitted. If it did, Swift speaks the actual C++ message. If not, it can treat it like a standalone state update. That avoids Swift inventing its own story about what happened.

The key thing is that Swift does not calculate whether the circuit is complete. Swift does not calculate whether the next piece can be placed. Swift does not maintain its own track list. Swift only projects the C++ state into controls and speech.

That is the architecture in miniature.

The key move the astute reader might have noticed by now is that you keep everyones job isolated. We don’t let too many cooks in the kitchen. SDL: visual rendering. C++: game logic and game state. Swift: accessibility. One source of truth for each event.

This not only keeps things clean. It lets you find out where the problems are and also lets you take the data from the last spot it is cleanly available.

But Do We Get To Build A Coaster?

You know by this time I’ve drank enough coffee while spending my token budget to understand what 1 software engineering week truly means. “I’m just going to add accessibility to Open RCT 2” - thats where time goes to die. I told myself I’ll do this for 2 hours. Which by the time I had spiked the architecture above, it had already been 2 hours. So you know, I had to think in software engineering hours, which if you somehow don’t know follow their own cursed exchange rate. When a software engineer tells you it will take an hour, it will probably take 7.

That aside, once the plumbing was designed and tested and working, this next part was easy for the agent. It saw the shape of the API, it saw the extracted Hyper-Twister code, it saw the Apple Accessibility API and the rule set I gave it, and in under 10 minutes it had a buildable accessible test version.

Beyond this was all testing. Questions like Why is there a lag? Why does the game have no sound? But most importantly, how on earth am I as a blind person supposed to know how crazy my coaster truly is? How will I ever complete this coaster?

And this is where I had to truly design my way out of the problem, this couldn’t be engineered out of. How do you communicate to a blind person that the thing on screen is a certain path and needs a sequence to happen to complete?

This is where I built Guidance.

How The Guidance Works

The guidance is not coming from the screen. It is coming from the builder model.

At any point, C++ knows three important things:


gState.cursor      // where the open end of the track is
gState.direction   // which way the open end is facing
gState.occupied    // which grid cells already contain track

So the question becomes very concrete: from this cursor, facing this direction, using legal track pieces, can I get back to the station?

The spike models possible next pieces as small guidance steps:


struct GuidanceStep
{
    size_t pieceIndex{};
    int slope{};
    int bank{};
    bool lift{};
};

Then it generates candidate pieces based on the current state. If the track is too high, try drops. If there is room, try straights, turns, brakes, maybe lift hills.


std::vector<GuidanceStep> completionCandidates(const Point3& cursor)
{
    std::vector<GuidanceStep> candidates;
 
    if (cursor.z > 0) {
        candidates.push_back({ 5, 0, 0, false });  // Steep drop
        candidates.push_back({ 1, -1, 0, false }); // Straight track down
    }
 
    candidates.push_back({ 1, 0, 0, false });   // Straight
    candidates.push_back({ 2, 0, -1, false });  // Left turn
    candidates.push_back({ 3, 0, 1, false });   // Right turn
    candidates.push_back({ 10, 0, 0, false });  // Block brakes
 
    if (cursor.z < 7) {
        candidates.push_back({ 1, 1, 0, false }); // Straight up
        candidates.push_back({ 4, 0, 0, true });  // Chain lift hill
    }
 
    return candidates;
}

Each candidate is turned into a possible segment from the current cursor and direction:


auto segment = makeSegmentFrom(
    kPieces[step.pieceIndex],
    node.cursor,
    node.direction,
    step.slope,
    step.bank,
    step.lift
);

Then the spike asks whether that segment is legal:


if (!canPlaceWithOccupied(segment, node.occupied, true)) {
    continue;
}

That checks bounds, collisions, and whether returning to the station is allowed:


bool canPlaceWithOccupied(
    const Segment& segment,
    const std::set<std::string>& occupied,
    bool allowStationClosure
) {
    if (!isSegmentInBounds(segment)) {
        return false;
    }
 
    if (occupied.find(pointKey(segment.end)) == occupied.end()) {
        return true;
    }
 
    return allowStationClosure &&
        !isStationPoint(segment.start) &&
        isCompleteStationConnection(segment.end, segment.endDirection);
}

The actual search is a small breadth-first search. It tries legal next pieces, tracks visited cursor/direction states, and stops when one route gets back to the station correctly.


RouteSearchResult findLegalCompletionRoute()
{
    constexpr int kMaxSearchDepth = 10;
 
    std::queue<SearchNode> queue;
    queue.push({ gState.cursor, gState.direction, {}, gState.occupied });
 
    std::set<std::string> visited;
    visited.insert(searchStateKey(gState.cursor, gState.direction));
 
    while (!queue.empty()) {
        auto node = queue.front();
        queue.pop();
 
        if (node.route.size() >= kMaxSearchDepth) {
            continue;
        }
 
        for (const auto& step : completionCandidates(node.cursor)) {
            auto segment = makeSegmentFrom(
                kPieces[step.pieceIndex],
                node.cursor,
                node.direction,
                step.slope,
                step.bank,
                step.lift
            );
 
            if (!canPlaceWithOccupied(segment, node.occupied, true)) {
                continue;
            }
 
            auto nextRoute = node.route;
            nextRoute.push_back(step);
 
            if (isCompleteStationConnection(segment.end, segment.endDirection)) {
                return { true, std::move(nextRoute) };
            }
 
            auto nextOccupied = node.occupied;
            nextOccupied.insert(pointKey(segment.end));
 
            auto key = searchStateKey(segment.end, segment.endDirection);
            if (visited.insert(key).second) {
                queue.push({
                    segment.end,
                    segment.endDirection,
                    std::move(nextRoute),
                    std::move(nextOccupied)
                });
            }
        }
    }
 
    return {};
}

Once a route is found, the accessibility layer does not just get “track exists.” It gets useful semantic guidance:


info.status = "Closeable";
info.suggestedNext = firstStepInstruction(legalRoute.route.front());
info.estimatedPieces = static_cast<int>(legalRoute.route.size());
info.detailedGuidance =
    "Closeable. End is " + info.distanceSummary + ", " +
    info.heightSummary + ", " +
    info.headingSummary + ". Legal route in " +
    std::to_string(info.estimatedPieces) + " pieces: " +
    routePhrase(legalRoute.route) + ".";

That is how VoiceOver can say something like:

Closeable. End is 2 east, 1 north from station. Height matches station. Facing west. Legal route in 3 pieces.

The important part is that this is not a caption for the picture. It is a query against the model.

That is the part I did not expect. RCT3 had an autocomplete feature and you could see it in real time trying permutation combinations of what would work, so in a way this logic was always sitting somewhere inside the shape of the game. But nobody had exposed it as a query. Once it exists as a query, even sighted users could want it. You could say, keep excitement above 90 percent as a constraint and make sure the path closes.

The accessible layer is not a lesser version of the visual layer. In this one way, it is better.

How To Avoid Being A Test Monkey For Your Agent: Write A Validation Harness

This is the part I care about more now than I did when I started. And I didn’t want to sit here every 5 minutes testing a broken version and sending it back with a reason why it doesn’t work.

I cannot read the screen. The agent can write the code and tell me it works. The overlay can say “Complete circuit.” Those are two narrators with the same incentive, and they can be wrong in the same direction.

A pretty accessible field saying “Complete circuit” while the actual track graph is broken is exactly the failure mode.

Verifier

So the spike has a headless validator.

No UI. No speech. No rendering. No agent summarising what it thinks happened.

Just commands into the model.

There is a known recipe that should build a closed circuit. The validator runs it and checks the actual final state:

VALIDATE_RECIPE_RESULT closed=true status="Complete circuit" segments=9
VALIDATE_RECIPE_PASSED

Nine segments. closed=true. Or it did not happen.

That is not a testing anecdote. That is the epistemology.

The completion sound only matters because the validator and the model agree first. Then the feedback means something. The moment the structure and the feedback agreed, the demo stopped being theatre. The track was closed. The model knew it. The validator proved it. Then VoiceOver and audio were allowed to celebrate.

This is the same shape as every other serious thing I am learning to trust: prove the invariant away from the interface. Then let the interface report it.

What This Proves

This does not prove OpenRCT2 is accessible now. It does not touch save files, scenery, scenario play, multiplayer, camera control or the thousand other systems inside an actual game. The real open question is whether this survives being put back into the full game, with every other system arguing over the same keypress.

What it proves is narrower:

A native macOS accessibility surface can sit over an SDL renderer.
Swift can own keyboard and VoiceOver interaction while C++ owns the builder state.
SDL can keep rendering without being the gameplay input path.
The accessibility layer can project from the same source of truth that drives the visual scene.
A headless validator can prove the model before the interface gets to brag.
Once accessibility is model-backed, it can expose information the visual layer usually leaves implicit.

Remember kids, pixels CANNOT talk.

Blinding Pixels · GitHub