In Defense of the Visual Cortex

Some time ago, in hopes to augment my programming penmanship, I set out on a solo endeavor to develop a game engine in C++ (source here) complete with a custom renderer, custom asset system, a scripting system and a custom editor. After two years of work and 16,000 lines of code, the weight became difficult to manage; adding a new asset type, for instance, would have involved opening several different files and touching across serialization, script engine registration, the asset manager, and others. In development, one must always strike a fine balance between over-abstracting (premature optimization is the root of all evil, or so I'm told...) and "just getting the features out"; in the face of this dilemma, I became increasingly displeased at having to take days or even weeks to properly architect new features or refactor existing code.

I therefore began looking into architecture planning and diagramming canvases, searching for any tool that could give me some insight into the structure of my codebase and provide visual and mechanical functions to plan changes and modify code. My search having come up empty, I began to theorize what this editor could look like. I shall henceforth refer to this purely imaginary and theoretically perfect apparatus as The Tool. While prototyping and researching, I came across an article that truly baffled me in the almost prophetic manner it spoke to me.

The Shoemaker Has No Shoes

In 1999, Roedy Green wrote the famous essay titled "How to Write Unmaintainable Code", a satirical piece about ensuring job security by purposefully writing bad code that would constantly be out of shape (and therefore always in need of a maintainer). The article's closing, however, enunciates a dilemma that has remained unresolved for decades: while software dev is a logical task in obvious need of structured visualization, programming is still done by reading source files as text.

Green states that maintenance programmers (if anyone cared to ask them) would demand three things of their software tools:

• ways to hide housekeeping detail so they can tell the forest from the trees
• shortcuts so they type less and see more of the program at once on screen
• relief from the myriad of petty time-wasting tasks programs demand

Green then goes on to list solutions that become possible only by representing code as structured data. Here are a few:

• view the same source many alternate ways (decision table, flow chart, loop-structure skeleton with detail stripped)
• ask structural queries: what methods produce an object of type X? what accepts X as a parameter? what variables are accessible at this point?
• transparent overlay sets of changes you can apply or remove
• do quite a bit of code through point-and-click

The satirical tone of the article as a whole, paired with this ending, makes the essay's conclusion abundantly clear:
since no one is willing to build this, we will continue to have unmaintainable software, and you will get to keep your job ;)

When I came across Green's essay, I was astounded that these imaginings that had been keeping me up at night for months had been written about over 25 years ago, and I was greatly angered that The Tool the essay described had still yet to exist!

The Cognitive Model

Peter Naur, in his 1985 essay Programming as Theory Building, argued that software is not the lines of code on disk but rather the theory which lives in the programmers' minds; he further stipulates that, if, say, the team members possessing that arcane knowledge were to leave, the software is as good as dead and should be reconstructed from a new theory. This is to say, conserving the mental model is crucial; it is the software.

I can hear many of you experienced developers now clammer in unison, explaining that though this conclusion may work well in theory, it often breaks down in practice, especially on larger codebases, and I'd be forced to agree with you. Very often, we are tasked to work on code which behaves quite differently from what we may intuit, code whose "theory-formulating" process we may not been involved in: an inherited codebase, an external dependency, a personal project from 3 weeks ago... There are things we can now, but what often bites us are the things we think we know (you don't know what it is you don't actually know).

The gap between the codebase's behavior and our understanding of it is cognitive debt, and while Naur's framing may seem quite idealistic, I'd argue it's simply a problem statement in need of a solution: How can we reduce cognitive debt and ensure we are working with a more complete theory of the code, as opposed to a partial/incorrect one? How can we become aware of what it is we didn't know we didnt't know? Let's think about how we understand code to begin with.

A well-written function can present a seeming clean view, but looks are often deceiving: it is between the lines that logic lies, and bugs hide; one cannot rely on naming alone. To truly understand a function's behavior, one must consult other functions and recurse down far enough, building out the map mentally as the web grows.

AssetHandle AssetManager::Load(const std::string& path) {
    auto id = NormalizeAssetID(path);        // ← and now you need to hold this too
    if (m_Cache.contains(id))
        return m_Cache.at(id);
    auto asset = m_Loader->LoadFromDisk(id); // ← and this
    m_Cache.emplace(id, asset);
    return RegisterHandle(asset);            // ← and this
}

The snippet above is problematic in that the small local view is in sharp contrast with the larger global context needed to understand it. Despite being the typical example of a "good" function (concise, single-responsibility, well-named variables and subroutines), we cannot confidently claim to know what's under the hood just by looking at it. We have no real insight into how interactions with the other functions and global resources could affect this function's behavior.
Too many questions remained unanswered, say, in the case of an non-existant file path:

• What would NormalizeAssetID return?
• Does RegisterHandle properly check the asset exists?
• Would any of these functions throw an exception, and if yes, where and how is it caught?
• Would the program crash, silently fail, or gracefully present the error somehow?

We won't have answers without reading more code, context switching, and tracing the logic flow, all in our heads.

If all the relevant code (and how it connects) could be visible at once, understanding even vast amounts of code across the entire project becomes a trivial task. In a traditional text editor, the solution would be to switch files or open up multiple files side-by-side, but really, how much text can be fit on screen at once?

Cognitive debt differs greatly from technical debt, which is tangible and visible; you can point at an ugly function and schedule the refactor, whereas you can't easily point at a mental gap (again, you don't know what you don't know). In this example, the code isn't even sloppy (the ideal scenario!) and yet the understanding remains elusive.

Given that code can be modified only when we realize the mental model we wish to project into the world is somewhat different from the codebase's current behavior, we'd do ourselves a favor of not having to archeologically reconstruct said behavior every time it needs to change.

A Not-So-Novel Innovation

A 16K-line codebase is the narrative equivalent, in size and complexity, of an intricately connected and expertly plotted 350-page detective novel (At 100K LOC, we have a book series longer than The Lord of the Rings). Software's not static, so this novel gets changes in random places every so often, sometimes in ways that break the plot. Happy reading!

Realistically, no one holds an entire novel in their head, especially one which shifts so often. It's clear to see we are in dire need of some intelligent solutions to manage these complications.

Patterns and insights only emerge once the data is visualized. The Datasaurus dozen research paper is a rather well-known example of this principle being applied specifically to the field of data analysis. The paper, with its central theme of "Always Visualize Your Data", illustrates that identical summary statistics can result from wildly different source data; the true story can only be revealed once the source data is visualized.

We programmers have no care, it seems, for summary statistics at the minimum, and go straight to reading the raw data, when really the same visualization principle should apply here too.

Here's the code snippet from before, visualized as a graph:

It now becomes abundantly obvious what occurs in the error path. We can see where the error is thrown, where it is handled, and how the global state of the app changes.

A note on visual programming languages and low-code platforms, and clarifying how they are different from what is being proposed here. Many such tools exist, and all suffer from a similar predicament: since the source of the program logic itself is now the diagram, all our grievances with text-based programming now get shifted into our visual processing neural pathways: lines everywhere, symbols that are hard to keep track of, and colors that mean all sorts of nothings.

The Tool should derive the visualizations from the source text, not become the source. Just like a statistician has a great number of different graphing aids at their disposal (bar graph, histogram, pie chart, etc.) to immutably peer into the same underlying data, programmers ought to have multiple lenses with which to view their code. It's not about representing everything everywhere all at once, but picking optics and motions with context-specific goals.

Here we see multiple such "lenses" over the same underlying code:

Here, we leverage our new spatial dimension to debug much more interactively:

And here, we find the fractal nature of code:

These aren't hairballs; they convey a context-specific meaning by their format and motion.

Happy Are Those Who See

Green concluded his essay by stating that only by experimenting with these ideas could we eventually get a sense for what was useful and valuable. I think it's time we developers started experimenting. Keeping a grasp on our codebases and truly understanding their behavior is now more important than ever. We humans have incredible dedicated hardware for comprehending shapes and space, and software dev tools should utilize these channels and transform the unstructured text into something we can more naturally parse.

Any editor claiming to operate effectively on software must attempt to present it and manipulate it as what it truly is: a shifting novel, a breathing structure, a fractal graph.