visualizing the system

Software is invisible and unvisualizable.

Fred Brooks, “No Silver Bullet” 1

1 motivation

Let’s take the liberty of quoting Mr. Brooks at some length:

Software is invisible and unvisualizable. Geometric abstractions are powerful tools. The floor plan of a building helps both architect and client evaluate spaces, traffic flows, and views. Contradictions become obvious, omissions can be caught. Scale drawings of mechanical parts and stick-figure models of molecules, although abstractions, serve the same purpose. A geometric reality is captured in a geometric abstraction.

The reality of software is not inherently embedded in space. Hence it has no ready geometric representation in the way that land has maps, silicon chips have diagrams, computers have connectivity schematics. As soon as we attempt to diagram software structure, we find it to constitute not one, but several, general directed graphs, superimposed one upon another. The several graphs may represent the flow of control, the flow of data, patterns of dependency, time sequence, name-space relationships. These are usually not even planar, much less hierarchical. Indeed, one of the ways of establishing conceptual control over such structure is to enforce link cutting until one or more of the graphs becomes hierarchical.

In spite of progress in restricting and simplifying the structures of software, they remain inherently unvisualizable, thus depriving the mind of some of its most powerful conceptual tools. This lack not only impedes the process of design within one mind, it severely hinders communication among minds.

I’m not sure what to make of this. It sounds discouraging. It sounds like Brooks is saying that our efforts to make the program more tractable by making it more visible are doomed.

Well, one of those “directed graphs” will be better than nothing. How about something like this?

program-graph.svg
Figure 1: A graph of the documents

So how do we do that? In the rest of this section, we’ll create that graph from the program itself.

2 the graph

As Brooks says, a software system constitutes “not one, but several, general directed graphs.” Willshake is no exception. Interesting graphs would include the subsystem structure, the global build dependencies, the file structure of the product, the DOM structure of the web site, and the logical structure of the web site, just to name the obvious ones. Those also happen to be formal projections of the system—ones that are expressed by the program, and can be derived by a program, with no manual effort.

But for the primary view of the system, the map that I first show to visitors, those formal graphs won’t do. For one thing, they tend to be too complicated, and too focused on one aspect. Computer systems—formal systems—tend to homogenize things, with no idea of the relative importance of the parts. In many cases, the things of importance to people aren’t formally represented in the system at all. I have tried to combat that by making topical documents the primary unit of programming. But I’m talking here about the system level.

So while all of those graphs might be useful and interesting, they aren’t necessarily conceptual maps, since they’re all ruled in some way by technical considerations. I want at least one map of the system that is governed only by a wish to make the system as clear as possible for both myself and readers.

What I really want is a map of the system as I see it—an aspirational view of the system, that allows me to express how I prefer to think of the system, whether or not that matches reality at the time. Being able to see such changes before actually making them real is very useful for getting a reaction to them. If it becomes part of how I think about the system, then it’s succeeded. (Of course, something can succeed too well, and then I have to destroy it.)

Such a graph is necessarily informal, curated, hand-crafted. To that end, each document includes a few lines indicating where and how (and whether) it fits into this graph. This informal graph merely represents the author’s opinion, which definitely doesn’t represent any objective property of the system at all (since, if it did, I would use that instead). Also, of course, that metadata has to be maintained manually.

So although I may eventually create some of those formal graphs—which can be generated automatically, they will really be subordinate to this one.

2.1 objective

What are the requirements?

  • it has to be pretty small. Not necessarily small enough to fit on one screen, but possible to constrain in at least one dimension (width or height).
  • it has to be self-explanatory. It has to function as a map: an orientation for newcomers to an unfamiliar place. It can’t be something that itself requires explanation.
  • it has to be acyclic. Circular references would not only be confusing here, they would make no sense.
  • it should build up cumulatively. In other words, it should be directed.
  • further to that point, it has to be composed of meaningful units.

Documents are the unit that we’re working with. Any smaller than that would be too detailed for a complete overview. If this proves at any point to be inaedequate, we should change the documents until the map is satisfactory.

Nodes are the “objects” in a graph—in this case, the documents.

Note that, although we’re talking about visualizing “software,” the documents in question are not all part of the program per se. Some documents are merely philosophical. We include them in this set because they all inform the decisions made in creating the program.

2.2 relationships among documents

Graphs are all about relationships. A graph is only better than a list if it shows meaningful relationships among the documents.

In the curated graph, a document can “build on” another document. That’s the only kind of direct relationship between documents that I identify.

I also identify at most one “subgraph” that the document belongs to. There are only top-level subgraphs, no nested ones.

2.3 creating the graph

Let’s give each document a chance to identify the things that it builds on. Suppose that we had metadata for each document describing what it builds on. Then we could say

<xsl:template match="documents">
	strict digraph {
	<<graph attributes>>
	<xsl:variable name="q">"</xsl:variable>
	<xsl:for-each select="document[title][not(invisible)]">
		<!-- We use the key as a short title, instead of the "actual" title.  Poor
		     man's word wrap: treating each word as a line, which gives "squarer"
		     (less horizontally-biased) shapes. -->
		"<xsl:value-of select="@key" />" [
		label = "<xsl:value-of select="translate(@key, '_', '&#10;')" />"
		href="/about/<xsl:value-of select="@key" />"
		<xsl:if test="status = 'shelved'">fillcolor = dimgray</xsl:if>
		]
		<xsl:for-each select="builds-on">
			<!-- Prevent LaTeX crash from underscore in case the title is missing.
			     Temp workaround Inkscape bug. -->
			"<xsl:value-of select="." />" [label = "<xsl:value-of select="translate(., '_', '&#10;')" />"]
			<xsl:value-of select="concat($q, ../@key, $q, ' -> ', $q, ., $q)" /> [
			<<edge attributes>>
			]
		</xsl:for-each>
	</xsl:for-each>

	<<subgraphs>>
}
</xsl:template>
<xsl:output method="text" />

To run that against such a document index (which we’ll create below), we just

: $(PROGRAM)/graph/to_dot.xsl \
  | $(PUBLISHED)/<documents.xml> \
|> ^o make program graph^ \
   xsltproc %f %<documents.xml> > %o \
|> $(PUBLISHED)/dot/program-graph.dot

3 subgraphs

Indeed, this needs to be revisited to connect with build subsystems.

Subgraphs, I’m finding, are more and more useful.

A subgraph would simply cluster all of the documents that say they belong to it.

<xsl:key name="subgraph" match="document" use="string(subgraph)" />
<xsl:template name="subgraph">
	<xsl:param name="subgraph" select="subgraph" />
	<xsl:param name="label"  select="translate($subgraph, '_', ' ')" />

	subgraph cluster_<xsl:value-of select="$subgraph" /> {
		id = "<xsl:value-of select="$subgraph" />_subgraph"
		label = "<xsl:value-of select="$label" />"
		style = "rounded"
		fontcolor = white
		fontsize = 48
		fillcolor = black
		<xsl:for-each select="key('subgraph', $subgraph)[not(invisible)]">
			<xsl:value-of select="concat('&quot;', @key, '&quot; ')" />
		</xsl:for-each>
	}
</xsl:template>
<xsl:for-each select="*[subgraph][not(invisible)]
											[not(subgraph = preceding-sibling::*/subgraph)]">
	<xsl:call-template name="subgraph" />
</xsl:for-each>

If this is a link between two subgraphs, attach to the subgraph edges.

<xsl:variable name="remote" select="../../document[@key = current()]/subgraph" />
<xsl:if test="../subgraph and $remote">
	<xsl:if test="../subgraph != $remote">
		arrowtail = dot
		penwidth = 15
	</xsl:if>
	ltail = cluster_<xsl:value-of select="../subgraph" />
	lhead = cluster_<xsl:value-of select="$remote" />
</xsl:if>

This is necessary for the lhead and ltail attributes above to work.

4 top-down or bottom-up?

Which way is up? Good question. There are advantages to either way. If you think the edges as meaning “depends on,” then top down (like tree roots) is what you see. If you think of “builds on,” then it’s bottom up (like tree branches). By default, dot will give you a top-down graph. This (along with using arrowtail instead of arrowhead above), flips the graph so that it’s top-down.

Somewhat counterintuitively, you need to se the label location to “bottom” if you still want the cluster labels on top in the top-down graph.

edge [
	dir = "back"
	arrowtail = inv
]

5 refining the graph

Graphviz is known for the beautiful, legible graphs that it produces. Even with the defaults, you generally get good results. But we want the graph to feel like part of our structure, so we take an interest in its looks.

We can control the appearance of the graph using the many attributes recognized by Graphviz.2.

This sets the id attribute of the SVG node. This is a very generic setting… we could almost set it in the command line that generates dot diagrams. I’m not sure why it’s not a default in dot itself.

At the moment, I’m not using it at all, but it would allow me to target specific nodes with scripts or styles.

I was trying this out for setting a serif font on the node text, but it wasn’t working consistently.

fontname = "Arial"
fontnames = svg

By the same token, the nodes are material. But in Graphviz, nodes are not filled by default.

node [
	arrowhead = inv
	style = "filled"  
	shape = circle
	/* The default is 0.11,0.055 */
	margin = "0.2, 0.055"
]

We use circular nodes for several reasons. For one thing, it gives them a pleasant uniformity. As it is, Graphviz sizes the nodes based on their text content, so they already vary in size arbitrarily based on the length of their titles. But this way, at least they don’t also vary in aspect ratio. Rectangular or elliptical nodes are indeed more fitted to their text, and that efficiency makes them harder to touch—yet without making the whole graph any smaller. These moon-like discs are also echoes of the site logo, which is appropriate for this role of reflecting on the site itself.

We also double the default side margins, which gives a little more breathing room between the label and the bounding shape.

Edges are the connections between graph nodes. They’re represented in the picture by lines (or curves, when necessary), and since this is a directed graph, they include arrowheads. Now, as we said above, this is about parts that build on other parts. So Graphviz will draw an arrow from the new thing to the older thing. This is a bit counterintuitive, although it is the convention for “dependency” graphs. As long as there’s no text explaining what the lines mean, an inverse arrow is a little less suggestive, while still indicating a direction in the relationship. 3

edge [
	color = "#777777DD"
	penwidth = 10
]

We use gray edges so that they’ll be visible whether the setting is dark or light. The default pen width is 1, meaning one pixel. When you’re drawing angled lines or curves on a screen, a one one-pixel-wide stroke will look weak and (on most screens) badly aliased. Anything larger helps mitigate this effect.

5.1 further refinements for web

Since the graph is created as an SVG graphic, and since SVG graphics recognize CSS, we could also use stylesheets to control how it looks. But remember that our documents are supposed to remain portable to different environments, where SVG may not be available. So the Graphviz attributes that we have been using are the preferred way to manipulate the output, since they will take effect regardless of the output type.

Still, we can use a stylesheet for special features in environments where it is supported.

stylesheet = "/static/style/document-graph-rules.css"

So now we can drop into CSS rules (keeping in mind that the SVG style properties are somewhat different than those used on HTML elements).

service make_stylesheets \
: $(CSS)/document-graph-rules.css \
|> !copy_to |> $(SITE_STYLE)/%b

You might wonder why we can’t just add the necessary rules to our “main” stylesheets—the ones that we use for the web site itself. This is because an SVG document is a completely new context—even when it’s “embedded” as an object. The SVG graphic doesn’t know anything about the styles defined in the containing document, including any web fonts that we’ve loaded. So if we want to use those fonts in the graph (and we do), we have to load them into the SVG document itself:

@import "/static/style/font-face-rules.css"

With that, we can have the nodes use the same fonts as the site logo.

Regarding these rules (which are also applied in the HTML document), see signs.

Seems the above doesn’t work if you use a CSS @import, but does work if included inline.

@import colors
.cluster polygon
	fill rgba($aboutColor, .8)

@import signs
.cluster > text                                 // Should target only the cluster label
	fill $signTextColor
	font-weight 200
	letter-spacing -.04em

Most of the document names (such as “the collection” and “the space”) are such that the emphasis may be placed on the last word. Meanwhile (I think because I put line breaks between the words), Graphviz is putting each word into a separate text element. As a result, we can use :last-child to target the last word.

:not(.cluster) > text:last-child
	font-size 125%
	font-weight 700

The word is resized in-place, without any adjustments to the line spacing. So in some cases there’s a slight overlap with the previous words. This creates a nice effect which is also reminiscent of the site logo.

.node ellipse
	transition fill .1s

	// The border is good for print, but doesn't look good on screens.
	// 
	// UPDATE: all stylesheets are currently targeting screen, so if you think
	// this one has some use for print, you may need to link it separately.
	@media screen
		stroke none

The .current selector is provisional. Its intention is to mark the node when a document is opened. At the moment, I don’t have a way to do this using getflow, since change rules cannot target elements in an embedded document.

Also, we use user_pointing_at because it’s the preferred way to set hover rules, but I’m pretty sure that the special javascript handling of touch events will not include SVG elements.

The nodes are links, so they should change a little when they’re pointed at.

@import colors
highlighted_node()
	& ellipse
		fill $contextColor

@import pointing
.node
	+user_pointing_at()
		highlighted_node()
	&.current
		highlighted_node()

I wouldn’t be opposed to a bit of drop-shadow on these, but alas, it’s not as simple as just adding a box-shadow rule, since box-shadow is not supported on SVG elements. It can very much be done with SVG, but it would require direct injection into the SVG exported by dot.4

.cluster
	polygon, path
		fill rgba($aboutColor, .6)

#Shakespeare_subgraph
	polygon, path
		fill rgba($worksColor, .6)

#the_catalog_subgraph
#the_collection_subgraph
#media_subsystem_subgraph
#the_world_subgraph
	polygon, path
		fill rgba($resourcesColor, .6)

6 give it a home

What’s the use of visualizing the system if it’s not put on display? You’ve got something there, man! Don’t hide it under a bushel!

Everything before this point can be seen as a simple input-output between the document metadata and the graph (or graphs) that it produces.

Everything that follows could be split into a separate document, if I could think of what to call it.

The “about” region is reserved for anything about willshake itself, so let’s put it there:

Couldn’t we put the img fallback for SVG object’s into a transform for change rules? As we do for exports? Though in this case a markup fallback might be better, anyway.

Also note that SVG is currently being refused to IE.

#about-main-sheet <after>
<section id="more-about" class="about-sheet">
	<div class="program-graph-description region-message">
		<p>Willshake is an experiment in <a
		href="/about/literate_programming">literate programming</a>—not because it’s
		about literature, but because the program is written for a human audience.</p>
		<p>Following is a <a href="/about/visualizing_the_system">visualization of
		the system</a>.  Each circle represents a document that is responsible for
		some part of the system.  You can open the documents by touching the
		circles.</p>
		<p>Starting with the <a href="/about/the_project">project philosophy</a> as
		a foundation, the layers are built up (or down, as it were): the programming
		system, the platform, the framework, the features, and so on.  Everything
		that you see in the site is put there by these documents—even <a
		href="/about/visualizing_the_system#sec-6">this message</a>.</p>
		<p>Again, this is an experiment.  The documents contain a lot of “thinking
		out loud” and a lot of old thinking.  The goal is not to make it perfect,
		but to maintain a <i>reflective</i> process that supports its own
		evolution.</p>
	</div>
	<object id="program-graph" type="image/svg+xml" data="/static/images/program-graph.svg">
		<img src="/static/images/program-graph.svg" alt="graph of the program" />
	</object>
</section>
</after>
#more-about
	padding-top 4rem

.program-graph-description
	@import centering
	@import typesetting
	centered-block($readingRems)

	p:first-child
		// Prevent bubble
		margin-top 0
@import beacon-metrics
#program-graph
	display block
	margin 0 auto
	// Leave room for the beacon, so it doesn't block
	//margin-top $beaconMarginRems + $beaconRems + 1rem
	max-width 100%

7 advertise it

This new content is added after the main introduction to the about region, and that introduction takes up the the whole screen. So how would anyone know that anything is there? There needs a signs that says, “But wait… there’s more!”

#about-willshake-message <after>
	<a id="about-more" href="/about#more-about" />
</after>

It’s your basic circle with arrow.

@import signs
@import arrows
@import colors
@import centering
@import thumb-metrics
@import pointing

#about-more
	$width = unit($thumbRems, em)

	// Occupy that space
	//font-size 150% only for large screen
	display block
	centered-block(width: $width)
	text-align center
	circle($width)
	sign-text-color()

	background rgba($aboutColor, .8)
	+user_pointing_at()
		background rgba($contextColor, .8)

	+arrow-after-pointing(down)
		display block
		margin 0 auto
		margin-top .5em

	// We use a pseudo-element for the sign text because the link only makes
	// sense in a styled environment.
	&:before
		content 'more'
		display block
		padding-top 1.5em

Footnotes:

1

Fred Brooks, “No Silver Bullet” (1985) http://worrydream.com/refs/Brooks-NoSilverBullet.pdf (This copy is hosted by Bret Victor.)

2

“Node, Edge and Graph Attributes” http://www.graphviz.org/doc/info/attrs.html

3

From Graphviz attributes documentation2.

a_inv.gif
4

For an example, see “SVG drop shadow using css3.” The answer by Erik Dahlström (one of the outstanding contributors on the subject of SVG) shows how this can be done with SVG filters. http://stackoverflow.com/a/6094674

about willshake

Project “willshake” is an ongoing effort to bring the beauty and pleasure of Shakespeare to new media.

Please report problems on the issue tracker. For anything else, public@gavinpc.com

Willshake is an experiment in literate programming—not because it’s about literature, but because the program is written for a human audience.

Following is a visualization of the system. Each circle represents a document that is responsible for some part of the system. You can open the documents by touching the circles.

Starting with the project philosophy as a foundation, the layers are built up (or down, as it were): the programming system, the platform, the framework, the features, and so on. Everything that you see in the site is put there by these documents—even this message.

Again, this is an experiment. The documents contain a lot of “thinking out loud” and a lot of old thinking. The goal is not to make it perfect, but to maintain a reflective process that supports its own evolution.

graph of the program

about

Shakespeare

An edition of the plays and poems of Shakespeare.

the works