self documenting

Programs are made of bits. The manner in which we present these bits greatly affects our sense of how they are put together.

Brian Foote and Joseph Yoder, “Big Ball of Mud”

Most of willshake’s master files are either documents or data. This document is about publishing the documents, where “publishing” means getting them from the form in which they are written into the form best suited for reading.

1 motivation

The whole point of literate programming is that you create something readable for humans (not just programmers). But it’s not just about being readable. Where will humans get the thing to read in the first place? If the answer is “by cloning the repository and building the documents,” you’ve defeated the purpose, because non-programmers will never do that. So it seems to me that literate programming also requires you to publish the documents in a way that is just as easy to get to as the product itself.

And what better place to do that than the web site itself? It’s already designed for the viewing of documents.

What I write as the Literate Programming source text is not the human-readable output (articles on Literate Programming tend to show only this beautiful output) but the rather messy source to the weaving, markup and all.1

In this section, I’ll create this “beautiful output.” I don’t currently publish the “messy source,” but I do see it as analogous to the relationship between modern editions of Shakespeare (the “beautiful output”) and the source editions (the Folio, also beautiful but perhaps messy to modern eyes).

2 logical location

It’s always been useful to have a way to reference a specific part of a specific book. Yes, you have page number, but that changes with each new edition. You’d need a scheme for referencing the content, not the arbitrary properties of the printing. The Bible has had this feature for a long time, but most other books have not followed suit. Since the Internet and the web, our lack of such a system has become even more noticeable.

Meanwhile, in programming, it’s also useful to have an exact way to reference any part of the program. It’s useful for all the same reasons as it is for other books, but programs can take formal referencing schemes even further. If one part of a program can locate and use other parts of a program as data, then it can extend the program through transformations.

The documents we’re talking about here are not related to Shakespeare per se; they are about willshake. As such, they all belong under the “about” section.

<route path="/about/{document}" to="about_document" />

3 viewing a document

We want to embed the document itself into the site. That’s what the document layer is for.

This was also removing scroll-layer from #exhibit, but that’s not right, right?

#about-layer <class remove="active" />
html <class remove="not-scroll-layer" />
#document-layer <prepend>
	<article id="top-of-paper"
			class="about document" data-program="{document}">
		<xslt name="identity"
					input="/static/doc/about/{document}.xml" />
	</article>
</prepend>

<!-- The document file name is usable as a title. -->
<title><eval>translate(document, '_', ' ')</eval></title>

The document has already been pre-processed to death, so it’s ready to drop in as-is.

4 presentation

So what does a document look like? This section covers that, in two parts.

  1. Document “hosting,” i.e. the integration of documents into space.
  2. Document content, i.e. formatting of the documents themselves.

4.1 integration

How does a document fit into the space?

document-in-space.svg
Figure 1: A document in space

This is what we’re going for.

The left side is reserved. Presumably—everything else being equal—whatever was there before will still be there. So being able to see it will amount to “preserving context,” which is useful for staying oriented.

The right margin is reserved for sidenotes, or blocks (such as figures and code) that may benefit from a greater width than the main text column.

We could also do edge-to-edge blocks, which would temporarily cover the context on the left. It might be worth doing in some cases, but for now I’m just trying to get these right.

The document view is centered around a column of text, which should be a comfortable reading width, though not quite as wide as a “normal” book.

@require typesetting
@require centering
$flowRems = $readingRems * 2/3
$flowMargin = 2em
centered-block(width: $flowRems)

This establishes what 100% means within the document.

padding $flowMargin
box-sizing border-box
position relative                       // support layering
document-width.svg

So with that you can define a “wide” element as occupying half of the viewport’s width, plus and additional half of the document flow’s width.

.wide
	margin-left -2em  // match document padding
	margin-right calc(50% - 50vw)

Special case for the source container.

.org-src-container
	width calc(50vw + 50%)
	box-sizing border-box
// From Tufte
$documentColor = #FFFFF8 // off white towards yellow

The document is a material thing.

@require document_colors
background $documentColor
box-shadow 0 0 .5em

The shadow extends out on all sides, so that the document appears to float above any material below it, regardless of what edge you happen to be looking at.

Now, as the viewport gets smaller, the margins on either side of the main flow will get smaller. When the viewport is narrower than the preferred width, then the flow itself will shrink, and there will be no margins at all.

// Create compositing layer in Chrome
transform translateZ(0)

margin-bottom 2em

But if we’re going “out of the box,” why not go all the way?

.full
	margin 1em calc(50% - 50vw)

With that, you can do the “edge-to-edge” imagery that’s so hot now, without giving up the context.

But for that to look good, it’ll take a little cleaning up.

@require fonts
@require colors
@require document-map-colors
.wide
.full
	box-sizing border-box
	box-shadow 0 0 1em #111
	background $documentMapDarkColor
	background blend($resourcesColor, $documentMapDarkColor, .25)
	border-radius 1em
	overflow hidden
	background-clip border-box

	img
		box-shadow 0 0 1em #111

	p:first-child
		margin-top 0

	// Probably a caption
	p:last-child:not(:first-child)
		body-font()
		color #FFE
@require responsive
+narrow-screen()
	padding 1em

4.1.1 BUG “wide” and “full” classes not working on figure

You get the border radius and box shadow, but not the larger size.

4.2 advertisement

DISABLED. I’d rather link to the PDF.

As another pointer to the presence of more material, let’s add to the about region message:

#mission-statement <append>
<span>You can read our <a href="/about/the_project#top-of-paper">philosophy</a>.</span>
</append>

4.3 signs for the documents

This is disabled right now… it was a plain-text list of the documents. The SVG diagram is more useful generally, but this could have a place, too—for example as a fallback for non-stylesheet browsers (better, in this case, even where SVG is supported by such browsers, since the links will function).

Also note that if you do restore this, you’ll need to ship documents.xml.

#about <append>
	<section id="the-program">
		<h2>the program</h2>
		<p>willshake.net is a literate program</p>
		<xslt name="program-links" input="/static/doc/documents.xml" />
	</section>
</append>

That assumes the document index has been copied to the web site.

: $(PUBLISHED)/<documents.xml>  \
|> ^o ship document index ^ \
   cp %<documents.xml> %o \
|> $(SITE_DOCS)/documents.xml

Now for that transform:

<xsl:template match="/">
	<ul><xsl:apply-templates /></ul>
</xsl:template>

<!-- Only match documents with titles.  This skips "included" files. -->
<xsl:template match="document[title]">
	<li>
		<a href="/about/{@key}">
			<xsl:value-of select="title" />
			<xsl:if test="description">
				<xsl:text> </xsl:text>
				<span class="description">
					<xsl:value-of select="description" />
				</span>
			</xsl:if>
		</a>
	</li>
</xsl:template>

5 hypertext integration

I know, “integration” sounds scary. This is about making the documents work, particularly with respect to links, in the context of the web site where it will live, which has a somewhat different logical structure than the physical documents.

5.1 adjust link addresses

Part of the value of literate programming is in cross-referencing. As long as we can address locations in the document, we can do so by using links. During HTML export, Org will render inter-document links with the assumption that each document is exported to an equivalently-named .html file. We are instead placing them under /about, with no extension.

<!-- Links to the about documents themselves -->
<xsl:template
		mode="org-copy"
		match="a/@href[
					 '.html' = substring(., string-length(.) - 4, 5)
					 and not(starts-with(., 'http'))]">

	<!-- The link may include a subsystem path, which is ignored for this
	     purpose. -->
	<xsl:variable name="with-path" select="substring-before(., '.html')" />
	<xsl:variable name="file">
		<xsl:for-each select="strings:tokenize($with-path, '/')">
			<xsl:if test="position() = last()">
				<xsl:value-of select="." />
			</xsl:if>
		</xsl:for-each>
	</xsl:variable>

	<!-- Support included documents -->
	<xsl:variable name="name"
								select="substring-before(concat($file, '__'), '__')" />
	<xsl:attribute name="href">
		<xsl:value-of select="concat('/about/', $name)"/>
	</xsl:attribute>
	<<validate document reference>>
</xsl:template>

We just assume that any link ending with .html and not starting with is a link to a document—as long as it doesn’t also start with http, which would indicate that it’s a link to an external site.

Regarding the double-underscore business. Willshake’s literate document system supports the use of Org’s INCLUDE feature. For export purposes, included documents are part of the containing document, and have no independent existence. But some links will point to included documents. Rather than making the links “incorrect” (by pointing instead to the containing document), I rely on the naming convention that included documents use, which is {parent}__{child}, where parent is the name of the containing document.

5.2 rebase site references

Anything in the org document that points to the site folder can be retargeted trivially to its child directory:

<xsl:template mode="org-copy" match="img/@src[starts-with(., '../site/')]">
	<xsl:param name="attribute" select="'src'" />
	<xsl:attribute name="{$attribute}">
		<xsl:value-of select="substring-after(., '../site')"/>
	</xsl:attribute>
</xsl:template>

This will generally be used for images.

5.3 support development domains

The willshake web site lives at https://willshake.net, of course. But when working on willshake, you’ll usually run a local copy of the site, at an address like localhost:8000. But what if a link uses an absolute path to the real domain, like this:

https://willshake.net/plays/AYL/4.1#This_looks

That will take you to a different site, which is not what you want. So we change it to a relative path here,

/plays/AYL/4.1#This_looks

This way, the link will be treated as internal.

You might ask, why not just use a relative path in the first place? Because we also publish the documents to non-web-based formats, such as PDF, where URI’s need to be absolute. This transformation is only for the web.

But if this is only needed for development, why do it in production, too? Because it yields shorter paths where the full address is unnecessary.

<!-- Rebase link paths. -->
<xsl:template mode="org-copy" match="a/@href[starts-with(., 'https://willshake.net/')]">
	<xsl:attribute name="href">
		<xsl:value-of select="substring-after(., 'https://willshake.net')"/>
	</xsl:attribute>
</xsl:template>

6 document features

This section is mostly about the formatting of the actual document contents. It should be relatively agnostic of the container.

Everything in the document should be hyphenated.

@require fonts
book-font()
hyphens auto

6.1 headings

h1, h2
	font-weight normal

Sections headings are links to themselves.

<xsl:template mode="org-copy" match="h2|h3|h4|h5|h6">
	<xsl:variable name="id" select="@id" />
	<xsl:copy>
		<xsl:apply-templates mode="org-copy" select="@*" />
		<a href="#{$id}">
			<xsl:apply-templates mode="org-copy" select="node()" />
		</a>
	</xsl:copy>
</xsl:template>

6.2 images

I think “a thousand words” gets a bad rap in that old saying about pictures. I am happy to have both, and I’m not alone.

Number one, it has to be easy to use images from the document, where “use” means that they show up on the site.

And as long as we’re using Org documents, it’s convenient if the image can also be seen while editing. The simplest way to achieve that is to use image references that are relative to the document. But of course that breaks the first priority, since those references will be no good when the document is in another context such as the web site. By rebasing certain paths, we have it both ways.

6.2.1 marginal images

Thanks to the collection, it’s easy to get and use images around here, and there’s no reason that it should be limited to Shakespeare-related content.

Further to the notion of “a picture and a thousand words,” it would be nice if we could set pictures in the margin—the “side” area that will be reserved for such things (see reading a document). It would be hard to overstate how useful this technique can be, given the right images and enough labor.2

We kind of have that with our “TBD” thing already. Here is the medium-sized image projection example.

Sargent-Terry-Lady_Macbeth@margin.jpg

Then, we’ll need a way to put things into the margin. You can do that with an #+ATTR_HTML: :class marginal before an image link. Org will export this as a figure with the marginal class.

.marginal
	position absolute
	left 100%
	width 100%
	max-width s('calc(50vw - %s)' % ($flowRems / 2))
	margin-left ($flowMargin * 3/4)

6.3 links

Give some indication that links are links, which we don’t by default. This should only apply within narrative text.

p, ul, ol, blockquote
	a
		text-decoration underline

Yeah, this doesn’t play well with descenders, and so on.3

Of course, I’ve generally avoided using links in the text directly, anyway, favoring links in footnotes, and not hyperlinked text at that (favoring plain URL’s, since these can still be used from printed PDF’s). It’s a good time to revisit that. You can extract the URL during PDF export, without polluting the web documents.

This is kind of a temporary measure. Some code blocks (which don’t wrap) are very long and this bubbles up to affect the computed size of the whole layer.

This is more or less for the same reason as above, but it’s less of a hack.

img, object
	max-width 100%

Figure images in documents should be centered horizontally.

figure
	img, object
		display block
		margin 0 auto
	// why this?
	&:not(.marginal)
		img, object
			border-radius .25em

The link is rendered as plain HTML of course.

6.4 quotes

Block quotes come from other documents—or at least other sources. Here, we make them look like a separate material from the document in which they are quoted.

The risk is that these sheets won’t look like part of the document “flow.” Unlike asides, block quotes are not flow-interrupting; they are supposed to be read in sequence. That’s a downside to this visual cue, which should keep us from using too dramatic of an effect.

blockquote
	line-height 1.25
	box-shadow 0 0 .5em #111        // make a sheet
	// see source block
	background tint($documentColor, 50%)
	padding .5em 2em
	box-sizing content-box
	border-radius 1em
	border-bottom-right-radius 0

We do occasionally quote some verse around here. Org actually has a special construct for it, which differs from a regular blockquote in that line breaks are preserved as written.

.verse
	line-height 1.25
	margin-left 4em

6.4.1 TODO formally indicate source

HTML doesn’t require you to provide a citation for a blockquote, and Org doesn’t even have a structure for this.

But I want to adopt a convention that will let me formally indicate the source of quotes. It may be that a footnote at the end of the quote would be the best way to do it.

6.5 lists

Do we actually use dl’s anywhere?

6.6 special constructs

As I look over the blocks I’ve demarked, several categories arise:

tbd
Notes (essentially to myself) about unsatisfactory things that are actionable—or should be soon.
apologias
Explanations about (usually) unsatisfactory things that have no immediate or even forseeable remedy.

The tbd’s can always be marginal, because the expectation is that they’ll be removed soon! So you don’t have to worry about how they negotiate a place with other content.

There is another way to look at all of the remaining notes, including the apologias, which is orthogonal to that grouping, and that is whether they must be in the flow or whether they can be marginal.

Another consideration is how long the note is. Longer passages in the margin just don’t work well. It’s not just the fact that absolutely positioned boxes can end up overlapping one another (which you could avoid by judicious use); the length is an indication that it’s probably not really an aside.

It’s possible that with a better layout strategy, there could be a way to preserve the aside-nature of the blocks that are currently longish aside’s, but as things stand, they clearly go back into the document flow.

6.6.1 asides

See note in publish the documents. The class version (.aside) is needed in addition to the tag version (aside) because we’re not really exporting html5 elements.

An aside is a remark that is not addressed to everyone. Hamlet’s first line is an aside. Of course, we’re talking here about asides in documents, but the idea is similar: that asides are not for everyone. The HTML5 specification says

The aside element represents a section of a page that consists of content that is tangentially related to the content around the aside element, and which could be considered separate from that content. Such sections are often represented as sidebars in printed typography.4

Bring together the aside-related code, and generally organize all this by construct, not by language.

So, if we use an aside (which we go out of our way to do), we are signaling that it may be taken out of the document flow. Still, it is part of the document, and moreover—unlike block quotes—its original source is the document. So it really means something different than a block quote, and should not use the same visual cue.

6.6.2 tbd

@require pointing
@require document-map-colors // see note in map section
.aside
aside
	padding-left 1em
	padding-right @padding-left
	border-left 4px solid $documentMapColor
	&.tbd
		position absolute
		left calc(100% - 1em)
		//margin-left 1em
		width calc(50vw - 50% - 1em)
		line-height 1.2
		box-shadow 0 0 2px
		background $documentColor                       // since we can't inherit
		z-index 1 // get in front of positioned things in main flow
		// In case they do overlap
		+user_pointing_at()
			z-index 2

6.6.3 notes

Very often in the course of these discussions, I find myself having to express a rather numbing amount of detail. In many cases, a passage cries out as being especially contingent on ephemera, if not trivia. Although I’m calling these sections “notes” right now, my general sense is that they are mostly just demarkers of incidental complexity—/apologias/ for the way things are.

They may be seen as removable in some way. And yet they aren’t quite unimportant enough to be footnotes.

So what? How should they look?

They aren’t “affordable” in that you can’t do anything with them, so they shouldn’t have any signifiers that would suggest that they did.

But really, why couldn’t these be “side notes”? They could be side notes without being “asides.” I think this would be a distinction within the existing notes. Some of them are just too long to be put in the margin, whatever their semantics. Also, those longer-form notes could conceivably have marginal contents of their own (if, for example, we put footnotes in the margin, which I’m planning to).

Most ways of accentuating these sections call more attention to them, which is kind of the opposite of what I want.

They’re not “asides,” since they’re part of the narrative flow, albeit “parenthetically.” So they’re shown as if the whole block were in parentheses.

@require document-map-colors // see note in map section
.note
	padding 0 1em
	text-align justify
	font-size 90%
	line-height 1.2

	// "Parentheses"
	border-left .25em solid $documentMapColor
	border-right @border-left
	border-radius .75em / 50%

6.7 dialogs

—Hey, what’s better than a monologue?

—A dialog?

—What’s better than a dialog?

—Shakespeare?

—Okay, well this ain’t Shakespeare, but it’s a nice break from the narrative, don’t you think?

—If you say so.

Just for fun, let’s use dialogs. It’s kind of a zen teacher/student thing. For the structure itself, no extra code is needed. Just by using a DIALOG block,

#+BEGIN_DIALOG
---Who's there?

---Nay, answer me.  Stand and unfold yourself.

---Long live the king.

---Barnardo?

---He.
#+END_DIALOG

Org export will create a div with class dialog. Everything inside of it will be formatted as elsewhere.

To create the back-and-forth effect, the paragraphs are set alternately to the left and right, much like a typical text messaging app.

speaker(set, side)
	p:nth-child({set})
		float side
		text-align side
		border-bottom-{side}-radius 0

Otherwise, they are the same: little speech bubbles.

@require document-map-colors
.dialog
	> p
		margin .25em 0
		padding .5em 1em
		line-height 1.25
		border-radius 1em
		clear both
		max-width 66%
		background lighten($documentMapColor, 80%)

		// Assume first character is a dash, and hide it
		text-indent -1em
		&:first-letter
			color transparent

	&:not(.flipped)
		speaker(odd, left)
		speaker(even, right)

	&.flipped
		speaker(even, left)
		speaker(odd, right)

	// The old "clearfix" trick
	&:after
		content ''
		display block
		clear both

As of now, the bubbles just have a light gray background. Use any stronger color, and you have to reverse the text, which I think is too “dramatic” for this. So I don’t even use two different shades for the speakers, because there isn’t much of a range that is workable here. The same goes for using a background color on the whole section, and making the speech bubbles white, as we do in the plays. It just creates a break in the document flow that is too abrupt.

Because of the max-width, these can sometimes wrap in an ugly way.5

Also, the :first-letter selector isn’t working in Firefox. I’ve seen this happen before when the character is a non-ASCII character. Not sure what to do about it.

—But wait, what if you wanted to start on the other side?

—Then you mark the dialog as “flipped”.

—How do you do that?

—With #+ATTR_HTML: :class flipped.

—Nice.

For other formats (such as PDF), the contents will be completely unaffected.

6.8 figures

Figures are first-class in HTML (as of HTML5). Mozilla Developer Network, my unofficial reference of choice, describes it this way:

The HTML <figure> element represents self-contained content, frequently with a caption (<figcaption>), and is typically referenced as a single unit. While it is related to the main flow, its position is independent of the main flow. Usually this is an image, an illustration, a diagram, a code snippet, or a schema that is referenced in the main text, but that can be moved to another page or to an appendix without affecting the main flow.6

At the moment, all images in these documents are automatically considered figures. It’s an open question as to whether I should be able to distinguish between these cases. As the official specification notes,

When a figure is referred to from the main content of the document by identifying it by its caption (e.g. by figure number), it enables such content to be easily moved away from that primary content, e.g. to the side of the page, to dedicated pages, or to an appendix, without affecting the flow of the document.

If a figure element is referenced by its relative position, e.g. “in the photograph above” or “as the next figure shows”, then moving the figure would disrupt the page’s meaning. Authors are encouraged to consider using labels to refer to figures, rather than using such relative references, so that the page can easily be restyled without affecting the page’s meaning.

Although I agree with the general point, I have not reviewed whether I have legitamite cases of the latter kind.

figure
	margin 0                                        // beat figure default
	// already set elsewhere, right? to auto, which is effectively the same
	//img
	//      margin 0

figcaption
	margin .5em 1em
	line-height 1.2
	font-size smaller
	text-align center

.figure-number
	display block
	font-style italic

Marginal figures are floating off of the document sheet, so they need to be their own material.

figure.marginal
	border-radius .25em
	background lighten($documentMapColor, 66%)
	box-shadow -.25em .25em .5em #111

6.8.1 Org and HTML5 figures and captions

This can be filed under “compensating for Org export,” a category which at some point will start to rival the complexity of implementing my own export. To be fair, Org may do a better job of this if “fancy” elements are enabled, but they’re not. See HTML publication options.

Figures are currently rendered by Org export as a div.figure with the contents wrapped inside of a p, followed by another p for the caption. So a bit of schlepping is needed to translate that into HTML5.

<xsl:template mode="org-copy" match="*[@class='figure']">
	<figure>
		<xsl:apply-templates mode="org-copy" select="node()|@*" />
	</figure>
</xsl:template>

<xsl:template mode="org-copy" match="*[@class='figure']/p[2]">
	<figcaption>
		<xsl:apply-templates mode="org-copy" select="node()|@*" />
	</figcaption>
</xsl:template>

The image is also needlessly wrapped inside of a paragraph element.

<xsl:template mode="org-copy" match="*[@class='figure']/p[img]">
	<xsl:apply-templates mode="org-copy" select="node()|@*" />
</xsl:template>

Also, when I use HTML_ATTR: :class before an image, Org applies the class to the img, not the container. This bubbles the class from the img to the containing figure.

<xsl:template mode="org-copy" match="@class[.='figure']">
	<xsl:copy-of select="..//img/@class" />
</xsl:template>
<xsl:template mode="org-copy" match="*[@class='figure']//img/@class" />

6.9 section status

I am using keywords for bugs and proposals and such, but I don’t think this is working now. Not because of this, but because the keywords I use aren’t set at export time.

Mostly for internal use, but no harm in publishing.

@require colors
keyword($color)
	color $color
	text-shadow 1px 1px 0 #555

	&:before
		content ''
		position absolute
		right 100%
		margin-right 1em
		width 1em
		height 100%
		background $color
		box-shadow 1px 1px 1px #555

$badColor = desaturate(red, 50%)
.tbd
	@extend .document .aside
	border-left-color $badColor

.outline-2
.outline-3
.outline-4
	position relative               // support section status markers

#table-of-contents li
	.todo:before
		display none

.todo
	font-family sans-serif;

.STUB
	keyword($badColor)

.DRAFT
	keyword(desaturate(yellow, 50%))

.PUBLIC
	keyword(desaturate(green, 50%))

6.10 IDEA epigrams

I use epigrams a lot, and it seems that they could do with some special formatting. At the least, they are usually right-aligned.

Footnotes:

1

David M. MacMillan and Rollande Krandall, “‘Minimalist’ Literate Programming in graphein

2

See, for example, the document version of Maciej Cegłowski’s fantastic presentation “Web Design: The First 100 Years” (2014). http://idlewords.com/talks/web_design_first_100_years.htm

3

Dave Liepmann & Edward Tufte, “Tufte CSS.” https://edwardtufte.github.io/tufte-css/

4

HTML5 § 4.3.5 “The aside element”. W3C Recommendation 28 October 2014. http://www.w3.org/TR/html5/sections.html#the-aside-element

5

This problem and its lack of a solution are addressed by the question, “max-width adjusts to fit text?” http://stackoverflow.com/q/13672760

6

<figure>”, Mozilla Developer Network. See also the somewhat more turgid official specification of § “The figure element”.

about willshake

Project “willshake” is an ongoing effort to bring the beauty and pleasure of Shakespeare to new media.

Please report problems on the issue tracker. For anything else, public@gavinpc.com

Willshake is an experiment in literate programming—not because it’s about literature, but because the program is written for a human audience.

Following is a visualization of the system. Each circle represents a document that is responsible for some part of the system. You can open the documents by touching the circles.

Starting with the project philosophy as a foundation, the layers are built up (or down, as it were): the programming system, the platform, the framework, the features, and so on. Everything that you see in the site is put there by these documents—even this message.

Again, this is an experiment. The documents contain a lot of “thinking out loud” and a lot of old thinking. The goal is not to make it perfect, but to maintain a reflective process that supports its own evolution.

graph of the program

about

Shakespeare

An edition of the plays and poems of Shakespeare.

the works