literate programming

Why bother with such a difficult method of programming? Because worthwhile programs should “live.”1

Donald_Knuth-2006-Mekhitarian@margin.jpg
Figure 1: “Donald Knuth in front of statue St. Mesrop Mashtots (inventor of the Armenian, Georgian and Caucasian Albanian alphabets in the 4th century), Matenadaran, Yerevan, Armenia, 9 June 2006”

Like all discussions of literate programming, this one has to begin with Donald Knuth. http://www.literateprogramming.com/knuthweb.pdf

1 origin

If you are a software developer who feels that the “state of the art” is “not entirely satisfactory,” you wouldn’t be the first. And you’d be in good company. In 1984, Donald Knuth, perhaps the greatest benefactor that the realm of computing has ever known—and no mean writer—introduced a concept he called “literate programming,” saying,

Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do.2

Literate programming inverts the traditional relationship between code and commentary. Instead of code files with scattered remarks, Knuth’s system (which he called WEB, because the pieces of the program were woven together) made the document the first-class citizen, with the code interpolated as needed for the sake of explanation.

Knuth wonders whether literate programming

may be only for the subset of computer scientists who like to write and to explain what they are doing. My hope is that the ability to make explanations more natural will cause more programmers to discover the joys of literate programming, because I believe it’s quite a pleasure to combine verbal and mathematical skills; but perhaps I’m hoping for too much.

2 motivation

I’ve only been practicing literate programming for a year, so I don’t pretend to have it all figured out. But in that time, I’ve found several good reasons why literate programming is a good fit for this project.

2.1 sustainability

Software systems are like sand castles: they’re wonderful, silicon-based creations that will be gone tomorrow. Maybe you’ll come back and rebuild it. At some point, you won’t. The only way to keep software alive for generations is to teach people how to rebuild it.

And in literate programming, it is culturally unacceptable to change the system without first “teaching the change.” Starting with yourself: you can’t explain something until you understand it. Literate programming forces you to explain everything. Literate programming is mostly about writing, and writing is a thought process.

2.2 document orientation

Consider how a program is like Shakespeare:

  • they both come from a set of source documents
  • they both may call for some explanation, for a general audience today
  • they both have internal structure
  • they both benefit from illustrations

In other words, any system whose specialty is the amplification of documents, will have a lot to offer, not only for Shakespeare and related works, but also for the system itself, if it should choose to use document form.

2.3 humanistic

As things stand, there is a great divide between design and implementation. This divide limits the depth of collaboration between those who understand the subject matter and those who understand the technology. This is familiar to anyone who’s ever been involved in a software project. It’s critical to communicate to a general audience, or at least to people without technical knowledge, and literate programming works to bridge that divide.

2.4 conclusion

It’s not clear that Knuth was thinking explicitly about any of these particular benefits when he first proposed programming as a literary act. But it is clear today that we need to embrace the practice. All non-trivial programs are multi-language systems communicating with other multi-language systems. The dictates of compilers and frameworks are as unconducive to understanding the full picture of a system now as they were in 1984.

3 composing features

Your features! Lord warrant us! What features!
As You Like It

Willshake’s literate programming system is used to help make the system more understandable. One way that it does that is by focusing on features.

After trying this method for a while, I was convinced that it was a good thing and set out to write about it. In trying to find anyone else pursuing this idea, I was lucky to find one person—Kartik Agaram—who explains the situation very well in his post called “A new way to organize programs."3 He uses a different technique (which he calls “directives”) to achieve the same result, but that’s not important. His explanation of the underlying situation is perfect on every point, and I will rely on it here.

That said, I’m not sure what to call this concept. I have called it by variously unfriendly terms, including “additive programming” and “lexical feature scoping.” As I read his article, Agaram uses the the term “layers” to mean the composable parts of a system, and “features” to mean a certain kind of layer.

This is closely related to literate programming. First, it favors a topical view of the system, by asking that the program be organized in the units that are meaningful to the person (features), rather than the units that are meaningful to the computer (bits of code).

we can now organize a program as a series of self-contained layers, each containing all the code for a specific feature, regardless of where it needs to go

For our purposes, a “feature” is an atomic unit of deployment. It is a set of changes that can be added to or removed. A feature is defined by a set of documents. Adding the documents adds the feature; removing the documents removes it.

A feature should be the kind of thing that you can propose. So you have to start from the most basic thing. There are some things that you can’t really break down any further, or that are so ingrained into the structure that you can’t isolate them in any meaningful way. The latter are generally called “cross-cutting concerns.”

3.1 the idea

Suppose you have a useful product. People use it, and people like it. Then someone proposes a new feature. You consider it, and you consider it a good idea. So you write the necessary documents to implement it, and add them to the program. Does the existence of this new feature change the fact that the program was good and useful without it? No.

The philosophy of additive building is that adding means adding. That’s it. From this, some things fall out:

  • adding doesn’t mean breaking what you have
  • you can remove what you’ve added, and you’ll get what you had before.

By “add” and “remove” we are talking about lexical scopes, and particularly files. That is, a feature consists of a set of files. By adding those files to the project, you get the new feature. By removing those features, you get what you had before.

There is a special relationship between this method and the practice of literate programming, since you can bundle multiple outputs into a single document. As such, you can add the feature simply by placing the document with the rest of the program, and you can remove it by removing the document.

Does this mean that you can’t change anything outside of the new document when adding a feature? No. It just means that you can’t make breaking changes. You might need to add some invisible metadata, or some kind of hook or entry point that didn’t exist before. Or you might need to restructure how things are put together to make the necessary modification possible. But such restructuring should not break what you had before. (Such changes are often called “refactoring.”) In those cases, “removing” the new feature may mean leaving those changes in place, since they are benign and may even be generally useful for the addition of other features. Whether this may include some observable adjustments to the product is a subjective matter.

This is a way of composing a product from features that are specified in document form. It makes the “composition” of a program from atomic parts a first-class concept.

Yet another way to think of this is that you can undo anything that you do. Of course, there are lots of mechanisms for undoing, not least being a revision control system. But there’s a difference. Assume that at certain points the product reaches a stable place. These checkpoints represent usable products. Then people propose new features, and people work on them, checking in changes. The older state of the program is available through the revision control system by going back in time to it. But in additive building, the older version is always with you, representing the fact that the new feature hasn’t changed the fact that it was valuable and usable in that state.

Finally, does this mean that you are always adding things? That you can’t just change something in place? Of course not. If you don’t like the way something is, change it. Or get rid of it.

3.2 limitations

Does this mean that every document defines a feature? In other words, can every document be treated as safely removable?

No, I think not. Indeed, you might use that question as a test for whether the document represents a feature. A document may be used to add some other mechanism that is used by a number of other documents. In that case, it’s more like a “service” (a build service, assuming you’re talking about a build-time mechanism, which is the kind that documents—as opposed to the programs that they produce—can use).

3.3 as applied to user interfaces

The “additive” philosophy also extends to the design of user interfaces. This is not strictly a consequence of additive programming, but it’s in the same spirit.

New features should be new features. But this is obvious, right—/of course/ new features are new features, what’s the alternative? The alternative is that new features tamper with old features, either by removing them, or by altering them. Why would this be bad? Because you know that the product was working before. If there was a better way for those controls to work in the earlier version, then the earlier version should be made better, and then the new feature considered. This way of thinking also discourages the use of “modes,” which is a common way to provide lots of features in a small space.4

Doesn’t this limit how many features you can add? Only to the extent that they are all competing for the same space. But if they are added through “expansion points,” which grow the overall space without re-packing particular views of it, there is much more latitude for expansion. Even then, it’s a challenge to keep adding to something without overloading it. As Rich Hickey puts it, “more stuff implies more stuff."5 So to the extent that an additive design encourages additions to be made carefully, that is a feature.

So, in general, new features should not change the meaning of old features, even though they may need to be rearranged.

4 origin of “program”

program (n.) — 1630s, “public notice,” from Late Latin programma “proclamation, edict,” from Greek programma “a written public notice,” from stem of prographein “to write publicly,” from pro- “forth” (see pro-) + graphein “to write” (see -graphy).

General sense of “a definite plan or scheme” is recorded from 1837. Meaning “list of pieces at a concert, playbill” first recorded 1805 and retains the original sense. That of “objects or events suggested by music” is from 1854. Sense of “broadcasting presentation” is from 1923. Computer sense (noun and verb) is from 1945. Spelling programme, established in Britain, is from French in modern use and began to be used early 19c., originally especially in the “playbill” sense. Program music attested from 1877.

program (v.) Look up program at Dictionary.com 1889, “write program notes;” 1896, “arrange according to program,” from program (n.). Of computers from 1945. From 1963 in the figurative sense of “to train to behave in a predetermined way.” Related: Programmed; programming.

http://etymonline.com/index.php?term=program

5 more quotes on literate programming

“considering programs to be works of literature.” from Knuth’s original paper

[Literature is the software of the print era]

—Alan Kay

Closest I’ve found to a source for this: http://www.rheingold.com/texts/tft/1.html

https://groups.google.com/d/msg/alt.quotations/VacnM0nO_ok/lx53-k1I_OcJ

Literature is first and foremost about having ideas important enough to discuss and write down in some form. So you have to ask, “What is the literature that is best written down on a computer?” One answer is to make a dynamic simulation of some idea that you think is important, a simulation that you can play with and that you can learn from. (Alan Kay)

Also:

http://www.scholastic.com/browse/article.jsp?id=5

Some links at the bottom http://howardism.org/Technical/LP/introduction.html#fnr.1

Despite its ability to produce software of higher quality and maintainability, the technique is not widely used.

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.12.1775&rep=rep1&type=pdf

Without wanting to be elitist, the thing that will prevent literate programming from becoming a mainstream method is that it requires thought and discipline. The mainstream is established by people who want fast results while using roughly the same methods that everyone else seems to be using, and literate programming is never going to have that kind of appeal. This doesn’t take away from its usefulness as an approach.

—Patrick TJ McPhee

Quoted at http://www.cs.tufts.edu/~nr/noweb/

Or, for that matter, at http://www.softpanorama.org/SE/literate_programming.shtml

Which is intelligible despite apparent language barrier

http://www.literateprogramming.com/best/prettyp1.html

This article begins with a fairly gentle introduction to Literate Programming, although

One thing that Literate Programming allows you to do is to focus on the interesting parts first. A standard principle in writing big poetic epics is to start not at the beginning, which can be dull, but in medias res, in the middle of things where it’s exciting. So the Iliad doesn’t start by recounting the years of inconclusive battles between the Achaeans and the Trojans which characterized most of the Trojan War. Instead, it starts nine years into it when Achilles throws a hissy fit and things start to get interesting.6

—David M. MacMillan and Rollande Krandall, “Minimalist” Literate Programming in graphein

http://www.circuitousroot.com/graphein/literate-programming.html

Footnotes:

1

Tim Daly, “Literate Clojure” (a large PDF)

3

Kartik Agaram, “A new way to organize programs.” October 7, 2013

4

For more, see Larry Tesler, the “no modes guy”. (Wikipedia)

5

feedback wanted: Instant literal for Clojure”, Clojure Dev Google Group

6

For Shakespeare’s telling of this story, see Troilus and Cressida.

about willshake

Project “willshake” is an ongoing effort to bring the beauty and pleasure of Shakespeare to new media.

Please report problems on the issue tracker. For anything else, public@gavinpc.com

Willshake is an experiment in literate programming—not because it’s about literature, but because the program is written for a human audience.

Following is a visualization of the system. Each circle represents a document that is responsible for some part of the system. You can open the documents by touching the circles.

Starting with the project philosophy as a foundation, the layers are built up (or down, as it were): the programming system, the platform, the framework, the features, and so on. Everything that you see in the site is put there by these documents—even this message.

Again, this is an experiment. The documents contain a lot of “thinking out loud” and a lot of old thinking. The goal is not to make it perfect, but to maintain a reflective process that supports its own evolution.

graph of the program

about

Shakespeare

An edition of the plays and poems of Shakespeare.

the works