live programming

Here we deal with things that are particular to willshake development. We’ll add some features that are useful while you’re making changes to the program. This document should not be included in production.

Development is a “cross-cutting concern” for our purposes. Elsewhere, we are sensitive to dependencies—not here. In other words, this document may assume the presence of any other feature at any time. So this document is said to “build on” the system, but everything is fair game.

This should not be done at all in production. As it is, it just exits when on live.

The first thing we’ll do is inject a “development” script into the document:

<require module="development" />

Using zz because scripts should go at the end; this ensures that the append occurs last.

Now we’ll write that script, using a placeholder that we can reference in this document. When this starts getting unwieldy, we’ll use the old “folder of scripts” approach instead:

define([], () => {

if (/willshake\.net/.test(window.location.hostname))
	return;

console.log("development mode");
<<development script>>
}());

The hostname check is of course a cheap way to prevent this from happening on live. Better would be to exclude this document completely for production, so that the script is not shipped, not to mention downloaded.

1 “live reload” changes in the browser

As part of our ongoing effort to poorly approximate the power of live programming, we use a technique that has been unabashedly called “live reloading.” That is: you use a browser to look at your web site; you change the program; and the browser automatically updates what you see. In some cases—such as style and image updates—the change can be applied in-place; for everything else, the whole page is requested, rendered, compressed, transmitted, decompressed, parsed, interpreted, laid out, composited, and displayed again. The beauty of “live” reloading is that you don’t have to press Ctrl+R to make this happen; you can just sit and watch.

To serve us in this noble cause we enlist the livereloadx package1, which you can install like so:

This package is based on the old LiveReload system by Andrey Tarantsov.2 It uses WebSockets to communicate between a special script and a special server so that it can reload assets (or the entire page) when the site’s disk files change. By default, it only tracks common file types (html, css, etc). Since we use a lot of xsl, we have to add that explicitly.

cd site; livereloadx --include '*.{xsl,svg}'

Once the server is running, the LiveReload client script is available for download. Generally, it will run on the same machine that is serving the site, but on a different port number (so it doesn’t conflict with the actual site). So, if we want to remain agnostic of the hostname, we have to load the script dynamically. The hostname will often be localhost, but the beauty of this method is that it also works when accessing the site from a remote client. Thus, we just use whatever hostname was used to view the site.

var livereload_script = document.createElement('script');
livereload_script.src = "//" + window.location.hostname + ":35729/livereload.js?snipver=2";
document.body.appendChild(livereload_script);

That’s it. Whenever a file in the site’s directory changes, any browser that has it open will hear about it and act accordingly.

Special exceptions to the security policy are needed for this to work. Right now we don’t bother to do this either additively or conditionally. See content security policy. Further to that, note that although we are “protocol agnostic” above (by using //), those security exceptions are in fact explicit as to protocol.

1.1 preserve scroll position when reloading page

The whole point of live reloading is to see changes applied in-place. But as we’ve said, this often means reloading the whole page. One defect of the livereload.js script is that it doesn’t retain the scroll position when reloading. Instead, a page that is partly scrolled at the time of a change will jump back to the top. This defeats the purpose of live reloading, since action is needed to return to the place of interest (which is presumably where you were looking). The browser itself will often restore your scroll position when you return to a page, but not, as a rule, when you reload it.3 Whether this is ideal for ordinary usage is not our concern; for development reloads, it’s clearly needed.4

Fortunately, the LiveReload object can be patched at runtime to effect this change.

First, we need a way to pass information (in this case, the scroll position) from one incarnation of the page to the next. Variables won’t work, since the javascript execution environment is thrown away upon each load. Fortunately, HTML5 introduces a couple of persistence mechanisms: localStorage and sessionStorage. The only difference between them is that localStorage will survive closing and reopening the browser, so for this purpose, sessionStorage is adequate. These storage API’s are designed to share the same permissions as cookies for a given domain. Trying to access them on a domain where cookies are not allowed will throw a SecurityException. So for this technique to work, cookies must be enabled, and we need to catch the exception so that the script will not always crash when they aren’t. Note that it’s not quite enough to simply reference the storage object; you actually have to use it to confirm that it’s working.

function get_session_storage() {
	const KEY = 'test_value', VALUE = 1;
	var storage;
	try {
		(storage = window.sessionStorage).setItem(KEY, VALUE);
		if (storage.getItem(KEY) == VALUE) {
			storage.removeItem(KEY);
			return storage;
		}
	} catch (exception) {
		console.log("Session storage is not available:", exception);
	}
}

If we should ever want to use persistent storage for user-facing features, this method can be moved into production.

Remember, we use layers. Only one layer is scrollable at a time, but it changes from place to place. (See “the-layers”).

function get_scroll_layer() {
	return document.querySelector('.active.scroll-layer') || document.documentElement;
}

Now, we replace LiveReload’s reloadPage function with our own version, which first saves the current scroll position.

livereload_script.addEventListener('load', () => {
	const reloader = LiveReload.reloader;
	const __base_method = reloader.reloadPage;
	reloader.reloadPage = function() {
		const storage = get_session_storage();
		if (storage)
			storage.setItem('last_scroll_y', get_scroll_layer().scrollTop);

		__base_method.apply(reloader, [].slice.call(arguments));
	};
});

Finally, when the page loads, we’ll check for a saved scroll position and then try to restore it.

(function() {
	const storage = get_session_storage();
	if (storage) {
		const y = storage.getItem('last_scroll_y');
		if (y) {
			console.log("restoring persisted scroll position", y);
			get_scroll_layer().scrollTop = y;
		}
		storage.removeItem('last_scroll_y');
	}
}());

1.2 a program change should only cause a reload when you’re looking at it

We integrate the program itself with the web site. It’s neat. But this causes a problem for live reloading. Suppose that I change a style rule in one of the documents. One of the nice features of livereload is that it will update stylesheets in-place. But the program in which I changed the style rule will also be updated, and since this is not a CSS change, it will trigger a full reload. We don’t want that.

But what’s the alternative? We could tell livereloadx to ignore the directory where programs are published. But we don’t want that either, because it’s useful to see the published program while you’re working on it—it’s a kind of dogfooding for the whole concept. So what do we do?

We’ll try something similar to the approach used for scrolling. We’ll hack LiveReload at runtime.

livereload_script.addEventListener('load', () => {
	const reloader = LiveReload.reloader;
	const __base_method = reloader.reload;

	reloader.reload = function(path, options) {
		var about_file = path.match(/^static\/doc\/about\/([^\/]+)\.xml$/);
		if (about_file
			&& window.location.pathname != '/about/' + about_file[1]) {
			console.log('Skipping reload of document', about_file);
			return;
		}

		__base_method.apply(reloader, [].slice.call(arguments));
	};
});

1.3 reload object elements

LiveReload does not support reloading of object elements, which can be used to “embed” external entities into the page. If nothing else, this is useful for “interactive” SVG graphics (that is, SVG graphics with functioning links and scriptable elements). In fact, the current version of LiveReload doesn’t recognize SVG as a type of image to reload at all. So we may eventually need to deal with both cases here. For the moment, this just reloads objects.

livereload_script.addEventListener('load', function() {
	const reloader = LiveReload.reloader;
	const __base_method = reloader.reload;

	reloader.reload = function(path, options) {
		if (path.match(/\.svg$/)) {
			// Note that the path sent to this method is not rooted.  Also, we
			// use "starts with" instead of an exact match because we'll be
			// adding a query to the end.
			const objs = document.querySelectorAll('object[data^="/' + path + '"]');
			[].slice.call(objs).forEach(obj => {
				// Yeah, reloader has a generateCacheBustUrl method, but it
				// complains about the `this' reference no matter how I invoke
				// it.
				// 
				// Also, yes, the object has a `data` property that you can set
				// directly.  But this takes the liberty of setting the `data`
				// *attribute* to an absolute path, making it harder to find
				// this element again next time it changes (since we're using
				// attribute selectors).
				obj.setAttribute(
					'data',
					obj.getAttribute('data')
						.replace(/(\?.*)?$/, '?t=' + new Date().getTime()))
			});
			return;
		}

		__base_method.apply(reloader, [].slice.call(arguments));
	};
});

Footnotes:

1

NPM, “livereloadx: An implementation of the LiveReload 2 server in Node.js” https://www.npmjs.com/package/livereloadx

2

LiveReload: “The Web Developer Wonderland (a happy land where browsers don’t need a Refresh button)" http://livereload.com/

3

Bugzilla Bug 838523: “When reloading a page, Firefox doesn’t retain the scroll position” https://bugzilla.mozilla.org/show_bug.cgi?id=838523

4

See, e.g., Bugzilla Bug 979989: “Maintain scroll position after page reload” https://bugzilla.mozilla.org/show_bug.cgi?id=979989

about willshake

Project “willshake” is an ongoing effort to bring the beauty and pleasure of Shakespeare to new media.

Please report problems on the issue tracker. For anything else, public@gavinpc.com

Willshake is an experiment in literate programming—not because it’s about literature, but because the program is written for a human audience.

Following is a visualization of the system. Each circle represents a document that is responsible for some part of the system. You can open the documents by touching the circles.

Starting with the project philosophy as a foundation, the layers are built up (or down, as it were): the programming system, the platform, the framework, the features, and so on. Everything that you see in the site is put there by these documents—even this message.

Again, this is an experiment. The documents contain a lot of “thinking out loud” and a lot of old thinking. The goal is not to make it perfect, but to maintain a reflective process that supports its own evolution.

graph of the program

about

Shakespeare

An edition of the plays and poems of Shakespeare.

the works