It’s always been possible to “drop in” on a web site. Just go to the URL.

But there has never been a first-class notion of change. All you can do is define new states. Every transition is like blindfolding you and putting you on an airplane, even if you’re just going next door.

Well, AJAX happened, and unleashed a flood of incalculable pain on the web programming world. Sure, it was worth it.

But imagine for a moment what it would mean to define a web site in terms of changes.

But we still don’t have a first-class notion of change—change between two states—and we never will. Everyone will keep doing it their own way. Well, getflow is me doing it my own way.

In getflow, there are two first-class things you can do in a domain. One is dropping in. That’s already first-class. The other one is traveling.

published/images/getting_from_one_place_to_another.svg

For nothing more than my own ease of authoring (which is admittedly subjective), the blueprints (routes and change rules) are expected in an XML format. This is based on the fact that the majority of the content will be HTML templates, which are practically isomorphic with XML.

Here we transform the getflow XML format into something we can use to actually implement the rules.

function read_blueprints(element) {
    return make_array(children_of(element))
	.filter(node => node.nodeName === 'route')
	.map(read_route);
}

And here is the Python version. I still call the blueprints the “plans” here.

from lxml import etree
from ..getflow.changes import ClassChange, TemplateChange

<<server read change rules>>

def read_route(node):
    return {
	'route': node.get('path'),
	'change_rules': list(read_change_rules(node)) }

def read_plans(file):
    return list(map(read_route, etree.parse(file).xpath('route')))

One of getflow’s design goals is to support an immediate response to the person. One thing that makes such responsiveness difficult is network latency. If you have to contact the (remote) server every time you want to visit a new location, then you can never hope for immediate reponses as a rule. Even in the best conditions, there is a limit on how fast you can make a roundtrip to the server, regardless of how quickly the server responds. Yet, the only way around this would be for the browser to already know how to get to every other place on the site, which for a site of any size would clearly be impossible.

The approach taken by getflow is to consolidate the essential structure of the site into a (potentially) very small set of rules, all of which can be downloaded up-front and in short order. Then, even if some of the details still have to be retrieved from the server on-demand, there will be at least something we can do while we’re waiting; that is, the broad outlines of the movement will be available, and we may proceed immediately with those.

This, then, is the part where we get these “blueprints,” which at present are in the form of an XML document containing routes and the accompanying change rules.

function read_the_blueprints(doc) {
    return state.routes = read_blueprints(doc.documentElement);
}

// Let's start reading the blueprints immediately.
const get_the_blueprints = 
	GET_XML("/getflow.xml")
	.then(read_the_blueprints, debug_message);

We also save the “blueprints promise”, which we can reference anywhere we want to guarantee that the blueprints have arrived.

A route consists of a path pattern and a set of accompanying change rules.

<<path pattern>>

function read_route(node) {
    return {
	change_rules: read_change_rules(node),
	matcher: get_path_pattern_matcher(node.getAttribute("path"))
    };
}

Whereas in principle the web does not impose any relationship between the content at a parent and a child path, getflow’s entire model is based on the assumption that sites are defined by those relationships. As such, we require a formal definition of what a “parent” path is. In short, it is the path with its query and last segment removed. This is sometimes called “hacking” the path, because you “hack off” the end of it.

function hack_path(path) {
    return '/' + path
	.split('?')[0]
	.split('/').filter(x => !!x)
	.slice(0, -1).join('/');
}

And here is the Python version.

def hack_path(path):
    return '/' + '/'.join(
	list( path.split('?')[0].rstrip('/').split('/')  )[1:-1]  )

And its tests.

from getflow.paths import hack_path
from test_helpers import check

def do_test(case):
    path, expected = case
    check(case, expected, hack_path(path))

[do_test(case) for case in (
    ('/', '/'), 
    ('/abc', '/'), 
    ('/abc/', '/'), 
    ('/abc/def', '/abc'), 
    ('/search?q=123', '/'), 
    ('/abc/def/ghi', '/abc/def'), 
    ('/abc/def/ghi?q=123', '/abc/def'), 
    ('/abc/def/ghi?q=some/url', '/abc/def') )]

<<hack path>>

return table.map(function(row) {
    var path = row[0],
	expected = row[1],
	got = hack_path(path),
	pass = got == expected;
    return ['~' + path + '~', '\\rightarrow', '~' + expected + '~',
	    pass ? '\\checkmark' : '\\xmark' + ' got ' + got];
});

Note that slashes are valid in a URL query.¹

Sometimes we want to leave a blank that will be filled in later. We’ve seen this with path patterns, and we’ll also use it in several other ways, including selectors and throughout templates.¹ Generally, we’ll use curly braces {} to demark placeholders. It should work like this.

So, how do we do it?

This pattern will match anything contained in curly braces (including the braces), and also make a separate capture of the contents.

var placeholder_pattern = /\{(.+?)\}/g;

function fill_placeholders(text, fn) {
    return text.replace(placeholder_pattern, fn);
}

function placeholders_in(text) {
    var placeholders = [];
    fill_placeholders(text, function(_, placeholder) {
	placeholders.push(placeholder);
    });
    return placeholders;
}

We can describe a huge—nay, infinite—number of locations in just a few lines. How can we do this? Using patterns.

    <<test helpers>>

return table.map(function(row) {
    var pattern = row[0];
    var description = row[1];
    return [format_cell(pattern), description || ''];
});

When someone asks to go to a certain address, we’ll see if it matches any of these patterns. If it does, that means there’s (maybe) something there, and we’ll see about going to that place. For now, we’re just interested in how the matching works.

Those names inside the braces are placeholders. The “real” addresses may fill in those blanks with anything (any single thing, i.e., no slashes). If a path fits the pattern, we’ll remember how names in braces were replaced. In practice, it looks like this:

    <<test helpers>>
    <<path pattern>>

function format_context(context) {
    return !context ? format_cell(context)
	: Object.keys(context).map(function(key) {
	    return format_cell(key) + ' = ' + format_cell(context[key]);
	}).join(', ');
}

var directory = test_patterns.map(get_path_pattern_matcher);

function look_up_address(path) {
    for (var i = 0; i < directory.length; i++) {
	var matcher = directory[i];
	var match = matcher(path);
	if (match) {
	    return match;
	}
    }

    return null;
}

return [
    ["/dog", null],
    ["/songs", "/songs"],
    ["/songs/Help", "/songs/{song}", {song: "Help"}],
    ["/songs/Freebird", "/songs/{song}", {song: "Freebird"}],
    ["/songs/Freebird/listen", "/songs/{song}/listen", {song: "Freebird"}],
    ["/life/Animalia", "/life/{kingdom}", {kingdom: "Animalia"}],
    ["/life/Animalia/Chordata", "/life/{kingdom}/{phylum}", {kingdom: "Animalia", phylum: "Chordata"}],
    ["/chapter", null],
    ["/chapter-one", "/chapter-{chapter}", {chapter: "one"}]
    // nice
    //["/life/Animalia/Chordata/Mammalia", "/life/{kingdom}/{phylum}/{class}", {kingdom: "Animalia", phylum: "Chordata", "class": "Mammalia"}],
].reduce(function(acc, test_case) {
    var path = test_case[0];
    var expected_pattern = test_case[1];
    var expected_context = test_case[2];
    var expected = !expected_pattern ? null : {
	pattern: expected_pattern,
	context: expected_context || {}
    };
    var got = look_up_address(path);
    var pass = deep_equals(expected, got);
    var out_rows = [[
	format_cell(path),
	'matches',
	!expected ? '/(nothing)/' : format_cell(expected_pattern),
	'',
	pass ? '\\checkmark' : '\\xmark got ' +
	    (got ? format_cell(got.pattern) + ' ' + format_context(got.context) : '=null=')
    ]];
    if (expected_context) {
	var context_cells = Object.keys(expected_context).map(function(key, i) {
	    return (i == 0 ? '/with/ ' : '/and/ ') +
		format_cell(key) + ' = ' + 
		format_cell(expected_context[key]);
	});
	out_rows[0][3] = context_cells[0];
	out_rows = out_rows.concat(context_cells.slice(1).map(function(cell) {
	    return [' ', ' ', ' ', cell];
	}));
    }
    return acc.concat(out_rows);

}, []);

Of course, if the path doesn’t have any placeholders, then it only matches that one path exactly, and thus only represents a single location. This is useful to create a “branching out” point for a set of places.

A “change rule” is basically a record with three fields:

selector

the CSS selector saying what element(s) to target

verb

(or “method”), one of the supported DOM operations to apply to the target

argument

either a list of class names (for class changes) or an HTML template

Currently, getflow defines two kinds of changes: class changes, and template changes. Class changes are much simpler.

function ClassChangeRule(selector, verb, argument) {

    // DUPLICATED!!! in TemplateChangeRule

    // Collect the (unique) inputs that we use.
    var _inputs = {};
    function register_input(input) {
	_inputs[input] = 0;
    }

    // From selectors
    for_each(placeholders_in(selector), register_input);

    // From argument
    for_each(placeholders_in(argument), register_input);

    return {
	selector: selector,
	verb: verb,
	argument: argument,
	inputs: Object.keys(_inputs)
    };
}

Template rules are not as simple. Like class changes, templates support context evaluation, inside of (braces in) attribute values, as well as within the content of eval nodes. But “worse,” templates can have xslt nodes that represent the result of an XSL transform (whose input path also supports context evaluation). For this reason, template changes must be processed asynchronously, since it may be necessary to retrieve a number of files.

Templates are how you get lots of things from one rule. Context is how you tell a template where you are.

Getflow uses a simple template language.

Templates are XML fragments. Certain bits have special meaning to the template evaluator. When given a context, those places will be expanded.

• evaluation of content of eval elements
• evaluation inside of {} in attributes
• evaluation of xslt elements

Within those rules you can leave certain things undecided (“variable”).

Variables remain undecided until a choice is made, often by a person, while the program is running. So the exact way that things look won’t be decided when you write the code. You yourself won’t see it unless you “run the program” and, typically, make choices.

The above example shows a simple “fruit tree.” The split boxes show where we might use a class to consolidate “siblings” that can be described in a regular way. Whenever you have a common data structure representing a collection of things, these are well served by a combination of context and templates.

context

where you are now, usually a choice that the person has made

template

the shape of a type of thing, which will never be seen as such, but will be “filled out” based on a context, then shown to the person

Great, when can I start?

The context is bascially a key-value map from the routes. In practice, it’s an XPath expression where the route groups are elements and their values are text.

You can use context in a number of places. Currently, context evaluation is supported in

• selectors
• eval elements inside of templates
• within {} expressions in attributes in templates (including xslt inputs and arguments)

function Context(map) {
    // 'd' is a dummy root element
    var doc = document.implementation.createDocument('', 'd');
    for_each(Object.keys(map), key => {
	var ele = doc.createElement(key);
	ele.appendChild(doc.createTextNode(map[key]));
	doc.documentElement.appendChild(ele);
    });

    function evaluate(expression) {
	return doc.evaluate(expression, doc.documentElement, null,
			    XPATH_STRING, null)
	    .stringValue;
    }

    return {
	map: map,
	evaluate: evaluate,
	expand: function(text) {
	    return (text || '').replace(placeholder_pattern, function(_, expression) {
		return evaluate(expression);
	    });
	}
    };
}

And here it is in Python.

import re
from lxml import etree

class Context:

    placeholder = re.compile(r'{(.+?)}')

    def __init__(self, dictionary):
	self.dictionary = dictionary # only used for "change keys"
	self.node = etree.Element('dummy')
	for key, value in dictionary.items():
	    item = etree.Element(key)
	    item.text = value
	    self.node.append(item)

    def evaluate(self, expression):
	return self.node.xpath('string('+expression+')') if expression else ''

    def expand(self, text):
	return self.placeholder.sub(lambda mo: self.evaluate(mo.group(1)), text)

Now, here are the Python tests.

from getflow.context import Context
from test_helpers import check

def test_evaluate(case):
    expression, dictionary, expected = case
    check(case, expected, Context(dictionary).evaluate(expression))

def test_expand(case):
    text, dictionary, expected = case
    check(case, expected, Context(dictionary).expand(text))


print('------ evaluate:')
[test_evaluate(case) for case in (
    ('', {}, ''),
    ('', {'x': 'abc'}, ''),
    ('x', {'x': 'abc'}, 'abc'),
    ('y', {'x': 'abc'}, ''),
    ('y', {'x': 'abc', 'y': 'def'}, 'def'),
    ('xy', {'x': 'abc', 'y': 'def'}, ''),
    ('"pq"', {}, 'pq'),
    ('concat("pq", "qp")', {}, 'pqqp'),
    ('translate("hello", "aeiou", "AEIOU")', {}, 'hEllO'),
    ("translate('hello', 'aeiou', 'AEIOU')", {}, "hEllO"),
    ('concat(x, z)', {'x': 'abc', 'y': '123', 'z': '890'}, 'abc890'),
)]

print('\n------ expand:')
[test_expand(case) for case in (
    ('a{x}b', {}, 'ab'),
    ('a{x}b', {'x': '123'}, 'a123b'),
    ('a{x}b{x}', {'x': '123'}, 'a123b123'),
    ('a{x}b{y}', {'x': '123', 'y': '456'}, 'a123b456'),
    ('a{a}b{a}', {'a': '123'}, 'a123b123'),
    ('a{a}b{b}', {'a': '123', 'b': '456'}, 'a123b456'),
    ('a{abc}b', {'abc': '789'}, 'a789b'),
    ('({concat(x, z)})', {'x': 'ALPHA', 'y': 'BRAVO', 'z': 'ZED'}, '(ALPHAZED)'),
    ('({translate(x, "aeiou", "AEIOU")})', {'x': 'hello'}, '(hEllO)'),
    ("({translate(x, 'aeiou', 'AEIOU')})", {'x': 'hello'}, '(hEllO)'),
    ("({translate(x, '.', '-')})", {'x': '1.1'}, '(1-1)'),
)]

function promise_to_apply_bound_change(change) {
	return new Promise(resolve => {
		let i, stuff_to_insert, node;
		const context = change.context,
			  path = change.path,     // for marking inserts
			  keys = change.keys,
			  selector = change.selector,
			  verb = change.verb,
			  argument = change.argument,
			  // "Removal" selectors are constructed internally and may contain
			  // context expressions, which should be treated literally.
			  nodes = document.querySelectorAll(
				  verb == 'remove'? selector : context.expand(selector));

		// Could stop now if nothing matches.

		// Is this a template operation?
		if (/insert|prepend|append|before|after|html/.test(verb)) {

			// If this is a call from a prior iteration
			if (argument instanceof DocumentFragment)
				stuff_to_insert = argument;

			else {
				stuff_to_insert = change.rule.fragment.cloneNode(true);

				// Evaluate context in "eval" nodes
				for_each(stuff_to_insert.querySelectorAll('eval'),
						 eval_node => eval_node.parentNode.replaceChild(
							 eval_node.ownerDocument.createTextNode(
								 context.evaluate(markup_of(eval_node))),
							 eval_node));

				// Evaluate context in attributes.
				scan_attributes(stuff_to_insert, attribute => {
					const value = attribute.value;
					if (/\{.*\}/.test(value))
						attribute.value = context.expand(value)
				});
			}

			// Does this thing have transforms?
			const xslt_node = stuff_to_insert.querySelector('xslt');
			if (xslt_node) {

				// Yes, this thing has transforms.      Resolve one transform, and
				// call this routine back.
				const transform_file = xslt_node.getAttribute("transform");
				const document_file      = context.expand(xslt_node.getAttribute("input") || "");

				Promise.all([
					GET_XSLT(transform_file),
					GET_XML(document_file)])
					.then(results => {
						// Destructuring here would be nice, right?      But then
						// you get some stupid shim from the transpiler.
						var proc = results[0];
						var input_document = results[1];

						// Set transform arguments
						for_each(xslt_node.attributes, attribute => {
							if (!/^(transform|input)$/.test(attribute.nodeName))
								proc.setParameter(null,
												  attribute.nodeName,
												  context.expand(attribute.value));
						});

						xslt_node.parentNode.replaceChild(
							proc.transformToFragment(input_document, document),
							xslt_node);

						// When those things are both loaded, then call apply_change
						// with the reified template.
						resolve(promise_to_apply_bound_change({
							selector,
							verb,
							argument: stuff_to_insert,
							context,
							keys,
							path,
							rule: change.rule
						}));
					}, error => {
						throw error;
					});
				return;
			}

			// Now the template is "resolved" and ready to add to the document.
			// But first, since we know that we may want to undo this change
			// later (to back out of this path), we mark all of the top-level
			// elements being inserted.  Note that this won't work for plain
			// text nodes.
			for_each(children_of(stuff_to_insert),
					 top_level_element =>
					 for_each(Object.keys(keys), key =>
							  top_level_element.setAttribute('data-getflow-' + key,
															 keys[key])));
		}

		for (i = 0; i < nodes.length; i++) {
			node = nodes[i];

			if (is_class_verb(verb)) {
				var classes = context.expand(argument).split(' ');
				for (var j = 0; j < classes.length; j++) {      
					node.classList[verb.replace(CLASS_VERB_SUFFIX, '')](classes[j]);
				}
			} else if (verb == 'html') {
				while (node.firstChild) {
					node.firstChild.remove();
				}
				node.appendChild(stuff_to_insert);
			}
			else if (verb == 'remove')
				node.remove();

			else if (verb == 'append')
				node.appendChild(stuff_to_insert);

			else if (verb == 'before')
				node.parentNode.insertBefore(stuff_to_insert, node);

			else if (verb == 'after')
				node.parentNode.insertBefore(stuff_to_insert, node.nextSibling);

			else if (verb == 'prepend')
				node.insertBefore(stuff_to_insert, node.firstChild);

			else
				debug_message("getflow: unknown verb :" + verb);
		}

		resolve(true);
	});
}

Whew. That’s the gnarliest part of the client. It’s much more straightforward on the server, which is still in happy-synchronous land, reading files from the disk!

from lxml import etree

def inner_xml(node):
    return (node.text or '') + ''.join(etree.tostring(e, encoding='utf-8').decode('utf-8') for e in node)

# Treat rooted paths as filenames relative to the site root.
class SiteResolver(etree.Resolver):
    def resolve(self, url, pubid, context):
	return self.resolve_filename(url.lstrip('/'), context)


class Template:
    parser = etree.XMLParser()
    parser.resolvers.add( SiteResolver() )

    # Although the template is a "fragment," it should be pre-wrapped for (our)
    # convenience.  That is, the string accepted by the constructor should be a
    # well-formed XML element, whose outer tag will be ignored.
    def __init__(self, xml):
	self.xml = xml
	# Just for "change keys"
	self.inner = inner_xml(etree.fromstring(self.xml))

    def evaluate(self, context):
	clone = etree.fromstring(self.xml)

	# We must convert to a list because we're modifying the tree while
	# traversing it.  Since we just parsed the thing, consider an iterparse
	# based approach, as here http://stackoverflow.com/a/22495071
	for node in list(clone.iter()):

	    if node.xpath('boolean(@*[contains(., "{")])'):
		for key, value in node.attrib.items():
		    node.set(key, context.expand(value))

	    if node.tag == 'eval':
		node.text = context.evaluate(node.text)

	    elif node.tag == 'xslt':
		document = etree.parse(node.get('input'), self.parser)
		transform = etree.XSLT(etree.parse(node.get('transform'), self.parser))
		arguments = {k: etree.XSLT.strparam(v) for k, v in node.attrib.items()}

		result = transform(document, **arguments)
		node.clear()

		# Replace content.  See note.
		if result:
		    node.getparent().replace(
			node, etree.fromstring('<xslt>'+str(result)+'</xslt>'))

	etree.strip_tags(clone, 'eval', 'xslt')

	return clone

Well, there is one wrinkle. That “replace content” part is inefficient, assuming that the result tree was generated in-memory. But I can’t find any way to access the full result through the result tree, when the result is a fragment. This gets close, but it doesn’t include leading text:

first_result = transform(document, **arguments).getroot()
if first_result is not None:
    node.extend(first_result.itersiblings())
    node.insert(0, first_result)

The server version does have unit tests, unlike the client. I’m not sure these are worth it.

from getflow.templates import Template, inner_xml
from getflow.context import Context
from lxml import etree
from test_helpers import check

def do_test(case):
    fragment, context, expected = case
    check(case, expected,
	  inner_xml(Template('<r>'+fragment+'</r>').evaluate(Context(context))))

[do_test(case) for case in (
    ('<a/>', {}, '<a/>'),
    ('<a>text</a>', {}, '<a>text</a>'),
    ('<a><!--comment--></a>', {}, '<a><!--comment--></a>'),
    ('<a>text and <!--comment--></a>', {}, '<a>text and <!--comment--></a>'),
    ('<a><eval></eval></a>', {}, '<a></a>'),     # Should this throw?
    ('<a><eval>expr</eval></a>', {}, '<a></a>'), # Should this throw?
    ('<a><eval>expr</eval></a>', {'expr': 'value'}, '<a>value</a>'),
    ('<a x="{x}" y="{z}">text</a>', {'x': '123', 'z': 'def'}, '<a x="123" y="def">text</a>'),
    ('<a x="{x}"><eval>expr</eval></a>', {'expr': 'value', 'x': '123'}, '<a x="123">value</a>'),
    ('<a x="0{x}"><eval>expr</eval></a>', {'expr': 'value', 'x': '123'}, '<a x="0123">value</a>'),
    ('<a x="{x}"><b y="{z}"/></a>', {'z': 'q', 'x': '123'}, '<a x="123"><b y="q"/></a>'),
    ('head<a/>', {}, 'head<a/>'),
    ('<a/>tail', {}, '<a/>tail'),
    ('head<a/><b/>', {}, 'head<a/><b/>'),
)]

Back to the client…

Once we know the locations that lie between where we are and where we’re going, we can use the site definition (here called routes) to collect all of the change rules that will apply.

function reverse_class_verb(verb) {
	return verb[0] == 'a' ? 'remove_class' : 'add_class';
}

// The "change keys" will be used to identify anything that is inserted, so that
// it can be removed later.
function reverse_forward_change(rule, change_keys) {
    var verb = rule.verb;
    var selector = rule.selector;

    // TODO: reversing HTML changes is much more involved
    if (verb == 'html')
	return {
	    fragment: rule.fragment,  // TEMP
	    selector,
	    verb,
	    argument: ''        // TBD: lookup content in parent
	};

    // Class change
    if (is_class_verb(verb))
	return {
	    selector,
	    verb: reverse_class_verb(verb),
	    argument: rule.argument
	};

    var remover =
		Object.keys(change_keys).map(
			key => '[data-getflow-'+key+'="'+change_keys[key].replace(/"/g, '\\$&') + '"]')
		.join('');

    // Template change
    return {
	// Assumes that all nodes inserted by getflow were marked thus
	selector: remover,
	verb: 'remove'
    };
}

Regarding the escaping of quotes in attribute selectors, see “Strings”, CSS2 Specification.

That function is supposed to behave like this:

Change rules themselves cannot be applied directly. They have to be done in a context. Moreover, they can also done in reverse.

Before we can actually apply a change, we have to first:

1. (possibly) reverse it, i.e. turn it into an “undo” rule
2. resolve the rule in some context

Step 2 is an asynchronous operation, since it may require looking up external resources (not to mention other processing).

function BoundChange(rule, backwards, context, path) {
    var keys = get_bound_rule_keys(rule, context);
    var actual_change = backwards? reverse_forward_change(rule, keys) : rule;

    return {
	rule: rule,
	backwards: backwards,
	context: context,
	path: path,
	keys: keys,
	selector: actual_change.selector,
	verb: actual_change.verb,
	argument: actual_change.argument
    };
}

function reverse_class_verb(verb) {
	return verb[0] == 'a' ? 'remove_class' : 'add_class';
}

// The "change keys" will be used to identify anything that is inserted, so that
// it can be removed later.
function reverse_forward_change(rule, change_keys) {
    var verb = rule.verb;
    var selector = rule.selector;

    // TODO: reversing HTML changes is much more involved
    if (verb == 'html')
	return {
	    fragment: rule.fragment,  // TEMP
	    selector,
	    verb,
	    argument: ''        // TBD: lookup content in parent
	};

    // Class change
    if (is_class_verb(verb))
	return {
	    selector,
	    verb: reverse_class_verb(verb),
	    argument: rule.argument
	};

    var remover =
		Object.keys(change_keys).map(
			key => '[data-getflow-'+key+'="'+change_keys[key].replace(/"/g, '\\$&') + '"]')
		.join('');

    // Template change
    return {
	// Assumes that all nodes inserted by getflow were marked thus
	selector: remover,
	verb: 'remove'
    };
}

function changes_between(from_path, to_path, routes) {
    return steps_between(from_path, to_path).reduce((all, step) => {
		// Yeah a little destructuring here would be nice but stupid transpiler
		// emits some junk when you use it.
	const path = step[0];
	const backwards = step[1];
	const route_match = find_route(path, routes);
	const route = route_match[0];
	const match = route_match[1];
	const context = Context(match.context);

	const bound_changes =
			  route.change_rules.map(
				  rule => BoundChange(rule, backwards, context, path));

	// When going "backwards," we not only reverse the operations
	// themselves, but also the order in which we do them.  Note that this
	// applies only to this batch, i.e. the rules for one path step.
	if (backwards)
	    // MUTATION: JavaScript's Array.prototype.reverse() reverses the
	    // array *in-place*.
	    bound_changes.reverse();

	// This creates an intermediate (throwaway) array for each batch.
	return route? all.concat(bound_changes) : all;
    }, []);
}

This is what it’s all about.

As noted elsewhere, getflow is all about links. The one basic thing that getflow does is handle links. When someone indicates that they want to follow a link, getflow might be able to do its thing.

How do we do that? We have to listen. We listen for an “event” indicating that someone is trying to act on a link.²

document.addEventListener("click", the_person_touched_the_site);

TODO: the following notes are a little tangled up between the notion of stopPropagation and that the of the (currently implicit) false in the above listener. Some of this explains why we use bubble instead of capture, but it’s rather tied up with the other matter.

The astute reader may observe that, getflow not being the only piece of software in the world, it’s possible that even here on this page, some other piece of software may also be interested in “clicks” on links. So at this point, we have to ask, do we want first dibs on this click event? See, there are two methods of capturing events in the DOM. A pretty good explanation of this (with ASCII art!) is at Peter-Paul Koch’s QuirksMode’s article “Event Order” (from his book ppk on JavaScript) (http://www.quirksmode.org/js/events_order.html), (although note that notwithstanding the heading “Page last changed today”, this content is rather old and mostly of historical interest).

So we have the option to “capture” this event before lower-down elements (such as the link itself) see it, or we can be polite and let any such handlers go first, after which the event will “bubble” up to us. We’ll assume that a more specific handler (as such low-down handlers would be) knows more about the the intended outcome than we do. Besides, stopping propagation is just generally rude, as it will cause unexpected results. As Philip Walton puts it,

If you’re ever unsure about what to do, just ask yourself the following question: is it possible that some other code, either now or in the future, might want to know that this event happened? The answer is usually yes.³

We here at getflow don’t place a high premium on playing well with others, but we’re not sociopaths, either; we don’t go out of our way to be hard to live with. So we’ll be polite, strictly for the reason that someone else might not be; that is, some lower-level handler may want to respond to the event and then stop propagation.

function the_person_touched_the_site(event) {
    var link = event.target;

    <<shall we respond to this touch>>
    <<okay try to handle it>>
}

It all comes down to this. Give me an address, and I’ll take you there.

Note that this is a state-manipulating function.

<<simplify changes>>

function go(to, source) {
	const url = parse_url(to);
	const to_path = url.pathname;
    const from_path = state.path;

	// TODO: I'm going to get rid of one of the following methods.  If the
	// latter, then I'll need a PubSub here.  Note that although
	// window.dispatchEvent is used below, that's a special case, since it has
	// to be catchable before you know that getflow has loaded.
	if ('function' == typeof _on_going) {
		try {
			_on_going(from_path, to_path);
		}
		catch (e) {
			debug_message(e);
		}
	}
	// Another way of doing this that allows multiple dispatch without a
	// separate PubSub,
	if (Event)
		window.dispatchEvent(
			new CustomEvent(
				'getflow-going', { detail: { from_path, to_path, source, } }));

    function update_path_state() {
	state.path = to_path;

		// Popstate is usually triggered by the "back" button (but also by
		// history.back()).  Either way, the location is already in history.
		if (source != 'popstate')

			// state has non-serializable things in it now, triggering Firefox's
			// "cannot clone" error if you pass it here.
			window.history.pushState(url, '', to);
    }

	function go_there(blueprints) {

		const changes =
			  changes_between(from_path, to_path, blueprints);

		const promises =
			  simplify(changes)
			  .map(change =>
				   () => promise_to_apply_bound_change(change));

		return chain(promises)
			.then(() => maybe_scroll_to(url.hash))
			.then(update_path_state, debug_message);
	}

	return get_the_blueprints.then(go_there);
}

At the end of a state transition, the web site should always be in the state that it would have started in if you’d gone directly to that same address. And if the address included an “anchor” (or “hash”), then a specific part of the document is supposed to be scrolled to the top.

So after the DOM-manipulating is done, there may still be more to do.

<<scroll to>>

function maybe_scroll_to(hash) {
	let id = hash.slice(1), target;

	if (id && (target = document.getElementById(id)))
		return scroll_to(target, 500);

	return resolve_now();
}

Since the whole point of getflow is to provide fluid transitions, it’s assumed that the scroll should be “smooth” rather than abrupt.

Indeed, if not for that, this would be a one-liner.

target.scrollIntoView();

So it’s not altogether surprising that smooth scrolling is coming to the platform.⁶

What good does that do here? The scroll-behavior: smooth property will only have any “natural” effect if the user clicks a link whose anchor points to the same page. But getflow has to deal with the scrolling “long” after the initial click. CSS isn’t going to help here.

Fortunately, there’s an API.⁷ When it’s not available, a custom animation is used.

function scroll_to(target, duration) {
	let layer = target, is_main_layer;

	// Force abrupt jump if signaled by a special attribute.
	if (target.getAttribute('getflow-scroll-behavior') == 'auto') {
		target.scrollIntoView();
		return resolve_now();
	}

	while (layer = layer.parentElement) {
		is_main_layer = layer == document.documentElement;

		if (is_main_layer
			|| window.getComputedStyle(layer).overflowY == 'scroll')
			break;
	}

	// Best method: Smooth scrolling API.
	if (layer.scrollBy) {
		layer.scrollBy({
			behavior: 'smooth',
			top: target.getBoundingClientRect().top});

		// Worst method: Abrupt jump to the element if we can't animate.
	} else if (!queue_frame) {
		target.scrollIntoView();

		// Compromise: Roll your own animation.
	} else return animate(duration, ratio => {
		const change = ratio * target.getBoundingClientRect().top;

		is_main_layer?
			window.scroll(0, window.scrollY + change)
			: layer.scrollTop += change;

		return true;
	});

	return resolve_now();
}

Regarding the ambiguity about which layer is the “main layer,” the browsers have not agreed on whether document.body or document.documentElement is supposed to be the main scrolling container. As near as I can tell, the “spec” indicates document.documentElement [citation needed]. But Chrome doesn’t play along (and Safari, too, AFAIK). Rather than trying to detect which browser we’re running in, we write this in a “cross-browser” way. That means treating the “main document” as a special case. (The browsers work the same way for scrolling child elements).

Further reading:

• https://miketaylr.com/posts/2014/11/document-body-scrollTop.html
• https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/75hVThfrJ08/YD9kSgQljFEJ
• https://code.google.com/p/chromium/issues/detail?id=345592

About that getflow-scroll-behavior hack. Some transitions can only be made seamless by applying CSS properties and scrolling more or less instantly. In such cases, of course, you don’t want to animate the scrolling. But how do you know when it’s one of those cases? The most “semantic” way would be, perhaps, to detect scroll-behavior auto. But that property won’t even be reported by browsers that don’t support scroll-behavior in the first place. Besides, any CSS class or property on the layer would be problematic, since, by the time you get to this point, the DOM changes have already been applied. So if you want smooth scrolling sometimes, or only for some transitions, how would you get the element in the desired state at that point? Whereas this method allows you to use an alternate anchor for transitions in which you know that scrolling should be bypassed. Yes, it’s a hack. Email me a better idea and I’ll change it.

Like it or not, we will have internal state that we need to keep track of. In particular, we need to maintain a “current path” that is separate from the browser’s “actual” current path (given by window.location), because, unlike in the standard GET model, we don’t change location all at once.

// TODO: why do you decode the search here?
// normalizePathname(window.location.pathname) +
//         decodeURIComponent(window.location.search)

var state = {
    path: normalizePathname(window.location.pathname) +
	decodeURIComponent(window.location.search)
};

If this site is not being run over HTTP, but over a file system, then the initial path will be assumed as the root. This is only needed for the Android app, where, inside of Cordova, window.location.pathname gives us some implementation-specific nonsense (see “fix paths” plugin for “the app”).

if (window.location.protocol == 'file:')
	state.path = '/';

Between any two locations in the site, there is one way to go—at least in getflow’s view. These functions calculate that route. We know that every location has one direct route to the “top” (or “root”, or “home”), namely by hacking off pieces until there’s nothing left. The route between two locations, then, is to go “upwards” from where you are until to reach a path that is in your destination’s ancestry, which in the “worst case”, will be the very top of the site.

Here’s how we expect path traversal to behave.

In finding the shortest path between two places, it helps to think of each location as seeing a ray from home.

Of course, the place we’re going also connects to home this way:

    <<steps from home>>

// The route will go "backwards" from the starting point, and, when it hits the
// ray pointing to the destination, it will get on board.
function steps_between(A, B) {
    // These are the "rays" from home.
    var home_to_A = steps_from_home_to(A);
    var home_to_B = steps_from_home_to(B);

    // How many steps are in common?
    var common = 0;
    while (common < home_to_A.length
	   && common < home_to_B.length
	   && home_to_A[common] === home_to_B[common]) {
	common = common + 1;
    }

    // Get the two legs of the route
    var steps_to_common = home_to_A.slice(common).reverse();
    var steps_from_common = home_to_B.slice(common);

    return steps_to_common.map(step => [step, true])
      .concat(
	steps_from_common.map(step => [step, false])
      );
}

This function behaves as follows:

What is the data structure for path traversal?

It must depend on the data structure that you use to define the edges, the shape of the thing. But that’s based on the our (implicit) model.

You might say, it simply lists the paths that you travel through. That’s the way that steps_between has worked so far.

But that doesn’t take into account edges. In other words, that takes for granted that there exists an edge between each of the nodes. But I think that that’s what the function is asserting—that you can travel between those nodes.

So the question is, is there a difference between the way we’d express the path if we had to be agnostic of where the edges lay (i.e., it might not be a tree), versus how we’d express it if it were a given that traversal were always up/down in a tree?

Surely, we could express things differently in the latter case. The result would be a list of tuples: location and direction (up or down). The latter bit is necessary in that you have to know whether the changes are to be reversed or not, but that again is specific to the method of traversal, where you know that you’re constructing each path as a set of changes to its parent. Of course, we have to cross the line into those assumptions at some point, the question is, where?

Also, related, should/can the return value assume that you know where you are to start with? Suppose it were a series of relative steps. Given that the caller must know what the start node was, is there any semantic difference or otherwise anything clearer about expressing the path not as having two absolute endpoints? Indeed, can you be agnostic of the starting point? I don’t think so, because the changes you’re reversing will be context-dependent. In other words, the path has to be in effect reversible itself, even though you won’t be traveling it in reverse.

The way that getflow works in practice is that you apply the router to each path in order to figure out what change rules to apply. But when you’ve “zipped” the steps, so that you have pairs, there’s an irregularity to it. You apply the router to the “deeper” path. Whereas, if you had just the location/direction pairs, you’d never actually need to reference the common path (the junction) explicitly. In other words, to go from /a to /b you would say reverse /a and forward /b, without ever mentioning /. Does that make sense? Ultimately it may, given that we often “cancel out” rules for lateral steps against the root. But that is optional, and there is a sense in which / is undisputably a path on the way from /a to /b (as we compute it).

Are the semantics of this data structure going to be helpful here, then?

The way that we go about implementing the transition takes some things for granted.

At this point, we could argue that getflow is “correct.” It will get you from point A to point B, making all of the changes indicated by the blueprints.

But we can do better. Insert story about my first day as a bike courier.

Sometimes the “shortest route” between two points can be made even shorter with a little off-roadin’. Consider once again the matter of going between two sibling places.

Yes, this is the shortest “official” route. But in some cases—especially cases like this one—you may be able to skip some of those changes.

How? Consider the gray node. It has three children. Remember that you can use placeholders to describe many places using one set of rules. Let’s suppose that the rules for gray’s children look like this:

1. prepare to make a colorful place
2. set the color to {color}

When an actual path is chosen, the “color” placeholder gets filled in, and that’s what allows the “red” and “blue” nodes to look different while sharing the same template.

But note that the first rule doesn’t use a placeholder for color. That means that it’s the same exact bit of work for all of the children. So going from red to blue will look like this:

1. unset the color to red
2. unprepare to make a colorful place
3. prepare to make a colorful place
4. set the color to blue

Look at changes 2 and 3. What a waste of work! Why should we undo something only to immediately do it again! Instead, we really could just do this:

1. unset the color to red
2. set the color to blue

This would give us kind of a “shortcut” between those places:

So before we actually apply the changes that we’ve collected, we’re first going to identify changes that cancel each other out (or can otherwise be skipped), and get rid of them!

We’ll start by scanning through each of the changes. This setup allows us to traverse the list while remembering the indices of any change that we know we want to exclude.

<<compare changes>>

function simplify(changes) {
    var i, j, change, other, skips = [];

    for (i = 0; i < changes.length; i++) {
	change = changes[i];
	<<maybe skip this change>>
    }

    return changes.filter((_, i) => skips.indexOf(i) < 0);
}

For each change, we’ll apply the following logic.

First, duplicate “idempotent” changes will be removed. Class changes, specifically, can be applied over and over again with no further effect.

But I use scare quotes because we don’t check that the changes are successive, so intervening changes could impact which elements are matched, not to mention the content of the document. In fact, even the change itself could do so, if a class selector is being used. So this is not “safe” at all.

let rule = change.rule,
	verb = rule.verb;
if (is_class_verb(verb)) {
	let reverse_verb = reverse_class_verb(verb);

	for (j = i + 1; j < changes.length; j++) {
		other = changes[j];
		let other_rule = other.rule,
			other_verb = other_rule.verb;
		if (change.rule.selector == other_rule.selector
		   && change.keys.context == other.keys.context) {

			if (change.rule.argument == other_rule.argument
				&& (verb == other_verb
					|| (change.backwards != other.backwards
						&& other_verb == reverse_verb)))
				skips.push(j);

			break;
		}
	}
}

I admit that that’s a rather horrid way to do something incorrect. See the test called “toggling state” to see why this is needed to cope with the following incorrect shortcut in some cases.

We also “wash out” changes that reverse each other. This, too, is done very loosely.

// TODO: should also check for relativedepth = 0
if (change.backwards) {
    for (j = i + 1; j < changes.length; j++) {
	other = changes[j];
	if (!other.backwards &&
	    changes_match(change, other)) {
	    skips.push(i, j);
	    break;
	}
    }
}

When you travel, you build up a history of the places you’ve been.

And in a web browser, your history is treated as a stack.

A stack is what it sounds like: a bunch of things on top of one another. But pretend that you can’t touch the middle of the stack—only the top. If you’ve ever seen those spring-loaded plate dispensers in a restaurant kitchen, it helps visualize how a stack structure works. Pushing onto the stack means adding a new item, which always goes on top. Popping off of the stack, means removing an item, which is always from the top. You don’t get to lower-down things without first going through the upper (i.e. more recent) ones. This is also known as last-in, first-out access.

Browsers treat history as a stack because it’s close to how people experience exploration. You take a sequence of steps, and you can retrace those steps. When you go to a new place, it gets pushed onto the end of the list. When you go back, the place you’re leaving is popped off the end of the list, so that your next-most-recent place becomes your new current place.

In the old model, web sites generally didn’t have to worry about history, because the browser handled it. History was a freebie. But in order to take advantage of AJAX to create fluid spaces—whose states are still addressable—you need programmatic control of the path. In other words, your web site has to decide what forward and back mean, and make them happen.

The History API (one of the first HTML5 features to be widely implemented) uses pushState (something the site can do) and onpopstate (something the user triggers by going “back”). Again, when you use the history API, it becomes your responsibility to make it work.

All of this to say, that if you use pushState (to go forward), you must handle the “pop state” event, which occurs when the user goes “back” (or when history.back() is called).

window.addEventListener('popstate', go_back);

So, what does it mean to “go back”?

Well, by the time the pop state handler is called, the document’s location has already changed. So it’s just a matter of telling getflow to go to the location that the browser already thinks is current.

function go_back() {
	go(document.location.href, "popstate");
}

See the note in update_path_state about the need for the source argument.

Finally, we need to let the world know that we’re here! In truth, this is just for the system tests.

// Some browsers (that *always* means IE) don't support custom events using
// Event.
if (Event) {
	window.dispatchEvent(new Event('getflow-ready'));
}

The GET part of getflow is only done on the server.

from ..getflow.routes import make_path_pattern_matcher, look_up_address
from ..getflow.paths import hack_path
from ..getflow.plans import read_plans
from ..getflow.context import Context
from ..getflow.selectors import css_selector_to_xpath
from ..getflow.cache import Cache
from lxml import html
from io import StringIO
import os.path

# Poor man's "import as", which I'm avoiding because the code is actually
# concatenated.
route_matcher = make_path_pattern_matcher

class Site:
    def __init__(self, plans):
	self.directory = [route_matcher(rule['route']) for rule in plans]
	self.route_rules = {the['route']: the['change_rules'] for the in plans}

PLANS_FILE = 'getflow.xml'
site_cache = Cache(
    lambda: (Site(read_plans(PLANS_FILE))
	     if os.path.exists(PLANS_FILE) else None),
    lambda: (os.path.getmtime(PLANS_FILE)
	     if os.path.exists(PLANS_FILE) else 0))

def read_content(filename):
    with open(filename) as the_file:
	return the_file.read()

INDEX_FILE = 'index.html'
index_html_cache = Cache(
    lambda: read_content(INDEX_FILE),
    lambda: os.path.getmtime(INDEX_FILE))

def apply_changes(doc, rules, context):
    for selector, change in rules:
	for node in doc.xpath(css_selector_to_xpath(context.expand(selector))):
	    # Selector is only needed for "change keys"
	    change.apply_to(node, context, selector)

    return doc