the mobile app

This document describes why and how we create an app—yes, an app—for willshake.

TL;DR: it lets you use willshake offline.

1 rationale

I believe in the web. I “want the web to win."1 Willshake is mainly a web site. Long live web sites!

And yet—there’s something about web sites… what is it?

web_requires_internet.svg

Ah, now I remember. Web sites require the Internet.

No matter how small your web site, it requires an Internet. It’s like Joe Armstrong said,

You wanted a banana but what you got was a gorilla holding the banana and the entire jungle.2

Whether you’re visiting willshake.net, or the Library of Congress, or reading the menu of the nearest pizzeria, using the web means summoning the largest continuously-operating machine that humankind has ever created—that’s right, the Internet3—and set in motion a “dizzying” process that nobody quite understands anymore.4 Bon apetit!

Of course, some web sites really need the Internet, at least to accomplish what they’re attempting, which may call for the real-time coordination of people and events all around the world. But many web sites don’t require that, and could be used offline. Alas, most “modern” web sites wouldn’t work offline, anyway, since they need the server to do special things that your computer isn’t set up to do. But in the “old days,” a web site was simply a directory of files, and no special processing was required. In that case, you could just “get” the entire web site (a snapshot of it, anyway) and use it later, even offline. Sure, it might take a while to download, but once it was finished, you would have instant access to the whole site with no network required.

In fact, we do have ways of doing this. The wget command will recursively download a location that you specify and (potentially) all of the other resources that it references. But this is already more than a “mainstream” operation, and the site still may not work locally (depending on whether the links could be rewritten correctly (I’m assuming wget does that)).

Another way is the “app manifest.” This is kind of a TODO item for me because I’ve read enough to know that it’s relevant here (you can tell supporting browsers, for instance, which files to download for offline use), but not enough to know what its particular issues are. (Update, it’s the W3C “App Package” (a.k.a. “widget”) specification, and only Firefox really implements it. Chrome has its own similar (but different) thing.)

Bottom line, willshake is designed to support offline use, simply because it doesn’t need real-time global coordination of any kind, nor any special processing beyond what can be done in a browser. And “mobile apps” are one way to provide that for “ordinary people” (i.e. people who will die before they ever type wget).

2 Cordova

Cordova is a program maintained by the Apache Foundation (although originally donated to them by Adobe(?) as a fork of the PhoneGap project). It is designed to solve this very problem: we’ve already made this web site, now we want to package it as an app.

The promise of Cordova is “cross-platform” development. What platforms do they promise? Here’s a list of supported platforms from their official documentation:5

  • Amazon FireOS
  • Android
  • Blackberry
  • Firefox OS
  • iOS
  • Ubuntu
  • Windows Phone
  • Tizen
what-cordova-doesnt-promise.svg
Figure 2: What Cordova doesn’t promise

Making the Cordova app can be considered in two stages:

  1. one-time setup
  2. incremental build (each time you change something)

2.1 setup

The one-time part of the setup is what makes this a little messy, because it mostly has to be done “out of band.”

2.1.1 get Cordova

Cordova is distributed as a node.js package, so getting it is “easy” once you have npm. I will spare both of us the details of that for right now.

2.1.2 create a Cordova project

As with most subjects, the matter of building will require us to talk about Tup. Ideally, we’d do everything through Tup, so that all you need to do is get a copy of the program (willshake), and type tup in the directory, and you’re done.

But that ain’t gonna happen. There’s no way that you can create a Cordova project from within Tup. Cordova projects have far too many ins and outs to track as a rule. Just initializing a Cordova project creates tons of files, none of which do I care about; yet Tup would require me to list each and every one explicitly (and then it would probably fail for some obscure reason related to FUSE). We’ve reached the “clean subset of the build” stage.6

Fortunately, this is not a big deal, since I don’t need to rebuild the app on every single change. Indeed, because Cordova’s build system is so vastly inferior to Tup, it would be completely impractical to do so, anyway.

So, the build is done “off tree.” Where exactly? By default, it’ll go into a directory next to the project called cordova-{variant}, where variant is the name of the project directory. This prevents it from conflicting with another (sibling) copy of the project that also happens to be building the app.

project_dir=`pwd`
variant="${project_dir##*/}"
echo "../cordova-$variant"

Again, creating the project is a “one-time” thing. All of the up-front setup is done in one script, create_cordova_project.

# For the moment, this should be run from the project's root directory.
default_directory="$(app/default_directory)"
app_dir="${1:-$default_directory}"
project_dir=`pwd`

echo "Create Cordova project in $app_dir"

mkdir -p "$app_dir"
cd "$app_dir"

<<create empty project>>
<<add platforms>>
<<add plugins>>

The main business of the script is to run cordova create. It wants to know the location of the project (which is here), its identifier (which is important), and its name.

cordova create . net.willshake.app willshake

New Cordova projects are not “empty”; they are initialized with some basic code for testing. If you want to make sure that Cordova is working, you can run cordova build android in the directory of a new Cordova project and get the “Hello Cordova” app. But willshake provides all of its own content, so that default content needs to be cleared.

I was originally using the Crosswalk plugin, which basically bakes an entire Chromium runtime into the app in order to replace the default web view. If you don’t do that, you end up getting whatever stock browser the user has on their system. A couple of years ago, that was a problem even for basic usage, especially CSS3. Now I’d have to go out of my way to tell the difference. Meanwhile, Crosswalk adds a good 20MB to the APK. So I’m disabling this for the initial release.

cordova plugin add cordova-plugin-crosswalk-webview

2.1.3 configuration

Several important bits about your app can be set through the configuration file, config.xml. If you don’t specify one, you’ll use the default, which will be incorrect in some cases.

<widget id="net.willshake.app" version="0.0.1"
				xmlns="http://www.w3.org/ns/widgets">
	<name>willshake</name>
	<description>
		The beauty and pleasure of Shakespeare for new media
	</description>
	<author email="contact@willshake.net" href="https://willshake.net">
		willshake developer
	</author>
	<content src="index.html" />
	<plugin name="cordova-plugin-whitelist" spec="1" />
	<access origin="*" />
	<allow-intent href="http://*/*" />
	<allow-intent href="https://*/*" />
	<allow-intent href="mailto:*" />
</widget>

The “whitelist” plugin lets you specify what hosts should be allowed to provide content. The whole purpose of this app is to prevent the need for the internet, and the basic version must work with no internet permission whatsoever.

cp "$project_dir/app/config.xml" .

2.1.4 add Android support

Android is a discussion for another time. Bottom line, you need (1) Java, (2) the Android SDK. It’s actually not too horribly bad, unless you measure horrible badness in gigabytes.

cordova platforms add android

2.1.5 Android problems

You’d think that willshake were a textbook case for Cordova. It’s designed for standalone use with nothing but a static file server.

And yet, willshake does not work with Cordova out-of-the-box.

There are two problems, both having to do with addresses.

deal with rooted paths

An app, of course, is not a web site. Locations within an app don’t have “addresses” the way that web sites do—or at least, the way that web sites should. In recent years, many of the “apps” that happen to live on the web are created without any regard to addresses whatsoever, the so-called “single page applications” (SPA’s). You can sit around all day playing with a “single-page” app without the address in your address bar ever changing. This view of a “web app” is well suited to a system like Cordova, where there are, again, no addresses.

However, if you do actually use addresses, then you understand that references have to be portable. When you’re making a link to something, you don’t know, can’t know, and don’t want to know where it will be used. The simplest way to deal with this is to “root” all of your paths, that is, give the full path to everything. For example, instead of referring to style.css, you refer to /style.css. For this to work, you have to assume that site lives at the root of the domain, so the tradeoff is that the site itself is no longer portable (e.g. to a subdirectory).

But this leads to a subtle problem when packaging the site for Cordova, although I confess that I don’t quite understand why, as the workaround is quite simple and not the least bit astonishing. Cordova “serves” files from a static location using a file:// protocol with some arbitrary (and platform-specific) directory prefix. Rooted paths, then, are resolved to a nonsensical location instead of being resolved relative to the location of the index file, as any human would expect.

A few people noticed this problem, starting around 20127 , 8 , 9 , 10 , 11 , 12, observing that “the problem crops up in those deeper pages” and asking, “But what about a file located deep in my directory structure?”

Clearly, I don’t think much of the suggestion that one just “always use relative paths."13 Huh? You always know where that code will be used? Changing willshake itself to use relative paths is a complete non-starter. Thus, we have to make Cordova work with willshake as it is.

Regarding the Android assets folder.

Files that you save here are compiled into an .apk file as-is, and the original filename is preserved. You can navigate this directory in the same way as a typical file system using URIs and read files as a stream of bytes using the AssetManager.14

ASCII assets

The other problem is that Android is very restrictive about the characters allowed in “asset” filenames. And since assets are basically a file system that is accessible via URI, it’s a natural fit for apps that emulate a web server.

But if you have any “funny” characters in any filename, the Android build won’t allow you to add it as an asset, saying, “Invalid filename: Unable to add.” It turns out that

Android requires ASCII filenames for assets. This doesn’t appear to be publicly documented anywhere, but the `aapt` source code has an ASCII check15

So the file Chassériau-Mac-1.3.jpg, for example, would not be allowed as an asset in Android, on account of that pesky é. Which I find surprising, since Android is supposedly the most popular operating system in the world. What if your app is Greek or Japanese? Doesn’t matter. Your asset names will be ASCII.16 , 17 , 18

One way to deal with this would be to avoid having such filenames in the first place. Well, the last time these things came under threat, it set in motion a process that led to my ultimately porting willshake to another operating system.19 So, no, that’s a non-starter.

Instead, I’ll leave the site untouched, and adjust the app so that it will still work. This will require a two-part solution:

  1. Change the filenames in some predictable way during the build.
  2. Reroute requests in the app, using the same logic.
solution

For both problems, the solution must be to intercept and rewrite all network requests. Fortunately, this can be solved fairly straightforwardly with a Cordova plugin, which has a virtual remapURI method.

package net.willshake;

import android.net.Uri;
import org.apache.cordova.CordovaPlugin;

import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;

public class FixPaths extends CordovaPlugin {

	@Override
	public Uri remapUri(Uri uri) {
		if (uri != null) {
			String path = uri.toString();
			<<fix path>>
		}

		return null;
	}

	<<make ascii path>>
}

Java has an amazing amount of noise. This is the essence of the function.

final String FILE_PROTOCOL = "file://";
final String ASSETS_PATH = "/android_asset/www";

if (path.startsWith(FILE_PROTOCOL)) {
	String site_path = path.substring(FILE_PROTOCOL.length());

	if (!site_path.startsWith(ASSETS_PATH)) {

		if (site_path.startsWith("/static/images")) {
			site_path = make_ascii(site_path);
		}

		return Uri.parse(FILE_PROTOCOL + ASSETS_PATH + site_path);
	}
}

The path munging function is basically percent-encoding.

static String make_ascii(String path) {
	final int slash = path.lastIndexOf("/");
	if (slash < 0 )
		return path;
	final String dir = path.substring(0, slash + 1);
	final String file = path.substring(slash + 1);
	try {
		return dir + URLEncoder.encode(file, "UTF-8").replace('%', '_');
	} catch (UnsupportedEncodingException __) {
		return path;
	}
}

That’s it, except for the required manifest. Note that

Plugins are not instantiated until they are first referenced by a call from JavaScript, unless <param> with an onload name attribute is set to “true” in config.xml.20

This is important, since the plugin doesn’t have any JavaScript component.

<?xml version="1.0" encoding="utf-8"?>
<plugin xmlns="http://apache.org/cordova/ns/plugins/1.0"
	id="net.willshake.FixPaths"
	version="1.0.0">
  <name>fix paths</name>
  <description>thing to deal with stupid cordova paths</description>
  <keywords>paths, stupid</keywords>
  <license>Apache 2.0</license>
  <platform name="android">
    <source-file src="FixPaths.java" target-dir="src/net/willshake" />
    <config-file target="config.xml" parent="/*">
      <feature name="FixPaths">
	<param name="android-package" value="net.willshake.FixPaths" />
	<param name="onload" value="true" />
      </feature>
    </config-file>
  </platform>
</plugin>

Now we just add the plugin during the project setup.

cordova plugins add "$project_dir/app/fix-paths"

2.2 build

Once the project is set up, you shouldn’t have to revisit those steps.

If you really have a static web site, there’s not much to using Cordova. Basically, we want to copy our site directory into the Cordova project’s www folder.

It’s your job to be sure that the build is up-to-date before running this. Normally, tup monitor will be running during development, which will update both the site and this script.

# Should be run from project root

if [ ! -d ".tup" ]; then
    echo "abort: This is supposed to be run from the project root."
    exit 1
fi

site="site"
app="$(app/default_directory)"
app_site="$app/www"
images="static/images"

# Mirror it to app
rsync \
    --recursive \
    --times \
    --copy-links \
    --verbose \
    --force --delete \
    --exclude="$images" \
    "$site"/* "$app_site"

app_images="$app_site/$images"
mkdir -p "$app_images"

app/link_images "$site/$images" "$app_images"

The link_images business deals with the ASCII assets problem.

import sys, os
from urllib import parse

(_, in_dir, out_dir) = sys.argv

in_dir = os.path.abspath(in_dir)

for name in os.listdir(in_dir):
    link = out_dir + '/' + parse.quote(name).replace('%', '_')
    if not os.path.islink(link):
	os.symlink(in_dir + '/' + name, link)

At the moment, this is ridiculously fast. Still, I wonder if we couldn’t just symlink the directory. Cordova definitely shouldn’t be writing those files, just copying them into the platform directories.

2.3 app icons

Android apps are supposed to have icons in various sizes for different displays. Table 1 lists the names and sizes of the expected icon resources.

Table 1: Android icon resource sizes
hdpi 72
ldpi 36
mdpi 48
xhdpi 96

Assuming you have (dummy) files in the form {name}.{size}, the icons can be created using a build rule.

: foreach $(ROOT)/app/icon-size/*.* \
  | $(ROOT)/assets/images/logos/logo.svg \
|> ^ make app icon %b^ \
   inkscape \
    --without-gui \
    --export-png "%o" \
    --export-width %e \
    --export-height %e \
    --export-background-opacity 0 \
    $(ROOT)/assets/images/logos/logo.svg \
|> $(ROOT)/app/icons/%B/icon.png

Normally I’d reach for ImageMagick to do this kind of thing. But I had the same experience as others, that it did not do well at all for this purpose, while Inkscape worked perfectly.21

Those files need to be in the corresponding locations of the res (resources) folder.

cp -r app/icons/drawable-* "$app/platforms/android/res/"

This replaces the default images provided by Cordova.

3 releasing

You can do all of the above and still never release an app. Believe me.

3.1 signing the app

If you want to distribute the app through the Google Play, store, you need to “sign” it with a cryptographic key.22

This is a one-time operation. I’m just putting it here for reference.

keytool -genkey -v \
		-keystore gavinpc-release.keystore \
		-alias gavinpc_key \
		-keyalg RSA \
		-keysize 2048 \
		-validity 10000

Unless I’m very wrong, it doesn’t represent any risk to me for this information to be public.

jarsigner -verbose \
		  -sigalg SHA1withRSA \
		  -digestalg SHA \
		  -keystore ~/ws/android/gavinpc-release.keystore \
		  ~/ws/cordova/platforms/android/build/outputs/apk/android-release-unsigned.apk gavinpc_key

Note that this has to be done on a “release” build. The debug builds are already signed with a key that is only useful for development (as I understand it).

4 roadmap

4.1 TODO determine whether there’s any use in making an app manifest

In any of the existing formats. It sounds like a small effort to get some possible benefits (persistent caching) in some cases (i.e. browsers).

4.2 could this be done without Cordova?

Cordova is a very convenient hairball.23 As shown here, it allows you to build an Android app using very little code—that is, if you’re not counting Cordova itself. And it also shields you from the volatility of the Android SDK, which is a win if Cordova itself is less volatile.

But how much of Cordova does willshake actually use? What would it take to build the Android app without it?

I don’t know. I know that Cordova is doing more than I give it credit for, without having studied it. Even if you’re not using client-side plugins (which willshake doesn’t), being able to, e.g. use Crosswalk (an even bigger hairball) with just one line of code is far worth the dependency as long as the focus is on the web site, which may be forever.

But being full of the kind of self-sabotaging curiosity that has waylaid this project for so many years, I had to at least scratch the surface.

First of all, if you’re not building an Android app that’s based on a WebView, then I would highly recommend starting with a minimal approach if only for pedagogical reasons. In only twenty short steps, you, too can build an Android program without any “build system,” such as Ant or Gradle, which usually have their own learning curve that has nothing to do with the system being built. I agree with the author of one such tutorial that

there are several reasons why you could benefit from utilizing simple command line utilities and build scripts instead. Actually understanding what’s going on is one of them.24

Even more interesting is a little unmarked project I found, which purportedly builds an Android APK using Tup.25 You heard me right.

And now that the stock browser is up to snuff, meaning the app can run without Crosswalk in most places on Earth, it’s all the more interesting to note how little is needed to make a functioning WebView-based app, compared to the huge operation that Cordova has turned into.26 (Warning, though, in reference to the issues people have reported when using WebView with hardware acceleration—which would be tempting—I can attest that those bugs have affected me (on other apps) prior to using Crosswalk.27)

Between these three resources, I expect to spend an idle day (in 2017) testing whether the APK build can in fact be done solely through willshake’s system. Why? Because the current build is very slow and processor-intensive. And because, like I said, I’m a massochist.

Footnotes:

1

James Long, “Radical Statements about the Mobile Web,” (2015) http://jlongster.com/Radical-Statements-about-the-Mobile-Web, which he “sort of regrets posting” http://jlongster.com/Stop-Trying-to-Catch-Mef

2

From Coders at Work, by Peter Siebel. Great book, by the way. And yes, Joe was talking about “object-oriented languages,” but it’s even more extreme (and still true) in this instance. http://www.codersatwork.com/

4

Jean-Baptiste Quéru, “Dizzying but invisible depth”, Google Plus (2011) https://plus.google.com/+JeanBaptisteQueru/posts/dfydM2Cnepe

I can’t share this post enough.

5

“Platform Support,” Apache Cordova Documentation https://cordova.apache.org/docs/en/4.0.0/guide_support_index.md.html

6

Tup insanely fast build system,” Reddit, programming (2013)

See also the discussion “recursive foreach,” which covers ways to apply Tup rules to directory trees. This could be relevant to the update (and other cases), though not the Cordova directory itself.

7

“Absolute paths in PhoneGap” StackOverflow (Jan 18, 2012) http://stackoverflow.com/q/8909577

8

“PhoneGap and root url paths for assets” StackOverflow (Jan 10, 2012) http://stackoverflow.com/q/8811263

9

“PhoneGap on iOS with absolute path URLs for assets?” StackOverflow (Jan 31, 2012) http://stackoverflow.com/q/9073953

10

“Phonegap: how to use absolute paths on both Android, iOS and browser?” StackOverflow (Apr 2, 2012) http://stackoverflow.com/q/9969451

11

“Does Phonegap support root relative path? What are the best practices?” StackOverflow (Jul 30, 2014) http://stackoverflow.com/q/25036288

13

The Jan 10, 2012 question was also posted to Google Groups, where it got somewhat different replies. https://groups.google.com/d/msg/phonegap/xyY7Y_yvRhA/qyDQAqiH3a4J

14

Managing Projects Overview”, Android Developers documentation

15

“Can’t deploy Unicode Assets.txt,” Xamarin Bugzilla, Bug 5017 https://bugzilla.xamarin.com/show_bug.cgi?id=5017#c5

16

Unicode Folder names in Asset folder cause bad build”, Issue 9935 for “Android Open Source Project” (Posted July 2010, re-confirmed May 2015).

17

“cordova build fails in packaging resources with German Umlauts” Apache Cordova Jira issue CB-8393 (February, 2015, open an no replies as of December) https://issues.apache.org/jira/browse/CB-8393

18

Some dispute about the reason for this at “Mono for Android: Unicode Assets file names can’t be packaged” (May 10, 2012) http://stackoverflow.com/q/10543713

19

“Inconsistent decoding of filename on Windows” Tup users group, April 22, 2015. (The permalink thing isn’t working right now.) https://groups.google.com/forum/embed/#!topic/tup-users/PVHzhGQc7fw

20

“Android Plugins,” Apache Cordova Documentation http://cordova.apache.org/docs/en/3.1.0/guide_platforms_android_plugin.md.html

22

Signing Your Applications”, Android Developers documentation

23

For more on hairballs, see “Simplicity Matters,” a presentation by Rich Hickey at RailsConf2012. https://www.youtube.com/watch?v=rI8tNMsozo0&t=15m39s

24

Building Android programs on the command line” Version 1.0, June 2011. Geotechnical Software Services Copyright © 2011

25

nshepperd / tup-java-tools. GitHub.

The author, Neil Shepperd, is also the author of “Shake,” a Haskell-based build system that I encountered when researching around Tup. It was far too heady for me, although I would have of course loved the poetry of using it for willshake.

26

Building Web Apps in WebView”, Android Developers documentation.

27

CSS3 and Javascript animations don’t work with GPU acceleration”, Issue 17352 for “Android Open Source Project” (Posted June 2011). This was confirmed through June 2013, and marked as “Obsolete” In December 2014. There is no explanation, though, and even if “Obsolete” means that the latest Android update resolves the problem, it presumably still affects the millions of devices running earlier versions.

about willshake

Project “willshake” is an ongoing effort to bring the beauty and pleasure of Shakespeare to new media.

Please report problems on the issue tracker. For anything else, public@gavinpc.com

Willshake is an experiment in literate programming—not because it’s about literature, but because the program is written for a human audience.

Following is a visualization of the system. Each circle represents a document that is responsible for some part of the system. You can open the documents by touching the circles.

Starting with the project philosophy as a foundation, the layers are built up (or down, as it were): the programming system, the platform, the framework, the features, and so on. Everything that you see in the site is put there by these documents—even this message.

Again, this is an experiment. The documents contain a lot of “thinking out loud” and a lot of old thinking. The goal is not to make it perfect, but to maintain a reflective process that supports its own evolution.

graph of the program

about

Shakespeare

An edition of the plays and poems of Shakespeare.

the works