Systeme D

3.

It all starts with an editor: the blog edit.

I gave a talk today at State of the Map US - officially the US local conference, but pretty much the second international one - about OpenStreetMap editors. Lots to say, and as ever, not enough time to say it in. So here’s my notes in slightly more prolix form. (Great word. Blame Catch-22 for that.)

The talk was called “It all starts with an editor” because that’s how OSM works: use editor to add data; it goes into a database; and a renderer makes a nice map out of the end of it. Incidentally, both the default editor and the default renderer come from the same tiny Oxfordshire town, Charlbury (population: 3,000). Go figure.

And my aim? To enthuse people to help make OSM’s editors better. It’s the single best thing you can do to help the OSM dataset grow, and I hope to explain why.

Part 1: why we need good editors

So let’s look at the editor of another crowdsourcing project: Wikipedia. Internet life without Wikipedia is now pretty much unthinkable, so this isn’t meant as excoriating criticism.

But still...

Wikipedia’s editor is a word-processor. We all know how to use a word-processor, so there’s a head-start right there. It has ‘bold’, ‘italic’ and ‘link’ buttons, just like your usual word-processor.

Except when you click ‘bold’, you get some text with asterisks either side. When you add a link, you get this vast amount of markup with ‘|cite ref’ and other obscure incantations. It’s like having bold and italic buttons in your word-processor, but what’s displayed on-screen is the raw RTF code. Seriously, I used word-processors on the Amstrad PCW in the mid-1980s that were more visual than this.

It’s not just the technology you have to get to grips with. Wikipedia has rules. Lots of them. I showed a slide of the names of Wikipedia’s policies - not the policies themselves, just their names - and it took up an entire four-column page of 10pt type. That’s a huge barrier entry.

Despite that… Wikipedia is world’s the sixth most popular website. So they must be doing something right.

If we were to take the same approach in OSM, we should ban Potlatch, mandate JOSM, and introduce lots more rules. Compulsory JOSM and lots of rules. Well, I guess the Germans would love it.

But it’s not enough. OSM is at its best when we harvest local knowledge. The principle for admittance to OSM should be “I know stuff that can go on this map”, not “I’m good at using an OSM editor”. The more local knowledge that can be added to OSM, the better the map is.

It’s already happening. I stumbled across a really good set of tutorials telling cyclists how to use Potlatch 2 to communicate their local knowledge to bikeplanner.org.

But it’s a moving problem. A typical OSM mailing list post in 2004: “Well, I got my patched Orinoco driver working, and I also got Kismet working and so it outputs what it finds by speaking to me with the Festival speech synthesis system”. We had a geek community for geeks, and the tools should be written to suit.

And in 2012: “Yes, I am new to osm. I want to add street names to the map in my town. The opening screen in Potlatch 2 shows on the left a list of points of interest: shopping, food and drink, etc. How do I add street names?”

The first guy clearly is going to be at home with JOSM (in fact, he’s Jono Bacon, now Ubuntu Community Manager). But the second guy. How do we help him?

The answer is that we write a more friendly editor. It sounds easy. It isn’t. Challenge 1: unlike Wikipedia and its word-processor, most people have never used a vector graphics app. Challenge 2: left unchecked, the tendency is for OSM’s barrier to entry to get higher, because people are inventing more and more complex tagging schemes. Challenge 3: our documentation sucks; the editor’s UI is the user’s sole guide. Challenge 4: browser-based and phone technologies are moving targets; five years ago you’d use Flash, today it’s all HTML 5.

And challenge 5 is that mapping is too much fun. Should I go out and survey a cycle route (yay!) or should I tackle one of those Hard Trac Tickets I’ve been putting off for months?

Traditionally there have been two schools of thought in writing editors. One is to minimise abstraction and give a raw-metal experience. Traditionally JOSM core tends to take that approach and any abstraction happens through plugins, whereas Potlatch has always abstracted - we even abstracted away a whole part of the data model (segments) in early versions of Potlatch 1.

Essentially, JOSM is a big powerful German-built 4x4, a raw, exhilarating driving experience, incredibly fast, very robust, and with all the bells and whistles you could ever want. Potlatch is a bicycle: simpler, not as speedy, essentially human-powered, but you can go anywhere with it and you don’t have to pass a test to use it.

I like bicycles.

Part 2: the Potlatch family

So, a brief history of Potlatch. Named after the newsletter of the Lettrist International, the predecessors of the Situationists. I wrote the first version because I wanted something that would let me edit OSM on my Mac (everything else was in Java, and OS X Java sucked), and because the whole nodes/segments/ways rigmarole was hard work in editors with no abstraction, and I wanted something as efficient as Illustrator’s line-drawing mode.

It wasn’t expressly conceived as a beginners’ editor. If there’s a design philosophy, it’s “a straightforward editor that doesn’t get in the way”. You should be able to do anything with it - i.e. it should give access to the full data model - but that’s not to say it has to be optimised for every single use case.

OSM needs a beginners’ editor and the Potlatch codebase is capable of providing that. But it wasn’t originally designed as such, and it’s not there yet.

Yet.

It started Potlatch 1 back in 200mumble. It grew like topsy, as did OSM. Perhaps its, and my, favourite hour was Tim Berners-Lee, doing his linked data spiel, demonstrating using P1 on-stage and saying “this is the future”. I shoud have just retired then; it doesn’t get better than that.

The UI is very much designed for the sort of mapping we were doing then. It’s more about creating, less about maintaining/refining. So, when you click the end-point of a way, it automatically assumes you want to extend it, rather than adding tags to the end-node. That was absolutely the right decision then. It’s probably not so appropriate now.

Potlatch 2 came along a few years later. Despite the name, it’s pretty much a complete rewrite; there’s virtually no shared code. Big changes include completely abstracting the tagging behind a friendly UI; a new MapCSS-based WYSIWYG renderer; and lots of tools to work with third-party data sources, because they’re a big part of mapping these days. (Lots of the credit for this goes to Andy Allan, who’s built an excellent SnapshotServer feature for this.)

Perhaps Potlatch 2’s greatest strength is that it has really good internal architecture, and that’s almost entirely the work of Dave Stubbs. Potlatch 1 was a bit of (well, a complete) lashup. (It was the first time I’d ever written any OO code. Go easy on me.) P2 is pretty much MVC, clean, and robust.

Another big thing with P2 was enabling custom deployments: customised versions for CycleStreets, HikeBikeMap, the Japanese community, and many other places. Even the USGS are using a version to update the National Map of America. That’s amazing and a real opening for the future.

So, chapter 3, and that’s iD.

iD is the P2 architecture ported to JavaScript. All the good stuff about the architecture is either ported or being ported. The one fundamental exception is the XML tagging preset architecture, which has proved to be a bit too complex for people, and which I’m replacing with a lighter, simpler, JSON-based system. But essentially, the underlying base is P2 in JavaScript.

Instead, I’d anticipate that the main change will be the UI.

I used to be an absolutely religious believer in modeless UI. I’m a long-term Mac user - my first one was a Mac IIsi in 1995 - and Apple’s Human Interface Guidelines, back then, extolled the virtues of modeless UIs. I love modeless UIs. No need to select a ‘draw tool’ or a ‘delete tool’ or a ‘move tool’ or any of that nonsense; instead, select item, do something with it.

It still works well, I think. Potlatch 2, for me, is very close to the way I like to work. But I know how P2 works (obviously I do, I wrote big chunks of it).

But put yourself in the shoes of a novice user. You create an OSM account, zoom in, and click ‘Edit’. Right… there’s a map.

No buttons.

What do I do now?

There’s two ways you can fix this. You can build a tutorial mode to hold the user’s hand; again, to take a Mac analogy, something like Apple Guide could be implemented and I’ve actually started work on that in P2.

The other thing you can do is (gasp) a modal UI. Modal UIs, if done well, can guide the user through what they want to do. You want to add a POI? Click the “Add POI” mode button, and follow the instructions.

I used to run scared from that because you end up creating wizards, and wizards are horrible; dialogue-based workflows that are fine for one-off config changes and first-time users, but really wasteful every time you want to draw a way or make a junction.

But my bolt-of-lightning realisation is that you can do both. If one of the modes (say, ‘Edit Object’) includes enough shortcuts to do P2-style modeless editing… then you’ve got the best of both worlds. The newbies get obvious modal editing, the experienced users get something seamless and fast.

iD is being written using the Dojo framework. Dojo is an MVC framework and toolkit with four main strengths. It’s very well suited for writing large applications, which an OSM editor certainly is. It has a great graphics library, and an OSM editor spends most of its time in graphics (and interactions with graphics) and not much in the traditional DOM. It has good UI widgets. And it’s an all-in-one solution: combining JQuery with Backbone with Require with JQuery UI is an alternative, but it’s nice that Dojo gives you the whole lot.

Funnily enough, those four strengths are exactly the strengths of the AS3/Flex framework which Potlatch 2 is written in. So porting the internals becomes very appealing.

(Incidentally, there are several reasons why it’s called iD. One is that doing anything in JavaScript involves getElementById. Another is that people find it extraordinarily, puzzlingly difficult to spell Potlatch. Another is that the 'i' is because I want something I can use on my iPad, and the 'D' is the D of Système D. And it’s also, continuing the French 1960s theme, a tribute to the Citroen iD.)

I keep going on about good architecture. Why is this important? Basically, because you really don’t want to reinvent this wheel every six months.

Take the ‘split way’ logic. This should be three lines of code; take original way, remove half the nodes, make a new way from those nodes. Except if you have a turn restriction, at which point you have to only keep that in one of the member ways. Or a route relation, in which case you have to preserve ordering, which is harder than it sounds. Or the way is P-shaped. Or whatever.

Andy wrote the 100 lines of code to do this and it took him a day. No-one else should have to waste that day. Reuse is good. The internals are very easy to get wrong… and they’re also incredibly boring to write.

Part 3: the future of OSM editing

Can we use this to build one single ultimate editor?

Google Map Maker is pretty exemplary. The UI is great for the beginner. It has some of the smartest programmers in the world behind it. But still, look at the GMM discussion groups, and you find comments like “I completely updated the island in OpenStreetMap in about 6 or 7 weeks. Changing roads in Google Map Maker is nearly impossible. Takes too long, the process is too unwieldy.” Or: “This editor is terrible. After working with the Potlatch editor in Open Street Maps I am very disappointed with GMM.”

I’m not damning GMM. It’s great at holding your hands into your first edit. There’s a whole lot we can learn from it. But you couldn’t build OSM with GMM alone. So we need more than one editor.

So, can we start to draw up a vision for future editor development?

Here’s a start. Three targets: desktop, online, mobile. Three levels: power-user, full-featured, targeted. And here’s an image of the grid.

You don’t have to fill every space. There’s not a lot of call for a version of JOSM that will run on the iPhone. Nor is there much point in writing single-purpose, targeted POI editors for desktop when you might as well fire up a browser and do it via the web.

I’m assuming that JOSM will continue, and long may it do so, with its precision German engineering. That fills the ‘power-user’ level perfectly, with a good second in Merkaartor.

And at the other end, there’s interesting developments in the ‘targeted editor’ space. Pushpin OSM is one. MapBox, with their Knight Foundation work, are also making really interesting noises in this space. It should be as easy for people to integrate an editor into their site as it is to integrate a slippy map, and having written that in the notes for my talk I was really pleased to hear that repeated later on.

But there’s still a space for a full-featured editor. As the bikeplanner.org tutorial showed, adding new cycle paths is a newbie task; and that requires a full geometry editor and relation support. Renaming a street - well, if only part of the street needs changing, you’ve got to do all that split way code.

This is the space that Potlatch, in its three incarnations, occupies. So the challenge for Potlatch is to remain as functional, but become easier to use.

We have to look at iD, rather than just incrementing P2, because Flash has two years left in the browser. We can do an AIR version for some platforms; iD is an answer to how we continue to supply it on the web.

But we shouldn’t just have two editors. To quote Chairman Mao, “let a thousand flowers bloom”. There should be more editors taking more new approaches. Let’s not reinvent the wheel by all targeting the same space, but rather, build different tools to cater to the many different potential audiences.

Part 4: Over to you

So, why should you get involved? Because, to return to the talk title, it all starts with an editor. I love mapping, but I’m aware that one day spent improving P2’s UI will have a better overall effect on the project than one day spent gathering housenumbers in my town.

We need code. But code doesn’t magically happen. It needs people to write it. And in terms of bang per buck, coding editors is the single most effective way to improve the quality of OSM.

Small patches are just as welcome as big stuff. I’m a big fan of “little things that mean a lot”. For example: last week, I changed the P2 toolbox palette to hide the advanced stuff at first, and run an extra sanity check before running a potentially destructive operation. It took half an hour or so, and the French mappers are really happy.

If you don’t code editors, there are other ways you can help. Stop overloading new concepts onto the existing data model; we don’t want the ‘split way’ code to become any more complicated. (Just as a rough guide, if your tagging solution involves uuencoding image data into tags, for example, you’re doing it wrong.) Yes, we do need a proper area datatype.

Write some docs. Film tutorial videos. Or write tagging presets. Editor coders are not the best qualified people to write tag config files, and it’s a distraction from their main task.

And be nice to those who put the effort in to writing editors. There are only two types of people who write editors right now: the bloody-minded and those who are funded. So if a newbie makes a mistake using an editor, don’t condemn the editor out-of-hand and demand it’s banned. Because, believe me, if you put a 12-year-old behind the wheel of a fast, powerful, German-engineered 4x4, they can do a lot more damage than if you put them on a bike.

(Fortunately the sysadmins are calm, unemotional people and have only ever once succumbed to the call to ban an editor from OSM. That was version 1722-1727 of JOSM, with the famous bug that transposed lats and longs.)

To hack on Potlatch 2, you need the Flash debug player (a free download from Adobe); the Flex command-line compiler (free download from Adobe or Apache); and a browser. The code is in git; clone mine at github.com/systemed/potlatch2, not OSM’s. There are readmes on the wiki and in the source. You don’t have to bother with ant if you don’t want to; just follow the instructions for debug builds.

And to hack on iD, you don’t really need anything special at all. Just check out the code from github.com/systemed/id, check out the Dojo docs, and enjoy.

Dive in. OSM will be better for it.

Posted on Sunday 14 October 2012. Link.

Previous post: TomTom: from PND to FUD.