Canalplan AC V9 Development Blog

Over the years CanalPlanAC has accumulated a host of data formats for various purposes. At least two – a custom C-like syntax (surprisingly similar to another I’ve adopted!) and “quick load” place and link files – are entirely obsolete but support for them still lurks in the interpreter.

Here are the main ones that anyone playing with the system might come across:

• SQL – since summer 2008 the main database for CanalPlanAC, the users, the information about photographs, some of the program flow control and state information during a session have been stored in SQL databases. I’m using SQLite – this is small and can be directly integrated into the interpreter making setup very easy.
• XML – until then XML was used for the master data file, and quick-read varieties in a sort of unstructured text file were generated. The unstructured files are also obsolete (although some support may still exist in route.c until I rip it out) but XML is still used although much more round the edges these days. In particular, it’s used for:
  • Sending data to some Javascript utilities.
  • Storing the configuration file /cgi-bin/config.xml which tells the program where all the data should be stored and similar things
  • To store the default user options in a file, and the user options (in a blob) in the user SQL database
  • To define options for gazetteer plugins (to be described some other time!)
• JSON – not used statically, but used for a number of inter process and code communications (particularly those involving Javascript)
• HTML – static pages are written in HTML
• CST - ClearSilver Templates. Since summer 2007 pages are generated from template files. This is an extended form of HTML with ClearSilver directives built into it – see the website for more.
• HDF – this is the configuration file language for ClearSilver. I didn’t have to use this, I’ve integrated ClearSilver so tightly I can easily pass it any format I like, but it seemed sensible. The most important HDF file is /templates/default/config.hdf which controls all the menus and buttons on all the pages.

So there you are. A pile of formats, but many of them widely known, and many of them only used in a few places. In the next few articles I’ll show how to add a sophisticated new feature (the Virtual Cruise) to CanalPlanAC and you’ll see how these various languages and formats are used.

OK, at the moment no-one is developing the program apart from myself. But it struck me that if I documented a few things as if people were, then I’d have a lot of stuff ready if ever I get a co-developer or even hand on the project.

The Canal Planner situation has, I think, shown us the danger of a project like this being all in the mind of one individual.

So what I’ll be doing from time to time is documenting how I did particular things, in a sort of “tutorial” way.  But before I do that, I thought I’d start with a sort of apology for the number of languages (big and small) used in CanalPlanAC.  I thought it might be fun to document them all – but don’t let this put anyone off: you don’t need all of them at any one time.

• C – the Interpreter that underlies CanalPlanAC is written in C.  This can remain unchanged for months, until I discover a bug or find the need to add a feature right into the language.  What tends to happen is that I write something in the scripting langauge but find it too slow or too cumbersome, so add it as a language feature instead.    There is – as yet – no modular extensibility to the C: there are about 35 thousand lines of C in 87 source files that I’ve written,and a chunk more in various pieces of “foreign” code that are linked in.
• CanalPlanAC scripting language.  I have no name for this language, which can be invoked interactively by /cgi-bin/canal/ or to execute a file as /cgi-bin/canal file.can.  This is a basic like language that deserves another dozen items on the syntax.  It was vaguely inspired by Superbasic which could give you a few pointers to some of the syntax.  Key things about it are:
  • An intermediate typing system where all values are one of a number of types, and get coerced from one to the other when assigned.
  • Support for associative arrays (called “lookup” tables) and lists
  • Inline operators: this is a mixed blessing – a line like «if a contains “hello” then …» is easy to read, but precedence sometimes goes wrong.  (So «print ‘this is me and you’ after ‘me’ contains ‘you’» produces the whole string (because it parses as evalutate «’me’ contains ‘you’» which is 0, then returns the original string after character 0.
  • Built-in comprehension of a waterways network with the ability to plan routes aroung it and travel along them.
  • Built-in read/write of SQL, JSON and XML
• There are a few small bash scripts, mainly to kick off CanalPlan programs.
• There is the whole automake/autoconf nest of snakes in /source/ but that shouldn’t need touching.
• Many web pages use Javascript (both home written and some appropriately licensed utilities) to do place name lookup, display maps etc.

Next – the data formats that these programs use to store data, communicate with them selves and talk to the user

Here are the bug fixes and enhancements for Version 9.2.8.0

• Extended that to all the SQL command interface, since it seemed to be working for that, but still getting problems elsewhere
• And to the SQL database open as well
• Added code to check database integrity. Started adding places for which I had photos but no places.
• Prevented the “moderator’s” box for adding places from emails displaying for everybody else (as I’d always intended)
• Started work on code to let places be added breaking lock flights
• Input page for places that break the lock flight added – nice new slider added for splitting locks
• Logon status added to new place page
• Fixed a bug in the SQL locking code from Monday
• The “moderators” box thing didn’t work – made it hard wired to me instead, as I’m the only person ever likely to do it
• All updates are now executed as single transactions, so will be rolled back if they fail leaving the database consistent

Here are the bug fixes and enhancements for Version 9.2.7.0

• Place with odd punctuation don’t break in “add a place” – failing to show anywhere else
• Local end doesn’t fail when multiple changes affect the same bit of waterway (distinction between version in database and version in memory)
• Added some random sleeping to the SQL code to prevent failing with database busy at the drop of a hat

Here are the bug fixes and enhancements for Version 9.2.6.3

• Prevented pasted places going in if there is one with that ID already
• Added a “COMMIT” stage to transaction logging, so failed transactions don’t clutter up the log
• Fixed some bugs I’d introduced into “create user”

Here are the bug fixes and enhancements for Version 9.2.6.2

• The simple edits to fields – text of places etc – are now tranasction logged

Here are the bug fixes and enhancements for Version 9.2.6.1

• Fixed a bug in javascript for new place adding
• Transaction logging working at both ends for new places. Added a number of recent new places to live version to test this – and because it needed doing.
• Restored photo adding – small typo in hacked-out code
• Typo in Makefile fixed so transaction logs don’t get sent

Here are the bug fixes and enhancements for Version 9.2.6.0

• Work on the local end of transaction logging. Tested for added photos. User creation is now logged.
• Added a feature to let me paste new places in, tweak them and have them added

Lots of changes rolled up into one blog

Here are the bug fixes and enhancements for Version 9.2.5.0

• Core code for adding places built – working off a small database of those changes that have been made over the past however many years. This is the bit of code that updates all the place and link databases with the new place and keeps a log so that it can be undone if found to be faulty. This works really well – it’s quite astonishing the first few times when you add a new place and it instantly is available to the drop down place selector and appears in the gazetteer. The next step is to add the “moderation” (it’s a form of retro-moderation) for these. Then the backlog can be added and moderated (there is, I’m afraid to say, a lot of good-intentioned rubbish in there), and then a front-end for the new system (but clearly made almost entirely of the old code) added.

• New places are now listed in the moderation display.
• Review changes link shown only for users with “moderator” status
• And moderation script will only run for them
• Can now accept changes as well (but not anything else!)
• Half a dozen new places now live in the database as a result of testing
• New “simple” wiki mode added for things that can’t include markup
• Place names can be edited

• New photo mode for gazetteer to show all photos in-line
• Editing of photo captions from this mode
• Support for photo caption changes in moderation

• Size of place added to gazetteer information
• Size of place field made editable

• Improved dimesionsions printing – added metres and inches All waterway dimensions were wrong! Fixed thme with new db script
• Basic “edit details of place” code added
• A new phase of the moon – both new C code to support it, and a new set of graphics created. The old “moongif” has been ripped out of the code base.

• Fixed a bug (added new code to route.c) in the waterways gazetteer where some canals came out with each leg in the wrong order (the B&F was one) and so also didn’t have their ends correctly described.
• Lock count in waterway gazetteer now works
• Place size descriptions are out by one
• Stray comma in wikiedit javascript (broke IE)
• Can no longer edit when not logged on (old testing hack removed)

• Moon made slightly smaller on index page
• A bit more padding on the left of the main box on the index page
• Fixed a bug where visiting the index page with an active but vague session caused a Javascript error
• Added a simple statistics (just place count) to index page
• Dodgy data made planning routes along the GU impossible.
• There’s a message about this being the development version on the index page

• Logging all SQL (to allow keeping multiple databases synchronised) was inefficient, and didn’t cope with (for example) adding photos. So I started to implement a new transaction system using a mini-language that can be acted on and written for later import.
• Doing this I hit a long-standing design flaw in the language: because I use () both as list delimiters and for expression grouping, single item lists tend to turn into atoms of the appropriate type. To fix this I:
  • Defined a new pair of characters that act as list delimiters only
  • Added some limited UTF-8 handing in the source code to allow me to use sloping double quotes (AltGr-V and B) for this purpose
  • Added traditional multiply and divide (AltGr-< and >) operators as a test while developing it
• Quicksearch on home page now preserves session IDs
• Transaction writing now works for new places
• And for adding photographs (complicated as it’s in three phases)
• And for moving photos

Here are the bug fixes and enhancements for Version 9.2.3.0

• Added facility for icons and tooltips to buttons in button bar at bottom of pages.
• Added such buttons and tips to pretty well everything. This is a good example of how much easier it is to work on this version – it only involved a change to one template file and one configuration file (plus finding a few new icons!).
• Made “reset to defaults” work for options
• Added new options to show only some parts of route output
• Added turns to itinerary
• Quick hack to allow internal links with the new wiki parser – but still needs work (contact developer?)
• Fixed bug where displaying empty pages looped forever
• Fixed wiki to allow editing of non-existant pages
• Re-organised the “what’s new pages” slightly
• Basic drop-down menu editing functionality on main planner page
• Floating boxes rather than table on main planner – should be better for different browser sizes

Please post bugs for this version as comments so we can pick them up and work on them.