Steve's Ramblings

Its been quite some time since I started work developing Wordbooker after I forked it off Robert Tsai’s Wordbook plug-in.

It became obvious about 6 months ago that the current code base (version 1.x) was far from ideal and it was starting to confuse me, and when the only coder starts getting lost in his own code you really have to start getting worried.

So I decided that I’d start version 2. I’d take the basic process flow from 1.0 and I’d also re-use code where it made sense but I’d clean things up and change how I modularised the code. I also set up a Google Code page for the plug-in, not for sharing the code on but really to use it as a bug tracker where people could also request new features.

Wordbooker 1.x has two major failings :

• Comment Handling
• Resilience to Facebook failures

Plus several other less important issues like not being WPMU/WP Networked Blogs friendly (in that it creates multiple tables per blog) and of course forgetting everything when people de-activate it for WP upgrades or plug-in upgrades.

Comment handling was something that I sort of bolted on and its never really worked properly, and due to the fact that Facebook have imposed a maximum number of feed posts an application can do in a 24 hour window its not something I’ve ever really taken much interest in, but its something that a lot of the users of the application want and it didn’t work with pages so that was top of the “user wants” for version 2.

Being more resilient to Facebook failures is very important – these range from sessions expiring if the user changes their Facebook password, through users hitting the dreaded feed post limit, to the complete breakdown of the Facebook platform : something which is actually all too common.

I’d worked out in my head how I was going to do the first of the two and sitting in the car on the outskirts of Stroud armed with a pad of paper and a pen had given me time to actually turn those vague thoughts into something more concrete.

The second big issue was something that had got me really stumped – the Facebook API is far from stable, and far from reliable, and sometimes it is so slow everything just times out. I’d done some work in the status update code to mitigate against Facebook failing but I’d not really got any clear idea on how to extend the process out to the whole of the Wordbooker plug-in, but it had become obvious that code that expected Facebook to be there, and that had no fall back if it wasn’t, wasn’t really the way forward.

It was a chance request by someone who does a lot of posts at once but only wants them to “trickle” out to Facebook that got me thinking and then I realised that the way I was planning to handle comments could actually be extended to how I’d handle posts.

So I guess I need to explain how comment handling works in Version 1 and how its going to work in Version 2

In V1 comments are handled in two different ways: Comments coming from WordPress are pushed to Facebook immediately with no moderation (even if you’ve got WordPress set to moderate comments). Comments coming to WordPress are handled by a WP-Cron based job which runs once an hour – this basically goes through each of the WordPress posts that Wordbooker “knows” have been published to Facebook, finds the time stamp of the last recovered Facebook comment, then fetches (using Facebook’s own query language) any new comments from Facebook and posts them to the appropriate blog post. Of course there is a problem here in that Wordbooker has to be able to recognise if its pulling comments down from Facebook that its actually posted up in the first place.

Pushing comments from the blog to Facebook counts against the feed action limit so if you’ve done a blog post you are then limited to 25 other post transactions in the next 24 hours – so if you get 26 comments on your blog then that last comment will fail because the post limit has been reached.

So in version 2 when comments are posted on your blog they aren’t posted to Facebook. All Wordbooker does it make a record that a comment has been posted. Then in the background it will check if the comment has been approved and if it has it will push it up to Facebook. If the comment has been marked as spam or deleted then Wordbooker will remove it from its list of comments to be processed. When a comment is posted to Facebook it will record the unique Facebook ID for that comment

Comments coming from Facebook will be checked by Wordbooker to see if the FB ID exists in its comment table (i.e. it’s a comment that Wordbooker created) and if it is then it will quietly ignore it.

Using a process queue like this means that I can handle FB problems easily. If you max out your feed limit or FB isn’t responding properly. then Wordbooker will simply sit there in the background attempting to post and if it can’t then the comments will simply be processed at a later date.

So that’s how comments are going to be handled, well roughly how I plan to do it – no doubt I’ll refine it and change it when I really start working through it all. Now that I’ve explained comment handling I can now explain how post handling will be done.

At the moment when you publish a post to your blog Wordbooker posts it immediately to Facebook – so if you’ve made a glaring mistake in the first part of your post its now sitting there on your FB wall for all your friends to see. The work round for this is to leave the Wordbooker Default Post option turned off and then once you’re happy with the post go in and edit it and force its publication on Facebook.

In version 2 you’ll have the following options when it comes to publishing a post:

• Publish Immediately
• Batch for automatic publication
• Leave on queue

The first option “Publish Immediately” will behave in exactly the same way as publishing does in version 1 – and will be there for those people who are totally confident that their post is correct, and also for posts that have been scheduled for publication at a future date.

“Batch for automatic publication” will flag that the post should be pushed to Facebook at a later date – I’m still working on how the user will control when posts get pushed but it will allow users to basically change the order posts are pushed to FB.

“Leave on Queue” will do just that – the post will not get pushed to Facebook at all and you can edit/change the post to you hearts content and publish it later.

Of course this means that the options page will have to feature a “Queue Manager” which will allow users to manipulate their queued posts – so they can force a post to be published, or add it to, or remove it from, the batch list, or change the order in which posts should be sent to Facebook.

So by basically queueing all posts to Facebook (yes – even immediate post requests will be put onto the queue) it means that if the push to Facebook fails for any reason then the post isn’t lost, and Wordbooker will simply retry at a later date.

There will also be a “Queue process history” (but not with such a clumsy name) where you’ll be able to see the outcome of recent post activities – so rather than just getting a general “feed transaction limit reached” message you’ll be able to see which post failed and what Wordbooker has done since then.

I know that I’ve been telling people that the Options page was getting too complicated and scary, and this is just going to make it much worse. Well I’ve got a solution…the Wordbooker Options page will be divided into several sections. I did look at making it several tabs but it didn’t feel right – so instead I’m using collapsible sections like the ones on the WordPress admin menus.

Right – I think I’ve probably bored you all enough by now so I’ll stop here.

I know its been pretty quiet round here for a bit. Its just been one of those periods of time when I’d nothing really to write about, and even if I did I didn’t really have the time, or the motivation, or the energy.

When I’ve not been working I’ve been bashing my head against the wall over my Wordbooker plugin which I’ve been playing a continual catch up game with Facebook and WordPress and W3C compliance (some of which was nothing to do with me but actually Facebook’s decision to use non compliant tags and people having WordPress themes with not quite the right Doctypes and other things). I’ve been wanting for ages to start on Version 2 of the plugin – which takes a different approach to parts of the posting process and needs a complete re-write of the comment handling process too, but until I’d got the 1.x code base to the point that it was stable enough to leave alone so I could concentrate on 2.0 I couldn’t do that. I’d just got ready to start recoding when Facebook announced they were changing the method of authenticating users against applications… that’s right. Just a couple of months after they changed it totally they are proposing changing it all again. It didn’t help that they accidentally turned on the new method and broke everything!

But things seem to have settled down and so I’ve started coding it, but I also wanted to watch “Travelling Man” which Nick had bought me on DVD for my birthday after we’d both read Andrew Denny’s blog entry about it. It’s been quite enjoyable watching it and recognising a lot of the locations, even if they have changed a lot since it was shot : for example the derelict warehouses near Preston Brook have been turned in to rather up market apartments – but it was good to see that Claymore Navigation’s boatyard hasn’t changed much in the intervening years.

So today I’m having a day off work, and I could be coding but instead I’m listening to Bob Geldof’s solo albums and trying to rustle up enough motivation to do much more than slouch on the sofa all day. I know that some would say that NOT listening to them would be enough motivation to get off the sofa, and others might say that listening to them just sucks motivation away from you, but hey… it’s my day off and I’m waiting for the Dishwasher to go through a cleaning cycle!

Nick commented once about some of the code in CanalPlan AC and how it was suffering from “bit rot”. Basically code that you shoved in, often as a quick dirty fix and then it starts to fail but with no good reason.

Well Wordbooker is suffering from bit rot, its just that luckily no-one has noticed it yet, well apart from me that is.

Wordbooker took code written by Robert Tsai for his plugin (Wordbook) and extended it. That is why Wordbooker is called Wordbooker – because it stands for WordbookE[xtended]R[elease]. Robert’s code used to support the WordPress PHP libraries for PHP4 and PHP5. PHP4 support was dropped but the wrappers were left inside the code, adding a level of abstraction around the calls to the Facebook PHP libraries.

When I added all the new features such as posting to walls and FB widgets and so on, into his code I sometimes used his wrappers and sometimes made the calls directly.

Facebook then depreciated some functionality and so calls to functions supporting that got commented out and eventually I stripped most of the code out.

I’d added functionality to handle comments really as a sort of proof of concept and then I added posting support to pages, but comments to pages didn’t get processed properly due to slight differences inside FB on how pages are handled.

So that was the next big project – re-write comment handling to make it a lot more robust (and at the same time simpler).

But then Facebook came along with the Graph API and the Oauth authentication process and said that all applications would be force migrated to the new Authentication method on June 1st. The new Graph API would make my job simpler as I only use a few calls (post to wall, post comment, get comment, get status, and a few FQL calls) and so obviously I should think about moving my code to use it.

That wouldn’t be bad if the Graph API was complete and not riddled with bugs (for example if you include a link in a post it thinks you are doing a “share” style post and goes and scrapes that URL – something which is not right and not how the old API works), but as it is I can’t use the new API for anything but authentication, so I’ve had to work out how to use the Oauth method for signing on, but then use the old API for the rest of my code, all under the looming date of the end of the month and with me working a day job as well.

Facebook also changed their policy on posting to walls and I asked them for clarification, and they asked for more information which I’ve given them, and now I’m waiting to hear back from them. It could mean that I have to change my code, but actually it will make the process easier to code and make it easier for the user to understand, so I am going to make the changes any way.

WordPress is also changing – with 3.0 coming out really soon now – and I wanted to make my install more multiblog friendly : so 4 tables per WP install rather than 4 tables per blog which means some quite jolly logic to work out if you are in WP2 (or WPMU2) or WP3 (running in single or multiple blog mode)

So I’m revisiting my old code and wondering if I should really throw 80% of it away and strip away the crud and spaghetti that seems to have collected during my rapid development of the project. I’ve now got a much clearer idea of how things should work and how the code should flow, and how to code round the countless “failures” of the FB back end which occur at depressingly frequently intervals, some of which I did last night when a couple of users were basically stuck because FB were playing round with code on the live servers again.