Geoff Garside

Harsh highlighting

Initialising Cocoa NSValueTransformers

Just a quicky, I created my first custom Value Transformer this evening. Normally to register your transformer you have to do something like the following

+ (void)initialize
{
    [super initialize];
    [self initialiseValueTransformers];
}
+ (void)initialiseValueTransformers
{
    LoginStatusTransformer *loginStatusTransformer =
        [[[LoginStatusTransformer alloc] init] autorelease];
    [LoginStatusTransformer setValueTransformer:loginStatusTransformer
        forName:@"LoginStatusTransformer"];
}

which hurt that part of my brain which likes clean code because of all the repetition.

So I cooked up a nice little pre-processor macro to tidy it up

#define init_transformer(klass) \
    klass *vt##klass = [[[klass alloc] init] autorelease]; \
    [klass setValueTransformer:vt##klass forName:@""#klass]

so now I have

+ (void)initialiseValueTransformers
{
    init_transformer(LoginStatusTransformer);
}

which will be expanded out to

+ (void)initialiseValueTransformers
{
    LoginStatusTransformer *vtLoginStatusTransformer =
        [[[LoginStatusTransformer alloc] init] autorelease];
    [LoginStatusTransformer setValueTransformer:vtLoginStatusTransformer 
        forName:@"LoginStatusTransformer"];
}

it seems to be working for me at the moment, any bugs in the macro please let me know.

Tweaking RDiscount

As I mentioned in my previous post I’d had some issues with Pygments and RDiscount. I have been using Pygments to highlight my code blocks in my blog. Unfortunately right after using a code block with linenos none of the markdown was being parsed anymore.

The Discount library upon which RDiscount is based has the ability to output a debug tree which shows the different blocks within the document. Things like paragraphs, quotes, code blocks and so on. Using this I was able to determine that the HTML block which contained the highlighted code was the last block being detected.

The Pygments highlighted source with line numbers is actually rendered in a table. This table closes with

</td></tr></table>

and herein lay the problem. The Discount library detects the closing HTML tag with the following function

static Line *
htmlblock(Paragraph *p, char *tag)
{
    Line *t = p->text, *ret;
    int closesize;
    char close[MAXTAG+4];

    if ( selfclose(t, tag) || (strlen(tag) >= MAXTAG) ) {
        ret = t->next;
        t->next = 0;
        return ret;
    }

    closesize = sprintf(close, "</%s>", tag);

    for ( ; t ; t = t->next) {
        if ( strncasecmp(T(t->text), close, closesize) == 0 ) {
            ret = t->next;
            t->next = 0;
            return ret;
        }
    }
    return 0;
}

which as you might be able to tell, on line 17, checks the line for the presence of the HTML closing element. In the case of the Pygments closing element, this was not on its own line. While looking for </table> it would only read up to </td></t from the line before giving up.

Testing with a pre-pygmentised file and pushing the </table> onto its own new line confirmed that this was the problem.

But how do we fix it?

Well I started out looking for C string searching functions, of course strstr was the first candidate. Replacing line 17 of the aforementioned htmlblock() function with

if ( strstr(T(t->text), close) != NULL ) {

seemed like a good idea, but running the RDiscount test suite revealed my naivety of the solution. With my change text such as </table> was matched breaking HTML code examples, this wasn’t good enough.

I spent much longer than I really should have trying other solutions before having the critical Eureka moment. I should add that prior to this moment I’d learnt of the more suitable strcasestr() function to better match the strncasecmp() function originally used.

At any rate my Eureka was realising that I was only ever going to need to search a string of closing tags, and that these closing tags would never be preceded by white space. Initially was thinking of using the CTYPE isspace macro, but a far simpler solution also struck me.

A closing tag, or string of closing tags will always start with a <, so provided the line started with < and then contained the respective closing tag I could reasonably safely assume the HTML block was being closed.

So I next transformed line 17 of htmlblock() to

if ( T(t->text)[0] == '<' && strcasestr(T(t->text), close) != NULL ) {

and re-ran the RDiscount test suite to see if the tests passed, they did. Next I added a snippet of the Pygments table to one of the test files with some extra markdown after it, updated the expected output file as well and re-ran the test suite. Again it all passed perfectly.

Once this was all done, I committed my changes to a branch of the fork I’d made of RDiscount and pushed them up to my fork on GitHub.

As I final test I updated the Discount source as well and re-ran it over the file to output the debug tree again. I was pleased to see all the right blocks of the file being shown, lovely.

Update

My fork has been merged with the master and been refined as well.

Changing the blogging gear

I’ve recently been working on merging my two blogs together into one. One of them I host myself, the other is with Tumblr. One has stagnated quite severely with the last update April 1st, 2008, the other only seems to get periodic and occasionally small updates.

Twitter has played a large part in the lack of updates to both of these sites. Through Twitter I can post short thoughts, URLs to items of potential interest and frequently have a more live dialogue with a group of people. I also don’t plug either of my blogs to any degree really, I provide links to them, but I don’t go telling people to read them.

So what, you may ask, is the point in merging them at all? Well I still want to blog, but the point of blogging has changed. I now see blogging as a platform from which I can propose my thoughts or detail my current exploits when they merit it. My recent head injury was one such item mainly because I was asked for further details on it and it wouldn’t fit the Twitter 140 character limit. The post on MySQL Replication was asked for by a friend of mine, while it was simplistic it helped me iron the process down in my head as well.

So as the game is changing a bit I’ve been through all of my old blog posts, correcting spelling mistakes as well as changing some grammar here and there. Its a bit vain I know, but some of it read really poorly, I also went and wrapped nearly all of the code blogs so they’d be parsed by Pygments. I had some fun with that, especially with RDiscount which I am using to parse the Markdown post files. I managed to sort out those problems though and now I’m just down to sorting out some niggly little bits before I close down the other two and redirect them to one location.

Setting up MySQL Replication

Today I was asked to set up a MySQL Replication server to allow for failover on one of the services we operate.

A friend of mine also asked me to document the procedure as its been a while since he’d done it and couldn’t remember. So here goes….

The Setup

I’ll be doing all of this on FreeBSD 7.1 servers and I’ll be installing things from ports and then setting up the replication part.

Installing MySQL is pretty much a breeze, I do tend to do this first though

# cat <<EOF > /etc/make.conf
.if ${.CURDIR:M*/databases/mysql5*}
BUILD_OPTIMIZED=YES
.endif
EOF

to ensure that MySQL is built to run as quickly as possible. Then you have the simple commands to install the server, assuming you use ports-mgmt/portupgrade

# portinstall databases/mysql50-server

once that has been done on each server then we can begin setting up the replication service.

Instruct the Master

First we need to tell the master that it is a master, we also need to make sure that it’ll listen for slaves. So we need to add some things to the configuration file. You can use /etc/my.cnf which is pretty global but I’ve opted for /var/db/mysql/my.cnf

[mysqld]
bind-address      = 192.168.0.10
server-id         = 1

now, it will listen and it knows its the master.

Next we need to instruct it where to keep the binary log file for replaying to the slaves so in the same file

log-bin           = mysql-bin
expire-logs-days  = 20
max_binlog_size   = 104857600

so the binary log file will be written to /var/db/mysql/mysql-bin, entries older than 20 days should be dropped and it shouldn’t grow bigger than 100 megabytes.

Finally we tell it exactly which databases we want to replicate, can you ignore this if you don’t want to be specific. Again in the same file

binlog-do-db      = testdb
binlog-ignore-db  = mysql,test

so only our testdb table will be binary logged, additionally mysql and test databases will be ignored.

Now you can restart your MySQL master server, to check that things are working you can connect to the server and run

mysql> SHOW MASTER STATUS;
+------------------+----------+---------------+-----------------------+
| File             | Position | Binlog_Do_DB  | Binlog_Ignore_DB      |
+------------------+----------+---------------+-----------------------+
| mysql-bin.000001 |       98 | testdb,testdb | mysql,test,mysql,test | 
+------------------+----------+---------------+-----------------------+
1 row in set (0.01 sec)

and you should get something like above. You can also check to make sure the server is listening for connections with

# sockstat | grep mysql | grep tcp4
mysql    mysqld     90119 13 tcp4   192.168.0.10:3306   *:*

Creating a replication user

So we need to setup an account on the master server for the slave to use to authenticate itself. To do this we create a REPLICATION SLAVE using the following MySQL statement

GRANT REPLICATION SLAVE ON *.* TO 'slave'@'slavehost' IDENTIFIED BY 'slavepass';

we’ll set these values on the slave as well so that it can authenticate to the master.

Instructing the Slave

Again I’ll be keeping my configuration in /var/db/mysql/my.cnf, so for the slave we need to tell it that it isn’t top of the food chain and also who is. To do this we add

[mysqld]
bind-address          = 192.168.0.11
server-id             = 2

to tell the Slave to listen for connections, this is so that external services which perhaps only need read access to the database can use the slave and save the master for writes.

Next we must inform the Slave of its masters address and its authentication credentials for use when addressing the Master

master-host           = 192.168.0.10
master-user           = slave
master-password       = slavepass
master-connect-retry  = 60

Finally as we only want one database to be replicated, in my case at least, add the following line to the configuration file

replicate-do-db       = testdb

so that only the testdb database will be replicated.

Copying the database to the Slave

Firstly we need to know where in the binlog we are at the moment, so on the master server run the following command

mysql> SHOW MASTER STATUS;
+------------------+----------+---------------+-----------------------+
| File             | Position | Binlog_Do_DB  | Binlog_Ignore_DB      |
+------------------+----------+---------------+-----------------------+
| mysql-bin.000001 |   327128 | testdb,testdb | mysql,test,mysql,test | 
+------------------+----------+---------------+-----------------------+

remember the File and Position values as we’ll need these to resynchronised the databases once the slave has a copy of the data.

Using mysqldump create a copy of the master database to put onto the slave

# mysqldump --opt --databases testdb > /root/master-databases.sql

Now copy the /root/master-databases.sql file to the slave and then import it, assuming you copied it to /root/master-databases.sql on the slave as well

# mysql < /root/master-databases.sql

Now we need to synchronise the two copies. On the slave load up the mysql client

mysql> STOP SLAVE;
mysql> CHANGE MASTER TO MASTER_HOST = '192.168.0.10',
    -> MASTER_USER = 'slave_user', MASTER_PASSWORD = 'slave_pass',
    -> MASTER_LOG_FILE = 'mysql-bin.000001', MASTER_LOG_POS = 327128;
mysql> START SLAVE;

and the changes will be synchronised between the two servers.

A quick test

Now its a good idea to make some test insertions into the master database and then check to see if they exist on the slave as well. I’ll leave this part up to you…

So what happened to me on Friday the 6th of March?

As so many of these tales unravel it all started with alcohol. I’d been invited out by Claire for her brother Neils fiancé Alices birthday drinks do. This is only the second time I’ve met Alice, the first time at their engagement party. I’ve known Neil for a while as he used to work at the same company I’m at now.

At any rate over the course of the night quite a few drinks where consumed.

Towards the end of the evening, one or two people had started to go home, I felt a bit dizzy quite rapidly. Normally I get a slowly increasing buzz which I know well and use as a good indicator of when to stop drinking. I made my way to the toilets to get some water on my face, unfortunately there was a queue of people coming out of the toilets, so I decided to go back up the spiral staircase to get some air instead. The next couple of events are bit more patchy. I recall standing under the staircase, possibly letting some people past, something also makes me think I might have sat down under the stairs, maybe I was waiting to get into the toilets to splash my face. Things just sort of get a bit black and hazy from there.

The next thing I remember is seeing Claire and the woman from the coat desk standing over me. Claire was pressing some blue paper towels to my forehead. I don’t specifically recall the first couple of questions but they were along the lines of “are you alright” and “how many fingers am I holding up”. Then we came to the slightly more tricky questions of did I remember what happened. As you can probably guess at this point, I wasn’t completely sure. I’d apparently been unconscious for a minute or so as well.

After a few minutes the ambulance “drunk bus” arrived, and the paramedics had a look over me, took my blood pressure, checked my blood sugar and took my details. They said the wound seemed superficial but that someone should keep an eye on me and asked if I’d come with anyone else, I said no and that I’d met up with every body at the bar. Once they’d finished collecting my details the paramedic and I wandered back over to my friends and told them that I should be alright but that someone should keep an eye on me over night as it could be worse than it looked. If I started having blurred vision or started vomiting I should be taken to hospital to be checked out fully.

Everyone came back out of the bar a few minutes later to decide what to do next, I think the party was winding down so most opted to return home. So we said farewell to each other amidst various Harry Potter comments. Claire offered to take me back to her house and keep an eye on me so we headed off in the direction of the tube. On the way back, before we caught the tube, she checked with one of her house mates that was away for the weekend to see if I could sleep in their bed for the night. Fortunately the tube is quite direct though the whole time I was wishing I had a hat to cover up my stupid forehead. When we got off the tube we had a rather pleasant walk to her house, it was good to chat without the noise of the bar.

When we got to her house Claire stuck some bandages to my forehead and chin so that I wouldn’t bleed all over her house mates pillow. We each then went to bed, as I couldn’t quite sleep yet I posted a picture on twitter and then answered a few questions about it before finally putting head down to sleep.

Around eight in the morning I woke up and checked my bandages, it was good having them there as my chin had bled a bit during the night. I left them in place and then lay back down in the bed for a bit to rest and run through the evening. After a little while I checked the internet to find some tips on speeding up the healing process. Vitamin C, Zinc and protein all seemed like best things to get, so I decided that I’d have a fry up with Orange Juice when I got home. The best hangover cure if ever there was one. A tasty multivitamin can follow that up for the Zinc. I also did some “Brain Training” as I figured it might help.

Some time later I got up again and carefully remove the bandages from my face, I managed to loosen the scab on my chin from the bandage with water so I didn’t tear it all off. At the same time I noticed some abrasions on the side of my face, quite shallow though and they’d scabbed up pretty quickly by the looks of things, a bit swollen though.

Around 9.30 Claire came down to check on me, by this time I was feeling reasonably well, as she needed to pack up to go to a ball in the evening I decided I would make my way back home. She gave me some directions back to the tube station and I remembered most of it from the walk the opposite direction the night before.

I got a couple of strange looks from people on the tube but mainly I kept my head down to avoid everyone. When I got back to Mile End I went into the new Tesco got my fry up supplies and some plasters. Then I had a nice big fry up when I got back to my flat.

Later that day I discovered a couple other party injuries, a decent sized but shallow cut on my shoulder and a skinned and bruised knee. None of it, bar the knee, is giving me any real pain and I’ve not had any dizziness or vomiting since the event.

So what actually happened?

Well the best I can piece together is that my head collided with the spiral staircase by the toilets, though its difficult to work out exactly how I got the gash I did from that. The cut on my chin did swell up a little but, but as I’ve no pain in my jaw I don’t think I hit it very hard at all. I’ve also no neck pains so I don’t think I fell down the stairs at all. I suppose its possible that I just stood up into the edge of a stair, that would probably explain the vertical nature of the gash though its not a straight wound.

But I wanna see a picture!!!

Ok fine, Jason took a pretty good one of it the other night. Fortunately its healing up reasonably well, so with any luck it should be faded in a week or two.

Proxying through complex situations

I was recently in need of access to my Windows XP desktop machine which is has been left at my parents house due to lack of space anywhere I’ve lived in London. I had previously set up a port forwarding rule on their home router to push any RDP connections through to my desktop. Annoyingly as I found these didn’t want to work. I don’t know what is wrong with my parents router but it won’t forward any ports, this might have something to do with the DMZ which is also turned on. I’ve never been a big fan of remote router access so I got on to their file server via SSH, installed elinks with SpiderMonkey javascript support and attempted to connect to the router to see if I’d set up the port forwarding. This didn’t work as the javascript location.replace is apparently not a function. This left me a bit stumped as to how I might be able to pull this off.

After a bit of searching through the SSH manpage I gave the following a try

ssh -D 8080 -f -C -q -N parents-fileserver

this created a SOCKS proxy through which I could tunnel into their local network and view the routers management interface. It was after a bit of unsuccessful playing here that I found out their router doesn’t always forward ports properly. This left me in a bit of conundrum.

Netcat

I would say that the majority of Unix people would have heard of Netcat, aka nc. Netcat is a networking tool which lets you do just about anything you could possibly want to with TCP or UDP. So I wondered if I could create an instance of nc to listen on the RDP port and then somehow pass that through to my desktop machine. A very quick google turned up this command

$ nc -l -p 3389 | nc 10.0.0.10 3389 | nc -b -l -p 3389

though many of the command switches had disappeared or their combinations were now invalid I turned it into this

$ nc -l 3389 | nc 10.0.0.10 3389 | nc -l 3389

and tried to connect up to my desktop. This command should create an instance of nc that listens on the RDP port and pipes whatever it gets through to another instance of nc which is connected to my desktop machine, this again pipes whatever it gets back through the listening port.

This didn’t work… at all.

So a bit more googling turned up this on the freebsd mailing lists

$ mkfifo /tmp/fifo
$ nc -l 3389 < /tmp/fifo | nc 10.0.0.10 3389 > /tmp/fifo

this should again pipe anything hitting the server RDP port to the desktop RDP port, the desktop RDP traffic gets written into /tmp/fifo which is then sent back to the originating client.

This also didn’t work… at all.

So now I was at a bit of a loss, but the freebsd mailing list entry mentioned a website from which they were basing their experiment on, so I went to check that out. The bottom of the page mentioned an alternative solution using a program called socat. Ok, I thought, lets give that a try and installed that.

So I ran the following command to create the socat proxy

$ socat tcp4-listen:3389,reuseaddr,fork TCP:10.0.0.10:3389

This worked… very, very well.

So now when I need to get into my desktop machine I can SSH to the file server and setup the proxy, do what I need to and then shut it down when I’m done.

My first Nu script

I get bored, quite easily and my desktop picture is no exception to this. Thankfully in Mac OS X, I believe Windows too, you can have a rotating desktop picture. By rotating, I don’t mean spinning, rather I mean you provide a directory of images and the OS displays them in a random order. So every 5 minutes I get another image.

Occasionally it will display and image which I am very bored of, or just don’t like. In this case I then have to flip through the images in my Desktop Pictures directory to find the offending image and remove it. Every time I had to do this I was sure there was an easier way. A couple months ago I found that the ~/Library/Preferences/com.apple.desktop.plist file contained the information which I required to get the image file name, and more portably the path used for the Desktop Pictures. Originally I wrote up a small bash script to grep the information I needed from the defaults read output, but I never liked this. The reason I never liked this was that the ChangePath and LastName property list keys appeared something like three times in the file. So I was having to tail the grep output to only select the last one, as this is the one I cared about.

What I really wanted was a way to use the defaults program to access a slightly more complicated key structure than it would allow.

Along came Nu

Nu is a lisp like language written by Tim Burks which allows access to the Cocoa and Foundation API’s in a scripting like shell (through nush).

The most I’d really used Nu for at this point was just playing around with the Cocoa API’s, messing with functions and seeing how things worked. This is one of the most frequent uses I have of Nu because it lets me essentially debug interactively in a sensible way.

Given my rather amateur usage of Nu so far I wanted to try and use it in its own right. Given my issues with the defaults program for accessing complex key paths, and knowing an NSDictionary would do this easily I thought this would be a good little dip in the sparkling Nu waters.

The script below is what I came up with, it is pretty simple, and pretty self explanatory if you already know the Cocoa API’s. As a simple walkthrough though

1. Check to see if the script was called with the “-rm” argument
   1. Set shouldOfferToRemove to true if it did, nil if not
2. Read in the com.apple.desktop.plist file contents
3. Get the dictionary for the Background/default key
4. Extract the ChangePath value which is the path to the Desktop Pictures directory.
5. Extract the LastName value which is the file name of the image file
6. Print the name of the image file
7. Prompt the user to delete the file if requested
   1. Delete the file if instructed

The most complicated portion of the script is the part where the response to either delete or leave the file is read.

(set input (((NSString alloc)
    initWithData:((NSFileHandle fileHandleWithStandardInput) availableData)
    encoding:NSUTF8StringEncoding)
        stringByTrimmingCharactersInSet:
            (NSCharacterSet whitespaceAndNewlineCharacterSet)))

this section could be broken down a bit to something more like the following

(set stdin (NSFileHandle fileHandleWithStandardInput))
(set inputData (stdin availableData))
(set inputString ((NSString alloc) initWithData:inputData
  encoding:NSUTF8StringEncoding))
(set trimSet (NSCharacterSet whitespaceAndNewlineCharacterSet))
(set input (inputString stringByTrimmingCharactersInSet:trimSet))

which does give a better indication of the operations being carried out, firstly a file handle to the input stream is obtained, then the availableData is read. The availableData returns whatever the user types before hitting Enter. The next couple of lines convert the NSData into an NSString and then trim any whitespace or newline characters in the string. This lets the script more easily compare the input against “y” and “Y”.

So without further ado, here is the script (also on gist)

#!/usr/bin/env nush
; Useful script for when you have a rotating background on Mac OS X
; this script will tell you which image is currently being displayed.
; If you add -rm when calling the command then it will provide you with
; prompt to let you delete the background.
;
; Installation:
;  Copy this into something like ~/bin/current_background and then
;  $ chmod 750 ~/bin/current_background
;  then use :D
;
; Usage: (assuming you call it current_background)
; Show the current background image filename
;  $ current_background
;  Current desktop image is background-1.jpg
; Show and optionally delete the current background image file
;  $ current_background -rm
;  Current desktop image is background-1.jpg
;  Are you sure you want to delete background-1.jpg [y/N]: 
;  y
;  Background background-1.jpg has been deleted
;
; Bugs:
;  1. When prompting for user input I can't seem to flush stdout manually
;     so I have to print a new line character in order to show the prompt.
;

(set processInfo (NSProcessInfo processInfo))
(set args (processInfo arguments))

(if (and (eq (args count) 3) (eq (args objectAtIndex:2) "-rm"))
    (set shouldOfferToRemove t)
    (else (set shouldOfferToRemove nil)))

;; Get the dictionary
(set plist ("~/Library/Preferences/com.apple.desktop.plist" stringByExpandingTildeInPath))
(set prefs (NSDictionary dictionaryWithContentsOfFile:plist))

;; Drill down to the dictionary key we care about
(set default ((prefs valueForKey:"Background") valueForKey:"default"))

;; Extract the fields we need
(set changePath (default valueForKey:"ChangePath"))
(set currentImage (default valueForKey:"LastName"))

;; Print the information
(puts "Current desktop image is #{currentImage}")

;; Offer to delete
(if (eq t shouldOfferToRemove)
    (print "Are you sure you want to delete #{currentImage} [y/N]: \n")
    (set input (((NSString alloc)
        initWithData:((NSFileHandle fileHandleWithStandardInput) availableData)
        encoding:NSUTF8StringEncoding)
            stringByTrimmingCharactersInSet:
                (NSCharacterSet whitespaceAndNewlineCharacterSet)))
    (if (or (input isEqualToString:"y") (input isEqualToString:"Y"))
        (set fm (NSFileManager defaultManager))
        (set path (changePath stringByAppendingPathComponent:currentImage))
        (if (fm removeFileAtPath:path handler:nil)
            (puts "Background #{currentImage} has been deleted")
            (else (puts "Background #{currentImage} could not be deleted")))
        (else
            (puts "Background #{currentImage} has not been deleted"))))