Trip
It's been a couple of weeks now, and the details are fading, but I'll give a brief run down on what happened while I was away...

Nothing Technical (yet)
Of course, the Brisbane to LA trip was the hard part. I woke up at 4:30am Wednesday morning with Luc being sick. I thought that his irritability was just a reaction to the natural stress of preparations for my trip, but he cheered up immensely as soon as he threw up all over me. :-)

I made the mistake of reading and watching movies for the first part of the trip (I got to see "Robots" at last). So by the time I decided to do some "work" on my notebook I was too tired to think clearly. Sure, I can work while I'm tired, but I'm only really effective if I'm already in the middle of something. Picking up the notebook from a cold start just didn't work. I did some Javadoc, but that was about it. After that I went back to books and movies. This continued into my flight from LA to D.C.

The Atlantic flight was delayed a little, so I barely had time to clear customs and run across the airport to my next flight. I made the flight, but my luggage didn't. That was unpleasant when I got into Dulles. All I wanted to do was clean up, and get a change of clothes, but I had to wait for several hours first. By the time I landed it was 5:30pm on Wednesday (same day, since I'd crossed the dateline). I'd been in the air for over 18 hours, and I'd been up for 29 hours straight. I'm glad I don't do that trip too often!

The second day was at a relaxed pace, though it was a little depressing hearing about the bombings in London that morning. A couple of years ago I spent a month travelling in the London tube every day, and it's hard to imagine something so catastrophic imposing on such an everyday thing. I checked out of the hotel, went back to the airport and picked up a rental car (they "upgraded" me to an SUV. Yuck). Then I was off to David and Bern's place to use their internet and catch up on news. I also rang my brother Rowan, which ended up having consequences for the weekend.

That afternoon I drove to Baltimore through torrential rain, on roads I'd never seen before, on a side of the road that I'm not used to. Thank goodness for MapQuest (though now I've started converting to Google).

Finally, on Friday I started work. The centre was just a couple of hundred meters away, so I could walk down. The day was just an introduction, and an overview of what was to come over the coming weeks. We went over the directory structure of the project, what can be found at Sourceforge, etc.

Things like the directory structure were tricky to explain, because some of it is unintuitive, and I didn't agree with how it should be structured. It's kind of like debating, where you're given a point of view and you have to sound convincing when you defend it, regardless of whether you agree with it or not. I had to do that a few times while I was there. Sometimes the best argument is just that it was someone else's decision, and the benefits in changing it are not worth the effort required.

When I'd spoken to Rowan he reminded me that it was his 30th birthday that weekend, and asked if I could make it down to Houston to see him. I checked out prices, but it was all looking too expensive. When I rang him back to let him know I couldn't make it, I got my sister-in-law instead, and she was able to organise a cheap stand-by flight for me on Continental (where he Dad works). So my first weekend was spent in Houston. I cleared out of work and went to BWI with one of the guys from Florida (he was commuting every week).

At BWI I was singled out for special testing in a booth that shot me with puffs of air (presumably testing for the presence of Nitrogen compounds). It was kind of interesting, and I got through the line just as fast as the people who weren't singled out... only I got to sit down while they looked through my bag. It ended up being less inconvenient for me than if I'd been left in line, since they did all the work searching my bag while I watched. I noticed that most people seemed happier than usual with the delays, probably because of the events in London two days before. As a PR exercise, the TSA seem to be doing a good job. I'll let others comment on their effectiveness.

Houston was fun, and the look on my brother's face when he saw me was priceless (he thought I couldn't come). I enjoy visiting new places, so I had a great time over the rest of the weekend. I should comment though... the quality of Houston's roads is TERRIBLE. I understand why, but you'd think they'd look for a solution.

Sunday night I flew back into BWI for work that week.

Project Structures
Since my task was to describe the storage code at the lowest levels in Kowari, I decided to go with a bottom-up approach. This had advantages and disadvantages, but I hope it worked out OK in the long run. Every so often (particularly when DavidW suggested that he could hear the sound of people's brains frying) I'd step back and try to give an overview, often coming back in with a top-down approach to where we were in the code.

One thing that would have been nice to have, was a diagram of file structure. That would have made it easier to see what the code was trying to accomplish. Unfortunately, I have no idea where the original diagrams would be (they'd be 4 years old by now, and DavidM and myself never thought too much of them, as they were easy enough to work out again), and I did not have enough preparation time to write them out again (plus, I'd have missed a few things if I'd tried).

Most of the code was perfectly familiar, but while showing some of it I discovered a few new features I'd never seen before. It would normally take me a few minutes to work out what something was doing, and I probably floundered a little on occasion, but I'd get through it in the end. I comfort myself with the knowledge that I'm one of the only people who knew enough about the system to be capable of working these things out, even if I didn't know them when I started. By the time I finished presenting a section of the code I ended up understanding it better than I did when I started. Ideally I'd have had enough preparation time to be all over it, but that wasn't available, so I can't really kick myself for not knowing it perfectly. They got a few free hours out of me on most evenings as I refreshed myself on code in preparation for the following day.

Having documented this code, and presented it in nauseating detail, I came away with a very clear idea of every layer for the storage code. This allowed me to write up those diagrams on file structure very easily, during the last couple of days. A couple of people said that they would have liked to see them at the beginning (something I'd already lamented), but I think they all appreciated getting to see them in the end.

Now that I'm back in touch with this code, I've been thinking that I should write up an overview of most of it, and also provide the file structure diagrams. I've since explained it to a couple of people here in Brisbane, and I think I've structured my thoughts on it well enough to build some developer documentation on it. However, that will take some time to write up and I'm about to start a new job, so I won't commit to it all straight away. I'll do what I can.

Block Files and Free Lists
The first week was spent entirely in the util-xa package.

Starting at the bottom, this includes the BlockFile interface, along with AbstractBlockFile, MappedBlockFile and IOBlockFile. These handle files with fixed-length records called Blocks, and transparently hide if the file is being accessed with normal I/O operations, or if it is memory mapped. Almost all files in Kowari are accessed with these classes, with the only exceptions being IntFile from the util package.

On top of this file abstraction is built the FreeList class. This name is a little misleading, as it only describes a single aspect of the class. However, it is there for historical reasons and it's not going to change any time soon.

Free lists are really a resource manager. Resources in this context are identified by a numerical ID. A FreeList will hand out resources when requested, and will collect resources when they are no longer in use. Resources will be recycled whenever possible (there are a lot of reasons for this, regardless of the resource type, but it always comes down to performance), and so the list of "freed" items has to be maintained in order to hand them out again when it is permissible to do so. This internal list of free items is where the class gets its name. A better name would be something like ResourceManager.

The list of free items is kept in a BlockFile. The blocks in this file form a circular, doubly-linked, linked list. It is in this file that the concept of Phase first appears, which is the basis of Kowari's transaction support. At this level, phases are handled as pointers to the first and last freed items within a phase, and a set of heuristics which manage those items which may and may not be re-allocated, according to the currently opened phases.

Phases don't make a lot of sense until the higher level phase trees are examined, but these rely on the functionality of FreeLists so much that it becomes difficult to to describe phase trees without first describing FreeLists. I've tried to do a top down explanation to a few people in the past, but they invariably question how resources are managed between phases. Hence, my approach of describing phases from the ground up, starting with FreeLists.

I'll describe this all in more detail when I document the FreeList shortly. I'll just say here, that this code took the best part of the first full week, and forms the basis of all the storage code Kowari's XA datastore.