Time Bugs

A short while ago I received a bug report for Mulgara, where certain Date/Time stamps were being randomly changed. I dreaded to think what this could be, and avoided it in the hope that the reporter (DavidM - but not the DavidM who wrote the storage layer!) or someone actively working in the storage layer (like Andrae) would find the problem.

Another report yesterday made me realize that DavidM didn't know where in the system the problem was to be found. Rather than simply tell him to look at SPObject and SPDateTimeImpl, I decided to have a look myself. After all, I know this area reasonably well.

The latest bug report claims that whenever the following timestamp is entered:
  April 26, 1981, 02:56:00
Then the system always returns:
  April 26, 1981, 03:56:00
(one hour later)

First, I ran the test queries and confirmed that the problem occurs for me too, and that it is definitely the string pool causing the problem. To help me narrow things down more quickly, I copied out some of the relevant code from the string pool so I could play with SPDateTimeImpl in an isolated environment.

I quickly realized that the problem was occurring when a third-party library was parsing XSD dates. So I reported this to the developers list, and stopped looking. There were some alternative library suggestions, and Brian said he would fix it.

Third Party F/OSS

It was still bothering me that this library was failing for us. Given that it was open source software, I decided that I could at least report the problem to the developers. I don't know about taking the time to debug the code, but at least I could give them a heads-up.

(I won't mention the name of the library, because I don't want anyone to think there is anything wrong with their library. Read on.)

So I wrote an example piece of code that parses one of these bad date/times, and sent it to the developers of this library.

A couple of hours later a developer wrote back to tell me that the library works correctly for him, and that the problem may be related to timezone issues. I'm in CST and he is in BST. He gave me a test case, and asked for some details of my system.

So I ran the test, and discovered 2 things. First, the time was still being printed incorrectly, though it worked for him. Second, the tests that he wrote all passed.

Initially I thought that the tests must have been written poorly, but on inspection I discovered that they were perfectly correct. In this case, he was comparing the results of their parsed date/time with a Calendar object from the Sun JDK. Of course, since the printed data was incorrect, I expected it to compare incorrectly to the Calendar object from the Sun JDK. But this wasn't happening.

The next step was to work out why this was happening, so I asked the third-party library to print its internal long value, and compare it to the internal long value from the Calendar. But the Calendar was also giving the wrong value. That's when I realized something was screwy with the JDK, and not the third-party library.

Timeslip

A quick test class shows the problem:

import java.util.Calendar;
public class TimeTest {
  public static void main(String[] args) {
    Calendar cal = Calendar.getInstance();
    cal.clear();
    cal.set(1981, Calendar.APRIL, 26, 2, 56);
    System.out.println("Time @2:56 in millis: " + cal.getTimeInMillis());
    cal.clear();
    cal.set(1981, Calendar.APRIL, 26, 3, 56);
    System.out.println("Time @3:56 in millis: " + cal.getTimeInMillis());
  }
}

If you live in Australia or Great Briton, then this code will work perfectly. For instance, running this is Australia will give you:

Time @2:56 in millis: 357065760000
Time @3:56 in millis: 357069360000

Note that there is a time difference between 2:56 and 3:56 of 3600000 milliseconds.

However, if you happen to live in the Central American timezone, then you will get this:

Time @2:56 in millis: 357123360000
Time @3:56 in millis: 357123360000

Note that these numbers are identical! They both represent 3:56am on that day.

I tried this with Java 1.4, 1.5 and 1.6 on my Mac. I also tried with with Java 1.5 on x86/Linux and Windows. All of these configurations return identical results.

Could Sun have created a problem here? Why does it only affect timezones in the USA?

At this point it was getting late, and I was startled at what I was seeing. Andrae was online, so I showed him. I was too tired to look in the source for java.util.Calendar and java.util.GregorianCalendar, so he offered to look for me. But rather than letting me go to bed, he insisted I should blog about it so that it was written up somewhere.

Fortunately, it takes a little while for me to write a blog entry. This allowed Andrae time to discover what was happening. It comes down to the following:

The Uniform Time Act of 1966 (15 U.S. Code Section 260a) [see law], signed into Public Law 89-387 on April 12, 1966, by President Lyndon Johnson, created Daylight Saving Time to begin on the last Sunday of April and to end on the last Sunday of October. Any State that wanted to be exempt from Daylight Saving Time could do so by passing a state law.

According to comments in java.util.GregorianCalendar, the hour sequence on that night is:

  12 Std -> 1 Std -> 3 Dst -> 4 Dst

This means that there was no such time as 02:56 on that morning. So the Mulgara date/time bug is not a bug at all, but required behavior.

Disambiguation

The Mulgara behavior may be correct from one perspective, but it can still be troublesome. A time like this can be supplied from a non-US source, but it cannot be entered into a US system. This demonstrates that Mulgara date/times do not support timezones, even though they are described in the XSD specification that they implement.

I'm thinking that the default behavior in Mulgara should be to assume the local timezone (as it does now), but to allow for explicit timezones as well. That way any ambiguity can be removed.

Date/times on the input should allow the timezone to be included, as per the spec. I was disappointed to learn that we don't support that at the moment, but it should be easy enough to add the option to the parser.

That leaves the issue of the output. At the moment, it will be converted to the client's local timezone though it will be printed without timezone information. I'd rather not change the default behavior, so I think we should introduce an option of printing the timezone. Strictly speaking, the timezone used won't matter, as the times will be converted according the timezone being used. However, it would still be nice to specify the exact zone to be used. The time is stored relative to UTC, and the conversion to the time in a specific locality is only really relevant when it needs to be presented to a person. This means that we only need to worry about it at the client end. The infrastructure doesn't exist to store a timezone at the server, so it is only really feasible to make the conversion at the client end anyway.

I expect to put in a system property to control the output format in a few days. I have a lot of travel ahead of me tomorrow, so perhaps I'll get the time then.

Object Interfaces

Last year I wrote some code for work for managing objects in Mulgara. It started out as a quick hack, but quickly extended into something properly structured and significant. Unfortunately, most of my hours at work seem to be very inefficient, but I spent many long evenings on it, and got something quite useful going in just 2 weeks.

The idea was to define objects using standard RDFS, and to store instances of those objects all in the same model. This is similar to an object database, or Castor, or even Hibernate, but the object definitions are more malleable, and the instances need not be complete, and can even have fields filled with inferencing.

I thought the result was reasonably compelling, and it enabled someone else to build a nice little Rails application that we used a few times in a demo. Unfortunately, since then we've gone in another direction, essentially orphaning the code.

I'd love to take this code and run with it, but I did write it for work. Even though the majority of the development time was after hours, and the idea was my own, I still spent some work time on it, and it was definitely for a work project. But that's OK. Now I get to implement a Second System. I'll just have to be careful not to overdo it.

Interface Options

One of the biggest problems with objects and object definitions in the previous system was the manual detail needed to build simple things. Every field had to be defined up front, including the name and datatype (including lists). There are also optional attributes such as the field being a key, required, and even transitive. These were encoded using owl:InverseFunctionalProperty, owl:minCardinality and owl:TransitiveProperty, which left the door open to start doing some more interesting things with OWL.

Defining all these fields made for a powerful system, but hardly a friendly one. I'd like to keep an interface like this, but also allow for more natural creation of objects.

The original style of interface was inspired by Perl. In this interface object definitions use a simple map, where each field name is mapped to it's type data (this includes the optional modifiers). I've also added in a couple of methods for finding fields via the properties. The object instances are similar, in that the field names map to the data. It's easy, and it works well, but it involves a lot of messy calls to Map.get() and Map.put().

The first alternative that comes to mind is to build objects by example. By this, I mean to create a Java class definition (sans methods, and with annotations for some more interesting features, like transitivity), and to construct the RDF via reflection. With a system like this, it would be easy to pass any java.lang.Class to create a new definition, or just pass an object and have the object and definition stored at once.

Reading objects back out of the store would take only a little more work. First of all, the name of the object definition can be tested in the class loader. If it exists, then create the object via reflection, and populate it. Otherwise, use ASM (or BCEL, or whatever appeals most to me at the time) to create a new class definition, and hand it off to the class loader to build in the same way as above. I'm still figuring out the bytecode for annotations, but ASM seems to handle them just fine.

(If I wanted to get really fancy, I could even store the methods for a class by storing the AST in RDF. This has been something I've wanted to do for a while, but it really belongs in another project, and including it at this stage really sounds like an example of the second system effect.)

The main problem with this system is that it requires that objects to be stored are already statically defined (you don't want your users to use a bytecode manipulator to dynamically build their classes). That makes the system similar to Hibernate, when it should be much more dynamic.

A simpler alternative might be to allow object definitions with a simple syntax, which I parse as a string. That kind of appeals, since it makes the API easy for the developer, while passing the hard work into the engine, where it belongs.

For the time being, it may be better to stick to the map-style implementation, and add features only once I have it all going again in an open source way.

Pitfalls

Implementing this the first time around showed up some of the difficulties in actually building something like this.

Atomicity

One important problem was a difficulty in querying all the data needed to construct objects in an atomic way. I had hoped that subqueries could solve this, but I didn't find a way through it. If it isn't atomic, then a structure could be inconsistent if someone were modifying the data at the same time.

So far my code has all been "client side", but the requirement of atomicity has me wondering if I need to move to a server side API. I should talk with Andrae about transactions.

Updates

There is also the question of updating the data. This is an important question, while at the same time it is a non-issue for RDF.

RDF asserts simple statements of subject-predicate-object. A statement either exists, or it doesn't. If I have a statement like:
  [ S O P1 ]
and I want to change the predicate so that the statement becomes:
  [ S O P2 ]
then I have not changed this statement at all! Instead I have removed the first statement, and inserted a new one. So RDF only handle assertions and denials, nothing else. Keeps everything simple.

However, any structures built on RDF (such as RDFS) do require the ability to modify them. I suppose that it is possible to naïvley remove the whole structure, and insert a new one, but this is both inefficient, and possibly disastrous for any links to that object. However, keeping track of object deltas is not very easy. I also have concerns (that I haven't addressed yet) of how to ensure any changes are compatible with related structures.

Open World

Another problem is that the real world wants object structures to be in a "closed world" definition. This isn't the time to justify this assertion, but there are a lot of practical reasons for it. To date I've taken a couple of liberties with RDFS, where I've required that fields in an instance must have type definitions in the object definition, but this is not RDF. The closed world is certainly more practical for computer applications, but I'm thinking I should explore an open world interface, while still keeping the API practical.

Datatype Properties

I also ran into an unusual issue with strings.

While I could have used the org.mulgara.query.Query interface (like I know Andrae does) I have chosen to use iTQL instead. There were several reasons, but the most compelling at the time was the need to create the code quickly and to debug easily.

However, I have to say that I hate working with strings. They're messy, prone to typos, and reek like magic numbers. That's a problem with iTQL, where all the queries are constructed from strings. To counteract this effect, I created a series of classes and enumerations which did all the iTQL building for me. The irony was that I ended up with a similar looking interface to using org.mulgara.query.Query directly. I like to think that I gained a few important features though. :-)

One part of the code passes all subjects, predicates and object to a method to get the appropriate iTQL representation. For URIs this is simply the URI wrapped in angle brackets. For blank nodes during insertion, this becomes a variable. And for literals, the data is converted to a string, placed between single quotes, and gets followed by the datatype. It even spots the difference between a URI and a URI Literal.

Reconstructing data from a literal is also straightforward, using an enumeration which maps the XSD datatypes back to the required Java constructor.

The problem is that all strings end up looking like this:
  'A string'^^<http://www.w3.org/2001/XMLSchema#string>

Now strictly speaking, this is right. Unfortunately no one ever writes RDF data that uses typed literals for strings. It appears that untyped literals are always used for strings, and typed literals for anything else. Just to be clear, Mulgara considers a typed string to be distinct from the same value in an untyped string, making strings incompatible between imported RDF and the data I construct.

For the moment I've left the string datatype where I found it, as I haven't tried to integrate data from other sources yet. However, I will probably need to make an exception for strings, where they have the datatype stripped off during a write, and missing datatypes inferred as strings during a read.

This problem did point out that these two types of strings are incompatible in Mulgara. Subsequently I've been wondering for a while whether we need to address this. Maybe I should go and look at some more RDF theory. I suspect that they are intended to be distinct, but I'm not sure I see a practical reason for it.

New API

I've been thinking for a while that Mulgara needs an API at a higher level than RDF. I'm not talking about RDFS or OWL inferencing (which is what I'm usually discussing) but a solid way to manipulate structures at this level of abstraction. I'm even prepared to use a different abstraction to RDFS and OWL, though it makes sense to pursue these ones.

Everything I wrote here is really about object definitions and instances (MOF levels 0 and 1), rather than addressing RDFS or OWL directly. In fact, only a couple of OWL constructs are used, so I can't say that it supports OWL in any meaningful way, only that it uses a subset of OWL to represent some structural information.

I want to pursue this style of API for the time being, since many real applications want to refer directly to objects and their definitions. I think this is a very practical approach, and can even be extended to encompass most (or maybe all) of OWL. But I'm also thinking that I should consider a real honest-to-goodness OWL API as well. Perhaps something that resembles the OWL abstract syntax. After all, there are a lot of people using OWL out there, who may want direct access to OWL structures, but don't want to manipulate RDF (which can be verbose for simple OWL constructs). But before going too far down this road, I should take a closer look at others attempts at an OWL API (such as Sofa, which was integrated into Kowari at one point).

Late

As usual, I've stayed up much too late to write this, leaving no time to proof read. Caveat emptor.

Numbers

Here I was, hoping to get more blogging done on my "working vacation", when I suddenly discover that I'm leaving Australia tomorrow! Oh well, it was nice to get some family time in. We have more planned for today.

I got an email this morning telling me that a link to some code I wrote has gone stale in an old blog entry. Oops.

The code isn't a big deal, but if anyone wants to generate a lot of RDF based on cardinal numbers, then I've put the file up on Mulgara.

See you in the USA.

New Look

I had to change some of the elements on this page (I was still linking to Kowari!), so while I was at it I decided to change the look a little bit.

The color change is mostly because someone anonymous complained about it. I couldn't tell if the complaint was spam, but I figured I was due for a change. The change in width is because the narrow columns have been bugging me for years. Unfortunately, I know next to no CSS, so I fiddled until I liked it a little better. Please let me know if I killed the layout in some horrible manner.

Mulgara Progress

Mulgara is going ahead nicely at the moment, thanks mostly to Andrae. He has implemented some major changes and bug fixes which were sorely needed. My only problem with any of this has been that my main support has been administrative, rather than technical.

Once upon a time I was writing TKS/Kowari/Mulgara code every day. It was very challenging and very satisfying. I also found that the whole process of blogging was useful to help keep my thoughts focussed, and keep other people in touch with what I was up to.

These days I'm involved in a lot of other people's code, and I can't blog about at all. It also keeps me away from coding, which is very frustrating, as I don't get that rush of implementing something cool, and seeing it all the way through to completion. That leaves me to do the interesting work at night, but after a long day at work, and time with family, I don't have the motivation to put in long hours of coding. Of course, two young boys mean that my weekends are packed as well.

Any remaining time does go to Mulgara, but the less frequently I work on it, the more I have to re-learn the context of where I was the "last time I was here". Just yesterday someone sent me an email, where they included a message they received 2 months ago. The email was coherent, and apparently written by someone who really knew the issues. I found myself agreeing with a lot of it and thinking, "this guy knows more than I do". Unfortunately (or fortunately?) I soon realized that the original author was myself.

It can be difficult to make a lot of progress when you have to bring yourself back up to speed every time you look at something.

I'm not sure what to do about this lack of time for non-work activities, but I'll have to resolve it soon. I'm supposed to start back at university shortly, and without any free time available I won't be writing a thesis! That's particularly frustrating, since I've been making all sorts of progress in processing OWL recently.

RLog

So what have I been doing in Mulgara/OWL?

Some time ago I realized that I needed to get an OWL processor out there, even if it were only partly complete. Hey, I can call it OWL-Lite if I want to! The specification explicitly says that there is no list of requirements to fulfill, so I'm covered. Fortunately, there are a lot of simple OWL inferences that I can perform already, courtesy of Raphael Volz. A number of others also come to mind, given some of the operations in iTQL, so this looked like a good first step.

The problem is that most of this work is easy to code using description logic (as Raphael has done in the paper I mentioned), but converting it into the RDF format I've created for the Krule engine is tedious and prone to bugs.

I found myself wishing that someone would write a programs that would parse the description logic, and convert it into Krule syntax. Even better, the dependencies could be automatically generated, based on the the format of the output of one rule, and the input to another. This would avoid many of the bugs I could potentially introduce through manual transposition, and make all future rules MUCH easier to encode.

So I'm motivated by laziness (who wants to manually encode all those OWL/RDFS rules and axioms?), impatience (why should I have to find all those dependencies, when computers are better at that kind of thing?), and hubris (hey, I could write a program to do it for me, and it would be cooler than every other logic interpreter out there because it works on my database).

So it looks like I want to re-write Prolog (yes, I've even looked at an interruption function in the algorithm to permit "cut" operations, à la Prolog). That takes hubris to a whole new level for me. But the funny thing is that it would be so easy. The hard work is in the rule engine, and I've already written that!

The language isn't really Prolog, or even the simpler logic languages, like Datalog. The main difference is in the allowable syntax. For a start, I want to allow domains on the predicates (so I can write owl:sameAs(x,y), for instance). I also want to use lower case letters for variables (instead of the upper case letters required by other logic processors). But in essence it's all the same.

So it's a type of logic, and it's based on RDF. Without anything better, I decided to call it RLog (which I pronounce Arr - Log). Who knows, if I finish it, it might even take off. (OK, OK, I don't have that much hubris).

It's funny that I'm looking to implement this. A couple of years ago I realized that Kowari could form the basis of a Prolog engine, though I knew that there was a lot of coding needed before we could do it. Now I'm on the verge of having written it, and I never really intended that.

Where Am I? How Did I Get Here? And What Am I Doing In This Handbasket?

I started all of this some time ago, but for the reasons I've mentioned above, I still have some way to go.

I started out by encoding my first set of OWL rules in RLog. The language didn't need much definition, just standard predicate logic, along with variables, domained predicates, comments, and so on. So then I converted all the RDFS rules, and added in the OWL rules that I want for the first cut. This showed up some interesting cases that I had to consider.

RLog tricks

The first thing I found was in rule 4b on RDFS. Strictly speaking, it should be:
  rdfs:Resource(u) :- a(x,u).
Meaning that for any triple <x a u> the element labeled "u" is a resource (or rdfs:Resource). That's what the RDF documents say anyway. However, this violates RDF, since u might be a literal. Literals are unable to have types like this, since it requires that they be the subject in a statements, which is illegal in RDF.

The way I've been getting around this in Krule so far has been to use a "type" model in Mulgara. The result removed all resources which were literals (via the "minus" operator). I decided that the best way to achieve this in RLog is with the following:
  rdfs:Resource(u) :- a(x,u), ~rdfs:Literal(u).
So now RLog has negation. Wonder how I'll make that work in the general case?

The next issue is a little trickier. Rule XI (Why Roman numerals? I never worked that out, except that this rule is weird) states that if a predicate URI starts with the RDF domain and an underscore, then it is a ContainerMembershipProperty.
  eg. http://www.w3.org/1999/02/22-rdf-syntax-ns#_1
This isn't quite right, as anything after the underscore is supposed to be a number, but it's not too bad. Sesame seems to get away with this rule.

Since Mulgara is using "magic" predicates for certain operations, it made sense to extend this into RLog:
  rdfs:ContainerMembershipProperty(i) :- i(x,y), mulgaraprefix(i,"&rdf;_").

Fortunately, none of the OWL rules needed anything fancy, though I suspect I may need to when I get deeper into subsumption.

Beaver

I think that I've already discussed the various compiler-compiler options I looked at. To recap, I chose Beaver as an LALR parser, and JFlex as the lexer.

JFlex is under the GPL, which would normally be bad (since Mulgara is OSL), but fortunately they state that any code generated by JFlex can be under any license you like. I don't plan on changing or extending JFlex, so that's fine (I'd be more than happy to contribute back any changes I made).

The incompatibility of the OSL and the GPL means that I can't distribute JFlex to generate the lexing code, but I have two other options. The first is that I just provide the lexing code, and the original .lex file. That would let anyone use the code just fine, since the lexer can just be taken "as is". A developer would only need to download JFlex for themselves if they needed to change the lexing rules.

The other option is to add an Ant task to download JFlex if needed. This seems to be the better, option, but I haven't done it yet.

As far as using Beaver/Flex is concerned, I've finished the parser, and can easily print out the Abstract Syntax Tree (AST). Now I need to write the code that walks over the tree, finds the dependencies, and converts the AST into Krule.

None of it seems to hard, but it's a matter of finding time. Meanwhile, the boys woke up a short time ago, so I'd better go and play "Daddy" for a while...

Patents

Today Slashdot has a story on LSI patenting multiply linked lists. The patent was applied for in 2002 and granted in April this year.

I'm glad to see lots of comments on Slashdot referring to prior art. A decent lawyer could avoid the question of doubly linked lists, so I was pleased to see more complex examples from the 80's and 90's.

I was particularly bothered by this patent due to the fact that I would have inadvertantly violated it several years ago. My blog describes an RDF DB design with takes advantage of multiple links for each node in a list, so that the lists can be traversed in completely different directions. Unfortunately the posts were made in May 2004, which was well after the submission date of the patent (2002). So my work doesn't qualify as prior art.

You'd think that my own work (among others) constitutes the breaking of the "non-obvious" requirement. Apparently not. There must be a whole swathe of the software developers community who may be classified as "brilliant geniuses" by the USPTO, given that we all come up with the same non-obvious idea.

When learning programming, we are all inculcated with standard techniques. We are taught to not reinvent the wheel. Design patterns, algorithms, you name it... the state of the art in programming is to take someone else's system and build on it. Make it something new by connecting old things in new ways, or adding into existing systems. We are NOT taught which techniques to avoid due to patent problems. We are not taught how to search for possible conflicts with existing patents. In fact, the whole point of modern programming is an anathema to what patents try to enforce. So it can be really frustrating to see patents on common ideas. It can be even more frustrating when someone decides to patent something that you've worked with. I guess I should be grateful that I needed a more scalable design than the one described in that blog post.

Mashups

I like seeing the tools we write being used in the real world. After all, what is the point otherwise? We are trying to create something useful, so it has to be applicable to use-cases that people really want. While my own specialty is Mulgara, this is just a tool for the semantic web. So I'm interested to see these use-cases, regardless of the underlying engine.

With this in mind, I enjoyed reading a post from Danny Ayers today. He describes using various data sources to solve a useful real-world query that he got from yet another source, to wit:

"Where can I buy firewood as a function of price and distance from where I am now?"

While I agree with Danny about the ability to put all of this together, I think he is a little optimistic about the effort to do so:

"... all the tech to answer the question on the Web is readily available as almost-commodity tools, not a lot of effort needed."

It would be nice to think that he is right, but I'm skeptical. Maybe I'm overestimating the effort. After all, I'm still a beginner at programming Ruby. :-)

My favorite line was last:

"Why would anyone put such data on the Web? Perhaps they might want to sell their wood."

BTW, of all the words in the US spelling system, the one that bothers me the most is "favorite". It just looks wrong somehow. Why this word bothers me more than "color", "optimize", or "bank check" is something I can't answer. :-)

Lament for Open Source Work

I miss those heady days of working on open source code 90-100% of the time. In those days I could talk about work as much as I wanted, with no ramifications. In fact, discussing my work on Kowari was a good thing.

I initially started blogging about my work simply so others in my team could find out what I was doing. If they didn't have time, then they didn't have to read it, but if they were interested then it was all there for them to read. I quickly found that organizing (there's that American spelling again) my thoughts sufficiently to write about them helped me to keep track of what I was doing, and to plan for up coming work.

Somewhat surprisingly (for me), it didn't take long before people I didn't know started to read my blog as well. This was an interesting experience, and led to some interesting conversations and contacts. Ultimately it helped me get my current job!

Unfortunately, my work now is almost exclusively commercial. That means I can't write about it in public. I've also found that I like blogging too much to write about it privately (even in the local Wiki at my company).

I can still write about Mulgara outside of working hours, but then I run into a new problem. Back in the day, I was able to work on Kowari (now Mulgara) during the day, and blog about it at night. These days I work on "other things" during the day, and at night I can either blog, or I can work on Mulgara. It's hard to find the time to do both. It's a little frustrating.

Actually, most of the technical things I've been writing about have been in the Mulgara Wiki, and not in this blog at all. So at least I've written something, but it's disappointing to think that I've lost most of my audience (small as it was). The emails and comments I received were often insightful, very useful, and quite gratifying.

With all that audience missing, who am I writing for today? Well, mostly myself... just like it was in the beginning. I suppose I'll also be read by those people who forgot I was in their RSS feeds (hello to both of you!). :-)

Recently I've noticed that I miss the process of writing. I guess I'll have a flurry of activity in this blog for a little while, until I get overwhelmed with other things again. Hopefully it will be fun while it lasts!

Functional Programming

Is functional programming the next big thing? A lot of the reading I've done lately would seem to suggest so, but that might just be the nature of what I've been reading, rather than a genuine shift in the IT community.

Of course, I've been hearing about functional programming for years. I've always skirted around it, since it was not taught in my electrical engineering course and I never pursued computer science academically until recently. (Maybe that's another reason I've been hearing about it more?)

My first exposure to this style of programming was when I learnt the C++ Standard Template Library (STL). Of course, it's no where near pure functional programming, but I was amazed at how elegant code could become with the STL. An HP example that I remember seeing demonstrated a simple program which read strings from stdin, sorted them, removed duplicates, and printed the results, all in 2 lines of code. However, if I wanted terse I could just code in Perl. The beauty of this code lay in its simplicity.

Over time I encountered this approach in many different forms, but it wasn't until I watched the lectures of Abelson and Sussman that I got a formal appreciation of it. Now I understand why programmers recommend learning Lisp, simply for the different way that it makes you think.

Recently I've started seeing many programmers whose blogs I follow commenting on functional programming. It has taken a while, given that the approach was first really pushed by Backus way back in 1978.

One of the big advantages of functional programming is the ability to provide a proof. There are mechanisms for proving the correctness of imperative programs, but these are largely impractical on real systems. Functional programs by contrast have the form of a mathematical expression, thereby making proofs easier to derive and verify.

I used to think of proofs as only being of interest to academics, but I've since changed my tune. Everyone in this industry is aware of how buggy software can be, and how often projects fail to meet deadlines due to errors. Unfortunately it is all too common that these problems (among other things) can lead to projects failing altogether. With this in mind, several companies have been successfully programming using tools for converting formal specifications in Z-notation into running code. The reported bug rate is significantly lower, along with extraordinarily high rates of success. I understand that many clients with critical applications (such as military and space deployments) are choosing to only accept code written in this way. It makes sense to me.

The more I learn about this, the more I appreciate Dijkstra point of view, particularly on teaching programming. And I'm not the only one lamenting his lack of influence (I apologise to those non-IEEE members who cannot read that link). I understand the point of view that he was Ivory Tower idealist, but with projects constantly failing in this industry, and our inability to find developers with a proper understanding of programming, then I've come to revise my point of view.

With this in mind, some months ago I started learning Haskell. This language was created for academic purposes, and is considered the only completely functional language. For that reason, it may sound impractical, though Quake Haskell, along with many other projects, provided me with enough encouragement that it really can be used in the real world.

I'm not very productive in Haskell yet (for instance, I can follow monads, but I don't really get them - though I'm still reading). But I've found that it has improved my understanding of certain programming approaches, and made me a much better developer in the languages I am good at.

In the process of learning Haskell, I started to learn more about another functional language called Erlang. This language was developed in the late 80's by Ericsson, partly in response to some of the things I've been talking about here. Ericsson also released a short film, if you're interested in learning more about it. Erlang has a number of similar features to Haskell, including pattern matching, which is phttp://beta.blogger.com/img/gl.link.gifrobably the single most important feature of these languages.

If they're similar, then why would I start learning Erlang when I still haven't mastered Haskell? Well, regardless of the success of Haskell in some of the above projects, there are still barriers to its practicality. The best comparison I've seen so far was by Joel Reymont, when he attempted to build a client application for a poker server. Regardless of his complaints on Haskell, he still finished by saying:
"That said, because it forever twisted my brain into a different shape and I think I’m overall a much better coder now."
This reflects my own experience so far, so I'm continuing to make my way through my Haskell books slowly. It's a shame that it has to be as slow as it is, but work, study and Mulgara leave me limited time. Maybe one day I will get competent enough at Haskell and Erlang that I can move on to OCaml.

It's amazing how all of these things interconnect. I'm also spending time on Description Logic and how to do this in RDF/OWL, so I can create a programming engine in Mulgara. While assertions are different to function calls, there is enough similarity that I wish I could do much of this work in Haskell rather than Java. At least Haskell has taught me many of the techniques I need to make this job easier, and much more likely to be correct.

Morphisms

This won't make sense to most people, but I'm just taking notes for myself so I don't forget...

I notice in the Wikipedia entry on Category Theory the following:

"Relations among morphisms (such as fg = h) are often depicted using commutative diagrams, with "points" (corners) representing objects and "arrows" representing morphisms. The influence of commutative diagrams has been such that "arrow" and morphism are now synonymous."

This reminds me of the (recently neglected) idea of compositional statements in OWL. I require some formalism to make composition a viable suggestion, and this may be a starting point.

The problem with this approach is that predicates can't really be considered to be morphisms, as it is possible to repeat them many times on a given subject. (Does this mean that the predicates can't be morhpisms, or just that they can't be monomorphisms?) Still, I wonder how many category theory ideas are portable into RDF? In a moment of meta-meta insanity, I'm left wondering if there is a morphism from category theory to RDF?

Nothing

I didn't want to embarrass myself in public, so I looked at the whole owl:Thing vs. owl:Nothing again. Something I'd forgotten is that owl:Nothing cannot be instantiated. So it's OK if owl:Nothing seems to be a contradiction (I think), since it is only the instances of a class which create problems, and you can't have an instance of owl:Nothing.

It's after midnight. I'd better get to bed before my head hurts any more.

OWL

I'm finally getting off my rear end and writing some code to do OWL in Mulgara. (I hear the gasps of disbelief from the peanut gallery).

So I needed to start with some OWL axioms. You know, owl:Class is an owl:Class, it's a rdfs:subClass of owl:Class, etc. So I went to a handy little file I have named owl.rdf. Now I distinctly recall that the file was not originally named owl.rdf but I wasn't sure why I'd renamed it. All the same, I've had it for some time, and it looks official, so I decided to work with it.

The easiest way to convert it into Krule axioms was to load this file into Mulgara, and then have an iTQL client build the RDF for me. I had a one line bug (why is it always one line?) which took me a while to track down to a typo in a constructor where I effectively said:

  parameter = this.parameter;

But finally my output was as expected.

At this stage I noticed something odd in the axioms. Out of curiosity I went back to the owl.rdf file and discovered this:

  <Class rdf:ID="Thing">
    <rdfs:label>Thing</rdfs:label>
    <unionOf rdf:parseType="Collection">
      <Class rdf:about="#Nothing"/>
      <Class>
        <complementOf rdf:resource="#Nothing"/>
      </Class>
    </unionOf>
  </Class>

  <Class rdf:ID="Nothing">
    <rdfs:label>Nothing</rdfs:label>
    <complementOf rdf:resource="#Thing"/>
  </Class>

So on one hand it says that owl:Thing is the union of owl:Nothing and everything, and on the other hand, owl:Thing is the complement of owl:Nothing. I can see a rational behind both points of view, but they can't both be true at one. owl:Thing either includes owl:Nothing or it doesn't. (Or am I missing something really big here?)

So where did this file come from? Unfortunately, I picked it up with wget. If I'd used Safari, then it would store the URL in metadata which I could view with mdls. Apple metadata might be a nice patch to work on for wget. Fortunately, when I went to the W3C to search for the file, I suddenly remembered where it came from.

RDF uses Uniform Resource Identifiers (URI) to identify resources. Note that these are Identifiers and not the more familiar Uniform Resource Locators (URL). URLs are a subset of URIs, but their purpose is for location rather than purely identification. A URI may look like a URL, but it may not be specifying the location of anything at all. This is often confusing for first-time users of RDF, who expect that the identifiers they see should be taking them somewhere useful.

Now just because RDF says that you don't need something at the address of a given URI, that doesn't mean you can't put something there. In fact, many people do, including the W3C. For this reason, one day I decided to put the URI for the OWL domain into wget and see if I got anything. The result was this RDF file called owl. Once I remembered that, I also remembered changing the name to the more useable owl.rdf.

So here is this file sitting at the W3C, hiding in plain sight. I find it amusing that it can have such a contradiction in it.

Am I Getting Old?

This story has been written elsewhere, but I have my own contributions to make to it. Hopefully the catharsis will help me sleep better. :-)

I've been helping to interview several programmers recently, and I've really been surprised at the results.

To be clear, I do not have a degree in Computer Science. My first degree was in Computer Engineering. That means that I spent many long hours debugging breadboard with a logic analyzer, and the code I was writing in my final year was usually for device drivers to communicate with the hardware I'd built. That isn't to say that I didn't have subjects in computer science - I did.

I did the basic first year CS course, which taught us about records, how to build linked lists, recursive functions, simple sorting algorithms (bubble sort!), and the other essentials. I did subjects on networking, operating systems, concurrency (anyone care to dine with philosophers?), modular programming, complexity theory, and databases. Funnily enough, in my first database course we were not given access to computers, and never learnt SQL. Instead, Terry Halpin taught us how to model data using a system called NIAM. It seemed very theoretical at the time, but it ended up being a vital foundation for my later work in OWL. I also had Computer Engineering subjects which taught me Assembler (in several flavours) and C, how to use them to talk to hardware and/or the operating system, how to model hardware in software, sockets programming, and so on.

I never covered more advanced data structures or algorithms, but then I never did a degree in CS, did I? My physics degree gave me the opportunity to do a couple of extra subjects in programming, but these were vocational, and hardly worth the effort. Instead, I had to gain an understanding of professional programming in my own time. I had a very good mentor to start with, but his main role was in teaching me how to teach myself. Consequently, I now have a large bookshelf full of books covering topics from MFC and GTK, through to Knuth, Type Theory, and Category Theory. There are still holes in my knowledge, but I've covered a lot, and I'm still progressing. In fact, learning after university has meant that I've learnt better, as I learnt out of genuine interest, and not because some lecturer said I needed to learn something in order to pass an exam. For this reason, I have learnt to have less respect for certified courses, and a greater appreciation for enthusiastic amateurs.

The same willingness to keep learning also applies to most of my friends and colleagues. This is important in our industry, both because the field is too large to know it all, and because the systems and protocols implementing the theory are constantly expanding. Fortunately, we don't need to know it all. The fundamental principles in computer science always hold, and form a solid foundation for all future learning. Once you have these down, then any new language or paradigm can be understood and applied quite rapidly.

So when we interview candidates, I look to make sure that they have a grasp of the fundamentals. Instead I keep discovering people with years of experience in J2EE, who are completely unaware of the underlying principles of computer science. It is almost universal that these people have solid vocational training, but very little ability to work outside of established paradigms.

Why do I consider these things important? Isn't it acceptable to simply work with EJBs, SOAP and JDBC? Why should I expect more?

There are several answers to these questions.

To start with, staying within these narrow APIs has left many of these people unaware of significant APIs in Java proper. Most developers we have met do not even know what file mapping is, yet alone the fact that it is included in the java.nio package. Concurrency is a subject with subtle complexities, yet these developers are often unaware of the attributes which make up a Thread in Java, including the stack that each thread owns. How can they hope to correctly use a ThreadLocal object, or know how to debug a deadlock?

Another problem with vocational training with a narrow focus is that almost all systems have a chance to interact with each other in some way. Not understanding the surrounding systems leads to an inability to solve some problems. I have yet to meet a candidate who understands what significance a 64 bit operating system has for a JVM, when compared to running on a 32 bit system. This is so important that Mulgara has a completely different configuration for file management when operating on 64 bit systems, with vastly different performance. How can programmers working on scalable "Enterprise" applications be effective at their jobs if they can't understand this difference? The level of ignorance is surprising, given the general interest in 64 bit chips and operating systems.

This extends down to simpler issues as well. As with many languages, some understanding of 2's complement binary arithmetic is required for working with all of the numeric types in Java. For instance, for every signed integer type in Java, the abs(MIN_VALUE) is always 1 greater than the MAX_VALUE. Even when shown this fact, the people I've spoken to don't understand the significance of the expression -MIN_VALUE. This could easily cause problems in code that said:

  if (-x > 0) {...}

We also find an amazing ignorance of complexity. How can a programmer write a scalable application if they believe that linear complexity is to be preferred over logarithmic? We even had one person think that polynomial was best!

The next issue is that Java will not be the dominant paradigm for the rest of our careers. Computing moves fast. Java may be around for some time yet (look at C), but something new is guaranteed to usurp it, and it will happen in the next decade (I feel uncomfortable making a statement with a guaranteed timeframe, but I have history on my side). For instance, many clients of mission critical systems are looking to developers using functional languages with the ability to mathematically prove the correctness of a system. I'm only mentioning it briefly, but the importance of functional languages in this respect cannot be underestimated. Proof trees make it possible to prove the correctness of an imperative program, but the results are messier than practical in a commercial environment.

Finally, and this is one of the most important principles, other languages teach programmers to think differently. This allows programmers to write important functionality in Java that would not be possible with standard J2EE practices. Mulgara needed a C programmer like David in order to have data structures on disk which are as efficient as the ones we have. Standard J2EE techniques would not store anything directly to disk, but would use a relational database. Even if a programmer had chosen to write directly to disk, standard serialization techniques would have been used. Instead, Mulgara uses densely packed data structures that serialize and deserialize quickly, and are structured in complex ways that are made cleanly opaque to the calling object oriented code... and it is all done in Java. David's understanding of how operating systems use write-behind algorithms, which hard drive operations are atomic, and how best to interact with the operating system's buffer cache were all vital in Mulgara's performance.

Google needed Lisp programmers to write MapReduce, which is one of the most important and fundamental tools in their system. It was Lisp programmers trying to get some semblance of Lambda Calculus into C++ who introduced the templates, and ultimately the STL. This then led to template classes and functions in Java, again highlighting the need to understand these other systems in order to better program in Java.

If we want to write important code that does anything beyond standard business logic for commercial applications, then we need people who understand computer science, and not people who have received vocational training in the popular language of the day. They need to understand the mathematics behind programming, the pros and cons of both functional and imperative languages, the features of underlying operating systems, along with many other things.

Finally, if they want to be really good, they need a desire to go out and learn some of this stuff for themselves, because no course can teach it all. It's just a shame that they don't seem to teach the basics any more. And it's the fact that I typed that last statement that makes me realise that I'm getting old. :-)

One at a Time

I'd love to be writing more here, but instead I'm putting my Mulgara ideas up on the Mulgara Wiki. Anyone interested should look under Development/Indexes.

I can only really manage the time for writing in one place at a time. Not to say that I'm abandoning this blog, but it is more appropriate to put Mulgara design details on the Wiki.

Mulgara v1.0.0

Mulgara v1.0.0 has been released!

All my spare time has gone into this recently, which should explain the dearth of blogs posts. Now that the first release is out, I should have more time to:

• Write some code.
• Write more blog entries.
• Get OWL working in Mulgara.

I hope that future releases will require less effort, since the project is off the ground, and we've established several procedures for this first release.

Thanks to the people who helped me get it out. In particular:

• Brian, who coded and debugged.
• David, who reviewed everything from legal files through to web pages, and without whom I'd find myself in a world of trouble.
• Thomas, who volunteered out of nowhere to design the web pages for us.

For anyone missing my technical musings on the structure of Mulgara, then have a look at the Mulgara Wiki. I've just started using this area, so others can contribute, and also because it allows me to structure some of the details.

Forking Code

Of course, I can't really discuss work online, which has been a good reason to keep quiet lately. (Since I'm working so much of the time). The other reason is because my spare time is kept occupied with Mulgara.

Anyone looking at Mulgara recently would have some idea of what I'm been doing recently, instead of blogging. The short story is that NGC have come to a legal agreement acknowledging that Kowari is open source, and they should not have stopped progress in the project.

There are some who hope we can now abandon the fork, and go back to working on Kowari. In one sense, I'd certainly like that. Kowari has a name for itself, and it's a name I've worked with for years. It helps people who have a commitment to using Kowari, and for whom a name change would be an annoying distraction.

On the other hand, NGC didn't have a legal leg to stand on when they started all of this, and yet they demonstrated that you don't need one to cause problems. The problems have all come down to misunderstandings, but Open Source Software has always been difficult for NGC to understand, and it is always possible they will have difficulty understanding it in the future. I know they also offended some of the Kowari developers while mendaciously describing our work, and that never goes over well with people.

So while it would be a terrible shame to lose the name Kowari, there areenough people agreeing that there are enough reasons to go ahead with Mulgara. Besides, having started it up and got it running now (many thanks to Luigi and Herzum Software for the support) I feel I have some ownership over the process in the new project.

The other thing people might notice is that I've been working on code. Alas, there are no new features yet, but I'm getting close to a point where they will be forthcoming.

Roughly, the Mulgara work falls into 3 categories:

• Administration for the project.
• Forking the Kowari sources.
• Upgrading Mulgara from Java 1.4 to Java 1.5.

Administration means learning about, and setting up Mailman, learning how to configure and use Subversion, learning the latest changes in Apache configurations (I think I last configured Apache at version 1.3), and setting up HTTPS and WebDAV access. Yuck.

I still haven't worked out how to get WebDAV to force logins over HTTPS. I can enforce logins, and I can get it over HTTPS, but not both. Oh well. I guess it just means that administration has to be done by Jesse or myself for teh time being.

Forking Mulgara has had its own difficulties. We just about had it done a few weeks ago, but NGC started insisting that certain recent code belonged to them and should not have gone into CVS at Sourceforge. They've since retracted this claim, but in the meantime we went back to the 1st of August last year and forked from that point. So we had to repeat a lot of the work we'd already done. Very annoying.

One benefit to going back to August has been that we may get to avoid a bug that we think has been introduced since then. A couple of users have reported difficulty loading large RDF/XML files. Since the parsing has all changed since August (not as a fix, but in an attempt to avoid using Jena), then this could be the cause of the problem. It will be a good opportunity to check it out anyway.

A number of bugs were introduced when all the packages, files and variables were renamed in the fork. In the course of tracking down all these bugs, I've used the opportunity to move to Java 1.5. It just created a few more bugs to track down. Fortunately, I knew where to find most of these problems from earlier personal attempts to convert to Java 1.5.

Java 1.5

The most obvious issue moving to Java 1.5 has been the new keyword "enum". I've noticed in several other places that "enum" is a popular name for packages and variables. Kowari was no exception. This was easily remedied. Unfortunately Apache has a package called enum which is referenced by code generated by the WSDL compiler. The workaround is to compile just this generated source with the -source 1.4 flag.

The other problem I encountered was in the change to Unicode 4.0.

I never gave a lot of thought to Unicode before Kowari was released as Open Source Software. Until that point I'd been able to get away with ASCII. After all, I don't know any other spoken languages (I really ought to do something about that) and all my work had been in English.

After Kowari had been out for a little while we started getting bug reports and fixes from someone in China compaining about how Kowari's strings were not handling certain characters properly. This woke me up to how international OSS is. I was used to working with people in the United States, but by being involved with something like this on the internet we were exposed to a whole new international community.

This has made me a little more sensitive to the need to support Unicode in all projects. OSS projects need it for the reasons I mentioned above, and commercial projects need it if they ever hope to succeed internationally (a real concern for me when my employer has offices in Italy, the UK, Turkey, and France).

Now 16 bit characters are all well and good, but the latest version of Unicode requires a full 20 bits. These extra characters (called "supplemental" characters) are not used very much, but I've already learnt that we can't ignore the things we are ignorant of. Unfortunately, the previous mechanism for managing these characters would not work in Java 1.5.

Previously we had used a regular expression to look for a pair of "surrogate" characters, which together form a single Supplemental Unicode character. Unfortunately, the character classes which used to be supported for this in Java 1.4 are no longer available. Some research showed that the regex library now wanted to treat these characters as a single character (which is a definite improvement), but checking documentation, and finally Java source code showed that there is no "character class" for referring to the supplemental character set.

I'm short on time here, but for anyone interested, the old regular expression for this was:
"\\p{InHighSurrogates}\\p{InLowSurrogates}"

Now the correct expression is to use the numeric range for the surrogates, and use them to build a class of unicode literals which span this region (U00010000 to U0010ffff):
"[\ud800\udc00-\udbff\udfff]"

This seems to be the worst of all worlds. We have to use a pair of 16 bit codes to create a single 20 bit unicode (using the surrogate arithmetic for calculating the final value. e.g. UD800+UDC00 = U00010000). However, we can no longer refer to these codes as a "pair of surrogates", but as a single character (even though we DEFINE that character with a 2-part code). It might have been nice if Sun created a character class like this for us. Something like:
"\\p{InSupplementals}"

Well, at least it now works in Java 1.5. It just doesn't work in Java 1.4.

geo:(41.8851942,-87.6351607)
mylocalguru

Subversion

For anyone not familiar with subversion (I include myself in this group) then the 1.2 version of the code can be retrieved via:

  svn co https://mulgara.org/svn/mulgara/trunk/kowari-1.2

I just thought I'd mention this in case anyone tries to get the whole repository (which includes the entire history of the project). The entire thing comes in at 2.6 GB. The latest branch is a more modest 204 MB.

Mulgara

The two most important elements of Mulgara.org are now running: Mailing lists and Subversion.

I've discovered that Yum is not the way to install Mailman on Fedora Core 4. It isn't integrated with any MTA which means that it needs a lot of manual configuration. After getting it working with Exim, I discovered that it wouldn't send any messages due to a problem with Sendmail. Moving from Exim to Postfix fixed the problem, but involved reconfiguring Mailman. Overall, a frustrating experience. I'm very grateful to Jesse for his help.

Jesse is the official administrator of mulgara.org, but like me he has other work to get done. Since I'm the one who's actually developing for Mulgara, he was more than happy to give me sudo and let me install the things I needed. The only problem is that he knows what he's doing, and I don't. :-)

Services

So where are we? We have:

• Subversion.
• A Mulgara-general mailing list for announcements, user enquiries, and general discussions.
• A Mulgara-dev mailing list for developer discussion
• A web page stub.

Still to do:

• The web page is currently just a place holder. We need to come up with content, and we need to move all the documentation over to it.
• From a development point of view, we need to move all the org.kowari packages over to org.mulgara. This would be easier with the refactoring tools in Eclipse, but these won't use the required svn mv command. Much of the first round will be a simple search/replace, so I may just be dusting off my perl/sed skills. To anyone with suggestions on how to do this more intelligently, please get in touch!
• The site needs a signed certificate. Thawte and GoDaddy both have certificates for under $100, but I should check out with other what we really need. I'm guessing we only need the bare minimum.
• We need a Wiki and bug reporting system. Herzum Software has offered to provide licensed copies of Atlassian's Jira and Confluence for use by the project. I'll have to ask Jesse to install these, as I have no idea about either of them.

Feel free to write to me with any questions, advice or complaints. I'll answer, accept, and listen, respectively.

Work, work , work

Yeach. Long hours in the last few weeks have left me feeling wrung out. I tried to make time for family through this (not always successfully), so I didn't have time for anything else. The first thing I tried to catch up on afterwards was exercise. This blog came way down in the list of priorities.

Bugs

I discovered a couple of problems in Kowari in the last couple of weeks. Unfortunately I had goals to push towards, so I haven't been able to look at them yet. I'll get back to them, but in the meantime I'll write them down here for others to look at.

Full Text Searching

One thing I wanted to do (fortunately I didn't need to do it at the time) was full text indexing and querying. This is straightforward using a Lucene model. Unfortunately I couldn't get it to work! All my queries came up with empty results.

I figured that I must have been doing something wrong, so I went to the examples in the documentation. I couldn't get any results here as well. So now I think that something must be wrong. I haven't had an opportunity to run the full test suite lately (I really only have a notebook, and it's slow enough without having to run the tests in the background), but I'll have to run it soon to make sure everything is still OK. If it is, then I'll be running Kowari through a debugger to figure out why I can't get any results from Lucene models.

Normally I'd have my big Linux box doing these tests for me, but it's still in a shipping container, and I've been told that I won't see it before June. Ask me if I'm frustrated.

There is one other machine that I could use, and that's the Mulgara server. I haven't discussed Mulgara here, so I'll discuss it more later.

Datatypes

This bug doesn't affect me at all (at the moment), but it's surprising to see all the same. I discovered this bug when scanning an model which restricted a class using an owl:cardinality of "1". The data type for owl:cardinality is xsd:nonNegativeInteger, so that's what I used. So far, so good.

Then in some test code, I inserted a series of numbers into a Sequence. These were just 1, 2 and 3, and each of them was typed as xsd:int. However, when looking at the full contents of the model I was surprised to see the following numbers in the sequence:

  "3"^^<http://www.w3.org/2001/XMLSchema#int>
  "2"^^<http://www.w3.org/2001/XMLSchema#int>
  "1"^^<http://www.w3.org/2001/XMLSchema#nonNegativeInteger>

So if the number was already in the string pool as a "nonNegativeInteger", then it just picks up that value again, rather than re-inserting it.

This will be OK for simple cases like this, but it won't work in the general case. Looking at the hierarchy of XSD data types you can see that int and nonNegativeInteger are both subtypes of integer, but they are not in the same branch. So neither can be used in the place of the other, unless an application specifically calls for an integer (their closest common ancestor).

So having the string pool automatically convert one to the other is going to be a problem one day. I've already started checking literal data types, and need to make an exception for this case. I can certainly conceive of a system which would fail a necessary consistency check because of this.

Now I'm left with the option of fixing it now, or waiting to put it into the XA2 store. I still don't know when we'll have time to get to XA2, so maybe an interim fix is the way to go.

RMI Connections

By far my biggest problem is RMI connections. I've seen posts complaining about how slow Kowari is, which I find really frustrating. Queries in Kowari are nearly instantaneous, and insertions are pretty fast as well. So why do people think it's slow? The answer lies somewhere in the Session interface.

For some reason, when the first query is performed on a Kowari database, the Session can take several seconds to establish a connection. This happens whether or not the system is remote or local. I have yet to profile this delay, but it all seems to be in the RMI code. I can't imagine that RMI is that bad, so I'm guessing that it's doing a DNS lookup in there, though I may be wrong. Either day, it's an unacceptable delay. We need some way to speed this up, particularly when running on a local machine. I put a lot of work into making Kowari really, really fast, and the frustration I feel at these network layer delays nearly drives me to tears.

Another problem is the what happens with an error from the ItqlInterpreterBean. It seems that a single error (often avoidable as they are normally caused by programming bugs) can render the Session nearly useless. There is some kind of synchronization issue after an Exception is thrown, that a Session can't seem to overcome, at least for a couple of operations after that. I'm not sure why this is, but it bears investigating.

A lot of these problems seem to come down to RMI. I don't know what the best replacement technology would be (SOAP will also have DNS issues, etc) but we need to look at something. Whatever we choose, we will probably need to bypass it when accessing a local data store. At the moment we do this transparently, but perhaps we should consider something a little more optimized for local connections, since this forms the majority of the use cases I've seen.

Mulgara

As David Wood pointed out, we are forking the Kowari Project. The new project will be called "Mulgara". We felt this was necessary to keep developing the RDF database when there was uncertainty due to the relationship between Kowari and Northrop Grumman.

Some of the problems appear to be caused by a misunderstanding of Open Source Software by Northrop. This has made me reluctant to discuss Mulgara until now, as I didn't want further misunderstandings to develop. Fortunately, we've had legal advice from several quarters confirming our right to fork the project (since the project is licensed under the MPL).

My employer, Herzum Software, has offered to host the new project, putting up a new Linux box for this purpose (running Fedora Core 4). Unfortunately, the administrator for the machine has been overseas for the last couple of weeks, and with my recent long hours neither of us has had time to configure all of the necessary services. I hope to change that this week. In particular, I've discovered that Yum is not a very good tool for installing Mailman (it didn't even notice that there was no MTA installed!)

Some of the services are already going, such as Subversion, but I still need to organize a certificate (at the moment it just provides the "localhost" certificate that comes standard with Fedora Core).

Check back soon and we should have it all running.

Commitments

It's quite difficult to blog while working and moving residence at the same time. I've had some big things happening at work at the same time, so the preference is usually to get those things done rather than write.

On the other hand, I've let slip so many things that I wanted to talk about, that I'm getting frustrated with my lack of blogging. I don't have a lot of time tonight, but I thought I should at least get something written.

FAT_MAGIC

Following a link off Slashdot, I found an interesting discussion of otool in OS X. The author had been looking for an ldd replacement for finding the dynamically linked libraries that a binary uses, which is something I've wanted as well. I just didn't know about otool.

otool seems to be a Swiss-army-knife for inspecting Mach binary files. The array of switches seem quite large, particularly since each function seems quite unrelated to most of the others. It makes the tool feel a little overloaded.

The article points out that the file /usr/bin/crt1.o contains the entry point code for all programs in OS X. I've been interested in the structure of universal binaries, particularly dynamic libraries, so this seemed like a good library to apply otool to.

Looking at the numerous parameters, I discovered that:

  otool -f

and

  lipo -detailed_info

produce nearly identical output. I suppose this makes sense, given that both are looking at the fields in standard Mach binary structures, but it's so close that I have to wonder if they might share some code. If that is the case, then I wonder if any other otool flags duplicate the functionality of other programs?

While looking at all the information available from otool, I discovered that all of the programs I was looking at start with a magic number of: 0xcafebabe. I was shocked at this, as this is the same magic number used by Java class files. Why would they share this value?

I suspect that FreeBSD took this number first with their definition of the Mach file format, and Sun's Java team weren't aware of it when they were defining Java. Or perhaps they knew, but didn't care about a small, non-commercial operating system (little did anyone know what would become of FreeBSD in the future!). So where did FreeBSD get this format from? Did they define it themselves, or did they inherit it?

Whatever the reason, it obviously created conflict. I found a reference to it in the change log for the cctools package in the Darwin project:

Changes for the 5.16 release (the cctools-520 release):
- Fixed a bug that caused otool(1) to crash when it was called with a bad fat
  file (a java class file).  The calls to ofile_first_arch() in ofile.c was not
  checking its return value and later causing the code to crash when it should
  have returned in case of an error.  Radar bug #3670740.

Looking at the code in question, it now looks for bytes at a particular offset to try to avoid mistaking a Java class file for a Mach binary. I don't know if it's justified, but I always feel uneasy when a programmer leaves a note to say that the implemented approach is a hack.

These magic numbers are supposed to help a system differentiate between different file types, particularly executables. Having the same number on two major file types seems like a problem, but it's not one that could be changed now.